Clojurians Log v2

@pesterhazy how much time should unravel spend loading compliment?

I'm seeing either > 1 minute, or never (I've given up)

@dominicm in a fully local setup?

@cgrand What does "fully local" mean in this context?

server & client on the same machine

Yes.

I'm now at 3 minutes.

[:receive {:origin :aux} [:started-eval {:actions {:interrupt (unrepl.replG__156/interrupt! :session654 6), :background (unrepl.replG__156/background! :session654 6)}} 6]]
[:unknown-command [:started-eval {:actions {:interrupt (unrepl.replG__156/interrupt! :session654 6), :background (unrepl.replG__156/background! :session654 6)}} 6]]
[:receive {:origin :aux} [:exception {:ex #error {:cause "compliment.core", :via [{:type clojure.lang.Compiler$CompilerException, :message "java.lang.ClassNotFoundException: compliment.core, compiling:(unrepl-session:6:63)", :at [clojure.lang.Compiler analyzeSeq "Compiler.java" 7010]} {:type java.lang.ClassNotFoundException, :message "compliment.core", :at [java.net.URLClassLoader findClass "URLClassLoader.java" 381]}], :trace [[java.net.URLClassLoader findClass "URLClassLoader.java" 381] [clojure.lang.DynamicClassLoader findClass "DynamicClassLoader.java" 69] [java.lang.ClassLoader loadClass "ClassLoader.java" 424] [clojure.lang.DynamicClassLoader loadClass "DynamicClassLoader.java" 77] [java.lang.ClassLoader loadClass "ClassLoader.java" 357] [java.lang.Class forName0 "Class.java" -2] [java.lang.Class forName "Class.java" 348] [clojure.lang.RT classForName "RT.java" 2204] [clojure.lang.RT classForNameNonLoading "RT.java" 2217] [clojure.lang.Compiler$HostExpr maybeClass "Compiler.java" 1041] #__1]}, :phase :eval} 6]]
[:unknown-command [:exception {:ex #error {:cause "compliment.core", :via [{:type clojure.lang.Compiler$CompilerException, :message "java.lang.ClassNotFoundException: compliment.core, compiling:(unrepl-session:6:63)", :at [clojure.lang.Compiler analyzeSeq "Compiler.java" 7010]} {:type java.lang.ClassNotFoundException, :message "compliment.core", :at [java.net.URLClassLoader findClass "URLClassLoader.java" 381]}], :trace [[java.net.URLClassLoader findClass "URLClassLoader.java" 381] [clojure.lang.DynamicClassLoader findClass "DynamicClassLoader.java" 69] [java.lang.ClassLoader loadClass "ClassLoader.java" 424] [clojure.lang.DynamicClassLoader loadClass "DynamicClassLoader.java" 77] [java.lang.ClassLoader loadClass "ClassLoader.java" 357] [java.lang.Class forName0 "Class.java" -2] [java.lang.Class forName "Class.java" 348] [clojure.lang.RT classForName "RT.java" 2204] [clojure.lang.RT classForNameNonLoading "RT.java" 2217] [clojure.lang.Compiler$HostExpr maybeClass "Compiler.java" 1041] #__2]}, :phase :eval} 6]]
[:receive {:origin :aux} [:prompt {:file "unrepl-session", :line 7, :column 1, :offset 1082, clojure.core/*warn-on-reflection* false, clojure.core/*ns* #unrepl/ns user}]]
[:unknown-command [:prompt {:file "unrepl-session", :line 7, :column 1, :offset 1082, clojure.core/*warn-on-reflection* false, clojure.core/*ns* #unrepl/ns user}]]

Oh, actually, it might be more malignant than only that. I've added compliment as a dependency, and unless I do (require 'compliment.core) first, the completion hangs.

Hi tried it and I think that unravel doesn’t require compliment.core and hasn’t it on the sideloader path

@cgrand A bit of an aside, but how do you feel about vendoring depndencies in this context? Should unravel be using unravel.compliment.core, so that compliment 0.3.4 can be used by unravel, and 1.0.0 can be used by clientC?

Or, perhaps more importantly, so that oldProjectA can use 0.0.1, and unravel can use 0.3.4.

Yes because this last scenario breaks even in a richhickeyic world.

The other option, and I'm not sure if you've thought about this, is to use clj-embed: https://github.com/RutledgePaulV/clj-embed which is an independent form of boot's pods. For something like compliment this might not make sense as it's inspecting the classpath & running state. But for other tooling needs (perhaps bundling zprint) it would be a good idea perhaps. I mention this library because I wonder if this is a problem unrepl may want to solve in this way.

Maybe it should actually be further vendored than just unravel.compliment.core, but instead unravelv0_2_2.compliment.core to allow parallel clients of unravelv0_2_1 also.

I also question whether it should be scoped to the client's connection altogether, with the vendoring provided by unrepl.

@dominicm I thought about classloader isolation and even did a quick prototype in the early days of unrepl

Like you mention, classloader isolation is strong, too strong.

The “best” option if you set up a second clojure, is to have it communicate in text-based form with the user clojure.

Basically you may treat it as a low-latency unrepl client.

Not sure it’s a good idea.

Classloader isolation has really the problem of being too strong for our purpose — and it doesn’t work in cljs (starting a second JS env in Node?)

So, vendoring it is. We shall all become professional users of Mr. Anderson!

With @richiardiandrea we discussed a subtler and better vendoring

What was the approach considered?

vendor compliment.core as vnd.sha1_XXXXX.compliment.core

ah, so it works globally for namespaces which are the same, makes sense.

;; assuming that (do-vendor vendorable vendored) substitutes all refs to vendored nses in the source of vendorable
;; then computes its hash, then puts the hash in ns in the source and returns the vendor-hashed ns name.

(let [deps-graph XXX]
  (loop [vendored {}]
    (if-some [vendorables (seq (filter (fn [dep] (every? #(or (white-list? %) (vendored %))  (:deps dep)))))]
      (recur (into vendored
               (map (fn [vendorable]
                      [(:name vendorable) (do-vendor vendorable vendored)]))
               vendorables))
      vendored)))

I wonder if JarJar can do something similar

(https://code.google.com/archive/p/jarjar/)

can do or could do?

I suppose either.

Someone willing to implement that? 😉

It would make a great xmas present to the community...

A tool to vendor dependencies you mean?

All of this seems to be pretty much what Mr. Anderson does.

yup, maybe a patch to mranderson

@bozhidar the hashing rather than a specified prefix is the distinction I think.

Vendoring deps was one of the most painful things when building CIDER - something some basic should be easier to do.

Yeah, I get the point, but fundamentally it’s the same process.

Patching Mr. Anderson to support this would be trivial.

Just looked at it and it’s not that trivial because of deps ordering, which matters for hashing

if libA depends on libB, when hash-vendoring, you can’t independently hash them

@cgrand I was about to ask about that for hashing.

I guess a version-tied prefix is the simplest way.

if libC uses libA v1 and libB v1.1 if libD uses libA v1 and libB v1.2 then we should get two different libAs

$ unravel --version
Unravel 0.2.2 (Lumo 1.7.0)

$ lein --version
Leiningen 2.8.1 on Java 1.8.0_151 Java HotSpot(TM) 64-Bit Server VM

JVM_OPTS='-Dclojure.server.myrepl={:port,50505,:accept,clojure.core.server/repl}' lein repl

unravel localhost 50505

$ unravel localhost 50505evalmachine.<anonymous>:22var reader = cljs.reader.push_back_reader.call(null,s); ^TypeError: Cannot read property 'call' of undefined at unravel$lisp$safe_read_string (evalmachine.<anonymous>:22:42) at Transform._transform (evalmachine.<anonymous>:56:60) at Transform._read (_stream_transform.js:186:10) at Transform._write (_stream_transform.js:174:12) at doWrite (_stream_writable.js:385:12) at writeOrBuffer (_stream_writable.js:371:5) at Transform.Writable.write (_stream_writable.js:288:11) at Transform.ondata (_stream_readable.js:642:20) at emitOne (events.js:115:13) at Transform.emit (events.js:210:7)

$ lumo --version
1.7.0

@ghsgd2 built from Git or brew?

@cgrand brew install bfontaine/utils/unravel

@ghsgd2 you'll need to use latest master unfortunately because the version on brew is not compatible with lumo 1.7.0+

@dominicm, did you solve the compliment issue?

compliment is only used when you pass --flag compliment

@pesterhazy There's a bug or 2 there, I found a workaround. Bug 1) compliment.core is used without requiring it first (makes compliment pretty much unusable without knowing to require it first, causes total lockup) Bug 2) unravel doesn't bundle a copy of compliment / it doesn't vendor compliment in order to prevent conflicts with the project.

and that requires compliment to be on the class path

yup both are true (oops!)

could you file an issue?

Doing for both now, there's a possible 3rd "nice to have" here, handling exceptions that happen during tab completion. (and other aux exceptions)

yeah it's a good point. I wonder how we should catch those exceptions

@pesterhazy Got it. However, official installation docs say that brew could be used to install unrepl on macOS. And lumo was installed by brew unrepl formulae as a dependency. Looks like brew formulae is broken and needs fixing.

@ghsgd2 that's correct

we want to release a new version that fixes that, and brings all the new feature in master as well

haven't found the time lately

@pesterhazy Same way you do for other exceptions? 😄 Displaying them is a little more troubling. Did the code for split screen ever go in? That would be a great place to dump it OR display a message saying "An unexpected exception has occurred, a log has been written to /tmp/unravel_xxx.log" which shows perhaps in the same place documentation would normally show?

the split screen only works for developers (using scripts/debug)

good idea about the log file

the key is to continue with regular operation after the exception occurs, rather than blocking forever

Yeah, that's super important. Medium level of importance is allowing you to fix bugs which bother people.

(On this scale anyway)

@pesterhazy No worries, created the issue as a reminder.

thanks!

I'm actually working on this related problem in unrepl.el right now: figuring out the best way to present aux conn exceptions to users... I'm using a popup temp buffer, but would love to hear if you come up with a different UI

Question, If I wanted to communicate from the browser to a plain socket repl running the unrepl blob, would web sockets be the most appropriate strategy?

I imagine standing up some sort of web socket server to communicate with the client

An error log?

With parfix, is there a way to send the current line without hitting "End"? My brain went to Ctrl-Enter

@gcast why not? The sad part is having to have an intermediate server.

Ok good. I've started down that road. I spent some time trying to see if it was possible to have a direct stream to unrepl, but my research and current understanding suggests browser security may make other approaches problematic

@dominicm I wanted it to but apparently you can’t get CTRL ENTER in a term app. I don’t remember what got merged. Did you try ^J?

I’m unhappy with the current behavior and @pesterhazy and I have differing opinion of what a good behavior should be.

@gcast you want a real browser not a web view in a native app?

well I suppose if the underlying repl can live on a remote server than a native app is just fine as well

I believe that sending a http request to a repl would almost work… if you found a way to not have the browser barf on the garbage.

the key is that the compute can happen on an arbitrary networked machine

hmmm, garbage-handling aside, would http-requests be slower for repl communication?

@cgrand At least in my terminal, Ctrl-J didn't work

It’s not merged yet- you can try the branch though

It’s a PR

Ah, I thought I was cutting edge using master as it is 😄

The first protoclient for unrepl was electron-based

really? I'd never heard of electron before but just did some research and it looks pretty nice

how mature is the CLJS environment for electron?

I don’t remember what I used. I could look it up.

It worked ok. Too much reloading. I wasn’t familiar with node enough. Otherwise I would have made it expose a socket repl (well two)

I see. this looks quite promising because there is already an electron-based notebook-app for arbitrary jupyter-kernels https://github.com/nteract/nteract

Would simplify the problem to just writing a native clojure backend that support unrepl.