Clojurians Log v2

I have a question, related to this <https://clojure.org/guides/dev_startup_time>. Say I have a deps.edn that has an alias that pulls in a few other sources, via extra-deps, using a couple of :local/root declarations, i.e., one that loads in rebel readline (with little customisation) and another which launches a nrepl (with some middleware injected). Can I still benefit from what the guide says? Can I compile the extra-deps includes, so that my clj startup is reduced?

sure

it's all on the same classpath

Okay, but where would the classes directory live, i.e., if I have something like this...

:rebel {:extra-deps {local-rebel {:local/root "/home/david/.clojure/libs/local.rebel" :deps/manifest :deps}
                       local-nrepl {:local/root "/home/david/.clojure/libs/local.nrepl" :deps/manifest :deps}}
          :main-opts ["-m" "local.rebel.main"]}

I'd put it in its own alias and then use multiple aliases

ie, it's actually the same as what's in the doc, you just might have extra tool aliases in play

okay, let me have a play around. My clj startup at the moment is taking about 8 seconds to launch...

so clj -A:dev:rebel or whatever

or actually you've got main args in there

so you might want to do clj -R:dev:rebel when doing the compilation

so you get just the deps but not the main stuff

okay, will have a play 🙂 Thank you Alex! 🙂

🍺

I'd love one thanks 🙂

Oh.... Mine is ~2min on good machines and we really don't care about that

:shocked_face_with_exploding_head:

I only restart my REPL about once a week so even a few minutes startup wouldn't really bother me. I'm curious about workflows where startup times of eight seconds is problematic @dharrigan?

I open the repl less then once a day. (it remains open)

I have a need, a need for speed 🙂 I guess I'm just learning about if by applying some of the recommendations in that web page I can shave off some time! No harm in trying to optimise if I can 🙂

Interesting, got it down to 3 seconds 🙂

I'll send you my bill

Errrr....would an I.O.U suffice?

I.O.U one beer, fresh from the tap, if when you get to Ireland/UK 🙂

I hope I can leave my house again someday

@dharrigan’s avatar is a duck… he already has a bill!

@alexmiller So I went and tried that dev startup time stuff (for the first time). I compiled a bunch of big libraries that my code relies on that rarely change, but did not compile my own code, then a ran a "load file" on my app's main file and it threw an exception about a clojure.core.cache.CacheProtocol not found... so I'm guessing there's a possibility for this pre-compile step to mess up in some way?

depends how you compiled everything

He's the Egyptian Household God of Frustration.

if you're trying to compile independent libs then you might not be compiling in the same order you actually load them in via your app

and then you can get into the case of recompiling a protocol, which can definitely go wrong

When I tried to repro with a (require 'api.main) instead of the load-file that the editor does, that seemed to work, so maybe it's something specific to load-file?

require will load too under the hood, but maybe

Ah, no difference. I started a fresh REPL, with those libraries precompiled and require failed this time, just as load-file had before.

So, you have to transitively compile "everything" to make this work, so that compile order doesn't matter?

TIL!: https://www.youtube.com/watch?v=im29S6ZWRDI

transitively compiling everything should work (b/c it's exactly what your app does when it loads)

What about a project where you have multiple entry point nses, that represent separate applications?

other things might work :)

you could use more than one alias/classes dir if you find that to be a problem

or you could still use just compile with one and use from the other entry point. you might miss some stuff but presumably it's mostly the same in deps.

Currently, I start a REPL once a week (roughly) and just load code into it as needed so I usually end up with our entire code base loaded into the REPL eventually.

Best Show Evar!

you are a bit outside the target user of this page :)

Compiling two different entry points that overlap might fall into the "recompiling a protocol" area?

depends

I am shorthanding the actual problem (recompilation itself is not the actual problem)

So the problematic piece (that surprises me a bit) is that (compile 'clojure.core.memoize) does not seem to transitively compile clojure.core.cache which I would have expected... I think that's the root of my earlier problem perhaps?

https://clojure.atlassian.net/browse/CLJ-1544

it should transitively compile anything loaded by clojure.core.memoize

but not everything in core.cache is used by memoize

When I look in the classes tree after (compile 'clojure.core.memoize) there are no core.cache classes at all.

what about if you compile clojure.core.cache? my kind of guess would be it is already compiled/loaded somewhere else in your repl

any chance you've already loaded something that has loaded core.cache prior?

compile piggybacks load so if it's not being loaded (b/c it already is), it won't be compiled

Ah, that is a possibility, because something in my dev tooling might well be using that... If I explicitly (compile 'clojure.core.cache) I do see the .class files produced for it which would suggest it isn't loaded previously?

(or am I misunderstanding your point?)

that is interesting, I wouldn't expect to see those if clojure.core.cache was previously loaded

I suspect you're getting the equivalent of :reload at the top level namespace you compile

but not on the ones below

Yup, confirmed. A "bare" REPL with just my dev tooling loaded has clojure.core.cache clojure.core.protocols clojure.core.server clojure.core.specs.alpha clojure.data.priority-map ... (non-`clojure.*` stuff omitted).

ah, right, compile calls load-one directly, bypassing the already loaded check

So, yeah, I'd need to explicitly (compile 'clojure.core.cache) to make all this work, right?

you can do what's in the bottom of that guide (kind of), which is to address a similar problem (when user.clj has already loaded stuff)

(binding [*compile-files* true] (require 'clojure.core.memoize :reload-all))

the only thing that makes load do compile is that dynvar

and the :reload-all will force a reload

I'm going to stick a list of (compile 'x.y.z) forms in a compile.clj file so I can do load-file on it to compile "everything" that I want pre-compiled and I'll add to it as/when I run into problems. FWIW, just with the set of libs I've put in that list so far, a cold (require 'api.main) has dropped from 18 seconds to 10 seconds. Like I say, since I only restart my REPL occasionally, that's really not a concern but now I'm curious about this idea of pre-compiling dependencies that rarely change...

...and if a dependency does change, presumably you'd have to force a recompile of the newer version (and everything that depended on it) in order to pick those changes up?

you don't have to

a newer dependency should have newer .clj files and be preferred

it will just be slower till you recompile

clojure takes the newer .class or .clj

I guess if you changed deps to a newer version of a lib that was still older than your local .class files that wouldn't have that effect

I spent a half day trying to build this into clj so it could do it automatically (since it knows when your deps go stale). it was tricky for a bunch of reasons but maybe a future feature

the point where your .cpcache is stale is also the point when you want to recompile

so was trying to hook that

Yeah, discussions about it being a possible future feature of clj was what made me even curious to try it, once it was brought up here.

relevant for tools that replace the classpath too