Clojurians Log v2
integrant
@thenonameguy: care to continue this discussion here?: https://clojurians.slack.com/archives/C06MAR553/p1584046078221000?thread_ts=1584045635.219900&cid=C06MAR553
@imre: FYI I think if you have a problematic-ns as a dependency, you are also a problematic ns. Alex Miller seemed to indicate that clojure assumes all AOT’d namespaces have also AOT’d their transitive namespaces. I think this is the problem I had with class not found exceptions.
the loop recur should take care of the transitive problematic thing, however the possibility that there are errors in my code is non-zero
ok good to know — I didn’t read through the code; just thought I’d mention it incase it was the issue.
sure
So the goal is: reduce startup time by compiling namespaces which don’t change frequently
Yes
Assuming you have an integrant config which has a distribution of 80% library keys - 20% application keys
which I think is pretty common, ofc depending on application size
you only want to compile the 80%, the things that are supplied by :duct, your-common-service-lib
probably the easiest way it is to figure out all the ns’s from the integrant config, and remove ‘project’ ones with a set of regexes
However I’m not sure I agree with your point here: https://clojurians.slack.com/archives/C06MAR553/p1584092550229200?thread_ts=1584045635.219900&cid=C06MAR553 You might be right… but I’m not sure you are. I was thinking the point is to have a set of compiled namespaces for when you next start the repl. Any namespaces that are cached on disk, will only be recompiled if necessary.
reset after the first one is already pretty fast; and is already only recompiling when things change (in memory). I think what makes reset slow subsequent times is normally more the work done in the ig/init-key definitions; rather than any compilation caused by (re)load-namespaces.
This is fixed by using (suspend|resume)-key.
yup
Sorry I should’ve restated this:
> So the goal is: reduce startup time by compiling namespaces which don’t change frequently
I think the goal is to reduce startup time by compiling essentially all namespaces each time. compile already does a quick check of modified times to see if it’s necessary to rebuild.
So I don’t the see need to do what you suggest
yeah, that’s the difference between our goals
I was just taking Alex’s suggestion into account
Ok, I’d missed that… Not sure why that is necessary though. He doesn’t explain the reasoning.
Probably there is some overhead in doing this
But as always, I’m okay with a naive impl first
let’s see what the drawbacks are ourselves
If that’s the goal though why involve integrant at all? Just look at deps.edn and compile all your dependencies.
Admitedly that will give you 80% of the speed
because to do this efficiently, you have to compile the ns-es before they are loaded
otherwise you have to compile and require :reload all of them
like user.clj in the blogpost
(will return to this afk)
> If that’s the goal though why involve integrant at all? Just look at deps.edn and compile all your dependencies.
I was talking nonsense here; you obviously need to identify the root namespaces and the require’s for your dependencies.
You suggested a way to do it at the top of this thread; however I think it has a problem too, which is that it loses the structural tree of require’s; so if compile is expensive you’ll be compiling namespaces multiple times unnecessarily too, i.e. your sets are unordered so if a requires b you might compile b first, then a.
What I think you want to do is first find the roots of all dependency subtrees that aren’t in your :paths; then call compile on each of those.
I think to do that you might want to use rewrite-clj or perhaps the metadata emitted from clj-kondo; so you can do so purely syntactically without requiring at all.
yup, the clj-kondo way sounds good
new borkdude project in 3..2.. 😂
actually it’s pretty trivial to do just use clj-kondo to generate that list — and then manually paste the ns’s into compile
clj-kondo --lint src --config '{:output {:analysis true :format :edn}}' | jet --pretty --query ':analysis :namespace-usages' | less Gets you half way there
then the question is how do you integrate that into your running app without incurring the java clj-kondo dev dependency
just shell out to a babashka script which outputs the ns’es
or something like that
I agree that’s nicer in principle; but it’s not really much of a chore to run some process once to generate the class files. Would only need to re-run periodically, after deps.edn changes.
right, could be just a ~/.clojure/deps.edn alias
Yeah I suppose you don’t need to use the graal build of clj-kondo
clj -A:faster-startup
oh, well you probably want to compile :test/`:dev` alias ns’es as well
probably something more like clj -A:your:normal:aliases -M:compile-deps
jinx!
alright, which one of us is writing this over the weekend? 😄
unlikely to be me I’m afraid
😞
okay, I’ll take a stab at it
can’t leave home bc of covid anyway
me neither — but I have a 9month old daughter
good priorities 🙂 I’ll report back here and/or DM you
FYI: https://clojurians.slack.com/archives/C03S1KBA2/p1584103144055700
ok, so we don’t need anything integrant specific
then we can just load all the aliases, load all the namespaces + dynamically require ones from the config, compile them, then reload the ns’es/restart the jvm
Could do; but I was thinking it was potentially in support of my first thought.
i.e. that we may have read too much into Alex’s mention of not compiling project ns’s.
If you don’t care about the distinction, you can use integrant to find the root keys as I was suggesting, and use each of those as a starting point for compile.
Look in #clojure but I tried using clj-kondo and manually stripping out my app namespaces and a few other problematic ones; and ran into classnotfoundexceptions.
following this
I looked a bit into the clojure source and it turns out compile is just this:
(defn compile
"Compiles the namespace named by the symbol lib into a set of
classfiles. The source for the lib must be in a proper
classpath-relative directory. The output files will go into the
directory specified by *compile-path*, and that directory too must
be in the classpath."
{:added "1.0"}
[lib]
(binding [*compile-files* true]
(load-one lib true true))
lib)(defn reset []
(stop)
(binding [*compile-files* true]
(clojure.tools.namespace.repl/refresh-all :after `start)))reset did populate the classes folder however the next call to reset died with a
:type java.lang.NoClassDefFoundError,
:message "mount/core/DerefableState",I haven't investigated any further but it appears there are certain namespaces where compile might be problematic to use
yeah there certainly are namespaces where compile is problematic — ones where a side effect is done on load, can cause problems… though I’d argue they’re usually problematic anyway. Mount might well fall into that bracket; depends on the nature of the side effect — but IIRC when I last looked at mount (years ago) I disliked it for these kind of reasons (and others).
I know, I had similar reservations about mount, but I have to use it in a few projects where startup/reload time improvements would be welcome
Yeah we have a few legacy projects using it; though I certainly wouldn’t encourage its use in greenfield work when there is integrant.
agreed
this seems to be able to get a good coverage of all the namespaces a project uses:
(->> {:platform @(requiring-resolve 'clojure.tools.namespace.find/clj)
:add-all? true}
(@(requiring-resolve 'clojure.tools.namespace.dir/scan-dirs)
{} @(requiring-resolve 'clojure.tools.namespace.repl/refresh-dirs))
(:clojure.tools.namespace.track/deps)
((juxt #(->> % :dependents keys)
#(->> % :dependents vals (apply concat))
#(->> % :dependencies keys)
#(->> % :dependencies vals (apply concat))))
(into #{} cat)
(sort))you could apply a filter to that and compile what remains
nah, that will still try to transitively compile stuff that's potentially removed
not 100% sure about how good an idea a tools.namespace approach is for this — haven’t really thought it about. But you could possibly call clear to handle removals.
actually using tools.ns is probably a bad idea — Though I guess it depends on whether you run it in a different process to gen classes before the dev repl process. If so it’s probably ok, otherwise I doubt clear will unload compiled classes for example.
I think that bigass juxt is causing the problem,
why I used tools.ns is that I'm trying to retrofit this into the reset fn of a mount/tools.ns project
startup time is the smaller thing in this case as there aren't a lot of things that the user ns loads. most stuff is only loaded when I call reset the first time
of course — sorry slack collapsed the thread; and I forgot you’d mentioned that earlier!
and I'm not too keen on keeping a list of namespaces to compile somewhere when that could be parsed out
hmmmm
it appears namespaces that s/def specs could also be problematic
on a second thought, the issue I'm having now could be something else
All right, I'm abandoning this for now. Unfortunately I keep running into errors when I'm attempting a tools.namespace refresh-all after having started the system with compiled classes. Here's where I'm at if someone wants to pick it up: https://gist.github.com/imrekoszo/b4ee78d69e71c520b0392886a9d511b7