Clojurians Log v2

So, with the link you posted above, would eg compiling all my deps (and not my source code) help on startup time?

@slipset yes, we did this and saw improvement.

Thanks! I’ll check that out. I guess my next, naive, question would be: Is there a way to compile a jar?

Tried this out in both java8 and java11 with clojure 1.10.1 and [clj-commons/pomegranate “1.2.0”] - and find the behavior hard to explain - in the repl - using lein 2.9.3. Wonder if anyone has any deeper insight on the classloader trickery happening: ---- These DO NOT work. hickory isn’t found on classpath. (1)

(p/add-dependencies
 :coordinates '[[hickory "0.7.1"]]
 :repositories (merge {"clojars" "<https://repo.clojars.org/>"}
                      pa/maven-central))

(require 'hickory.core)

(p/add-dependencies
 :classloader (clojure.lang.RT/baseLoader)
 :coordinates '[[hickory "0.7.1"]]
 :repositories (merge {"clojars" "<https://repo.clojars.org/>"}
                      pa/maven-central))

(require 'hickory.core)

(def cl (clojure.lang.RT/baseLoader))

(p/add-dependencies
 :classloader cl
 :coordinates '[[hickory "0.7.1"]]
 :repositories (merge {"clojars" "<https://repo.clojars.org/>"}
                      pa/maven-central))

(require 'hickory.core)

(let [cl (clojure.lang.RT/baseLoader)]
  (p/add-dependencies
   :classloader cl
   :coordinates '[[hickory "0.7.1"]]
   :repositories (merge {"clojars" "<https://repo.clojars.org/>"}
                        pa/maven-central)))

;; NOTE: NOT INSIDE the `let`
(require 'hickory.core)

(let [cl (clojure.lang.RT/baseLoader)]
  (p/add-dependencies
   :classloader cl
   :coordinates '[[hickory "0.7.1"]]
   :repositories (merge {"clojars" "<https://repo.clojars.org/>"}
                        pa/maven-central))
  ;; NOTE: INSIDE the `let`.
  (require 'hickory.core))

it’s pretty weird to me. The case where the classloader is in a let-binding seems to matter. It makes it “used by require”

but no other time. Notably, not even a var holding the loader matters (case (3))

I would try #2 with both expressions in a let with no bindings

(let [] ...)

I forget the exact details but when evaluating/compiling code the compiler will often push a new dynamic classloader binding, which is sometimes what RT/baseLoader returns

so my guess is what you are seeing is differences in behavior based on if the add-dependencies expression is compiled and run with the same classloader on that stack as the require expression

ahhh ok

I think what your saying makes sense

I wasn’t thinking enough about clojure.lang.RT/baseLoader returning different things

you’r probably rigth I’m getting the wrong case

not 100% sure, wrapping in a let like that will make the two part of the same compilation unit, without all the explicit classloader stuff, so if that fixes the issue it is likely what is happening

yeah, and perhaps I should use clojure.lang.RT/makeClassLoader as well to test

which would always give me a dynamic loader

regardless of baseloader

I doubt that will do anything

but the compilation unit binding Compiler.LOADER is a prime suspect

the issue isn't likely with what you are explicitly passing to pomegranate, but what the compiler is implicitly doing behind the scenes

I've had to fiddle with requires with pomegranate in the past(the behavior changed around clojure 1.8 I think), but never dug into why

so doing case (2) in a let-binding as you said works

as expected

eg:

(let []
  (p/add-dependencies
   :classloader (clojure.lang.RT/baseLoader)
   :coordinates '[[hickory "0.7.1"]]
   :repositories (merge {"clojars" "<https://repo.clojars.org/>"}
                        pa/maven-central))

  (require 'hickory.core))

Then my repl loader permanently can refer to these new classes after too

(as before)

and

(p/add-dependencies
 :classloader (clojure.lang.RT/makeClassLoader)
 :coordinates '[[hickory "0.7.1"]]
 :repositories (merge {"clojars" "<https://repo.clojars.org/>"}
                      pa/maven-central))
(require 'hickory.core)

oh yes, I think I can

require is just not occurring in a compilation unit that would know of this new one

So basically, if we can get “loading” (eg. require) to happen in the same compilation unit as using clojure.lang.RT/baseLoader to add the deps; we’ll be dealing using that same mutated loader from pomegranate

I think once we do the load; it is safe to just use the stuff after that point within the repl

meaning, I don’t know of a reason I’d need to hold onto the loader and keep binding it

to close it out

you can also do

(def cl (clojure.lang.RT/makeClassLoader))
(with-bindings {clojure.lang.Compiler/LOADER cl}
  (p/add-dependencies
   :classloader cl
   :coordinates '[[hickory "0.7.1"]]
   :repositories (merge {"clojars" "<https://repo.clojars.org/>"}
                        pa/maven-central))

  (require 'hickory.core))

then again, using with-bindings may be making them both in the same compilation unit still

so probably should break all the way into push thread bindings around multiple separate forms to see it in most raw

no, I doubt you need to do any explicit classloader stuff at all

(def cl (clojure.lang.RT/makeClassLoader))
(with-bindings {}
  (p/add-dependencies
   :classloader cl
   :coordinates '[[hickory "0.7.1"]]
   :repositories (merge {"clojars" "<https://repo.clojars.org/>"}
                        pa/maven-central))

  (require 'hickory.core))

fails

so it does get compiled in different context

stop passing in a classloader

fails too

(with-bindings {}
  (p/add-dependencies
   :coordinates '[[hickory "0.7.1"]]
   :repositories (merge {"clojars" "<https://repo.clojars.org/>"}
                        pa/maven-central))

  (require 'hickory.core))

fail

wrap it in let instead

I get let works

I’m trying to show the underlying mechanism of why let works

the sharing of the clojure.lang.Compiler/LOADER

(let []
  (p/add-dependencies
   :coordinates '[[hickory "0.7.1"]]
   :repositories (merge {"clojars" "<https://repo.clojars.org/>"}
                        pa/maven-central))

  (require 'hickory.core))

you have to refer to baseloader

don’t think pomegranate does (by default)

@mikerod This seems to be all accidental complexity introduced by pomegranate -- using add-lib for this sort of thing "just works".

@seancorfield add-lib being an unreleased fn right?

(! 504)-> clj -A:deps -r
user=> (require '[clojure.tools.deps.alpha.repl :refer [add-lib]])
nil
user=> (add-lib 'hickory {:mvn/version "0.7.1"})
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See <http://www.slf4j.org/codes.html#StaticLoggerBinder> for further details.
true
user=&gt; (require 'hickory.core)
nil
user=&gt; 

:deps ; to get access to clojure.tools.deps.alpha.repl/add-lib
  {:extra-deps {org.clojure/tools.deps.alpha
                {:git/url "<https://github.com/clojure/tools.deps.alpha>"
                 :sha "19d197ab221d37db750423eb970880cb87a91100"}}

@seancorfield they use similar strategys. This doesn't just work with add-lib when using lein/add-lib

yeah pomegranate isn’t really doing much fancy here either

There's a long outstanding cider issue about this.

I think it recently got closed

just trying to add to a classloader when it has the ability to be added to - via a protocol

I'd love to know what the deal is with COMPILER, because that stumped me.

COMPILER?

Sorry, LOADER. hazy memory

It makes sense above

but it’s sort of digging

you can do similar with the context classloader instead

but you are somewhat relying on the fact that clj require (`load`) will use a RT/baseLoader

So the problem here is... what exactly? I'm confused. You show something related to Leiningen / Pomegranate that doesn't work in situations you think it should... and that's a bug/problem in... what? I'm not following...

which will in turn use the bound LOADER or context loader if you set that etc

But why does lein have particular issues with LOADER. I couldn't get it to trigger under clojure.

Right, this is feeling like a lein/CIDER problem, not a Clojure problem.

It’s just that in some repl’s the classloaders are new each eval

or something to that extent

so if you use something like pomegranate incorreclty - you may mutate the wrong loader

and not really have ability to use that loader for something like require subsequently

above there is outlined approaches for that to not be the case

it’s something that happens in lein, not really a “problem”

So that's... a bug in lein?

just something you sort of have to understand if mutating a loader

Hmm. I fixed that in nrepl though

you have to be sure you are using that mutated loader

I gave them a common base.

(sorry, I'm not trying to be difficult, I'm just trying to see how this relates to #clojure-dev )

I wanted to understand the underlying mechanism for why a given compile-context seemed to change my result

I don’t care that I was using a lein repl to get to that state

so to me - relates directly

you can have different repl impl’s some may or may not end up the same - depends on how they manage classloaders etc

I’m not saying there is even a problem, was just baffled by how let seemingly caused a difference for no reason

and conclusion was: it formed a “single compilation unit”

where the Compiler.LOADER was bound

which affects the clojure.lang.RT/baseLoader return val

could perhaps have been find in #clojure channel

hard to say when it’s pretty internal-digging

Why does that change the base loader? That's been my question for years :)

static public ClassLoader baseLoader(){
	if(Compiler.LOADER.isBound())
		return (ClassLoader) Compiler.LOADER.deref();
	else if(booleanCast(USE_CONTEXT_CLASSLOADER.deref()))
		return Thread.currentThread().getContextClassLoader();
	return Compiler.class.getClassLoader();
}

first if

then you can see in clojure.lang.Compiler.eval() how this LOADER is bound

so if you do some mutation to it - via pomegranate and also end up doing a load while still in that same eval - you are working against same loader at the right time

likely the reason lein's repl behaves different is the clojure.main/repl sets the context class loader to be a dynamic classloader

don’t have to rely on let though - you can just use with-bindings ; you also don’t even need LOADER; you can set the USE_CONTEXT_CLASSLOADER and achieve similar

Yeah, never looked into why lein does whatever it does with loaders

lein likely does nothing

and it isn't really on lein

the issue is with the server side of the nrepl it creates

https://github.com/nrepl/nrepl/commit/c3ec4c699e4c9ac439fda822daeb0dda17370c9a was merged recently

FWIW, of your original five cases, (1) does work in the regular clj REPL (and it also worked for me in a plain lein repl) (2), (3), and (4) do not work, and (5) does work. -- @mikerod Does (1) not work for you in lein repl?

yeah, the server is really from nrepl

and it does have middleware setting the ccl and LOADER etc - so things are being done

it likely all depends on the mix of nrepl and clojure versions being used

If (1) works for you in lein repl, then I'm much clearer about what you're asking -- which is about the (2), (3), and (4) cases which don't work in either REPL and so that points to an isolated class loader issue (am I close / caught up now?)

@seancorfield all above was for lein repl

and (1) was in the “doesn’t work” list

so whatever is happening in the repl there - which really relates to nrepl - didn’t end up sharing the loader across forms

perhaps do could work, but think that may be split

Ah, so you have some middleware/plugin that is breaking (1) in lein repl (as well as the three common cases)...

so have to use let or something to keep it grouped

yes, many variables involved concerning what repl is used and what that repl does

And again: I posted the examples trying to just get an idea of what mechanism was causing the the weird difference I saw when using let.

I’m not reporting like a bug or issue - to be clear

was exploratory

good news is - it makes sense how to make pomegranate work now in lein repl’s (to me at least)

I think add-lib will hit similar situations in lein

looks to do something quite similar with how it mutates the loader

I haven't used lein for much of anything since late 2015, so I was just very puzzled when I tried your example (1) in a plain lein repl and it worked... hence my confusion! 🙂

hmm weird if it worked

but yeah, I know you seem to prefer the alt stack

(! 513)-> lein version
Leiningen 2.9.3 on Java 14 OpenJDK 64-Bit Server VM
(! 514)-> lein repl
nREPL server started on port 52427 on host 127.0.0.1 - <nrepl://127.0.0.1:52427>
REPL-y 0.4.4, nREPL 0.6.0
Clojure 1.10.0
OpenJDK 64-Bit Server VM 14+36-1461

@mikerod next version of nrepl has a workaround for this

Same as me @seancorfield besides I tried java8 & 11 (but those shouldn’t change the loader setup of the nrepl server)

@dominicm interesting; I’ll have to see - the current repl still can work as is - just have to be more careful about things, but would be nice to not have to

still no idea why Sean’s is working haha

not for this case:

(p/add-dependencies
 :coordinates '[[hickory "0.7.1"]]
 :repositories (merge {"clojars" "<https://repo.clojars.org/>"}
                      pa/maven-central))

(require 'hickory.core)

the issue is the version of nrepl, the lein version is a red herring

oh, guess I somewhat cut off too [cemerick.pomegranate.aether :as pa] for the maven repo

@hiredman lein version directly chooses the nrepl version though?

it’s part of it’s deps

and I’m using 2.9.3

it is overridable

which is this https://github.com/technomancy/leiningen/blob/2.9.3/project.clj#L21

I’m not overridding

I would double check

unless Sean is or something (doubt it since he doesn’t use lein)

I am not sure how one would do that though, I try deps :tree, but the nrepl version might be part of some profile thing, dunno

yeah, I’m using nrepl "0.6.0" as well

in deps tree