Clojurians Log v2

Ran into an interesting issue while working on depstar 2.0 today: I'd been assuming that you could use :replace-deps to add it as a dependency now that it calculates the project basis from the deps.edn files (and the :aliases you provide it with), and that works great for almost everything... except where it does AOT compilation, which has to be done in the context of the project's code so that the necessary dependencies are available. I looked at what Leiningen does and it seems to set up a specific classloader context for running compilation (based on the project's dependencies etc).

Although calculating the project basis causes all the project's dependencies to become available (locally) for inclusion in the uberjar, they are not on the classpath used to invoke depstar if :replace-deps is used.

Thoughts/suggestions? What I've done for the time being is just documented that if you ask depstar to do AOT-compilation, you'll need to use :extra-deps instead of :replace-deps when specifying it as a dependency. What options do I have if I wanted to run clojure.core/compile in a different classloader context?

(that's not ideal because then you run the risk of depstar's transitive dependencies affecting your project's compiled code in some way)

start another JVM just for the compilation from within depstar

that’s the approach I’m taking on a similar tool

@ghadi Seems like a bit of a sledgehammer to crack a nut but it is also straightforward I guess. Are you shelling out? If so, how do you decide what command to use? Or is there a JVM-specific spawning process?

Hmm, with ProcessBuildler, that's not bad. And it can generally assume java -cp ... will work I guess?

Separate JVM is what I did in tools build

You can’t run clj if java doesn’t work so ...

And the clojure script sets JAVA_CMD so I can rely on that too.

Here's what I ended up with https://github.com/seancorfield/depstar/blob/develop/src/hf/depstar/uberjar.clj#L541 (it's all due for a big refactor -- most of today has been focused on just getting all the 2.0 functionality working and the next job is creating automated tests and then refactoring)

I see the output of -Stree has changed with the latest cli version. Are there some docs on what the notation used means?

@kenny X means that version of the dependency is not being used. The keywords at the end of some lines indicate why a particular version was used or not.

X means it's not being used because it's overridden by something else? It seems that last keyword is only every present on X's.

Example from one of our services:

X compojure/compojure 1.6.2 :use-top
    X ring/ring-core 1.6.3 :older-version
      X ring/ring-core 1.6.0 :older-version
      X ring/ring-core 1.6.0 :older-version
    X ring/ring-core 1.7.1 :older-version
  X worldsingles/lowlevel /Developer/workspace/wsmain/clojure/lowlevel :use-top
    . ring/ring-core 1.8.2 :newer-version
        X org.clojure/java.classpath 0.3.0 :older-version

Oh I see. What does it mean when a dep doesn't have a keyword?

That's just a normal selection of the version.

As in all deps have the same version of the dep?

The end result is just one set of versions. If there's no conflict of versions, there will be no annotation I believe.

Right. That's what I'm seeing. Just making sure it was the correct interpretation. This new output is great. How come it isn't documented anywhere? Alpha?

Although:

X org.clojure/java.classpath 0.3.0 :older-version
    . org.clojure/java.classpath 1.0.0

I expect @alexmiller can provide more detail when he's around...

It’s a bit more complicated than that, and there are some cases where inclusions list a reason too

Maybe I will doc it today :)

I omit it in cases where it’s uninteresting

Like finding a new dep

Or finding same dep version already selected

In that case, why wouldn't it be annotated with :newer-version like in your first example @seancorfield?

@kenny If it finds version N and later finds version N+1, I would expect it to annotate both, and then if it continues to find version N+1 it will not annotate those (since they are "finding dep version already selected")

If it only finds version N across all deps, it's not going to annotate either.

At least, that's my understanding of what @alexmiller just said.

it doesn't annotate both because it's a single pass breadth-first expansion

(or rather to be clear, all choices have a reason, I'm just suppressing some of them in the printing b/c they're not interesting)

when we first see a lib (version N) during expansion, it will have reason :new-dep

if you later find same lib version N+1, it will have reason :newer-version

but there are a lot of tricky cases

right now the reason codes :new-top-dep, :new-dep, and :same-version are suppressed

I'm getting a different resolution for a project that uses tools.reader 1.3.2 and pulls in a mvn lib that depends on tr 1.3.4 (I'm getting tools reader 1.3.4 in this case which I'm happy about), when using that lib via local/root (then it pulls in 1.3.2)

Is that expected?

(I did try -Sforce)

I will just remove the tools.reader dep from the project since I always want to use what the local/root project is using anyway

not enough info there for me to tell

local/root dep changes are not automatically picked up, so you do need to -Sforce if those change

if you have an output from -Strace or -Stree, happy to take a look

oh yikes, I see what is happening. the mvn lib has AOT-ed class files... which should not happen.

Found the culprit: apparently when you have a :main in the top-level of defproject lein will compile on deploy ... which might make sense. I just used that main for testing some stuff. Problem solved.

@kenny @seancorfield new page on dep expansion, tracing, tree printing https://clojure.org/reference/dep_expansion

undoubtedly more to say, but that is a dump of several things have only been in my head for a long time so felt good to get them out :)

How do you get to this page without the link?

It linked in two places on the deps/CLI reference page.

Ah. Isn't it odd how it's not in the left nav though?

It's not a reference in and of itself: it's just extra material for an existing reference. Not every page is directly linked in the left nav.

The deps/CLI reference page is the entry point. The new page is linked from dep expansion under the Resolve dependencies section of Operation and also from tree printing under Other Programs (for -X:deps tree).

Oh, okay. I could potentially see that being confusing for someone who is directly sent this link. That person has no way to "back out" to contextualize 🙂

hi, what is the equivalent in deps.edn of :java-source-paths ["src/java/"] from leiningen ? I would like to add deps.end for clj-antlr so I can use it from git. The java directory contains java sources: https://github.com/aphyr/clj-antlr

@eugen.stan no equivalent. You'll have to use javac yourself first.

thanks, I keep finding these edge cases 🙂 . Do you have any examples?

@eugen.stan He means you literally run the javac command yourself to compile the .java files to .class files.

This isn't an "edge case" -- this is just something that the Clojure CLI was not intended to do, by design. Leiningen is sort of a "kitchen sink": it has a whole bunch of tools included that have nothing to do with just "running Clojure code" (which is what the Clojure CLI is designed for).

People are building "tools" that mimic some of the stuff that Leiningen does (or that is provided by one of the many Leiningen plugins) and those "tools" are just small Clojure programs that you run via the Clojure CLI.

When you look at the general design principles behind Clojure -- and a lot of the things that Cognitect folks have produced -- there's a deliberate choice to keep things simple: each tool should do one job.

Someone could build a tool to compile Java source code -- a tool you could run from the Clojure CLI -- but you'd still need to run that tool and then run other tools and/or the CLI itself. There's still no "task pipeline" for the Clojure CLI. Maybe tools.build will provide that "task pipeline"? We'll have to wait and see what the Cognitect folks produce.