Clojurians Log v2

Hi, @seancorfield — I used depstar for the first time last week, and it’s so fantastic. I have no idea why I waited so long to try it! 🙂 (I had used lein for this before…) When creating an uberjar, depstar seems to re-download all the external dependencies from maven central, even though in an earlier Docker build step, I ran clj -Spath to pre-download them (presumably into .m2 directory?) Is my understanding of what is happening correct? And if so, is there a way to have depstar use .m2 for jars, instead of re-downloading? Thx!!!

@genekim depstar uses tools.deps.alpha under the hood so it’s behavior should be exactly the same as the Clojure CLI.

So I’d need more information about exactly what commands you are running and the project setup.

(and I’ve never seen it download deps that are already cached locally in ~/.m2 — which is what all of the underlying t.d.a stuff uses for both the Clojure CLI and for depstar)

What you might be seeing is tools.deps.alpha itself being downloaded (along with all of its transitive deps) since the CLI bundles all that into its uberjar.

So you probably want to run clojure -Sdeps -Spath to pre-download things.

…so strange… I’m staring at the build steps right now… Here’s what has me puzzled… Annotated build output: > Step #1: Step 15/17 : WORKDIR /src/pubsub-web > Step #1: ---> Running in 35dbc5d9db0f > > *clj -Spath* happened here... nothing happened, because deps.edn didn't change, so all jars are already downloaded... > > Step #1: Removing intermediate container 35dbc5d9db0f > Step #1: ---> 9781d01f38bc > Step #1: Step 16/17 : RUN make uberjar > Step #1: ---> Running in 4cbf3018663a > > uberjar generation happens here… note that all jars are downloaded again. Why? > > Step #1: clojure -X:uberjar :aot true :jar ./target/pubsub-web-standalone.jar \ > Step #1: :main-class web.server > Step #1: Downloading: com/github/seancorfield/depstar/2.0.206/depstar-2.0.206.pom from clojars > Step #1: Downloading: org/clojure/tools.deps.alpha/0.9.863/tools.deps.alpha-0.9.863.pom from central > Step #1: Downloading: org/clojure/tools.logging/1.1.0/tools.logging-1.1.0.pom from central > Step #1: Downloading: org/clojure/pom.contrib/1.0.0/pom.contrib-1.0.0.pom from central

In other words, you are not pre-downloading the dependencies that depstar itself needs.

Ohhhh…. -Sdeps -Spath !!! Trying it now.

Or clojure -A:uberjar -Spath to get everything depstar needs.

But the modern way to do that with the CLI is to use the “prepare” option.

clojure -P -X:uberjar ...

In other words, the exact same command you would use to build the uberjar except with -P at the beginning.

The CLI prepare option is designed for exactly this purpose (instead of using -Spath or similar).

OH RIGHT!!! I read about -P a couple of days ago, but I couldn’t find it this morning! (I spent an hour googling for “what is equivalent to ‘lein deps’ for clj”. 🙂

https://clojure.org/reference/deps_and_cli#_prepare_for_execution

I’ll add a note to the depstar README about CI usage and -P for future souls on this journey.

For the second time ever, I’m going to write up a gist to remind myself of an interaction we’ve had on Clojurians, in the hopes of helping something struggling with the same problem — the first time was for jdbc and mysql. 🙂

Search for stuff on Zulip chat — it’s all mirrored there and that has no limit on search history 🙂

https://github.com/seancorfield/depstar/issues/74

There’s a subtle issue there: if you run depstar and pass in aliases to it (perhaps to pin library versions, or some other “production” aliases for the JAR contents), you need to also have those aliases in the clojure -P command (as well as :uberjar or whatever your depstar alias is).

Make sense?

Yes indeed! That did the trick, and indeed, that was exactly what was required to eliminate the need to download jars at uberjar generation time:

# to pre-download dependencies, only done if deps.edn changes
RUN mkdir /tmp/pubsub-web
COPY ./pubsub-web/deps.edn /tmp/pubsub-web
WORKDIR /tmp/pubsub-web
RUN clj -P
RUN clj -A:depstar -P

Step #1: clojure -X:uberjar :aot true :jar ./target/pubsub-web-standalone.jar \
Step #1: 	:main-class web.server
Step #1: Downloading: com/cognitect/transit-clj/1.0.324/transit-clj-1.0.324.pom from central
Step #1: Downloading: com/cognitect/transit-java/1.0.343/transit-java-1.0.343.pom from central
Step #1: Downloading: javax/xml/bind/jaxb-api/2.3.0/jaxb-api-2.3.0.pom from central
Step #1: Downloading: org/msgpack/msgpack/0.6.12/msgpack-0.6.12.pom from central
Step #1: Downloading: javax/xml/bind/jaxb-api-parent/2.3.0/jaxb-api-parent-2.3.0.pom from central
Step #1: Downloading: net/java/jvnet-parent/5/jvnet-parent-5.pom from central
Step #1: Downloading: com/googlecode/json-simple/json-simple/1.1.1/json-simple-1.1.1.pom from central
Step #1: Downloading: org/javassist/javassist/3.18.1-GA/javassist-3.18.1-GA.pom from central
Step #1: Downloading: com/cognitect/transit-java/1.0.343/transit-java-1.0.343.jar from central
Step #1: Downloading: com/googlecode/json-simple/json-simple/1.1.1/json-simple-1.1.1.jar from central
Step #1: Downloading: org/javassist/javassist/3.18.1-GA/javassist-3.18.1-GA.jar from central
Step #1: Downloading: org/msgpack/msgpack/0.6.12/msgpack-0.6.12.jar from central
Step #1: Downloading: javax/xml/bind/jaxb-api/2.3.0/jaxb-api-2.3.0.jar from central
Step #1: Downloading: com/cognitect/transit-clj/1.0.324/transit-clj-1.0.324.jar from central
Step #1: [main] INFO hf.depstar.uberjar - Compiling web.server ...
Step #1: 2021-03-27 18:01:20.801:INFO::main: Logging initialized @6780ms to org.eclipse.jetty.util.log.StdErrLog

You’d have to dig into the dependency trees of things to figure that out I suspect… I don’t know what would bring those in… they are not from depstar.

I ran this in depstar’s repo folder:

(! 503)-> clojure -Stree|fgrep cognitect
  . com.cognitect.aws/api 0.8.505
    . com.cognitect/http-client 0.1.106
  . com.cognitect.aws/endpoints 1.1.11.969
  . com.cognitect.aws/s3 811.2.858.0

Is it possible you have dynamically required/resolved dependencies in your own code?

Huh. Maybe middleware is dynamically resolved? (This is definitely challenging my very shaky mental model of how dependency resolution actually works.) Filing this under “things to better understand someday.” 🙂

. metosin/reitit-middleware 0.5.12
    . metosin/reitit-ring 0.5.12
    . lambdaisland/deep-diff 0.0-47
      . mvxcvi/puget 1.1.2
        . mvxcvi/arrangement 1.2.0
        X fipp/fipp 0.6.17 :older-version
      X fipp/fipp 0.6.17 :older-version
      . org.clojure/core.rrb-vector 0.0.14
      . tech.droit/clj-diff 1.0.1
      . mvxcvi/arrangement 1.2.0
    . metosin/muuntaja 0.6.8
      . metosin/jsonista 0.3.1
      . com.cognitect/transit-clj 1.0.324
        . com.cognitect/transit-java 1.0.343
          X com.fasterxml.jackson.core/jackson-core 2.8.7 :older-version
          . org.msgpack/msgpack 0.6.12
            . com.googlecode.json-simple/json-simple 1.1.1
            . org.javassist/javassist 3.18.1-GA
          X commons-codec/commons-codec 1.10 :older-version
          . javax.xml.bind/jaxb-api 2.3.0

https://gist.github.com/realgenekim/97cd8da28b00761560bef06b8072eb9f