Clojurians Log v2

I’m getting up to speed with the deploy process using tools from the greater tools.deps ecosystem and I ran into a strange issue today. Essentially, I was finding that libs installed locally did not bring their transitive dependencies along with them. I used clojure.tools.cli.api/mvn-install to do the install.

A bit of investigation shows that mvn-install does not copy the pom file to the local repo -interestingly @seancorfield has observed that slipset/deps-deploy does install the pom file. So is that a bug or a limitation/design decision in c.t.d. mvn-install?

@alexmiller ^

@alexmiller I confirmed with @cch1 that running clojure -X:deps mvn-install omits the .pom file and when trying to use that dependency in a clean folder, t.d.a. reaches out to Clojars to look for the "missing" .pom file. When I re-ran the same scenario but using slipset/deps-deploy for the local install, it did copy pom.xml to <library>.pom in ~/.m2 and when trying to use that in a clean folder, t.d.a. did not reach out to Clojars to look for the .pom file. Definitely feels like a bug in t.d.a's mvn-install to me.

can you back up to the full command you're running?

are you supplying a pom? using the one in a jar file? supplying additional attributes?

There is a pom.xml in the same directory. I ran just clojure -X:deps mvn-install :jar '"the-lib.jar"' -- do we need extra args to also copy the pom.xml file? It reads the pom.xml file to get the group/artifact/version data...

The same pom.xml file is inside the JAR as well (so that's the one it reads, based on the reference docs).

Ah, if I say :pom '"pom.xml"' as well, then it does copy that to <library>.pom -- is that an intentional difference @alexmiller?

Here's a full repro session:

(! 692)-> rm -rf ~/.m2/repository/seancorfield/
(! 693)-> jar tvf next-jdbc.jar |fgrep pom
   110 Mon Oct 19 11:24:06 PDT 2020 META-INF/maven/seancorfield/next.jdbc/pom.properties
  1996 Mon Oct 19 11:24:06 PDT 2020 META-INF/maven/seancorfield/next.jdbc/pom.xml
(! 694)-> clojure -Sforce -X:deps mvn-install :jar '"next-jdbc.jar"'
Installing next-jdbc.jar 
Installed to /Users/sean/.m2/repository/seancorfield/next.jdbc/1.1.610
(! 695)-> ls -lR ~/.m2/repository/seancorfield/
total 0
drwxr-xr-x  4 sean  staff  136 Oct 19 17:51 next.jdbc

/Users/sean/.m2/repository/seancorfield//next.jdbc:
total 8
drwxr-xr-x  4 sean  staff  136 Oct 19 17:51 1.1.610
-rw-r--r--  1 sean  staff  305 Oct 19 17:51 maven-metadata-local.xml

/Users/sean/.m2/repository/seancorfield//next.jdbc/1.1.610:
total 96
-rw-r--r--  1 sean  staff    164 Oct 19 17:51 _remote.repositories
-rw-r--r--  1 sean  staff  44300 Oct 19 11:24 next.jdbc-1.1.610.jar
(! 696)-> rm -rf ~/.m2/repository/seancorfield/
(! 697)-> clojure -Sforce -X:deps mvn-install :jar '"next-jdbc.jar"' :pom '"pom.xml"'
Installing next-jdbc.jar and pom.xml
Installed to /Users/sean/.m2/repository/seancorfield/next.jdbc/1.1.610
(! 698)-> ls -lR ~/.m2/repository/seancorfield/
total 0
drwxr-xr-x  4 sean  staff  136 Oct 19 17:52 next.jdbc

/Users/sean/.m2/repository/seancorfield//next.jdbc:
total 8
drwxr-xr-x  5 sean  staff  170 Oct 19 17:52 1.1.610
-rw-r--r--  1 sean  staff  305 Oct 19 17:52 maven-metadata-local.xml

/Users/sean/.m2/repository/seancorfield//next.jdbc/1.1.610:
total 104
-rw-r--r--  1 sean  staff    188 Oct 19 17:52 _remote.repositories
-rw-r--r--  1 sean  staff  44300 Oct 19 11:24 next.jdbc-1.1.610.jar
-rw-r--r--  1 sean  staff   1996 Oct 19 11:16 next.jdbc-1.1.610.pom
(! 699)-> ls -l pom.xml
-rw-r--r--  1 sean  staff  1996 Oct 19 11:16 pom.xml

I see Installing next-jdbc.jar in the first case and Installing next-jdbc.jar and pom.xml in the second case so I suppose it is telling you what it is doing...

Without the .pom file in the repo, attempts to use the dependency will still reach out to Clojars/Maven for that .pom file and if it isn't found, it doesn't look like the pom.xml inside the JAR is sufficient for t.d.a to pick up the dependencies.

it will only use the pom if you pass it explicitly (doc string: https://clojure.github.io/tools.deps.alpha/clojure.tools.cli.api-api.html#clojure.tools.cli.api/mvn-install)

reaching out to maven is surprising though, so I'll look at that tomorrow

Downloading: seancorfield/next.jdbc/1.1.610/next.jdbc-1.1.610.pom from clojars

Feel free to DM me tomorrow if you need me to run more test cases.

(and there was no attempt to download the .jar file, only the .pom file)

sorry, maven looking for the pom file is normal behavior, I thought you were saying it was reaching out for the pom file when installing

Right, but that will fail if there is only a local install -- and the problem Chris was seeing was that if no .pom file existed, transitive dependencies were not being found.

well that's to be expected

the question is what you supply on install

you have to either supply a pom, or use the pom in the jar, or is this the case where you did neither?

There is a valid pom.xml inside the JAR -- I showed that above.

sorry, too much reading, and I'm a drink into the night :)

The problem is that the .pom file is not created in the local repo on the install. Unless you explicitly specify :pom '"pom.xml"'

ok, that's surprising to me, should be using the one in the jar in that case

OK, we can follow-up tomorrow. Feel free to DM me about it.

See my console session above.

yeah, that code is not there. it is using the pom in the jar to pull G/A/V but it's not actually installing it as a distinct artifact. not sure why I thought that would happen automatically, maybe tricked myself with some caching or something

So, definitely a bug? Want me to create a JIRA?

definitely a bug, sure on jira

should be easy to fix

https://clojure.atlassian.net/browse/TDEPS-169

seancorfield/depstar {:mvn/version "1.1.132"} -- calls shutdown-agents at the end to account for "badly-behaved" code being AOT'd that has side-effecting top-level forms that start threads (e.g., Neanderthal) -- follow-up in #depstar

Sorry I’m late catching up here. I don’t have much more to offer than what Sean noted above. Let me know if I can help test.

Clojure tools 1.10.1.723 (a prerelease) is now available: • Fix clj -X:deps tree adding tools.deps.alpha to tree • Fix clj -X:deps mvn-pom adding tools.deps.alpha to pom deps • Fix clj -X:deps git-resolve-tags not working • Fix clj -X:deps mvn-install on jar to also install embedded pom

the first 3 were actually all a bug in the default :deps alias using :extra-deps rather than :replace-deps (duh)