Clojurians Log v2

Hi all, I try to create an uberjar with depstar using the -X method. I.e. i updated my deps.edn to have an alias specifying a exec-fn and exec-arg. When running on my laptop clojure -X:uberjar a jar is generated just fine. However when I try to create the jar from a docker container it fails with

Execution error (FileNotFoundException) at java.io.FileInputStream/open0 (FileInputStream.java:-2).
-X:uberjar (No such file or directory)

clojure:openjdk-11-tools-deps-1.10.1.469
clojure:openjdk-11-tools-deps-1.10.1.561

you don't have the newest clj tools in your docker

you need clojure 1.10.1.697

@zetafish I'm planning an overhaul of my project's readme files this week to make that clearer (that some of the usage examples require 1.10.1.697). I also need to update all of the clj-new project stuff to generate projects that can work both ways.

what do you think about https://github.com/juxt/pack.alpha?

it has interesting feature skinny jar

Is clojure.libfile still present in 697?

I don't see it in the system properties when running code via 1.10.1.697 @dominicm -- there's clojure.basis now, which is the path to the combined EDN file.

I see it was still present in 590:

> /usr/local/Cellar/clojure\@1.10.1.590/1.10.1.590/bin/clojure
Clojure 1.10.1
user=> (System/getProperty "clojure.libfile")
"/Users/sean/.clojure/.cpcache/2731801087.libs"
user=> 

Ah, that could explain the bug I'm seeing. I thought it was being retained for backwards compatibility.

@dominicm Lots of breakage between the previous stable and the current stable 🙂 but if you change (System/getProperty "clojure.libfile") to (some-> (System/getProperty "clojure.basis") (str/replace #"\.basis$" ".libs")) you can still get at the old file (although, as I recall, the basis file now contains the libs list as an element?)

Yeah, (some-> (System/getProperty "clojure.basis") (slurp) (edn/read-string) :libs) should be the same data.

yeah, I removed it in 697. we've never publicly doc'ed it and I was not aware of anyone using it (other than add-libs which won't work with latest anyways) so I didn't think there was any reason to retain it

I was using it to find myself on the classpath and remove my paths

Thank you Alex. I presume the 697 image will become available on docker hub soon. In the meantime I built the image locally and the -X:uberjar works fine then.

I want a self contained jar that I can run with java -jar app.jar so skinny does not fit the bill.

I'd say separating the tool from the basis is the better long term path for that

we have an ongoing conversation about some details of that right now

I don't have anything to do with the docker stuff, so no control over that for me

If I could reliably identify my lib in the classpath, then that'd be fine 🙂. Happy to switch over to the newnew system when that comes out too.

A github search indicates I'm probably the only user anyway

We also use it in our internal tool based on the lib, but I can update that the same way

@dominicm removing yourself from the classpath: is this for an uberjar tool maybe? accepting a classpath argument that's built using -Spath may also work - depstar now also supports it

(although depstar still removes itself from the classpath even if you provide that option, I believe)

oh right - that doesn't seem necessary - building an uberjar with depstar in it could be useful

As the original author of depstar, I consider the approach of removing self from the classpath adorable, but a design mistake

Should use t.d.a API instead

I doubt it would cause any surprises if I changed the behavior to only omit depstar from the (current) runtime classpath but allow it as part of the -P / --classpath path 🙂

Using the t.d.a. API to read the various deps.edn files has problematic edge cases -- as we're seeing with -X:deps tree which includes t.d.a in the tree even when it isn't part of the project being analyzed.

It's also problematic for monorepo use cases -- where CLJ_CONFIG is used to point to a "master" deps.edn inside the repo, alongside each subproject's deps.edn -- which is something Datomic Ions folks do I believe (in addition to us relying heavily on it at work).

also, --classpath can be calculated using other build tools like lein, boot, maven, whatever, which makes it connectable with other ecosystems

Isn't -X:deps tree misreport deps because of the same design mistake?

clojure -Stree works correctly, clojure -X:deps tree does not.

Bugs aside, use the API

But that is because it uses the basis of its invocation instead of calculating the classpath using its capabilities to do so

(Aside, is CLJ_CONFIG isn’t public API?)

I don't have an argument against computing a classpath separately and providing it to a tool -- but that's orthogonal to the issue of figuring out the classpath for a tool to use. If t.d.a is enhanced to better support monorepos that need a way to have an additional deps.edn file in the mix (and then exclude the actual user deps.edn) I'll happily switch to that.

@ghadi Yes, CLJ_CONFIG is definitely a "hack", but it's a useful hack in the absence of t.d.a supporting an additional deps.edn in a monorepo context.

isn't the user deps.edn excluded by -Srepro?

as in user home dir

Right, but that excludes CLJ_CONFIG.

deps.clj has an option called -Sdeps-file: https://github.com/borkdude/deps.clj I haven't really thought about the consequences for relative paths, but if you don't use those, it should work

but ( cd ...; clojure ...) also works

my colleague also started using CLJ_CONFIG the same week he started using clj because of this limitation

just feedback, no criticism ;)

I wonder, if it's undocumented, he sure found it pretty quickly somewhere, I didn't tell him about it

The problem with that (getting a file from the command line) is that the desired order for repos such as ours is:

System -> User (happy to exclude with -Srepro) -> repo-master (this is missing) -> Project -> -Sdeps

Although, if you could specify an arbitrary list of files to read in order via -Sdeps-file (and they didn't have to be called deps.edn) then we could use -Sdeps-file ../versions/master-deps.edn:project-deps.edn and not have an actual deps.edn file in each subproject in the repo.

-Sdeps-path might be a more appropriate name and I'd be happy to support it in deps.clj (and change the name). It already does this now but for just one file

-Sdeps-overrides could also work as a name to indicate that it doesn't read any other files than these

but I'm relatively sure all of this can be done using scripting as well: merge the configs yourself, provide it via -Sdeps and move the local deps.edn aside (hacky)

also I'm relatively sure these things are in some row/column in an excel sheet somewhere on Alex's computer :)

Our deps files are pretty big -- too big to go on the command line I think as merged data -- and, as you say, that's a hacky approach.

(also, we're not about to switch from the official Clojure CLI to a third party tool at this point, esp when we would need multiple different binaries for different parts of our dev/test/staging pipeline, whereas we can have just one CLI install)

And, yeah, I've already brought this up with Alex (which was how I learned that it's similar to something the Datomic Ions users have asked for) so, yeah, I'll bet it's already in a spreadsheet. I don't know what I'd prefer in terms of a solution. As long as it addresses the deps ordering shown above and is officially supported, I'll be happy with whatever Cognitect/Nubank think is the best approach 🙂

@seancorfield small details: deps.clj also runs using clojure / JVM as a dependency but yeah I don't expect you to switch - I was just thinking out loud about this option. I'm blathering too much.

Ah, true, I'd forgotten there was a library version as well...

I just pushed 0.0.10 - there was something with the clojars deploy token, missed that. But now it works:

/tmp $ clojure -Sdeps '{:deps {borkdude/deps.clj {:mvn/version "0.0.10"}}}' -M -m borkdude.deps -Sdeps-file "$HOME/git/clojurescript/deps.edn" -Spath
src/main/clojure:src/main/cljs:resources:/Users/borkdude/.m2/repository/com/cognitect/transit-clj/0.8.309/transit-clj-0.8.309.jar:/Users/borkdude/.m2/repository/com/google/javascript/closure-compiler-unshaded/v20180805/closure-compiler-unshaded-v20180805.jar:...

(so I'm in /tmp, but I'm using deps.edn from clojurescript in $HOME/git/clojurescript)

Just putting this here for now, but there is a new prerelease clj available - 1.10.1.708. Has a small fix to -X error handling and more importantly an update on how dep expansion works when multiple transitive versions of the same dep are in the tree to fix a case where a dep can be incorrectly omitted.

The circumstances for the latter case are quite complicated so it's a little challenging for me to categorically say whether your deps are affected (they probably aren't), but if you were were interested in trying it, would be happy to have any feedback before I make it stable.