Clojurians Log v2

When building a uberjar using clojure -A:depstar -m hf.depstar.uberjar MyProject.jar, I get a long list of "clashing jar item" warnings like

{:warning "clashing jar item", :path "org/mozilla/javascript/ContextListener.class", :strategy :noop}
{:warning "clashing jar item", :path "org/mozilla/javascript/ContinuationPending.class", :strategy :noop}
{:warning "clashing jar item", :path "org/mozilla/javascript/DToA.class", :strategy :noop}
{:warning "clashing jar item", :path "org/mozilla/javascript/Decompiler.class", :strategy :noop}
{:warning "clashing jar item", :path "org/mozilla/javascript/DefaultErrorReporter.class", :strategy :noop}
{:warning "clashing jar item", :path "org/mozilla/javascript/DefiningClassLoader.class", :strategy :noop}
...

And why are the strategies ":noop"?

They're warnings, so you can ignore them. It's just saying that the file was found in multiple dependencies that you have. :noop means it didn't try to do anything. For some file type it will merge/concatenate the files.

You can use the -S / --suppress-clash option to hide those messages if you want.

I'm probably going to make that the default and and an option to display them only as needed since a lot of people seem to be very concerned about "warnings" if they don't understand classpaths and dependencies can contain duplicates.

Hmmm. The above command did generate a "MyProject.jar", however, trying to run it with java -jar MyProject.jar resulted in an error:

Error: Invalid or corrupt jarfile MyProject.jar

You didn't specify a main class in your JAR to execute, and you didn't compile it.

If you read the README, it explains the various ways to make JAR files and how you can run each one. In particular, for the command you ran

clojure -A:depstar -m hf.depstar.uberjar MyProject.jar
java -cp MyProject.jar clojure.main -m project.core

I just updated the README to make it clear that folks need to read more of the information there if they want to use java -jar to run their uberjar.

It seems to be working now, thanks!

Yeah, a bit of heads-up will be tremendously helpful for us newbies 🙂

See if the updated readme helps now.

(I'm working on making the clash warnings off by default)

Is there a way to manually deal with the item clashes? I just found out that another package's index.html appears in my public, which is going to create problems...

@seancorfield The update is very clear and helpful. Thanks!

Unfortunately, there are some libraries that have been incorrectly packaged -- they shouldn't contain things that would conflict with user's normal files (like public/index.html) so you should report a bug against the library that is doing that.

(and this is partly why depstar has previously defaulted to providing this warning)

Got it. Is there a way to find out which one is the offending package?

There was a discussion about this a week or two ago in one of the channels here because some library was causing exactly that problem, with a stray index.html file 😐

I mean in this case its relatively obvious once I read the index.html, but when it's not obvious is there a way to know which package it is?

You can use the -v option on depstar to tell you exactly what files are being copied in and that should tell you -- it will be large output so you'll want to redirect it to a file.

Got ya. Wouldn't it be more helpful if the warnings themselves can point them out? Just a thought

By the time it sees a conflict, it no longer knows where the previous file came from.

And conflicts are not always a problem.

The real issue is that the order of things on the classpath is not always deterministic.

So it's OK to have the same file present in multiple dependencies, but it's sometimes not OK to have the same filename with different contents.

(and this is true of Java classpaths in general -- this isn't a Clojure issue)

Got ya, so basically, if all is done correctly, namespaces should never clash unless they literally are the same thing

@seancorfield But I mean, that's exactly why it would be great if the warning can point me which package it is -- so that I can know who is being naughty (even just let me know the second one would be great, which I assume depstar has the reference when the clash happens).

@zhuxun2 The information about which dependency is being processed is not available by the time the warning is issued.

(and it's just as likely that the one that "causes" the conflict would be seen as your code, depending on where it was in the classpath)

So -v is the best way to debug that.

I'll probably cut 1.1.115 tomorrow with the changes made so far. That will make it harder to detect problematic index.html files, by the way, since the warning will be off by default.

(dinner time here)

@seancorfield Thanks a lot!

`seancorfield/depstar {:mvn/version "1.1.116"}` is available https://github.com/seancorfield/depstar/releases/tag/v1.1.116 -- it suppresses the clashing jar item warning by default and adds -D / --debug-clash to enable the warnings.

Getting an error with 1.1.116 (that I don't get with 1.1.104) building an uberjar w/ some exclusions on clojure 1.10.1.561: {:error "unable to copy file", :name "clojure/tools/reader/default_data_readers.clj", :exception java.lang.ClassCastException, :message nil}. I'm invoking it like this:

:main-opts ["-m" "hf.depstar.uberjar" "-X" "fluree/db/.*\\.clj[cs]?$" "target/fluree-ledger.standalone.jar" "-C" "-m" "fluree.db.server"]

Happy to turn all that into a GitHub issue if you'd like.

@cap10morgan Sorry. Roll back to 1.1.104 and I'll fix it a.s.a.p.

No worries. And thanks!

Can you share your deps.edn? I'm having a hard time reproducing the bug (although I think I know what is causing it).

I think that tools.reader might be ending up on your classpath twice (but I'm not sure how).

If you want to just share it privately, you can email details to <mailto:sean@corfield.org|sean@corfield.org>

And if you could run that failing command on 1.1.116 with -D and -v options added to produce all the debug output, that would be really helpful @cap10morgan

sure I can email it to you

I'll try it again w/ -v and -D too

(I think the clash strategy logic is seeing default_data_readers.clj and mistakenly treating it like data_readers.clj -- bug in my regex -- and then trying to merge it... which of course fails... but I'm not sure how you'd get tools.reader on your classpath twice 😐 )

emailed

Got it!

OK, yeah, it's finding default_data_readers.clj twice and trying to merge them which is the bug I thought it might be. I can't figure out why it's on the classpath twice (and, strangely, the verbose debugging output doesn't tell me so I'll need to dig into that some more and figure out what I'm not not displaying in verbose mode -- but that's a separate issue).

Try 1.1.117 which I just released to Clojars.

@cap10morgan ^

will do!

Looks like it's fixed! Thanks!

OK, seancorfield/depstar {:mvn/version "1.1.117"} is available https://github.com/seancorfield/depstar/releases/tag/v1.1.117 -- it suppresses the clashing jar item warning by default and adds -D / --debug-clash to enable the warnings. This fixes an edge case bug in 1.1.116 to do with tools.reader as a dependency.