Clojurians Log v2

@seancorfield Would it be possible to exclude clojure and clojure.spec.alpha sources/AOT-ed assets from an uberjar? The use case would be for babashka which already has clojure built-in (spec not yet, but maybe coming). So an uberjar with only (non-clojure) source files and no AOT-ed stuff would be leaner. https://github.com/borkdude/babashka/issues/535#issuecomment-674518153 Since you're reading from a system property here https://github.com/seancorfield/depstar/blob/b7e9e4537e33bbf78fbdbc0ef56d9675cb494bc2/src/hf/depstar/uberjar.clj#L283 it might be a small tweak to override the classpath with a command line arg or file reference (java itself uses the @ convention for that).

E.g.:

clojure -A:depstar -m hf.depstar.uberjar MyProject.jar -v -m proj --classpath "$(clojure -Spath)"

Btw, I think depstar will be able to compile with GraalVM so maybe I can even include it in babashka itself to create babashka uberjars

I’d like to distribute a deps.edn file of aliases for everyone to use in our company. But I want people to be able to have their own local ./clojure/deps.edn. Is there a way to specify a deps.edn file to merge in, either globally for all projects, or from a projects deps.edn?

no, not currently

Is it possible to add a list of jar files from a project sub-dir to the classpath without having to name them? I know about :local/root and I see this in the docs https://clojure.org/guides/deps_and_cli#local_jar but I have a lot of them and they aren't published anywhere. Do I just have to make up names for them and add them individually as :deps?

It’s kind of a hack, but java supports on the classpath for this (literal path/ so not shell expanded). I think if you added a :paths entry for “path/*” that should work

thank you @alexmiller I will try that out

amazing, thank you Alex, it works

@seancorfield Awesome: depstar works with GraalVM and can hence be included in babashka proper so it can make its own uberjars

$ bb -cp $(clojure -Spath) --uberjar foo.jar -m proj
Building uber jar: foo.jar
$ bb -jar foo.jar
:foo - failed: int?

@borkdude Cool. As for your earlier question, depstar deliberately just includes everything on the current classpath, aside from itself.

That was a fundamental design decision I regret

In retrospect I’d use t.d.a as a library to calculate what to tar up

rather than subtracting depstar itself from the classpath

yep. but that can be easily changed. I've already done that in my fork

You using t.d.a to calculate instead?

To be honest, using the classpath has made depstar a lot easier to use for people, and a lot more predictable.

@ghadi I just added a :classpath option that can be used to override

https://github.com/borkdude/depstar/blob/d3fb96c3731d4b7987e75927e6c13497da22fae2/src/hf/depstar/uberjar.clj#L360

Hmm, interesting approach. Since it's easy to get clojure to produce an arbitrary classpath from deps.edn aliases etc...

Exactly.

this is a similar thing to what I’m trying to avoid, vectors of strings

So the classpath given to babashka, produced by clojure is just passed down to depstar:

$ bb -cp $(clojure -Spath) --uberjar foo.jar -m proj

I understand. I’m just saying the shell is not leverage compared to a clojure library. One of the things I want to do is split a list of dependencies by their rate of change for smarter distribution to containers or Lambdas

You mean, make uberjars for things not changing often?

For containers, you don’t want every commit to result in a 60MB new uberjar, you want some structural sharing across commits

I guess one could use the -A or -T option for that, to make groups of deps and then push that classpath into depstar

Yeah I know you can hack it in shell, but why not program it directly in bb or clojure?

my hope with tools.build (not that I have anything to do with it currently) is that it encourages build tasks to compose better than the lowest common denominator of the CLI

bb will have an --uberjar option, but it can also have a bb -e '(babashka.uberjar/run {:classpath ...})-ish interface

can have them both

Wait are tools.build out?

no

depstar currently has zero dependencies, except Clojure itself, so it can't interfere with anyone's project dependencies -- that's worked very well in practice. I like @borkdude’s idea of just giving it an explicit classpath to build form 🙂

bb won't do AOT btw (maybe it could but I don't think it's worth it). for JVM uberjars (with AOT-ed classes), just use clojure / depstar

bb uberjars typically will contain sources and other assets, but no classes

Unless I want to have the whole build happen in a single vm, and it has to have other build-specific dependencies like data.xml for updating pom.xml. I use 2 vms for that now, but I'd rather not :)

I did have depstar compile with GraalVM as is and when I didn't remove the AOT-related code, it did compile, but the binary got somewhat bloated (+ 30MB, happens when it hits a specific part of Clojure). So maybe compile just works natively, since it's just going to transform .clj into classfiles, I haven't tried it.

I'll try it now... just for science

I'm not sure whether you're agreeing with depstar's approach or suggesting something different?

Interesting. Compilation of a class doesn't work in a GraalVM binary because the classpath inside the binary is not set correctly for clojure RT. I guess it won't work for that reason, although technically it could maybe work.

Going to try this:

(binding [*compile-path* (str tmp-c-dir)]
          (clojure.lang.RT/compile (str main-class ".clj")))

crap, it's private ;)

With a reflective hack it works. I see all kinds of AOT-ed sourced, even aot__init.class, but no aot.class. Maybe :gen-class isn't part of RT/compile? Eh.... enough for today.

So the AOT part could be useful when depstar had a binary CLI version, so to speak. I don't need it for bb, but it could be useful for fast build tooling in general.

To make this work clojure.lang.RT should also have a pluggable classpath resolver. It might work for Clojure files, but I wonder how it will work if code relies on .class files from the Java world. Does the Clojure compiler kind of treat .class files as data, without actually having to have them on the classpath per se, as long as you point it to the right file? I.e. no runtime need, just able to read and analyse statically?

I see, Compiler relies on reflection when dealing with interop, so there it will likely break, since the reflection API only works with classes that are part of the same runtime. OK, this was a nice tangent.