Clojurians Log v2

I've been trying to get graalvm compilation working, but constantly hit a problem where at runtime the binary cannot find java.math.BigDecimal class. Tried googling that, but don't see anyone having the same problem. Is this a known limitation, or am I doing something wrong? I've tried using the clj.native-image package, but also building an uberjar with depstar and then calling native-image manually with native-image --no-fallback --initialize-at-build-time --no-server -jar hello.jar

@tomi.hukkalainen_slac Can you give the specific error? It might be related to reflection

I can get basic stuff working, but the exception is thrown when I try to call clojure.core/bigdec

In bb it works:

$ bb -e '(bigdec 1)'
1M

** ERROR: **
Exception: #error {
 :cause java.math.BigDecimal
 :via
 [{:type java.lang.ClassNotFoundException
   :message java.math.BigDecimal
   :at [com.oracle.svm.core.hub.ClassForNameSupport forName ClassForNameSupport.java 64]}]
 :trace
 [[com.oracle.svm.core.hub.ClassForNameSupport forName ClassForNameSupport.java 64]
  [java.lang.Class forName DynamicHub.java 1308]
  [clojure.lang.RT classForName RT.java 2212]
  [clojure.lang.RT classForName RT.java 2221]
  [clojure.core$bigdec invokeStatic core.clj 3640]
  [clojure.core$bigdec invoke core.clj 3635]
  [hello$heka_with_cost invokeStatic hello.clj 91]
  [hello$heka_with_cost invoke hello.clj 85]
  [clojure.core$comp$fn__5825 invoke core.clj 2573]
  [clojure.core$map$fn__5884 invoke core.clj 2759]
  [clojure.lang.LazySeq sval LazySeq.java 42]
  [clojure.lang.LazySeq seq LazySeq.java 51]
  [clojure.lang.RT seq RT.java 535]
  [clojure.core$seq__5419 invokeStatic core.clj 139]
  [clojure.core.protocols$seq_reduce invokeStatic protocols.clj 24]
  [clojure.core.protocols$fn__8168 invokeStatic protocols.clj 75]
  [clojure.core.protocols$fn__8168 invoke protocols.clj 75]
  [clojure.core.protocols$fn__8110$G__8105__8123 invoke protocols.clj 13]
  [clojure.core$transduce invokeStatic core.clj 6886]
  [clojure.core$transduce invokeStatic core.clj 6881]
  [clojure.core$transduce invoke core.clj 6872]
  [hello$read_heka invokeStatic hello.clj 118]
  [hello$read_heka invoke hello.clj 115]
  [cli_matic.core$invoke_subcmd invokeStatic core.cljc 546]
  [cli_matic.core$invoke_subcmd invoke core.cljc 525]
  [cli_matic.core$run_cmd_STAR_ invokeStatic core.cljc 589]
  [cli_matic.core$run_cmd_STAR_ invoke core.cljc 560]
  [cli_matic.core$run_cmd invokeStatic core.cljc 601]
  [cli_matic.core$run_cmd invoke core.cljc 591]
  [hello$_main invokeStatic hello.clj 135]
  [hello$_main doInvoke hello.clj 134]
  [clojure.lang.RestFn applyTo RestFn.java 137]
  [hello main nil -1]]}

Where the line 91 in my code is the call to bigdec

Ah, sorry, I don't know where to reply 🙂 Here?

ok fine

I think the bigdec fun has a reflective usage in the last branch:

:else (BigDecimal. x)

and you might be hitting that branch

how are you creating that bigdec

Threading a datum from clojure.data.csv/read-csv

You can likely solve this by incorporating java.math.BigDecimal in a reflection config

it's probably using a string then

So, from a string

Yeah

But on babashka even that works :thinking_face:

bb -e '(bigdec "1")' that is

this is because babashka has this class in its reflection config

Ah!

@alexmiller It seems bigdec is using reflection to create a BigDecimal from a string. Perhaps the function could add a special case for a string?

Had to do a bit of reading, but it works now! Thanks!

Seems like babashka has some kind of way to automatically catch reflection problems to generate the config? Have to investigate that for robust builds

:thumbsup:

@tomi.hukkalainen_slac It doesn't have that, but I do generate the reflection config programmatically using a script

It has a curated list of classes it includes for interop

Ok, so just test the binary carefully to catch those 😕

Yes. There is an agent which can help you find reflective usages, but it emits a lot of false positives for Clojure programs

However, there might be some useful clues in there

Also the startup time improved from 1.5s→7ms but actual processing went from 19s→51s. Though I knew that the JVM is optimized for long running processes, I was surprised that it hits that hard even with less than a minute of runtime.

well yes, the JVM has JIT which optimizes at runtime. a native-image is AOT-ed instead of JIT-ed.

the enterprise version supports profile guided optimizations though, so you can optimize your native image for specific usages

for anything longer running than 5 seconds the JVM probably makes more sense

GraalVM is also a JVM implementation so you could try to compare its performance with normal OpenJDK. It should be more performant there too, if not, they consider it a bug

Good point, I'll do that too

A lot of Clojure's reasonable performance is thanks to the JIT. Consider this edge case I came across https://github.com/bsless/clj-fast/issues/19

Using graalvm as JVM gets the processing time down to 13s

Seems like I don't actually need to set the -Dclojure.compiler.direct-linking=true though I had copied that from many examples at first. What does that really do, and when is it needed?

it can make the binary size somewhat smaller in some cases. also it's a performance optimization

in general what it does: it avoids var indirection when calling functions

e.g. (defn foo []) (foo). The latter call goes via the var #'foo. When enabling direct linking, it will no longer go through the var, but will call the function directly

I've seen it used with native-image, but that sounds useful even with just normal uberjars

its useful in any final artifact

although it comes with trade-offs

we used to have it on in production

but this also takes away some power e.g. for patching some var real quickly without re-deploying, which is probably a very niche case, but can be useful for a quick debugging session

Yeah, I've heard that some people leave nrepl or something open on every deployment for such

yep, we do that

But that idea gives me the heebie-jeebies

I also use it for things you would normally expose to UI or endpoint, e.g. run special tasks

greater power comes with greater risk, make your own trade-offs

Sure

Hm, the included musl libs in GraalVM don't have libz.a, but choosing that as libc will still use -lz. Have to figure out how to pass extra paths

@tomi.hukkalainen_slac For musl : https://github.com/lread/clj-graal-docs#static-linking-with-musl Also see https://github.com/babashka/babashka/blob/master/script/setup-musl

@thiagokokada and @rahul080327 can tell you more about it perhaps

Yeah, you need to build zlib using musl-gcc and install it at the correct location so the linker can find it

Remember to do a static build of zlib

Ah, I passed over that too quickly. I somehow read it so that installing the musl-tools will be enough, and didn't read through the PR

But of course the distro version of zlib is for glibc, not musl

https://github.com/lread/clj-graal-docs seems like a critical resource to get anything working, thanks for that. Wish I had found that first, not the numerous old blog posts

Musl installation is a bit too much of a hassle to do manually, so I'll defer that until I have a container set up for building the whole thing, so that everyone else can then just use that to get everything working across distros

@tomi.hukkalainen_slac You can find it useful regarding zlib and musl: https://github.com/FieryCod/holy-lambda/blob/master/docker/ce/Dockerfile

It gives at least 5% throughput improvement from my experience It also significantly reduces native-image build time

Thanks, I'll get back to this soonish, I hope

Just FYI, I think @raspasov or @smith.adriane was asking about this (can't remember) Someone on the GraalVM slack just mentioned they are going to production with a GraalVM native-image based iOS app: https://apps.apple.com/us/app/cospaces-edu/id1224622426

https://graalvm.slack.com/archives/CN9KSFB40/p1622977000077900

I was definitely interested in investigating how to target mobile with native-image. I got some good answers in the graalvm slack. I was able to get the HelloGluon example running on my phone this weekend.

Very neat.

@smith.adriane were you using Clojure + GraalVM to target iOS/Android?

I would like to.

@borkdude do you happen to know what language they were using to make the CoSpaces Edu app? Was it Java?

@smith.adriane yes 🙂 That would be neat; I have a hard time imagining how the interop would look like with something like SwiftUI, but it would be cool if it’s possible.

I don't know, ask them :)

objective c is pretty dynamic. I think interop could actually be pretty usable

Objective-C, yes. Swift is what is more of a question mark.

SwiftUI is only available from Swift. And it’s kinda neat the way it works, it’s more or less similar to React in terms of state management.

Not a super impressive program, but I was able to get clojure to run on my iPhone using native-image!

(ns com.phronmophobic.mobiletest)

(defn clj-sub [a b]
  (- a b))