graalvm

Discuss GraalVM related topics. Use clojure 1.10.2 or newer for all new projects. Contribute to https://github.com/clj-easy/graal-docs and https://github.com/BrunoBonacci/graalvm-clojure. GraalVM slack: https://www.graalvm.org/slack-invitation/.
tvaughan 2021-01-30T01:13:50.074100Z

Not sure if anyone is interested in yet another example of how to compile a native image, but if so https://gitlab.com/tvaughan/kibit-runner#native-image. Only docker is required. For anyone looking to reproduce this without docker, Clojure 1.10.2 is required and for some reason this needs to be specified in deps.edn. A recent version of depstar is also required.

1๐Ÿ‘€1
ericdallo 2021-01-30T16:30:52.078500Z

ok, nrepl fixed using dynaload ๐Ÿ˜„ I tried to use SLF4j, log4j, log4j2 and logback, but all have issues with file appenders with graalvm... so I think timbre will be the only way I fixed all found issues with reflection, need to test all features still. I noticed that the first clojure-lsp classpath scan time using clj-kondo was increased, for a proect with 250 files to scan, while with JVM it takes 90s with graalvm takes 130s, not sure there is anything to improve that though. About the image size, I removed a lot of unnecessary reflection, still the image size is about 110MB while the jar has 30MB, I found https://github.com/ptaoussanis/timbre/issues/326 regarding timbre that adds 50MB to the image size but not sure is happening What you suggest to decrease the image size @borkdude?

borkdude 2021-01-30T16:32:45.078800Z

@ericdallo > while with JVM it takes 90s with graalvm takes 130s, not sure there is anything to improve that though. This is why I think the JVM makes most sense for projects like this, I keep recommending this for my own LSP server for clj-kondo

borkdude 2021-01-30T16:33:04.079Z

As for the image size, you can look at the analysis callgraph or use the dashboard options.

ericdallo 2021-01-30T16:36:58.079200Z

I see ๐Ÿ˜” Besides that, everything is pretty much fast, like diagnostics, code actions, refacotings which improve user xp IMO I still need to make the memory test, then we can think in the tradeoff.

ericdallo 2021-01-30T16:37:12.079400Z

How can I check the analysis callgraph?

borkdude 2021-01-30T16:37:50.079600Z

"-H:+PrintAnalysisCallTree"

borkdude 2021-01-30T16:38:01.079800Z

this will dump some information about the pulled in classes, methods, etc

ericdallo 2021-01-30T16:38:17.080Z

Nice, I'll try

borkdude 2021-01-30T16:38:23.080200Z

and there is also a dashboard option: https://www.graalvm.org/docs/tools/dashboard/?ojr=help%3Btopic%3Dgetting-started.md

borkdude 2021-01-30T16:38:41.080400Z

I haven't used the dashboard that much, but it seems useful. You can drill down into classes

lread 2021-01-30T18:50:09.082300Z

Just upgraded rewrite-clj v1 branch tests to use Graal v21.0.0. One more datapoint, @borkdude, for the https://github.com/borkdude/clj-reflector-graal-java11-fix not being needed!

lread 2021-01-30T18:52:46.084300Z

For my Graal sci test under JDK11, I did still need to include borkdude/sci.impl.reflector. Did not dig in at all, did you expect this still to be needed @borkdude?

borkdude 2021-01-30T19:04:22.084700Z

yes, this lib is needed, but already pulled in by sci itself?

ericdallo 2021-01-30T19:20:08.085200Z

probably yes, since I removed from clojure-lsp and still works

lread 2021-01-30T19:45:52.085900Z

Oh sorry, I meant the 0.0.1-java11 override version of borkdude/sci.impl.reflector for JDK11.

borkdude 2021-01-30T19:48:38.086300Z

Right, that one isn't needed anymore, since it has the same MethodHandle code

lread 2021-01-30T19:53:29.086800Z

Cool, thanks, Iโ€™ll retry without it.

ericdallo 2021-01-30T20:11:30.086900Z

@borkdude for some reason the dashboard flag just gives me OoO even with a -JXmx of 6GB :man-shrugging: I did some hacking to run Graalvm 21.0.0 on My NixOS (docker doesn't work) instead of 20.2.0 and the startup time fall from 130s -> 75s... it's faster than JVM classpath scan ๐Ÿ˜ฎ (~87-90s), I will make more tests, special the memomry one (not sure how to measure that yet)

ericdallo 2021-01-30T20:11:59.087100Z

Also, I added the new -H:+InlineBeforeAnalysis flag, not sure this could cause that performance improvement though

borkdude 2021-01-30T20:13:14.087300Z

@ericdallo Be sure to use the --no-fallback option so you won't get a fallback image, this is effectively just a JVM

borkdude 2021-01-30T20:13:41.087500Z

And yeah, the dashboard data may require moar memory

borkdude 2021-01-30T20:13:45.087700Z

it's a lot of data

ericdallo 2021-01-30T20:15:08.087900Z

yeah, I'm using the --no-fallback, I don't want that kind of lie anymore ๐Ÿ˜‚

borkdude 2021-01-30T20:15:43.088100Z

You could also post an issue about a performance regression on the graalvm issue tracker

ericdallo 2021-01-30T20:18:35.089Z

@borkdude sorry, I probably was not clear, I was using Graalvm 20.2 with all my tests because it's the only available for NixOS while this https://github.com/NixOS/nixpkgs/pull/105815 , also the docker image doesn't work with NixOS for some reason related with shared libs, but I managed to manually install the graalvm 21.0.0 in the nix way

ericdallo 2021-01-30T20:19:12.089700Z

so it's not a regression since I went from 20.2.0 -> 21.0 ๐Ÿ˜„

borkdude 2021-01-30T20:19:22.090Z

so 21 is faster right?

borkdude 2021-01-30T20:19:44.090200Z

ok, then it's good!

ericdallo 2021-01-30T20:20:23.090400Z

yep, at least for the only thing I noticed that it was slower than JVM, now it's not anymore ๐Ÿ˜„

lread 2021-01-30T22:18:33.092800Z

If I retry my rewrite-clj sci tests without overriding borkdude/sci.impl.reflector to 0.0.1-java11, native-image compile works fine, but when I interpret my tests I get java.lang.NoSuchMethodError: java.lang.reflect.AccessibleObject.canAccess exceptions. Iโ€™ll create a todo for myself to look at this sometime later.

borkdude 2021-01-30T22:19:18.093300Z

@lee I have encountered this error as well. Adding this method to the reflection config did the trick for me

borkdude 2021-01-30T22:19:48.093600Z

See comment here: https://github.com/seancorfield/next-jdbc/issues/157#issuecomment-770192263

lread 2021-01-30T22:21:02.094600Z

Huh, I thought I had tried that! Worth another shot, thanks!

borkdude 2021-01-30T22:21:09.094800Z

@lee I think this is because the method is looked up using reflection: https://github.com/clojure/clojure/blob/140ed11e905de46331de705e955c50c0ef79095b/src/jvm/clojure/lang/Reflector.java#L38

borkdude 2021-01-30T22:21:22.095100Z

(or something)

borkdude 2021-01-30T22:21:42.095400Z

so basically it's missed by the analysis and you have to help it a little bit

borkdude 2021-01-30T22:22:52.095800Z

This might become a very common config for projects that hit this path, so probably worth documenting

borkdude 2021-01-30T22:23:31.096300Z

[{"name": "java.lang.reflect.AccessibleObject",
  "methods" : [{"name":"canAccess"}]},
 ...

lread 2021-01-30T22:24:05.096600Z

Ya, Iโ€™ll add a note to clj-graal-docs.

borkdude 2021-01-30T22:29:52.097200Z

I think I'll copy your note to the reflector graal-fix repo then as well, since if you ran into that one, you'll probably now need this

lread 2021-01-30T22:32:53.097400Z

ah, good idea!

lread 2021-01-30T22:35:20.099800Z

Note that I did not need it when 1) compiling rewrite-clj and its tests, only when 2) compiling rewrite-clj for sci tests. But I donโ€™t use a reflection.json for 1, so maybe thatโ€™s the diff.

borkdude 2021-01-30T22:36:00.100400Z

yeah, sci uses this Reflector class to do interop

borkdude 2021-01-30T22:36:06.100700Z

so that makes sense

lread 2021-01-30T22:36:14.100900Z

Ah, right.

lread 2021-01-30T22:37:20.102100Z

reflection.json fix you shared above works for rewrite-clj sci tests, thanks!

borkdude 2021-01-30T22:38:17.102300Z

๐ŸŽ‰

lread 2021-01-30T22:39:53.103100Z

oh poo, I guess Iโ€™ll need to generate a different reflection.json for jdk8 than I do for jdk11.

borkdude 2021-01-30T22:40:28.103600Z

in babashka I generate these using a script

borkdude 2021-01-30T22:40:46.104200Z

different feature flags also need different reflection configs there

lread 2021-01-30T22:40:48.104300Z

yeah, I stole that at one point, maybe I just need to grab an update!

lread 2021-01-30T22:41:39.105Z

ok, good stuff, should not tax my wee brain too much! :simple_smile:

borkdude 2021-01-30T22:43:22.105500Z

there is only so much manual JSON or YAML my brain can handle before I switch to a clojure script

2๐Ÿ˜‚