Clojurians Log v2

pgmig was updated from clojure 1.8.0 (!) to clojure 1.10.2-alpha1: https://github.com/leafclick/pgmig/commit/68864abe8e94910402e6a01d7de88f5d0cdfc504

1.10.2-alpha1

yes

@mkvlr I have a frontend rest service running in native image:

[tiketguard:26315]    classlist:  10,791.61 ms
[tiketguard:26315]        (cap):   1,308.11 ms
[tiketguard:26315]        setup:   3,329.41 ms
[tiketguard:26315]   (typeflow): 323,689.19 ms
[tiketguard:26315]    (objects):  78,563.42 ms
[tiketguard:26315]   (features):  10,511.45 ms
[tiketguard:26315]     analysis: 419,706.99 ms
[tiketguard:26315]     (clinit):   1,737.86 ms
[tiketguard:26315]     universe:   6,046.41 ms
[tiketguard:26315]      (parse):  11,295.08 ms
[tiketguard:26315]     (inline):  13,751.14 ms
[tiketguard:26315]    (compile):  65,125.46 ms
[tiketguard:26315]      compile:  95,394.01 ms
[tiketguard:26315]        image:  14,677.93 ms
[tiketguard:26315]        write:   1,736.10 ms
[tiketguard:26315]      [total]: 552,193.88 ms

it seems to work normally with 1.10.2-alpha1 as well

hmm, no premature statement

it compiles though 🙂

I think it culprint is some of the deps having spec in its deps (and not in the latest version)

shouldn't 1.10.2-alpha1 pull in the latest spec, or does it get overriden somehow?

clj -Spath or lein classpath should tell you the answer

could also be that one dep of yours is AOT-ed with clojure 1.10.1 or so

spec.alpha-0.2.187.jar

what error are you seeing?

I'll have to investigate. But the message is what I see when "wrong" spec is in com.oracle.svm.core.jdk.UnsupportedFeatureError: Invoke with MethodHandle argument could not be reduced to at most a single call: java.lang.invoke.Invokers$Holder.invoke_MT(Object, Object, Object, Object)

@katox that's a JDK11 related thing.

there are deps that don't include stuff when using clj 1.8

try jdk8

graalvm-xxx-java8

yet pgmig works with it

yeah, it depends whether your program will hit that piece of code I think

It seems so. Bunch of code is spec related and thus not included when using clj 1.8.

See: https://github.com/lread/clj-graal-docs#jdk11-and-clojurelangreflector and https://github.com/oracle/graal/issues/2214

That'd make sense. I tried to avoid reflection calls in pgmig but that's not feasible in the larger app. I'll try to play again with it soon

the patch works well

for now I'll try again with graal on java8

@borkdude good tip, the reflector error is gone on java8 graal. There are still problems in the final binary but the build is on track now.

I think I found a better fix for the clojure.lang.Reflector issue with java11! https://github.com/oracle/graal/issues/2214#issuecomment-596598421 No need anymore to patch Clojure itself by overriding the class.

I updated some more deps and now the full app works with java8 graal!

I'll make some proper docs around this java11 fix later

@borkdude is the above a configuration class that can be included in the project along with json configuration? Because I tried the java configuration code before and ran into more troubles.

@katox This is the fix in action: https://github.com/borkdude/sci/commit/4c8190c0e95860ece8e8b195b05d31ad0866d763 you need to compile this with svm.jar on the classpath. The commit shows that.

Oh that’s interesting @borkdude, so with Java substitutions, we can monkey patch Clojure!

yes, at least the Java bits

so if Clojure is going to have more Java-version specific bits, we can maybe work around it this way

yeah… neat! Another tool in the chest!

I think oracle provided some maven svm artifact. You could put it as a native image profile dependence.

this one https://mvnrepository.com/artifact/org.graalvm.nativeimage/svm/19.3.1

ah nice. I'll use that then

Very impressive @borkdude.

it is GPL licenced. is this what the classpath exception is for, i.e. when I compile my binary with this, I can still distribute it?

So am I reading this right. This works both graal java 8 and java 11. And keeps full functionality in both.

yep

I think I could even make a library out of this

I'll try this later this week

You have been busy! How did you learn about @TargetElement? Just reading source code?

it came up in another issue. See https://github.com/oracle/graal/issues/2136 towards the end.

This is from the native-image chat:

Michiel Borkent  [4:22 PM]
Is the svm.jar available as a maven library so I can put it on my classpath via a package manager?


Michiel Borkent  [4:38 PM]
it seems it is, but not updated for newer versions? <https://mvnrepository.com/artifact/com.oracle.substratevm/svm> (edited) 

David Lloyd  [4:49 PM]
the group ID changed to org.graalvm.nativeimage
[4:49 PM]
<https://search.maven.org/search?q=svm>
<http://search.maven.org|search.maven.org>
The Central Repository Search Engine
The Search Engine for The Central Repository

Michiel Borkent  [4:50 PM]
oh thank you!

Peter Niederwieser  [4:50 PM]
afaik the jar can't currently be used with openjdk.

Ah yeah, I see. Very nice. This is so much better.

But you don't use it with openjdk. You use graal and the graal's native image to create and link the binary. And the binary on linux is linked to some gnu libc stuff.

glibc (just that lib) is also LGPL not GPL.

the gnu library that normally causes linking license problems is libreadline (its GPL)

You're right. Still there is no openjdk involved unless you use that on CI server.

I'll process this fix into proper docs and maybe a lib later this week(end)

Just FYI native-image compilation times for the FE service app - various -J-Xmx settings

16G
[tiketguard:21475]      [total]: 280,823.65 ms
10G
[tiketguard:30357]      [total]: 874,997.50 ms
8G
java.lang.OutOfMemoryError

for reference the uberjar size is about 40megs (got a heavy cut to actually compile and run as a native-image)

Here's the library: https://github.com/borkdude/clj-reflector-graal-java11-fix

Feedback welcome. If it also works for you, I'll drop the alpha label

It worked for sci.

https://github.com/borkdude/sci/commit/c7ee869aff204703e3abae52c4fba00880b9ed47

Interesting speedup going from 10 to 16 G