graalvm

Discuss GraalVM related topics. Use clojure 1.10.2 or newer for all new projects. Contribute to https://github.com/clj-easy/graal-docs and https://github.com/BrunoBonacci/graalvm-clojure. GraalVM slack: https://www.graalvm.org/slack-invitation/.
Karol W贸jcik 2021-06-11T06:19:21.088300Z

Is it possible to specify list of reflection entries which should be ignored when generating native-configurations using GraalVM agent? It's makes me crazy that I have to each time manually trim some of the entries. PS: I'm looking for automated way, so that I can built it into #holy-lambda

borkdude 2021-06-11T06:39:28.090400Z

@karol.wojcik there is something even better that I鈥檓 going to try out today. You can ignore the place where the reflection happened, so if you specify the place in the Clojure compiler it should only leave the real problematic cases

1馃憤
borkdude 2021-06-11T06:41:28.091300Z

But you can also have access based filters

Karol W贸jcik 2021-06-11T06:46:05.091500Z

Nice!

Karol W贸jcik 2021-06-11T06:46:47.091700Z

Can wait to see the progress in that area. Please notify me when you make any progress. I'm eager to help with this!

borkdude 2021-06-11T08:21:45.092400Z

@karol.wojcik This filter works reasonably well:

borkdude@MBP2019 ~/Dropbox/temp/refl $ cat filter.json
{ "rules": [
  {"excludeClasses": "clojure.**"},
  {"includeClasses": "clojure.lang.Reflector"}
]
}
borkdude@MBP2019 ~/Dropbox/temp/refl $ cat src/refl/main.clj
(ns refl.main
  (:gen-class))

(defn refl-str [s]
  (.length s))

(defn -main [& _]
  (println (refl-str "foo")))

borkdude@MBP2019 ~/Dropbox/temp/refl $ java -agentlib:native-image-agent=caller-filter-file=filter.json,config-output-dir=. -cp $(clojure -Spath):classes refl.main
3
borkdude@MBP2019 ~/Dropbox/temp/refl $ cat reflect-config.json
[
{
  "name":"java.lang.String",
  "allPublicMethods":true
},
{
  "name":"java.lang.reflect.Method",
  "methods":[{"name":"canAccess","parameterTypes":["java.lang.Object"] }]
},
{
  "name":"java.util.Properties",
  "allPublicMethods":true
}
]

borkdude 2021-06-11T08:22:12.093Z

not sure where the java.util.Properties thing comes from though, but it's only one thing

Karol W贸jcik 2021-06-11T08:22:17.093100Z

馃槷 馃く

Karol W贸jcik 2021-06-11T08:22:55.093300Z

I'll take a look how it behaves for some more complex cases. Probably we would need some kind of filtering for resources as well.

borkdude 2021-06-11T08:25:01.093600Z

I'm mostly interested in the reflection config though

borkdude 2021-06-11T08:37:50.094Z

I made an issue at graal here to make this process a bit easier: https://github.com/oracle/graal/issues/3461

borkdude 2021-06-11T08:47:48.094600Z

ah, this is probably already supported! trace-output=foo

borkdude 2021-06-11T08:49:23.095200Z

and indeed it logs:

{"caller_class":"clojure.lang.Reflector", "args":[], "function":"getMethods", "tracer":"reflect", "class":"java.util.Properties"}
Not sure why Clojure does this reflection but ok

borkdude 2021-06-11T08:55:33.095700Z

@karol.wojcik So using this trace-output option I found out you can get the same config using only:

{ "rules": [
  {"excludeClasses": "clojure.lang.RT"}
]
}

1馃く2鉂わ笍
borkdude 2021-06-11T08:55:40.095900Z

That's it :)

Karol W贸jcik 2021-06-11T08:55:51.096200Z

@borkdude You must be kidding me! 馃槃

borkdude 2021-06-11T09:00:15.096900Z

I would just leave out `{ "name":"java.util.Properties", "allPublicMethods":true } ` though, as this class pulls in XML libraries and bloats your image. I'm not sure why Clojure does this reflection during the lifecycle of a program, but I guess that's for another time.

borkdude 2021-06-11T09:02:34.097300Z

but this can be configured using an access filter as well

borkdude 2021-06-11T09:20:34.097800Z

Hm, this config is better it turns out:

{ "rules": [
  {"excludeClasses": "clojure.**"},
  {"includeClasses": "clojure.lang.Reflector"}
]
}

Karol W贸jcik 2021-06-12T10:23:35.106900Z

Huh. I read GraalVM configuration once again. I know what you mean now. I will try to implement this filtering! Thanks @borkdude

borkdude 2021-06-12T10:24:47.107100Z

I'm also experimenting with it

1鉂わ笍
borkdude 2021-06-12T12:46:00.107500Z

@karol.wojcik Got something here: https://clojurians.slack.com/archives/C8NUSGWG6/p1623501901111500

Karol W贸jcik 2021-06-12T13:06:16.109200Z

I will try it out tomorrow morning! Getting ready for some party 馃巿. Wish you a great day @borkdude !

borkdude 2021-06-11T09:30:39.098Z

Updated docs: https://github.com/lread/clj-graal-docs#reflection

borkdude 2021-06-11T11:11:26.100400Z

I think I found the spot where the false positive java.util.Properties reflection comes from. https://github.com/clojure/clojure/blob/b1b88dd25373a86e41310a525a21b497799dbbf2/src/clj/clojure/core.clj#L7085 Would be nice if this could be resolved, not sure if the core team is open for this change, because for Clojure itself it's not that important.

1馃憤
borkdude 2021-06-11T11:17:05.100700Z

cc @alexmiller

Karol W贸jcik 2021-06-11T11:21:05.101Z

Looks quite easy to fix. I'll try to make a patch for it

borkdude 2021-06-11T11:22:32.101200Z

I think the fix is not the issue, it's more if it will be accepted or not ;)

1馃槃
Karol W贸jcik 2021-06-11T11:26:20.101500Z

Ah right!

alexmiller 2021-06-11T12:21:08.101900Z

Sure, make a jira

borkdude 2021-06-11T12:22:56.102100Z

ok, I'll make a JIRA + patch once I confirm locally that this solves the "false positive" in the graal generated config

1馃憤
borkdude 2021-06-11T12:23:08.102300Z

thanks

Karol W贸jcik 2021-06-11T13:34:08.102700Z

It's not a silver bullet for everything it seems. For instance compiling project with cognitect.aws lacks some of the reflections:

{:cognitect.anomalies/category :cognitect.anomalies/fault, :cognitect.anomalies/message Class java.nio.HeapByteBuffer[] is instantiated reflectively but was never registered. Register the class by using org.graalvm.nativeimage.hosted.RuntimeReflection, :cognitect.http-client/throwable #error {
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 :cause Class java.nio.HeapByteBuffer[] is instantiated reflectively but was never registered. Register the class by using org.graalvm.nativeimage.hosted.RuntimeReflection
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 :via
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [{:type java.lang.IllegalArgumentException
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 :message Class java.nio.HeapByteBuffer[] is instantiated reflectively but was never registered. Register the class by using org.graalvm.nativeimage.hosted.RuntimeReflection
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 :at [com.oracle.svm.core.graal.snippets.SubstrateAllocationSnippets arrayHubErrorStub SubstrateAllocationSnippets.java 295]}]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 :trace
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [[com.oracle.svm.core.graal.snippets.SubstrateAllocationSnippets arrayHubErrorStub SubstrateAllocationSnippets.java 295]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [clojure.lang.RT seqToTypedArray RT.java 1754]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [clojure.lang.RT seqToTypedArray RT.java 1750]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [clojure.core$into_array invokeStatic core.clj 3440]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [cognitect.http_client$map__GT_jetty_request invokeStatic http_client.clj 108]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [cognitect.http_client.Client$fn__10581 invoke http_client.clj 226]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [cognitect.http_client.Client submit_STAR_ http_client.clj 226]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [cognitect.http_client$submit invokeStatic http_client.clj 213]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [cognitect.aws.http.cognitect$create$reify__10605 _submit cognitect.clj 10]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [cognitect.aws.http$submit invokeStatic http.clj 45]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [cognitect.aws.client$send_request$fn__10177$state_machine__5488__auto____10204$fn__10206$fn__10220 invoke client.clj 97]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [cognitect.aws.client$send_request$fn__10177$state_machine__5488__auto____10204$fn__10206 invoke client.clj 96]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [cognitect.aws.client$send_request$fn__10177$state_machine__5488__auto____10204 invoke client.clj 84]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [clojure.core.async.impl.ioc_macros$run_state_machine invokeStatic ioc_macros.clj 978]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [clojure.core.async.impl.ioc_macros$run_state_machine_wrapped invokeStatic ioc_macros.clj 980]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [cognitect.aws.client$send_request$fn__10177 invoke client.clj 84]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [clojure.lang.AFn run AFn.java 22]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [java.util.concurrent.ThreadPoolExecutor runWorker ThreadPoolExecutor.java 1149]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [java.util.concurrent.ThreadPoolExecutor$Worker run ThreadPoolExecutor.java 624]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [clojure.core.async.impl.concurrent$counted_thread_factory$reify__228$fn__229 invoke concurrent.clj 29]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [clojure.lang.AFn run AFn.java 22]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [java.lang.Thread run Thread.java 748]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [com.oracle.svm.core.thread.JavaThreads threadStartRoutine JavaThreads.java 553]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [com.oracle.svm.core.posix.thread.PosixJavaThreads pthreadStartRoutine PosixJavaThreads.java 192]]}}

borkdude 2021-06-11T13:44:46.102900Z

issue + patch: https://clojure.atlassian.net/browse/CLJ-2636 confirmed that this gets rid of the "false positive" in the generated graal config

Karol W贸jcik 2021-06-11T13:50:53.103100Z

cc @borkdude

borkdude 2021-06-11T13:51:39.103300Z

yeah, I think this is coming from their http client

borkdude 2021-06-11T13:51:44.103500Z

or something

borkdude 2021-06-11T13:51:49.103700Z

so the agent didn't catch this?

borkdude 2021-06-11T13:52:07.103900Z

ah well, it's not perfect then, but it can help

borkdude 2021-06-11T13:52:14.104100Z

that's how I always approached it

borkdude 2021-06-11T13:52:38.104300Z

ah I see

Karol W贸jcik 2021-06-11T13:52:52.104500Z

It's coming from RT

alexmiller 2021-06-11T13:52:58.104700Z

thx

borkdude 2021-06-11T13:52:59.104900Z

clojure.lang.RT is known to use reflection around arrays

Karol W贸jcik 2021-06-11T13:53:05.105100Z

Exactly!

borkdude 2021-06-11T13:53:12.105300Z

this is a known issue, but since we ignored it, you don't get to see it

borkdude 2021-06-11T13:57:50.105500Z

@karol.wojcik I think we can get around this by building on the output of the trace-output instead. E.g. for array creation you see this:

{"caller_class":"clojure.lang.RT", "result":true, "args":["[Ljava.lang.String;"], "function":"newInstance", "tracer":"reflect", "class":"java.lang.reflect.Array"}

borkdude 2021-06-11T13:58:12.105700Z

but for "false positives" you see this:

{"caller_class":"clojure.lang.RT", "result":true, "args":["refl.main__init"], "function":"forName", "tracer":"reflect", "class":"java.lang.Class"}

borkdude 2021-06-11T13:58:19.105900Z

so that's an easy filter operation

borkdude 2021-06-11T13:59:53.106100Z

or perhaps combine the info

Karol W贸jcik 2021-06-11T14:02:22.106300Z

I don't follow: > by building on the output of the聽`trace-output`聽instead What do you mean by that? Can I filter trace-output and feed GraalVM with it to generate reflect-config?

borkdude 2021-06-11T14:03:04.106600Z

I'll get back to this later, meeting