Clojurians Log v2

Is it possible to specify list of reflection entries which should be ignored when generating native-configurations using GraalVM agent? It's makes me crazy that I have to each time manually trim some of the entries. PS: I'm looking for automated way, so that I can built it into #holy-lambda

@karol.wojcik there is something even better that I’m going to try out today. You can ignore the place where the reflection happened, so if you specify the place in the Clojure compiler it should only leave the real problematic cases

https://www.graalvm.org/reference-manual/native-image/BuildConfiguration/#caller-based-filters

But you can also have access based filters

Nice!

Can wait to see the progress in that area. Please notify me when you make any progress. I'm eager to help with this!

@karol.wojcik This filter works reasonably well:

borkdude@MBP2019 ~/Dropbox/temp/refl $ cat filter.json
{ "rules": [
  {"excludeClasses": "clojure.**"},
  {"includeClasses": "clojure.lang.Reflector"}
]
}
borkdude@MBP2019 ~/Dropbox/temp/refl $ cat src/refl/main.clj
(ns refl.main
  (:gen-class))

(defn refl-str [s]
  (.length s))

(defn -main [& _]
  (println (refl-str "foo")))

borkdude@MBP2019 ~/Dropbox/temp/refl $ java -agentlib:native-image-agent=caller-filter-file=filter.json,config-output-dir=. -cp $(clojure -Spath):classes refl.main
3
borkdude@MBP2019 ~/Dropbox/temp/refl $ cat reflect-config.json
[
{
  "name":"java.lang.String",
  "allPublicMethods":true
},
{
  "name":"java.lang.reflect.Method",
  "methods":[{"name":"canAccess","parameterTypes":["java.lang.Object"] }]
},
{
  "name":"java.util.Properties",
  "allPublicMethods":true
}
]

not sure where the java.util.Properties thing comes from though, but it's only one thing

😮 🤯

I'll take a look how it behaves for some more complex cases. Probably we would need some kind of filtering for resources as well.

I'm mostly interested in the reflection config though

I made an issue at graal here to make this process a bit easier: https://github.com/oracle/graal/issues/3461

ah, this is probably already supported! trace-output=foo

and indeed it logs:

{"caller_class":"clojure.lang.Reflector", "args":[], "function":"getMethods", "tracer":"reflect", "class":"java.util.Properties"}

@karol.wojcik So using this trace-output option I found out you can get the same config using only:

{ "rules": [
  {"excludeClasses": "clojure.lang.RT"}
]
}

That's it :)

@borkdude You must be kidding me! 😄

I would just leave out `{ "name":"java.util.Properties", "allPublicMethods":true } ` though, as this class pulls in XML libraries and bloats your image. I'm not sure why Clojure does this reflection during the lifecycle of a program, but I guess that's for another time.

but this can be configured using an access filter as well

Hm, this config is better it turns out:

{ "rules": [
  {"excludeClasses": "clojure.**"},
  {"includeClasses": "clojure.lang.Reflector"}
]
}

Updated docs: https://github.com/lread/clj-graal-docs#reflection

I think I found the spot where the false positive java.util.Properties reflection comes from. https://github.com/clojure/clojure/blob/b1b88dd25373a86e41310a525a21b497799dbbf2/src/clj/clojure/core.clj#L7085 Would be nice if this could be resolved, not sure if the core team is open for this change, because for Clojure itself it's not that important.

cc @alexmiller

Looks quite easy to fix. I'll try to make a patch for it

I think the fix is not the issue, it's more if it will be accepted or not ;)

Ah right!

Sure, make a jira

ok, I'll make a JIRA + patch once I confirm locally that this solves the "false positive" in the graal generated config

thanks

It's not a silver bullet for everything it seems. For instance compiling project with cognitect.aws lacks some of the reflections:

{:cognitect.anomalies/category :cognitect.anomalies/fault, :cognitect.anomalies/message Class java.nio.HeapByteBuffer[] is instantiated reflectively but was never registered. Register the class by using org.graalvm.nativeimage.hosted.RuntimeReflection, :cognitect.http-client/throwable #error {
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 :cause Class java.nio.HeapByteBuffer[] is instantiated reflectively but was never registered. Register the class by using org.graalvm.nativeimage.hosted.RuntimeReflection
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 :via
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [{:type java.lang.IllegalArgumentException
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 :message Class java.nio.HeapByteBuffer[] is instantiated reflectively but was never registered. Register the class by using org.graalvm.nativeimage.hosted.RuntimeReflection
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 :at [com.oracle.svm.core.graal.snippets.SubstrateAllocationSnippets arrayHubErrorStub SubstrateAllocationSnippets.java 295]}]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 :trace
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [[com.oracle.svm.core.graal.snippets.SubstrateAllocationSnippets arrayHubErrorStub SubstrateAllocationSnippets.java 295]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [clojure.lang.RT seqToTypedArray RT.java 1754]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [clojure.lang.RT seqToTypedArray RT.java 1750]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [clojure.core$into_array invokeStatic core.clj 3440]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [cognitect.http_client$map__GT_jetty_request invokeStatic http_client.clj 108]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [cognitect.http_client.Client$fn__10581 invoke http_client.clj 226]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [cognitect.http_client.Client submit_STAR_ http_client.clj 226]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [cognitect.http_client$submit invokeStatic http_client.clj 213]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [cognitect.aws.http.cognitect$create$reify__10605 _submit cognitect.clj 10]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [cognitect.aws.http$submit invokeStatic http.clj 45]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [cognitect.aws.client$send_request$fn__10177$state_machine__5488__auto____10204$fn__10206$fn__10220 invoke client.clj 97]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [cognitect.aws.client$send_request$fn__10177$state_machine__5488__auto____10204$fn__10206 invoke client.clj 96]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [cognitect.aws.client$send_request$fn__10177$state_machine__5488__auto____10204 invoke client.clj 84]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [clojure.core.async.impl.ioc_macros$run_state_machine invokeStatic ioc_macros.clj 978]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [clojure.core.async.impl.ioc_macros$run_state_machine_wrapped invokeStatic ioc_macros.clj 980]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [cognitect.aws.client$send_request$fn__10177 invoke client.clj 84]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [clojure.lang.AFn run AFn.java 22]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [java.util.concurrent.ThreadPoolExecutor runWorker ThreadPoolExecutor.java 1149]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [java.util.concurrent.ThreadPoolExecutor$Worker run ThreadPoolExecutor.java 624]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [clojure.core.async.impl.concurrent$counted_thread_factory$reify__228$fn__229 invoke concurrent.clj 29]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [clojure.lang.AFn run AFn.java 22]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [java.lang.Thread run Thread.java 748]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [com.oracle.svm.core.thread.JavaThreads threadStartRoutine JavaThreads.java 553]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [com.oracle.svm.core.posix.thread.PosixJavaThreads pthreadStartRoutine PosixJavaThreads.java 192]]}}

issue + patch: https://clojure.atlassian.net/browse/CLJ-2636 confirmed that this gets rid of the "false positive" in the generated graal config

cc @borkdude

yeah, I think this is coming from their http client

or something

so the agent didn't catch this?

ah well, it's not perfect then, but it can help

that's how I always approached it

ah I see

It's coming from RT

thx

clojure.lang.RT is known to use reflection around arrays

Exactly!

this is a known issue, but since we ignored it, you don't get to see it

@karol.wojcik I think we can get around this by building on the output of the trace-output instead. E.g. for array creation you see this:

{"caller_class":"clojure.lang.RT", "result":true, "args":["[Ljava.lang.String;"], "function":"newInstance", "tracer":"reflect", "class":"java.lang.reflect.Array"}

but for "false positives" you see this:

{"caller_class":"clojure.lang.RT", "result":true, "args":["refl.main__init"], "function":"forName", "tracer":"reflect", "class":"java.lang.Class"}

so that's an easy filter operation

or perhaps combine the info

I don't follow: > by building on the output of the `trace-output` instead What do you mean by that? Can I filter trace-output and feed GraalVM with it to generate reflect-config?

I'll get back to this later, meeting