graalvm

Discuss GraalVM related topics. Use clojure 1.10.2 or newer for all new projects. Contribute to https://github.com/clj-easy/graal-docs and https://github.com/BrunoBonacci/graalvm-clojure. GraalVM slack: https://www.graalvm.org/slack-invitation/.
Karol W贸jcik 2021-05-05T14:01:25.211400Z

Karol W贸jcik 2021-05-06T07:27:58.220400Z

Hmm I will experiment with it a little

Karol W贸jcik 2021-05-05T14:02:44.211500Z

@brdloush This should help you ship native compiled version of your app faster 馃檪

Tomas Brejla 2021-05-05T14:03:11.211700Z

you definitely got my attention ! 馃槃

borkdude 2021-05-05T14:05:49.211900Z

@karol.wojcik That's a nice tool, but note that > You don't have to write configurations for native image by hand, since there is Agent which might trace all the calls you make. The agent usually generates a too broad config since clojure.lang.Reflector is used during compilation

borkdude 2021-05-05T14:06:17.212100Z

So depending on how you use this, you will get a too long config which makes your native image way too big

borkdude 2021-05-05T14:06:50.212300Z

I think if you AOT your sources before running this, then it may work correctly.

Karol W贸jcik 2021-05-05T14:08:20.212600Z

Hmm.. Actually this is what I'm doing for holy-lambda. AOT then run uberjar on agent.

Karol W贸jcik 2021-05-05T14:08:41.212800Z

Will it generate too broad config if used that way?

borkdude 2021-05-05T14:08:41.213Z

You actually need to run your code in order to detect reflections (at runtime). The uberjar only invokes code that is relevant to compile time

borkdude 2021-05-05T14:10:52.213200Z

E.g. when you have a -main that does (.length (first args)) you should get only a reflection config for the string class, and not any other classes

Karol W贸jcik 2021-05-05T14:13:17.213400Z

I meant to AOT compile the code and pack it in uberjar, then use agent to call -main from the uberjar.

Karol W贸jcik 2021-05-05T14:13:38.213600Z

If you use in-context without wrapping it in -main those will run as well.

Karol W贸jcik 2021-05-05T14:15:35.213800Z

Hmm I think I'm getting what you mean. There are some extra reflections entries which should not be there

{
  "name":"clojure.spec.alpha.Spec"
},
{
  "name":"clojure.spec.alpha.Specize"
},
{
  "name":"clojure.spec.alpha__init"
},
{
  "name":"clojure.spec.gen.alpha__init"
},
{
  "name":"clojure.string__init"
},
{
  "name":"clojure.uuid__init"
},
{
  "name":"clojure.walk__init"
},
{
  "name":"double[]"
},
{
  "name":"example.core__init"
}

borkdude 2021-05-05T14:19:20.214Z

yes

borkdude 2021-05-05T14:19:33.214200Z

probably a lot more than these?

borkdude 2021-05-05T14:20:03.214400Z

> then use agent to call -main from the uberjar. that sounds good

Karol W贸jcik 2021-05-05T14:21:04.214600Z

I will try to just call -main so that no extra reflections are produced

Karol W贸jcik 2021-05-05T14:30:30.214800Z

Meh. Still the same.

borkdude 2021-05-05T14:32:13.215Z

Maybe you can just filter out the __init classes ;)

Karol W贸jcik 2021-05-05T14:32:17.215200Z

@borkdude By AOT compile you mean this?

USE_AGENT_CONTEXT=true clojure -X:uberjar :aot true :jvm-opts '["-Dclojure.compiler.direct-linking=true", "-Dclojure.spec.skip-macros=true"]' :jar agent-output.jar :main-class example.core

borkdude 2021-05-05T14:33:08.215400Z

yes, this will create a jar agent-output.jar which contains the program right?

borkdude 2021-05-05T14:33:20.215600Z

(the name in that case is weird, it should be example.jar or so)

borkdude 2021-05-05T14:33:45.215900Z

and you should NOT run the agent during compilation

borkdude 2021-05-05T14:33:49.216100Z

that is what you NOT want :)

borkdude 2021-05-05T14:33:58.216300Z

you want to run the agent afterwards

borkdude 2021-05-05T14:34:23.216500Z

(by compilation I mean Clojure AOT compilation, not GraalVM compilation)

Karol W贸jcik 2021-05-05T14:37:29.216800Z

I don't want to have in-context calls in jar which will be native compiled that's why there is agent-output. Output.jar for the agent to run.

borkdude 2021-05-05T14:37:45.217Z

oh now it makes sense :)

Karol W贸jcik 2021-05-05T14:38:04.217200Z

> and you should NOT run the agent during compilation Am I doing it?

borkdude 2021-05-05T14:38:50.217400Z

I didn't understand how you were using the jar, but now I get it :)

Karol W贸jcik 2021-05-05T14:38:55.217600Z

Ahh ok 馃槃

Karol W贸jcik 2021-05-05T14:39:28.217800Z

Filtering __init classes is a good advice

Karol W贸jcik 2021-05-05T14:39:37.218Z

I'm still wondering how we can trim more.

Karol W贸jcik 2021-05-05T14:41:38.218200Z

As you can see in the example I'm AOT compiling project and still I have various entries like:

{
  "name":"java.io.Closeable"
},
{
  "name":"java.io.File"
},
{
  "name":"java.io.FileInputStream"
},
{
  "name":"java.io.FileOutputStream"
},
{
  "name":"java.io.FileWriter"
},
{
  "name":"java.io.InputStream"
},
{
  "name":"java.io.InputStreamReader"
},
{
  "name":"java.io.NotSerializableException"
}
Even though I'm not using it directly.

borkdude 2021-05-05T14:42:15.218400Z

HACK warning: Maybe redefine some functions in clojure which invoke the reflector to spit out the class name in case it's not actually doing reflection for reflection purposes? And then use that as a filter?

1馃憤
borkdude 2021-05-05T14:42:38.218600Z

I think the agent is triggered when the java reflect stuff is used

borkdude 2021-05-05T14:42:47.218800Z

and clojure does this maybe to just look some stuff up

chrisn 2021-05-05T15:31:13.219100Z

I also had major issues with agent along the same lines where the config it output wasn't specific enough and it listed lots of classes that weren't necessary for the executable to work thus bloating executable size.

Karol W贸jcik 2021-05-05T16:10:49.219300Z

Hmm.. I'm thinking whether we can trim configuration a little bit. For instance if we look at clojure.lang.RT/classForName we can see that not all classes have to be initialized thus those are probably not necessary for GraalVM native-image to work 馃檪 I've have printed all of those:

From 9e0ef90a3a48f74b7b3d1b368a466d398de1b7b4 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Karol=20W=C3=B3jcik?= <karol.wojcik@tuta.io>
Date: Wed, 5 May 2021 17:50:01 +0200
Subject: [PATCH] Log classes which should be ommited on GraalVM side
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Signed-off-by: Karol W贸jcik <karol.wojcik@tuta.io>
---
 src/jvm/clojure/lang/RT.java | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/src/jvm/clojure/lang/RT.java b/src/jvm/clojure/lang/RT.java
index 74399cf1..ba84cb3f 100644
--- a/src/jvm/clojure/lang/RT.java
+++ b/src/jvm/clojure/lang/RT.java
@@ -2209,6 +2209,10 @@ static public Class classForName(String name, boolean load, ClassLoader loader)
 			c = DynamicClassLoader.findInMemoryClass(name);
 		if (c != null)
 			return c;
+
+		if(!load) {
+			System.out.print(name + ",");
+		}
 		return Class.forName(name, load, loader);
 		}
 	catch(ClassNotFoundException e)
-- 
2.31.1

Karol W贸jcik 2021-05-05T16:12:16.219500Z

Then I've used followin script to remove those entries:

(require '[cheshire.core :as json])

(def CLASSES_TO_REMOVE
  (set (clojure.string/split "clojure.core__init,java.lang.reflect.Array,clojure.lang.ExceptionInfo,clojure.lang.IExceptionInfo,java.util.concurrent.BlockingQueue,java.util.concurrent.LinkedBlockingQueue,clojure.core_proxy__init,clojure.asm.ClassWriter,clojure.asm.ClassVisitor,clojure.asm.Opcodes,clojure.asm.Type,java.lang.reflect.Modifier,java.lang.reflect.Constructor,java.io.Serializable,java.io.NotSerializableException,clojure.asm.commons.Method,clojure.asm.commons.GeneratorAdapter,clojure.lang.IProxy,clojure.lang.Reflector,clojure.lang.DynamicClassLoader,clojure.lang.IPersistentMap,clojure.lang.PersistentHashMap,clojure.lang.RT,clojure.core_print__init,java.io.Writer,clojure.genclass__init,java.lang.reflect.Modifier,java.lang.reflect.Constructor,clojure.asm.ClassWriter,clojure.asm.ClassVisitor,clojure.asm.Opcodes,clojure.asm.Type,clojure.asm.commons.Method,clojure.asm.commons.GeneratorAdapter,clojure.lang.IPersistentMap,clojure.core_deftype__init,clojure.core.protocols__init,clojure.gvec__init,clojure.lang.Murmur3,clojure.lang.IHashEq,clojure.lang.Sequential,clojure.lang.Util,clojure.lang.SeqIterator,java.util.List,clojure.core.VecNode,clojure.core.IVecImpl,clojure.core.ArrayManager,clojure.core.ArrayChunk,clojure.core.VecSeq,clojure.core.Vec,clojure.instant__init,java.util.Calendar,java.util.Date,java.util.GregorianCalendar,java.util.TimeZone,java.sql.Timestamp,clojure.uuid__init,clojure.java.io__init,clojure.string__init,java.util.regex.Pattern,java.util.regex.Matcher,clojure.lang.LazilyPersistentVector,java.io.Reader,java.io.InputStream,java.io.InputStreamReader,java.io.PushbackReader,java.io.BufferedReader,java.io.File,java.io.OutputStream,java.io.OutputStreamWriter,java.io.BufferedWriter,java.io.Writer,java.io.FileInputStream,java.io.FileOutputStream,java.io.ByteArrayOutputStream,java.io.StringReader,java.io.ByteArrayInputStream,java.io.BufferedInputStream,java.io.BufferedOutputStream,java.io.CharArrayReader,java.io.Closeable,java.net.URI,java.net.URL,java.net.MalformedURLException,java.net.Socket,java.net.URLDecoder,java.net.URLEncoder,clojure.core.Eduction,clojure.core.server__init,clojure.edn__init,clojure.main__init,clojure.spec.alpha__init,clojure.walk__init,clojure.spec.gen.alpha__init,java.io.StringReader,java.io.BufferedWriter,java.io.FileWriter,java.nio.file.Files,java.nio.file.attribute.FileAttribute,clojure.lang.Compiler,clojure.lang.Compiler$CompilerException,clojure.lang.LineNumberingPushbackReader,clojure.lang.RT,clojure.lang.LispReader$ReaderException,clojure.lang.LineNumberingPushbackReader,java.net.InetAddress,java.net.Socket,java.net.ServerSocket,java.net.SocketException,java.io.Reader,java.io.Writer,java.io.PrintWriter,java.io.BufferedWriter,java.io.BufferedReader,java.io.InputStreamReader,java.io.OutputStreamWriter,java.util.concurrent.locks.ReentrantLock,example.core__init,fierycod.graalvm_agent_helper.core__init" #",")))

(def REFLECTION_CONFIG (json/parse-string (slurp "resources/native-configuration/reflect-config.json")))

(def REFLECTION_CONFIG_TRIMMED
  (filterv (complement (fn [entry]
                         (or (clojure.string/includes? (get entry "name") "clojure")
                          (contains? CLASSES_TO_REMOVE (get entry "name")))))
           REFLECTION_CONFIG))

(println (count REFLECTION_CONFIG))
(println (count REFLECTION_CONFIG_TRIMMED))
I have also made an assumption that are classes with clojure are probably not needed as well. Here is the trimmed config:
[{"name" "boolean[]"}
 {"name" "byte[]"}
 {"name" "char[]"}
 {"name" "double[]"}
 {"name" "float[]"}
 {"name" "int[]"}
 {"name" "java.lang.Boolean"}
 {"name" "java.lang.Character"}
 {"name" "java.lang.Class"}
 {"name" "java.lang.Double"}
 {"name" "java.lang.Float"}
 {"name" "java.lang.Iterable"}
 {"name" "java.lang.Long"}
 {"name" "java.lang.Number"}
 {"name" "java.lang.Object"}
 {"name" "java.lang.Object[]"}
 {"name" "java.lang.StackTraceElement"}
 {"name" "java.lang.StackTraceElement[]"}
 {"name" "java.lang.String", "allPublicMethods" true}
 {"name" "java.lang.ThreadLocal"}
 {"name" "java.lang.Throwable"}
 {"name" "java.lang.UnsupportedOperationException"}
 {"name" "java.lang.annotation.Annotation"}
 {"name" "java.lang.annotation.Retention"}
 {"name" "java.lang.reflect.Field"}
 {"name" "java.math.BigDecimal"}
 {"name" "java.math.BigInteger"}
 {"name" "java.time.Instant"}
 {"name" "java.util.Collection"}
 {"name" "java.util.Map"}
 {"name" "java.util.Properties", "allPublicMethods" true}
 {"name" "java.util.RandomAccess"}
 {"name" "java.util.Set"}
 {"name" "java.util.UUID"}
 {"name" "java.util.concurrent.ArrayBlockingQueue"}
 {"name" "long[]"}
 {"name" "short[]"}]

Karol W贸jcik 2021-05-05T16:13:13.219700Z

@borkdude I don't now though if it's correct approach.

chrisn 2021-05-05T18:26:49.220Z

Haha, it certainly is bold 馃檪.