Clojurians Log v2

@brdloush This should help you ship native compiled version of your app faster 🙂

you definitely got my attention ! 😄

@karol.wojcik That's a nice tool, but note that > You don't have to write configurations for native image by hand, since there is Agent which might trace all the calls you make. The agent usually generates a too broad config since clojure.lang.Reflector is used during compilation

So depending on how you use this, you will get a too long config which makes your native image way too big

I think if you AOT your sources before running this, then it may work correctly.

Hmm.. Actually this is what I'm doing for holy-lambda. AOT then run uberjar on agent.

Will it generate too broad config if used that way?

You actually need to run your code in order to detect reflections (at runtime). The uberjar only invokes code that is relevant to compile time

E.g. when you have a -main that does (.length (first args)) you should get only a reflection config for the string class, and not any other classes

I meant to AOT compile the code and pack it in uberjar, then use agent to call -main from the uberjar.

If you use in-context without wrapping it in -main those will run as well.

Hmm I think I'm getting what you mean. There are some extra reflections entries which should not be there

{
  "name":"clojure.spec.alpha.Spec"
},
{
  "name":"clojure.spec.alpha.Specize"
},
{
  "name":"clojure.spec.alpha__init"
},
{
  "name":"clojure.spec.gen.alpha__init"
},
{
  "name":"clojure.string__init"
},
{
  "name":"clojure.uuid__init"
},
{
  "name":"clojure.walk__init"
},
{
  "name":"double[]"
},
{
  "name":"example.core__init"
}

yes

probably a lot more than these?

> then use agent to call -main from the uberjar. that sounds good

I will try to just call -main so that no extra reflections are produced

Meh. Still the same.

Maybe you can just filter out the __init classes ;)

@borkdude By AOT compile you mean this?

USE_AGENT_CONTEXT=true clojure -X:uberjar :aot true :jvm-opts '["-Dclojure.compiler.direct-linking=true", "-Dclojure.spec.skip-macros=true"]' :jar agent-output.jar :main-class example.core

yes, this will create a jar agent-output.jar which contains the program right?

(the name in that case is weird, it should be example.jar or so)

and you should NOT run the agent during compilation

that is what you NOT want :)

you want to run the agent afterwards

(by compilation I mean Clojure AOT compilation, not GraalVM compilation)

I don't want to have in-context calls in jar which will be native compiled that's why there is agent-output. Output.jar for the agent to run.

oh now it makes sense :)

> and you should NOT run the agent during compilation Am I doing it?

I didn't understand how you were using the jar, but now I get it :)

Ahh ok 😄

Filtering __init classes is a good advice

I'm still wondering how we can trim more.

As you can see in the example I'm AOT compiling project and still I have various entries like:

{
  "name":"java.io.Closeable"
},
{
  "name":"java.io.File"
},
{
  "name":"java.io.FileInputStream"
},
{
  "name":"java.io.FileOutputStream"
},
{
  "name":"java.io.FileWriter"
},
{
  "name":"java.io.InputStream"
},
{
  "name":"java.io.InputStreamReader"
},
{
  "name":"java.io.NotSerializableException"
}

HACK warning: Maybe redefine some functions in clojure which invoke the reflector to spit out the class name in case it's not actually doing reflection for reflection purposes? And then use that as a filter?

I think the agent is triggered when the java reflect stuff is used

and clojure does this maybe to just look some stuff up

I also had major issues with agent along the same lines where the config it output wasn't specific enough and it listed lots of classes that weren't necessary for the executable to work thus bloating executable size.

Hmm.. I'm thinking whether we can trim configuration a little bit. For instance if we look at clojure.lang.RT/classForName we can see that not all classes have to be initialized thus those are probably not necessary for GraalVM native-image to work 🙂 I've have printed all of those:

From 9e0ef90a3a48f74b7b3d1b368a466d398de1b7b4 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Karol=20W=C3=B3jcik?= <karol.wojcik@tuta.io>
Date: Wed, 5 May 2021 17:50:01 +0200
Subject: [PATCH] Log classes which should be ommited on GraalVM side
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Signed-off-by: Karol Wójcik <karol.wojcik@tuta.io>
---
 src/jvm/clojure/lang/RT.java | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/src/jvm/clojure/lang/RT.java b/src/jvm/clojure/lang/RT.java
index 74399cf1..ba84cb3f 100644
--- a/src/jvm/clojure/lang/RT.java
+++ b/src/jvm/clojure/lang/RT.java
@@ -2209,6 +2209,10 @@ static public Class classForName(String name, boolean load, ClassLoader loader)
 			c = DynamicClassLoader.findInMemoryClass(name);
 		if (c != null)
 			return c;
+
+		if(!load) {
+			System.out.print(name + ",");
+		}
 		return Class.forName(name, load, loader);
 		}
 	catch(ClassNotFoundException e)
-- 
2.31.1

Then I've used followin script to remove those entries:

(require '[cheshire.core :as json])

(def CLASSES_TO_REMOVE
  (set (clojure.string/split "clojure.core__init,java.lang.reflect.Array,clojure.lang.ExceptionInfo,clojure.lang.IExceptionInfo,java.util.concurrent.BlockingQueue,java.util.concurrent.LinkedBlockingQueue,clojure.core_proxy__init,clojure.asm.ClassWriter,clojure.asm.ClassVisitor,clojure.asm.Opcodes,clojure.asm.Type,java.lang.reflect.Modifier,java.lang.reflect.Constructor,java.io.Serializable,java.io.NotSerializableException,clojure.asm.commons.Method,clojure.asm.commons.GeneratorAdapter,clojure.lang.IProxy,clojure.lang.Reflector,clojure.lang.DynamicClassLoader,clojure.lang.IPersistentMap,clojure.lang.PersistentHashMap,clojure.lang.RT,clojure.core_print__init,java.io.Writer,clojure.genclass__init,java.lang.reflect.Modifier,java.lang.reflect.Constructor,clojure.asm.ClassWriter,clojure.asm.ClassVisitor,clojure.asm.Opcodes,clojure.asm.Type,clojure.asm.commons.Method,clojure.asm.commons.GeneratorAdapter,clojure.lang.IPersistentMap,clojure.core_deftype__init,clojure.core.protocols__init,clojure.gvec__init,clojure.lang.Murmur3,clojure.lang.IHashEq,clojure.lang.Sequential,clojure.lang.Util,clojure.lang.SeqIterator,java.util.List,clojure.core.VecNode,clojure.core.IVecImpl,clojure.core.ArrayManager,clojure.core.ArrayChunk,clojure.core.VecSeq,clojure.core.Vec,clojure.instant__init,java.util.Calendar,java.util.Date,java.util.GregorianCalendar,java.util.TimeZone,java.sql.Timestamp,clojure.uuid__init,clojure.java.io__init,clojure.string__init,java.util.regex.Pattern,java.util.regex.Matcher,clojure.lang.LazilyPersistentVector,java.io.Reader,java.io.InputStream,java.io.InputStreamReader,java.io.PushbackReader,java.io.BufferedReader,java.io.File,java.io.OutputStream,java.io.OutputStreamWriter,java.io.BufferedWriter,java.io.Writer,java.io.FileInputStream,java.io.FileOutputStream,java.io.ByteArrayOutputStream,java.io.StringReader,java.io.ByteArrayInputStream,java.io.BufferedInputStream,java.io.BufferedOutputStream,java.io.CharArrayReader,java.io.Closeable,java.net.URI,java.net.URL,java.net.MalformedURLException,java.net.Socket,java.net.URLDecoder,java.net.URLEncoder,clojure.core.Eduction,clojure.core.server__init,clojure.edn__init,clojure.main__init,clojure.spec.alpha__init,clojure.walk__init,clojure.spec.gen.alpha__init,java.io.StringReader,java.io.BufferedWriter,java.io.FileWriter,java.nio.file.Files,java.nio.file.attribute.FileAttribute,clojure.lang.Compiler,clojure.lang.Compiler$CompilerException,clojure.lang.LineNumberingPushbackReader,clojure.lang.RT,clojure.lang.LispReader$ReaderException,clojure.lang.LineNumberingPushbackReader,java.net.InetAddress,java.net.Socket,java.net.ServerSocket,java.net.SocketException,java.io.Reader,java.io.Writer,java.io.PrintWriter,java.io.BufferedWriter,java.io.BufferedReader,java.io.InputStreamReader,java.io.OutputStreamWriter,java.util.concurrent.locks.ReentrantLock,example.core__init,fierycod.graalvm_agent_helper.core__init" #",")))

(def REFLECTION_CONFIG (json/parse-string (slurp "resources/native-configuration/reflect-config.json")))

(def REFLECTION_CONFIG_TRIMMED
  (filterv (complement (fn [entry]
                         (or (clojure.string/includes? (get entry "name") "clojure")
                          (contains? CLASSES_TO_REMOVE (get entry "name")))))
           REFLECTION_CONFIG))

(println (count REFLECTION_CONFIG))
(println (count REFLECTION_CONFIG_TRIMMED))

[{"name" "boolean[]"}
 {"name" "byte[]"}
 {"name" "char[]"}
 {"name" "double[]"}
 {"name" "float[]"}
 {"name" "int[]"}
 {"name" "java.lang.Boolean"}
 {"name" "java.lang.Character"}
 {"name" "java.lang.Class"}
 {"name" "java.lang.Double"}
 {"name" "java.lang.Float"}
 {"name" "java.lang.Iterable"}
 {"name" "java.lang.Long"}
 {"name" "java.lang.Number"}
 {"name" "java.lang.Object"}
 {"name" "java.lang.Object[]"}
 {"name" "java.lang.StackTraceElement"}
 {"name" "java.lang.StackTraceElement[]"}
 {"name" "java.lang.String", "allPublicMethods" true}
 {"name" "java.lang.ThreadLocal"}
 {"name" "java.lang.Throwable"}
 {"name" "java.lang.UnsupportedOperationException"}
 {"name" "java.lang.annotation.Annotation"}
 {"name" "java.lang.annotation.Retention"}
 {"name" "java.lang.reflect.Field"}
 {"name" "java.math.BigDecimal"}
 {"name" "java.math.BigInteger"}
 {"name" "java.time.Instant"}
 {"name" "java.util.Collection"}
 {"name" "java.util.Map"}
 {"name" "java.util.Properties", "allPublicMethods" true}
 {"name" "java.util.RandomAccess"}
 {"name" "java.util.Set"}
 {"name" "java.util.UUID"}
 {"name" "java.util.concurrent.ArrayBlockingQueue"}
 {"name" "long[]"}
 {"name" "short[]"}]

@borkdude I don't now though if it's correct approach.

Haha, it certainly is bold 🙂.