graalvm

Discuss GraalVM related topics. Use clojure 1.10.2 or newer for all new projects. Contribute to https://github.com/clj-easy/graal-docs and https://github.com/BrunoBonacci/graalvm-clojure. GraalVM slack: https://www.graalvm.org/slack-invitation/.
ericdallo 2021-01-24T04:02:33.006800Z

Hey all, I' trying to compile clojure-lsp (#lsp) with Graalvm, with the help of @borkdude we managed to make it work with sqlite ๐ŸŽ‰ ,but I'm still having other issues, this time related to log4j

kulminaator 2021-01-24T08:03:56.008Z

originally we hoped for interop with other java libraries logging that we use beneath but that never materialized since we need structured logs and not some freetext mayhem

Shantanu Kumar 2021-01-24T11:11:32.008200Z

@kulminaator @ericdallo Havenโ€™t tried with GraalVM native image, but it doesnโ€™t use any reflection or dynamic namespace loading https://cambium-clojure.github.io/ except discovering the current *ns*, which clojure.tools.logging also does.

Shantanu Kumar 2021-01-24T11:27:22.008500Z

A simpler alternative could be java.util.logging that is built into Java. The clojure.tools.logging wrapper loads classes at runtime https://github.com/clojure/tools.logging/blob/master/src/main/clojure/clojure/tools/logging/impl.clj#L52

borkdude 2021-01-24T11:36:15.008800Z

or don't use java logging at all and use timbre for example?

Shantanu Kumar 2021-01-24T12:04:04.009Z

Thereโ€™s a page on logging in native image: https://www.graalvm.org/reference-manual/native-image/Logging/

ericdallo 2021-01-24T14:06:46.009300Z

Thanks for the help @kulminaator @kumarshantanu @borkdude I think replacing the log with another lib it'd be faster indeed, we just need a lib that logs to a custom file and that's it. I'll take a look at timbre, it seems to nave no issues with graalvm Otherwise test https://cambium-clojure.github.io/

ericdallo 2021-01-24T14:56:14.009800Z

@borkdude I made a simple clojure sample using timbre to log to a file (using spit-appender) ang got https://pastebin.com/g9wqxqbv when compiling with graalvm

borkdude 2021-01-24T15:27:35.010400Z

@ericdallo > Error: Frame states being merged are incompatible: unbalanced monitors - locked objects do not match This seems to indicate you are again not using 1.10.2-rc3

borkdude 2021-01-24T15:28:20.010600Z

or hmm timbre is using monitor-enter and monitor-exit by itself? hmmmm

ericdallo 2021-01-24T15:29:19.010800Z

yes, they are using it!

borkdude 2021-01-24T15:29:19.011Z

this is the same problem that Clojure had. You should probably make a PR to timbre to use clojure's locking and not use monitor-enter/exit manually

ericdallo 2021-01-24T15:29:32.011200Z

i'm using latest clojure 1.10.2-rc3

ericdallo 2021-01-24T15:29:40.011400Z

oh, nice

ericdallo 2021-01-24T15:30:10.011600Z

I don't know what should I change, could you provide some materia/links?

ericdallo 2021-01-24T15:31:40.011800Z

I think I should follow this example: https://clojuredocs.org/clojure.core/locking

borkdude 2021-01-24T15:31:42.012Z

see https://github.com/clojure/clojure/commit/f5403e9c666f3281fdb880cb2c21303c273eed2d#diff-313df99869bda118262a387b322021d2cd9bea959fad3890cb78c6c37b448299 this is a very technical low level change, to ensure byte code analyzers can verify that the monitor enter and exit match up. your PR would replace monitor enter/exit to just use clojure's locking and not the current try catch

ericdallo 2021-01-24T15:32:50.012300Z

I see, I'll give a try, thanks @borkdude!

borkdude 2021-01-24T15:33:05.012500Z

you can refer to CLJ-1472 for details

1๐Ÿ‘
borkdude 2021-01-24T15:37:06.012800Z

@ericdallo timbre allows using your own appender, so for now you could copy this code, fix it and use your custom appender

borkdude 2021-01-24T15:37:16.013Z

an upstream PR is of course nicer long term

ericdallo 2021-01-24T15:37:49.013200Z

Oh, good idea, I'm coding the fix for timbre right now, but I'll proceed with a custom adapter until the PR get merged

borkdude 2021-01-24T15:38:42.013400Z

if that code is still reachable native-image might still complain, but you can patch with alter-var-root for the moment being

1๐Ÿ‘
ericdallo 2021-01-24T15:39:59.013700Z

(fn self [data]
        (let [{:keys [output_]} data]
          (let [output (force output_)]
            (locking (Object.) ; For thread safety, Ref. #251
              (try
                (with-open [^java.io.BufferedWriter w (jio/writer fname :append append?)]
                  (.write   w ^String output)
                  (.newLine w))

                (catch java.io.IOException e
                  (if (:spit-appender/retry? data)
                    (throw e) ; Unexpected error
                    (do
                      (jio/make-parents fname)
                      (self (assoc data :spit-appender/retry? true))))))))))
Does that makes sense?

ericdallo 2021-01-24T15:40:24.013900Z

I removed the monitor usages replaced by the locking defmacro

borkdude 2021-01-24T15:41:10.014400Z

should work

ericdallo 2021-01-24T15:45:31.014600Z

https://github.com/ptaoussanis/timbre/pull/334

ericdallo 2021-01-24T16:37:08.015Z

@borkdude would this code be enough to fix the issue right now? https://pastebin.com/y1L3HEPU WIth that I'm still having the same unbalanced monitors - locked objects do not match

ericdallo 2021-01-24T16:45:38.015200Z

This is the https://pastebin.com/gTB7QtB6. Not sure the alter-var-root is having effect

borkdude 2021-01-24T17:10:47.015400Z

The thing with direct linking is, is that this default appender may be used in multiple places, so alter-var-rooting it only at the source spot won't suffice if that's the case

borkdude 2021-01-24T17:11:31.015600Z

what about using a custom appender?

ericdallo 2021-01-24T17:19:53.015800Z

didn't work too, but I released a local version of my change on timbre and got other issue now, will paste the output

ericdallo 2021-01-24T17:23:16.016Z

Oh, actually it worked after adding --allow-incomplete-classpath

ericdallo 2021-01-24T17:23:28.016200Z

noice, another issue solved ๐Ÿ˜„

ericdallo 2021-01-24T17:58:00.016600Z

@borkdude I think this is the only missing issue: https://pastebin.com/JPv6vZ0s About the clojure.lang.Agent

ericdallo 2021-01-24T17:58:30.016800Z

It seems related to the threads that clojure-lsp spawns during the initialize

borkdude 2021-01-24T18:03:29.017Z

yeah, you should not do that on the top level

borkdude 2021-01-24T18:03:32.017200Z

but in the -main

ericdallo 2021-01-24T18:08:29.017400Z

Hum, but we start that on main already: https://github.com/clojure-lsp/clojure-lsp/blob/master/src/clojure_lsp/main.clj#L427

borkdude 2021-01-24T18:45:53.017700Z

you don't launch any agents at the top level?

borkdude 2021-01-24T18:46:45.017900Z

can you pinpoint who is starting this thread?

ericdallo 2021-01-24T18:57:26.018100Z

Yep, that's what I'm trying to understanding why it's not working, we just call this function run from the -main https://github.com/clojure-lsp/clojure-lsp/blob/master/src/clojure_lsp/main.clj#L431

borkdude 2021-01-24T19:04:18.018400Z

@ericdallo if you avoid the run call, do you still get the error?

1
ericdallo 2021-01-24T19:11:50.018800Z

does not work even commenting the run @borkdude ๐Ÿ˜ฎ

ericdallo 2021-01-24T19:14:47.019Z

I'll keep commenting server declaration until the erros is gone, then we ll know what line is the offending one

borkdude 2021-01-24T19:15:15.019200Z

maybe some other lib is spawning a future at the top level

borkdude 2021-01-24T19:15:30.019400Z

try eliminating some libs from the ns decl

borkdude 2021-01-24T19:15:51.019600Z

I've recently encountered this with some lib, don't remember which one it was

ericdallo 2021-01-24T19:18:01.019800Z

Yeah, probably some lsp4j class

borkdude 2021-01-24T19:19:03.020Z

you can try --initialize-at-runtime=<the name of the class>

ericdallo 2021-01-24T19:37:03.020200Z

Ok, found the offender require:

[nrepl.server :as nrepl.server]

ericdallo 2021-01-24T19:37:49.020400Z

weird, since we start the nrepl server here, but requiring it's enough to graal compile blow up: https://github.com/clojure-lsp/clojure-lsp/blob/master/src/clojure_lsp/main.clj#L414

borkdude 2021-01-24T19:39:43.020700Z

which library is this?

ericdallo 2021-01-24T19:39:58.020900Z

nrepl/nrepl {:mvn/version "0.8.3"}

ericdallo 2021-01-24T19:40:25.021100Z

https://github.com/nrepl/nrepl

borkdude 2021-01-24T19:41:18.021400Z

could be one of these... https://github.com/nrepl/nrepl/blob/master/src/clojure/nrepl/server.clj#L5-L15

borkdude 2021-01-24T19:41:30.021800Z

why do you have an nREPL server in LSP?

ericdallo 2021-01-24T19:41:42.022Z

Just searched graalvm and found a issue where you are mentioned ๐Ÿ˜‚ https://github.com/nrepl/nrepl/issues/181

ericdallo 2021-01-24T19:42:10.022300Z

It's for debugging AFAIK @borkdude, @snoe maybe can confirm that

borkdude 2021-01-24T19:43:05.022800Z

it sounds like this could be moved to a dev dependency

borkdude 2021-01-24T19:43:37.023Z

btw, I have an nREPL server in babashka which works with graalvm :P

borkdude 2021-01-24T19:43:41.023200Z

built from the ground up

ericdallo 2021-01-24T19:44:06.023500Z

> it sounds like this could be moved to a dev dependency I think so

borkdude 2021-01-24T19:44:38.023700Z

oh yes, now I remember!

ericdallo 2021-01-24T19:44:41.024100Z

Does babashka just require the same nrepl.server? how does that work hahaha?

borkdude 2021-01-24T19:45:00.024500Z

this completion namespace has top-level futures, very bad

borkdude 2021-01-24T19:45:17.024700Z

I made a fork of this stuff to make reply (the lein REPL) work with graalvm

borkdude 2021-01-24T19:45:34.024900Z

and I turned the futures into delays

ericdallo 2021-01-24T19:45:49.025100Z

๐Ÿ˜ฎ

ericdallo 2021-01-24T19:46:15.025300Z

do you have the code, it'd be helpful to fix on nrepl too, right?

borkdude 2021-01-24T19:49:11.025800Z

I just added you to the repo

borkdude 2021-01-24T19:49:21.026Z

I haven't open sourced the code since it's quite hacky

1๐Ÿ‘
borkdude 2021-01-24T19:49:40.026200Z

and copy/pasted the entire reply lib to hack on it

ericdallo 2021-01-24T19:51:04.026500Z

you have to many repos ๐Ÿ˜‚ that's crazy and awesome at same time ๐Ÿ˜›

borkdude 2021-01-24T19:52:09.026700Z

@ericdallo what you can maybe do is copy paste the nrepl completions namespace into your own repo

borkdude 2021-01-24T19:52:17.026900Z

and then change the futures into delays

borkdude 2021-01-24T19:52:42.027100Z

(or move nrepl to a dev dev... or some feature toggle)

ericdallo 2021-01-24T19:55:06.027300Z

moving to dev deps maybe is enough and the recommended IMO, I'll try to move to another profile so

borkdude 2021-01-24T19:55:35.027500Z

this will also yield a smaller native image

ericdallo 2021-01-24T19:56:37.027700Z

Yes! I just noticed that now it compiles but it seems that the app starts and then finish, while when running via lein run it keeps the socket open waiting for the LSP IO

ericdallo 2021-01-24T20:24:26.027900Z

@borkdude it compiles clojure-lsp perfect, is just that it just exit the app, do I need to make anything more?

ericdallo 2021-01-24T20:26:56.028100Z

I pushed what I done so far here: https://github.com/clojure-lsp/clojure-lsp/pull/267

ericdallo 2021-01-24T20:45:38.028400Z

Oh, actually, it's throwing an exception at runtime and the log is going to the file, I didn't notice it ๐Ÿ˜…

2021-01-24T20:44:40.990Z gregnix-note INFO [clojure-lsp.main:415] - Starting server...
2021-01-24T20:44:41.000Z gregnix-note ERROR [taoensso.timbre:796] - Uncaught exception on thread: main
                                              clojure_lsp.main.main                               
                                                                ...                               
                                             clojure-lsp.main/-main                  main.clj: 436
                                             clojure-lsp.main/-main                  main.clj: 438
                                               clojure-lsp.main/run                  main.clj: 418
          org.eclipse.lsp4j.launch.LSPLauncher.createServerLauncher          LSPLauncher.java:  46
                  org.eclipse.lsp4j.jsonrpc.Launcher$Builder.create             Launcher.java: 321
             org.eclipse.lsp4j.jsonrpc.Launcher$Builder.createProxy             Launcher.java: 364
org.eclipse.lsp4j.jsonrpc.services.ServiceEndpoints.toServiceObject     ServiceEndpoints.java:  41
                                                                ...                               
     com.oracle.svm.reflect.proxy.DynamicProxySupport.getProxyClass  DynamicProxySupport.java: 113
                com.oracle.svm.core.util.VMError.unsupportedFeature              VMError.java:  86
com.oracle.svm.core.jdk.UnsupportedFeatureError: Proxy class defined by interfaces [interface org.eclipse.lsp4j.services.LanguageClient, interface org.eclipse.lsp4j.jsonrpc.Endpoint] not found. Generating proxy classes at runtime is not supported. Proxy classes need to be defined at image build time by specifying the list of interfaces that they implement. To define proxy classes use -H:DynamicProxyConfigurationFiles=<comma-separated-config-files> and -H:DynamicProxyConfigurationResources=<comma-separated-config-resources> options.

ericdallo 2021-01-24T21:01:52.028700Z

Ok, after adding H:DynamicProxyConfigurationResources with those classes it seems to work ๐Ÿ˜„ , let me test it

borkdude 2021-01-24T21:41:26.028900Z

cool

ericdallo 2021-01-25T00:51:00.029400Z

I still could not make it work ๐Ÿ˜” It compiles and start without any issues, but it seems to be some wrong classes from the same interface... It's using the https://github.com/eclipse/lsp4j/blob/master/org.eclipse.lsp4j.jsonrpc/src/main/java/org/eclipse/lsp4j/jsonrpc/services/GenericEndpoint.java#L38 instead of the https://github.com/eclipse/lsp4j/blob/master/org.eclipse.lsp4j.jsonrpc/src/main/java/org/eclipse/lsp4j/jsonrpc/RemoteEndpoint.java#L46, so when LSP client (lsp-mode) calls the server first methods (initialize), it returns this error:

Jan 24, 2021 9:28:05 PM org.eclipse.lsp4j.jsonrpc.services.GenericEndpoint request
WARNING: Unsupported request method: initialize
This is something that seems relevant, the dynamic_proxy.json:
[
    ["org.eclipse.lsp4j.services.LanguageClient", "org.eclipse.lsp4j.jsonrpc.Endpoint"]
]
BTW https://github.com/eclipse/lsp4j/blob/master/org.eclipse.lsp4j.jsonrpc/src/main/java/org/eclipse/lsp4j/jsonrpc/services/ServiceEndpoints.java#L38-L55 that creates the proxy dynamically that I needed to setup this โ˜๏ธ I already tried to change https://github.com/clojure-lsp/clojure-lsp/pull/267/files#diff-47adefe2f307a723b07cd0862a4478ec5d038fa6a8ba1733f92560c21e2f49bbR299 to reify instead of proxy also. I feel like this is the last error or something like that ๐Ÿคž

ericdallo 2021-01-26T23:09:09.031Z

I'm stuck with this last error โ˜๏ธ found this on lsp4j side: https://github.com/eclipse/lsp4j/issues/349 This is the only thing I get it on strerr:

Jan 26, 2021 8:06:04 PM org.eclipse.lsp4j.jsonrpc.services.GenericEndpoint request
WARNING: Unsupported request method: initialize
While other logs not using graalvm say something like that:
Jan 26, 2021 7:02:07 PM org.eclipse.lsp4j.jsonrpc.RemoteEndpoint handleCancellation
WARNING: Unmatched cancel notification for request id 9710
what makes me guess that for some reason, it's using the wrong implementation of the lsp4j Endpoint interface

ericdallo 2021-01-26T23:09:29.031300Z

I compiled with graalvm 21 to check if there anything, but still the same

borkdude 2021-01-26T23:10:23.031500Z

cool. I have no clue and I'm going to bed soon. keep me posted

1๐Ÿ‘‹1
ericdallo 2021-01-27T12:47:21.031900Z

I kind of found the issue, I made a simple reify of the LanguageServer from lsp4j :

(defn fake-server []
  (reify LanguageServer
    (^CompletableFuture initialize [_this ^InitializeParams params]
     (do
       (println "INITIALIZING!")
       (log/info "INITIALIZING!")))

    (getTextDocumentService [_this]
      nil)
    (getWorkspaceService [_this]
      nil)))
During the start of the app, I call:
(log/info "-->" (class server)
                    (:members (r/reflect server)))
That prints all methods from the class, it works perfect with lein run,but with graalvm it prints #{}

borkdude 2021-01-27T12:59:43.032100Z

So what does this tell you?

ericdallo 2021-01-27T13:01:55.032300Z

It means the for some reason, graalvm is not working with that reify

borkdude 2021-01-27T13:04:17.032800Z

@ericdallo what error do you get when using it?

ericdallo 2021-01-27T13:05:43.033Z

Jan 27, 2021 9:46:32 AM org.eclipse.lsp4j.jsonrpc.services.GenericEndpoint request
WARNING: Unsupported request method: initialize

ericdallo 2021-01-27T13:06:23.033200Z

because it can't find the implementation of one of the methods initialize

borkdude 2021-01-27T13:06:49.033400Z

because you haven't reified it?

borkdude 2021-01-27T13:07:01.033600Z

oh sorry I misread

ericdallo 2021-01-27T13:07:46.033800Z

this is what I get from lein run / java from graalvm home:

--> clojure_lsp.main$fake_server$reify__31521 #{#clojure.reflect.Method{:name initialize, :return-type java.util.concurrent.CompletableFuture, :declaring-class clojure_lsp.main$fake_server$reify__31521, :parameter-types [org.eclipse.lsp4j.InitializeParams], :exception-types [], :flags #{:public}} #clojure.reflect.Method{:name getTextDocumentService, :return-type org.eclipse.lsp4j.services.TextDocumentService, :declaring-class clojure_lsp.main$fake_server$reify__31521, :parameter-types [], :exception-types [], :flags #{:public}} #clojure.reflect.Field{:name const__4, :type java.lang.Object, :declaring-class clojure_lsp.main$fake_server$reify__31521, :flags #{:public :static :final}} #clojure.reflect.Method{:name meta, :return-type clojure.lang.IPersistentMap, :declaring-class clojure_lsp.main$fake_server$reify__31521, :parameter-types [], :exception-types [], :flags #{:public}} #clojure.reflect.Field{:name const__2, :type clojure.lang.Var, :declaring-class clojure_lsp.main$fake_server$reify__31521, :flags #{:public :static :final}} #clojure.reflect.Method{:name getWorkspaceService, :return-type org.eclipse.lsp4j.services.WorkspaceService, :declaring-class clojure_lsp.main$fake_server$reify__31521, :parameter-types [], :exception-types [], :flags #{:public}} #clojure.reflect.Field{:name const__1, :type clojure.lang.Var, :declaring-class clojure_lsp.main$fake_server$reify__31521, :flags #{:public :static :final}} #clojure.reflect.Field{:name const__6, :type clojure.lang.Keyword, :declaring-class clojure_lsp.main$fake_server$reify__31521, :flags #{:public :static :final}} #clojure.reflect.Method{:name withMeta, :return-type clojure.lang.IObj, :declaring-class clojure_lsp.main$fake_server$reify__31521, :parameter-types [clojure.lang.IPersistentMap], :exception-types [], :flags #{:public}} #clojure.reflect.Constructor{:name clojure_lsp.main$fake_server$reify__31521, :declaring-class clojure_lsp.main$fake_server$reify__31521, :parameter-types [clojure.lang.IPersistentMap], :exception-types [], :flags #{:public}} #clojure.reflect.Constructor{:name clojure_lsp.main$fake_server$reify__31521, :declaring-class clojure_lsp.main$fake_server$reify__31521, :parameter-types [], :exception-types [], :flags #{:public}} #clojure.reflect.Field{:name const__3, :type clojure.lang.Keyword, :declaring-class clojure_lsp.main$fake_server$reify__31521, :flags #{:public :static :final}} #clojure.reflect.Field{:name const__0, :type clojure.lang.Var, :declaring-class clojure_lsp.main$fake_server$reify__31521, :flags #{:public :static :final}} #clojure.reflect.Field{:name const__5, :type clojure.lang.Keyword, :declaring-class clojure_lsp.main$fake_server$reify__31521, :flags #{:public :static :final}} #clojure.reflect.Field{:name __meta, :type clojure.lang.IPersistentMap, :declaring-class clojure_lsp.main$fake_server$reify__31521, :flags #{:final}} #clojure.reflect.Field{:name const__7, :type java.lang.Object, :declaring-class clojure_lsp.main$fake_server$reify__31521, :flags #{:public :static :final}}}
It's printing correct all methods from that reify

borkdude 2021-01-27T13:08:09.034Z

@ericdallo Maybe this will work: Is it possible to make the reified object at the top-level, at compile time?

ericdallo 2021-01-27T13:08:46.034200Z

yes it's possible, changing from defn to def would be enough?

borkdude 2021-01-27T13:09:30.034400Z

think so

ericdallo 2021-01-27T13:09:38.034600Z

Let me try it :loading:

ericdallo 2021-01-27T13:23:41.034800Z

Same issue, this is the output of that print from above:

--> class clojure_lsp.main$reify__31521 #{}

borkdude 2021-01-27T13:24:15.035Z

also the same error when used?

ericdallo 2021-01-27T13:25:07.035200Z

yep, maybe try with gen-class? https://stackoverflow.com/a/8614783

borkdude 2021-01-27T13:30:48.035600Z

or add the class to the reflection config

Shantanu Kumar 2021-01-27T13:31:54.035800Z

@ericdallo Maybe try once without type-hinting the initialize method? Since itโ€™s the only method in the interface, it should work without hinting due to auto-inference.

ericdallo 2021-01-27T13:31:55.036Z

But how can I get the class name? or you mean add the interface to the reflection config?

ericdallo 2021-01-27T13:32:34.036200Z

good point @kumarshantanu, I'll need type hint for other methods later, but for that test it's valid, let me try

borkdude 2021-01-27T13:32:38.036400Z

yeah, the interface

1๐Ÿ‘
ericdallo 2021-01-27T13:44:06.036700Z

It gives another error, but it seems is a step ahead ๐Ÿ˜†

SEVERE: Unable to invoke no-args constructor for class org.eclipse.lsp4j.adapters.InitializeParamsTypeAdapter$Factory. Registering an InstanceCreator with Gson for this type may fix this problem.
java.lang.RuntimeException: Unable to invoke no-args constructor for class org.eclipse.lsp4j.adapters.InitializeParamsTypeAdapter$Factory. Registering an InstanceCreator with Gson for this type may fix this problem.
	at com.google.gson.internal.ConstructorConstructor$14.construct(ConstructorConstructor.java:226)
	at com.google.gson.internal.bind.JsonAdapterAnnotationTypeAdapterFactory.getTypeAdapter(JsonAdapterAnnotationTypeAdapterFactory.java:55)
	at com.google.gson.internal.bind.JsonAdapterAnnotationTypeAdapterFactory.create(JsonAdapterAnnotationTypeAdapterFactory.java:49)
	at com.google.gson.Gson.getAdapter(Gson.java:423)
	at com.google.gson.Gson.fromJson(Gson.java:887)
	at org.eclipse.lsp4j.jsonrpc.json.adapters.MessageTypeAdapter.fromJson(MessageTypeAdapter.java:329)
	at org.eclipse.lsp4j.jsonrpc.json.adapters.MessageTypeAdapter.parseParams(MessageTypeAdapter.java:249)
	at org.eclipse.lsp4j.jsonrpc.json.adapters.MessageTypeAdapter.read(MessageTypeAdapter.java:119)
	at org.eclipse.lsp4j.jsonrpc.json.adapters.MessageTypeAdapter.read(MessageTypeAdapter.java:55)
	at com.google.gson.Gson.fromJson(Gson.java:888)
	at org.eclipse.lsp4j.jsonrpc.json.MessageJsonHandler.parseMessage(MessageJsonHandler.java:119)
	at org.eclipse.lsp4j.jsonrpc.json.MessageJsonHandler.parseMessage(MessageJsonHandler.java:114)
	at org.eclipse.lsp4j.jsonrpc.json.StreamMessageProducer.handleMessage(StreamMessageProducer.java:193)
	at org.eclipse.lsp4j.jsonrpc.json.StreamMessageProducer.listen(StreamMessageProducer.java:94)
	at org.eclipse.lsp4j.jsonrpc.json.ConcurrentMessageProcessor.run(ConcurrentMessageProcessor.java:113)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(Thread.java:834)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
Caused by: java.lang.reflect.InvocationTargetException
	at java.lang.reflect.Method.invoke(Method.java:566)
	at com.google.gson.internal.UnsafeAllocator$1.newInstance(UnsafeAllocator.java:50)
	at com.google.gson.internal.ConstructorConstructor$14.construct(ConstructorConstructor.java:223)
	... 21 more
Caused by: java.lang.IllegalArgumentException: Class org.eclipse.lsp4j.adapters.InitializeParamsTypeAdapter$Factory is instantiated reflectively but was never registered. Register the class by using org.graalvm.nativeimage.hosted.RuntimeReflection
	at com.oracle.svm.core.graal.snippets.SubstrateAllocationSnippets.hubErrorStub(SubstrateAllocationSnippets.java:247)
	at sun.misc.Unsafe.allocateInstance(Unsafe.java:840)
	... 24 more


Process clojure-lsp stderr<1> finished

ericdallo 2021-01-27T13:44:27.036900Z

Not sure is related to remove the type hinting or to move to the reflection.json

borkdude 2021-01-27T13:44:41.037100Z

never change two things at a time ;)

ericdallo 2021-01-27T13:44:59.037300Z

yeah hahaha

ericdallo 2021-01-27T13:45:15.037500Z

it seems another issue now, it's serializing the json to respond to client

ericdallo 2021-01-27T13:45:31.037700Z

Probably I should do something to initialize Gson correctly

ericdallo 2021-01-27T13:46:21.037900Z

Oh this looks a good start:

Class org.eclipse.lsp4j.adapters.InitializeParamsTypeAdapter$Factory is instantiated reflectively but was never registered. Register the class by using 

ericdallo 2021-01-27T13:46:36.038100Z

should I add it to reflection.json?

borkdude 2021-01-27T13:46:44.038300Z

guess so

1
ericdallo 2021-01-27T13:49:23.038600Z

If that works, probably I'll need to add https://github.com/eclipse/lsp4j/tree/master/org.eclipse.lsp4j/src/main/xtend-gen/org/eclipse/lsp4j/adapters to the reflection too, if that works, I'll certainly open an issue on lsp4j with steps how to make it work with graalvm ๐Ÿคž

ericdallo 2021-01-27T14:50:04.038800Z

Good news, problem solved with the suggestion from you, @borkdude , I added some classes to reflection json and it's getting into the method initialize correctly now โœ… The next issue ๐Ÿ˜† is: clojure-lsp tries to parse Gson json objects via https://github.com/clojure-lsp/clojure-lsp/blob/master/src/clojure_lsp/interop.clj#L416-L432, and it's returning this error on runtime with graal:

com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine            PosixJavaThreads.java:  192                                                                
                              com.oracle.svm.core.thread.JavaThreads.threadStartRoutine                 JavaThreads.java:  519                                                                
                                                                   java.lang.Thread.run                      Thread.java:  834                                                                
                                     java.util.concurrent.ThreadPoolExecutor$Worker.run          ThreadPoolExecutor.java:  628                                                                
                                      java.util.concurrent.ThreadPoolExecutor.runWorker          ThreadPoolExecutor.java: 1128                                                                
                                                    java.util.concurrent.FutureTask.run                  FutureTask.java:  264                                                                
                                    java.util.concurrent.Executors$RunnableAdapter.call                   Executors.java:  515
                          org.eclipse.lsp4j.jsonrpc.json.ConcurrentMessageProcessor.run  ConcurrentMessageProcessor.java:  113
                            org.eclipse.lsp4j.jsonrpc.json.StreamMessageProducer.listen       StreamMessageProducer.java:   94
                     org.eclipse.lsp4j.jsonrpc.json.StreamMessageProducer.handleMessage       StreamMessageProducer.java:  194
                                       org.eclipse.lsp4j.jsonrpc.RemoteEndpoint.consume              RemoteEndpoint.java:  190
                                 org.eclipse.lsp4j.jsonrpc.RemoteEndpoint.handleRequest              RemoteEndpoint.java:  261
                             org.eclipse.lsp4j.jsonrpc.services.GenericEndpoint.request             GenericEndpoint.java:  120
                       org.eclipse.lsp4j.jsonrpc.services.GenericEndpoint.lambda$null$0             GenericEndpoint.java:   65
                                                                                    ...                                       
clojure_lsp.main.proxy$clojure_lsp.ClojureExtensions$LanguageServer$c8d5825a.initialize                                       
                                                                 clojure-lsp.main/fn/fn                         main.clj:  326
                                                       clojure-lsp.main/client-settings                         main.clj:  278
                                                          clojure-lsp.interop/json->clj                      interop.clj:  421
                                                                                    ...                                       
java.lang.IllegalArgumentException: No matching field found: isJsonNull for class com.google.gson.JsonObject

ericdallo 2021-01-27T14:50:34.039200Z

Should I add com.google.gson.JsonObject to reflection too?

borkdude 2021-01-27T15:36:56.039500Z

yes

borkdude 2021-01-27T15:37:07.039700Z

at least that method

ericdallo 2021-01-27T15:41:14.039900Z

After that I get:

java.lang.NoSuchMethodError: java.lang.reflect.AccessibleObject.canAccess(java.lang.Object)
Not sure what to add to reflection.json now

borkdude 2021-01-27T15:42:26.040100Z

Maybe java.lang.reflect.AccessibleObject ๐Ÿ˜†

ericdallo 2021-01-27T15:42:46.040300Z

Yeah, that was my guess ๐Ÿ˜› compiling :loading:

borkdude 2021-01-27T15:43:04.040500Z

you can also run with a graalvm agent and then it will spit out all the things it needed reflection for during the run of a program

borkdude 2021-01-27T15:43:15.040700Z

it will also spit out a lot of classes you don't need, but it might give you some idea

borkdude 2021-01-27T15:43:23.040900Z

be sure to run with direct linking, this might help

ericdallo 2021-01-27T15:44:40.041100Z

Yeah, I thought about that, but the issue is that is not trivial to test clojure-lsp on terminal, we need to send LSP spec jsons, I'll try to move it to a script that use the java from graalvm, hope it works

borkdude 2021-01-27T15:47:33.041300Z

I mean, this is just a setting when creating the uberjar

borkdude 2021-01-27T15:47:43.041500Z

-J-Dclojure.compiler.direct-linking=true

borkdude 2021-01-27T15:47:53.041700Z

oh

borkdude 2021-01-27T15:47:58.041900Z

I see, never mind

ericdallo 2021-01-27T15:51:33.042100Z

np, thanks for the tip ๐Ÿ™‚

ericdallo 2021-01-27T18:36:18.042500Z

Having this now:

Can't call public method of non-public class: public sun.nio.fs.UnixPath sun.nio.fs.UnixPath.getParent()
even with this on reflection.json:
{
        "name": "sun.nio.fs.UnixPath",
        "allDeclaredConstructors": true,
        "allPublicConstructors": true,
        "allDeclaredMethods": true,
        "allPublicMethods": true
    }
Related to this line: https://github.com/clojure-lsp/clojure-lsp/blob/master/src/clojure_lsp/crawler.clj#L199

Shantanu Kumar 2021-01-27T19:24:28.042900Z

@ericdallo Try hinting dir as a public interface? (.getParent ^java.nio.file.Path dir)

ericdallo 2021-01-27T19:25:12.043100Z

Hum, good guess again, let me try it, I was thinking in refactor that to use <http://clojure.java.io|clojure.java.io> instead of manually using java classes, similar how clj-kondo do to get the config dir

borkdude 2021-01-27T19:27:28.043400Z

@ericdallo You should never have to include OS specific things in your reflection config, this config won't work on Windows for example

borkdude 2021-01-27T19:28:12.043900Z

either for the return type or for the local

ericdallo 2021-01-27T19:30:51.044100Z

You are right, something like that?

[dir ^java.nio.file.Path  (shared/uri-&gt;path project-root)]
@borkdude

borkdude 2021-01-27T19:31:21.044400Z

yes, but if you place this type hint on your function it would be even better

1๐Ÿ‘1โž•
ericdallo 2021-01-28T14:00:15.044900Z

Ok, so the type hinting fixed that case, but It being really hard to make lsp4j work correctly, it looks it uses a lot of reflection and I'm not sure I'm on the right path... The https://github.com/clojure-lsp/clojure-lsp/pull/267/files#diff-736ec9c37d3ba754db0a1f0c6cc5df40e8f284fb11e4824e55920cce814c37d8R1 already looks huge and I started to think that adding some classes have caused inconsistency like request params objects come with some fields null and etc. Also I could not run the java from graalvm to check what should looks like my reflection.json since is not available for my NixOS ๐Ÿ˜•

Shantanu Kumar 2021-01-28T15:40:03.045100Z

@ericdallo If you add this to your project.clj, you might get all the reflection warnings with lein before even trying GraalVM:

:global-vars {*warn-on-reflection* true
                *assert* true
                *unchecked-math* :warn-on-boxed}

borkdude 2021-01-28T15:41:06.045300Z

this won't help for java code which uses reflection

1โœ”๏ธ
borkdude 2021-01-28T15:41:24.045500Z

for this you can use the graalvm agent

ericdallo 2021-01-28T15:44:06.046100Z

Yep, that's the agent that should be used with graal's java that is not available for NixOs :/

borkdude 2021-01-28T15:44:46.046300Z

you can just download graalvm to a Downloads directory. there is no need to "install". Or aren't you allowed to downloading anything in nixOS?

ericdallo 2021-01-28T15:47:54.046500Z

I can if it don't use any other lib

ericdallo 2021-01-28T15:48:06.046700Z

I will give a try so

borkdude 2021-01-28T15:51:16.046900Z

yes, it's standalone. just unzip it and set GRAALVM_HOME to the thing and then use $GRAALVM_HOME/native-image

borkdude 2021-01-28T15:51:42.047100Z

this is how I use multiple graalvm versions side by side

borkdude 2021-01-28T15:53:21.047300Z

also run $GRAALVM_HOME/gu install native-image

borkdude 2021-01-28T15:54:04.047500Z

$GRAALVM_HOME/bin/gu install native-image
to be precise

ericdallo 2021-01-28T16:12:06.047700Z

thanks! but to run with graal's java it should be something like $GRAALVM_HOME/bin/java I guess

borkdude 2021-01-28T16:12:57.047900Z

that's right. you can export GRAALVM_HOME on the path like export PATH="$GRAALVM_HOME/bin:$PATH"

borkdude 2021-01-28T16:13:09.048100Z

and then just run java

1๐Ÿ‘
ericdallo 2021-01-28T16:30:17.048400Z

It doesn't work, it depends on libs from SO ๐Ÿ˜•

borkdude 2021-01-28T16:32:33.048800Z

This is what I install on CircleCI:

sudo apt-get -y install gcc g++ zlib1g-dev

borkdude 2021-01-28T16:32:47.049Z

I'm sure there is a nixOS incantation for this?

borkdude 2021-01-28T16:33:15.049200Z

you can always run this in a docker I guess

ericdallo 2021-01-28T23:15:51.049400Z

I tried the docker before, but failed for some reason, I gave a try again and it worked run the java from graal with the agent on NixOS via docker:tada:

ericdallo 2021-01-28T23:18:15.049600Z

I got a huge relfect json config generated with a lot of classes from clojure and some with dynamic names (using the object reference Id)

borkdude 2021-01-28T23:19:05.050Z

Yeah, there is a lot of stuff in there that you don't need. I'm not sure why it's showing up. Maybe it helps if you run with direct linking.

1๐Ÿ‘
borkdude 2021-01-28T23:19:27.050200Z

-J-Dclojure.compiler.direct-linking=true

borkdude 2021-01-28T23:20:10.050400Z

I'm afk now

1๐Ÿ‘‹
ericdallo 2021-01-29T01:18:43.050800Z

Ok, use the direct-linking helped reducing some weird classes, I used the output config to the native image input and the result looks better than before ๐Ÿ˜„ now it seems to load LSP correctly and communication work (json ser/deser), but I'm having a issue when responding with clojure.spec.alpha/conform https://github.com/clojure-lsp/clojure-lsp/blob/master/src/clojure_lsp/interop.clj#L409. Basically we use this function everytime we want to respond to client, so we validate if server response is following the spec and do some clj-map->java transformations (nothing magical, just instantiating the class manually) https://pastebin.pl/view/6450ceea

ericdallo 2021-01-29T03:32:00.051500Z

It was related with some instances not being recognized in runtime, fixed it using some type hints ๐Ÿ˜„

ericdallo 2021-01-29T18:48:08.056500Z

Hey @borkdude Just got a corner case with sqlite ๐Ÿ˜• It seems to throw a segfault when the db does not exists: (missing .lsp/sqlite.db) https://pastebin.com/JqHJPJzk

ericdallo 2021-01-29T18:48:32.056700Z

Right on this line: https://github.com/clojure-lsp/clojure-lsp/blob/master/src/clojure_lsp/db.clj#L35

borkdude 2021-01-29T18:49:33.057Z

@ericdallo I don't know what causes this. Where did you get the suggestions for the reflection / JNI config for sqlite?

ericdallo 2021-01-29T18:50:27.057200Z

Right here: https://github.com/xerial/sqlite-jdbc/issues/413

ericdallo 2021-01-29T18:50:42.057500Z

From the example: https://github.com/xerial/sqlite-jdbc/issues/413#issuecomment-525926912

borkdude 2021-01-29T18:52:09.058Z

@ericdallo Does his Java example work correctly when the db does not exist?

ericdallo 2021-01-29T18:52:27.058200Z

Hum... I didn't test, good point, I'll test it

borkdude 2021-01-29T18:53:56.058500Z

And to double check, you added all these classes to the JNI config right? https://github.com/mageddo/graalvm-examples/blob/59f1f1bf09894681edfddaa100b4504770ad0685/sqlite/src/main/java/com/mageddo/sqlite/JNIReflectionClasses.java#L46-L55 and to reflection

borkdude 2021-01-29T18:54:51.058900Z

Also try graalvm java 8 instead of 11

ericdallo 2021-01-29T19:01:57.059100Z

Yes, they are on https://github.com/ericdallo/sqlite-jni-graal-fix workaround

ericdallo 2021-01-29T19:02:18.059500Z

reflection not really

ericdallo 2021-01-29T19:03:22.059700Z

Oh, just found some classes missing from the fix, I don't know why, I'll add them too

ericdallo 2021-01-29T19:03:54.060300Z

Oh, yeah, so it's enough probably

ericdallo 2021-01-29T19:04:02.060500Z

I realized it's missing the Throwable class

ericdallo 2021-01-29T19:04:16.060700Z

since when don't find the db it should throw an exception, may be related

borkdude 2021-01-29T19:04:28.060900Z

yeah

1
borkdude 2021-01-29T19:14:15.061200Z

Btw, Quarkus is a Java framework by Oracle of which all extensions can be compiled to native. Sometimes nice to borrow stuff from as well: https://github.com/quarkusio/quarkus/tree/master/extensions/jdbc

ericdallo 2021-01-29T19:15:25.061400Z

Hum, interesting

ericdallo 2021-01-29T19:49:05.061600Z

Yep, it fixed the issue ๐Ÿ™‚ Probably it was the missing Throwable... that is not on that repo master, only on that specific commit

borkdude 2021-01-29T19:49:21.061800Z

cool

ericdallo 2021-01-29T19:49:35.062Z

a segfault like that is really hard to debug...

borkdude 2021-01-29T19:49:42.062200Z

yep

borkdude 2021-01-29T19:50:32.062400Z

this is why you should probably not just switch to native, but offer this as something experimental ;)

borkdude 2021-01-29T19:50:40.062600Z

and keep offering the JVM jar as well

borkdude 2021-01-29T19:50:57.062800Z

among other reasons

borkdude 2021-01-29T19:51:28.063Z

but it's cool that you got sqlite to work. maybe I can switch my pod from go to clojure now ;) although the golang pod is really small (few megabytes)

borkdude 2021-01-29T19:52:11.063200Z

does LSP native not require you to have sqlite installed, some libsqlite thing on your system? maybe try the binary in a bare ubuntu docker image (if that hasn't got it by default)?

ericdallo 2021-01-29T19:58:19.063500Z

Yeah, My idea is if everything work provides that as experimental for some time

borkdude 2021-01-29T19:59:12.063700Z

Note that if you exclusively offer the native, you will probably have to provide it for the three major OSes and also a static one for alpine

ericdallo 2021-01-29T19:59:24.063900Z

yes, since I use NixOS there is no way to use a lib that I didn't tell it to use, but a test on a docker it'd really help to make sure

ericdallo 2021-01-29T20:00:05.064100Z

Yes, I have in mind setup a CI for the 3 OS building via GH actions, just like babashka

1
ericdallo 2021-01-29T20:19:29.064400Z

@borkdude how the nrepl just for dev would work? In the code should I check somehow what is the profile/some env var? This would affect graal?

ericdallo 2021-01-29T20:20:23.064600Z

I can add the dep for the dev profile only, but my question is how should I code that to use for dev only or something like that

borkdude 2021-01-29T20:22:07.064800Z

yeah, you can do it through an env var or just by trying to resolve it on the top level:

(when-let [server-var (try (requiring-resolve 'nrepl.server/server) (catch Exception e nil))]
  ...

borkdude 2021-01-29T20:22:29.065Z

note that it's best to keep this at the top level so it's not resolved at run time

borkdude 2021-01-29T20:22:39.065200Z

the native-image will compile this away

borkdude 2021-01-29T20:23:20.065400Z

and then of course you move the nrepl deps to a dev profile

borkdude 2021-01-29T20:23:23.065600Z

or alias

ericdallo 2021-01-29T20:23:46.065800Z

got it, do you use anything like that in your projects?

borkdude 2021-01-29T20:41:07.066Z

yeah, this is how all the features in babashka are implemented

borkdude 2021-01-29T20:41:17.066200Z

see build.md, feature flags

ericdallo 2021-01-29T20:43:00.066400Z

cool, thanks!

borkdude 2021-01-29T20:44:30.066600Z

@ericdallo I also have this library: https://github.com/borkdude/dynaload

ericdallo 2021-01-29T20:47:54.067300Z

that dynaload seems to fit perfect this use case

ericdallo 2021-01-29T20:49:00.067500Z

should I wrap the dynaload call with a try catch to check if nrepl is enabled? Or is better to check a env var?

borkdude 2021-01-29T20:49:57.067700Z

What you can do is do a require based on an env var. And then use dynaload to use the var(s)

ericdallo 2021-01-29T20:50:23.067900Z

what is the point to use dynaload if I can just require 'mylist ? I think I don't get it

ericdallo 2021-01-29T20:50:58.068100Z

Oh, it won't compile if I don't require it.. it makes sense

ericdallo 2021-01-29T20:51:08.068300Z

dynaload just use a symbol and do the magic

borkdude 2021-01-29T20:51:10.068500Z

this is more useful for libraries than apps probably. in libraries you can say: I can work with these vars, if you have required the lib already

borkdude 2021-01-29T20:51:33.068700Z

in normal JVM dynaload will also do the require for you

borkdude 2021-01-29T20:51:40.068900Z

but for GraalVM this has a negative impact on binary size

borkdude 2021-01-29T20:51:52.069100Z

although it will still work

ericdallo 2021-01-29T20:52:35.069300Z

I see

borkdude 2021-01-29T20:53:07.069500Z

To keep it simple: Just do a try catch where you try to require/resolve the lib/vars. If you then provide the library on your classpath, the vars will work. Else they won't do anything.

ericdallo 2021-01-29T20:53:47.069700Z

great

ericdallo 2021-01-30T16:30:52.078500Z

ok, nrepl fixed using dynaload ๐Ÿ˜„ I tried to use SLF4j, log4j, log4j2 and logback, but all have issues with file appenders with graalvm... so I think timbre will be the only way I fixed all found issues with reflection, need to test all features still. I noticed that the first clojure-lsp classpath scan time using clj-kondo was increased, for a proect with 250 files to scan, while with JVM it takes 90s with graalvm takes 130s, not sure there is anything to improve that though. About the image size, I removed a lot of unnecessary reflection, still the image size is about 110MB while the jar has 30MB, I found https://github.com/ptaoussanis/timbre/issues/326 regarding timbre that adds 50MB to the image size but not sure is happening What you suggest to decrease the image size @borkdude?

borkdude 2021-01-30T16:32:45.078800Z

@ericdallo > while with JVM it takes 90s with graalvm takes 130s, not sure there is anything to improve that though. This is why I think the JVM makes most sense for projects like this, I keep recommending this for my own LSP server for clj-kondo

borkdude 2021-01-30T16:33:04.079Z

As for the image size, you can look at the analysis callgraph or use the dashboard options.

ericdallo 2021-01-30T16:36:58.079200Z

I see ๐Ÿ˜” Besides that, everything is pretty much fast, like diagnostics, code actions, refacotings which improve user xp IMO I still need to make the memory test, then we can think in the tradeoff.

ericdallo 2021-01-30T16:37:12.079400Z

How can I check the analysis callgraph?

borkdude 2021-01-30T16:37:50.079600Z

"-H:+PrintAnalysisCallTree"

borkdude 2021-01-30T16:38:01.079800Z

this will dump some information about the pulled in classes, methods, etc

ericdallo 2021-01-30T16:38:17.080Z

Nice, I'll try

borkdude 2021-01-30T16:38:23.080200Z

and there is also a dashboard option: https://www.graalvm.org/docs/tools/dashboard/?ojr=help%3Btopic%3Dgetting-started.md

borkdude 2021-01-30T16:38:41.080400Z

I haven't used the dashboard that much, but it seems useful. You can drill down into classes

ericdallo 2021-01-30T20:11:30.086900Z

@borkdude for some reason the dashboard flag just gives me OoO even with a -JXmx of 6GB :man-shrugging: I did some hacking to run Graalvm 21.0.0 on My NixOS (docker doesn't work) instead of 20.2.0 and the startup time fall from 130s -&gt; 75s... it's faster than JVM classpath scan ๐Ÿ˜ฎ (~87-90s), I will make more tests, special the memomry one (not sure how to measure that yet)

ericdallo 2021-01-30T20:11:59.087100Z

Also, I added the new -H:+InlineBeforeAnalysis flag, not sure this could cause that performance improvement though

borkdude 2021-01-30T20:13:14.087300Z

@ericdallo Be sure to use the --no-fallback option so you won't get a fallback image, this is effectively just a JVM

borkdude 2021-01-30T20:13:41.087500Z

And yeah, the dashboard data may require moar memory

borkdude 2021-01-30T20:13:45.087700Z

it's a lot of data

ericdallo 2021-01-30T20:15:08.087900Z

yeah, I'm using the --no-fallback, I don't want that kind of lie anymore ๐Ÿ˜‚

borkdude 2021-01-30T20:15:43.088100Z

You could also post an issue about a performance regression on the graalvm issue tracker

ericdallo 2021-01-30T20:18:35.089Z

@borkdude sorry, I probably was not clear, I was using Graalvm 20.2 with all my tests because it's the only available for NixOS while this https://github.com/NixOS/nixpkgs/pull/105815 , also the docker image doesn't work with NixOS for some reason related with shared libs, but I managed to manually install the graalvm 21.0.0 in the nix way

ericdallo 2021-01-30T20:19:12.089700Z

so it's not a regression since I went from 20.2.0 -> 21.0 ๐Ÿ˜„

borkdude 2021-01-30T20:19:22.090Z

so 21 is faster right?

borkdude 2021-01-30T20:19:44.090200Z

ok, then it's good!

ericdallo 2021-01-30T20:20:23.090400Z

yep, at least for the only thing I noticed that it was slower than JVM, now it's not anymore ๐Ÿ˜„

ericdallo 2021-01-31T01:39:29.107600Z

Using the dashboard, I can't see anything that is ocupying a huge space... it's ood that summing all those, it won't be equal to 110MB

ericdallo 2021-01-24T04:03:12.006900Z

A huge context here: https://clojurians.slack.com/archives/CPABC1H61/p1611087562030000 Hey @borkdude I think moving the discussion to here would be better for others with same problems

ericdallo 2021-01-24T04:21:36.007200Z

So, with https://github.com/clojure-lsp/clojure-lsp/pull/267/files it seems it has 2 issues right now at build-time, one regarding log4j and another about some clojure.lang.Agent class. both graalvm says that I should try use --initialize-at-run-time, so after I added --initialize-at-run-time=org.apache.log4j.LogManager, it indeed fixed the log4j error (but probably it'll happen at run-time when I got to work), but I have no idea how to fix the clojure.lang.Agent error Log: https://pastebin.com/cKXDiQPv clojure-lsp uses log4j to log to a custom file /tmp/clojure-lsp.out , LMK If you think this can be replace with other lib that works with graalvm

kulminaator 2021-01-24T07:54:17.007600Z

started using log4j for a project 6 years ago, still regretting it but it's always waiting in the background to get the boot

kulminaator 2021-01-24T07:54:48.007800Z

my usecase is spilling out logs to file or stdout in json format so i can feed them to elastic .... and log4j is not helping me there in any way

kulminaator 2021-01-24T08:03:56.008Z

originally we hoped for interop with other java libraries logging that we use beneath but that never materialized since we need structured logs and not some freetext mayhem

Shantanu Kumar 2021-01-24T11:11:32.008200Z

@kulminaator @ericdallo Havenโ€™t tried with GraalVM native image, but it doesnโ€™t use any reflection or dynamic namespace loading https://cambium-clojure.github.io/ except discovering the current *ns*, which clojure.tools.logging also does.

Shantanu Kumar 2021-01-24T11:27:22.008500Z

A simpler alternative could be java.util.logging that is built into Java. The clojure.tools.logging wrapper loads classes at runtime https://github.com/clojure/tools.logging/blob/master/src/main/clojure/clojure/tools/logging/impl.clj#L52

borkdude 2021-01-24T11:36:15.008800Z

or don't use java logging at all and use timbre for example?

Shantanu Kumar 2021-01-24T12:04:04.009Z

Thereโ€™s a page on logging in native image: https://www.graalvm.org/reference-manual/native-image/Logging/

ericdallo 2021-01-24T14:06:46.009300Z

Thanks for the help @kulminaator @kumarshantanu @borkdude I think replacing the log with another lib it'd be faster indeed, we just need a lib that logs to a custom file and that's it. I'll take a look at timbre, it seems to nave no issues with graalvm Otherwise test https://cambium-clojure.github.io/

ericdallo 2021-01-24T14:56:14.009800Z

@borkdude I made a simple clojure sample using timbre to log to a file (using spit-appender) ang got https://pastebin.com/g9wqxqbv when compiling with graalvm

borkdude 2021-01-24T15:27:35.010400Z

@ericdallo > Error: Frame states being merged are incompatible: unbalanced monitors - locked objects do not match This seems to indicate you are again not using 1.10.2-rc3

borkdude 2021-01-24T15:28:20.010600Z

or hmm timbre is using monitor-enter and monitor-exit by itself? hmmmm

ericdallo 2021-01-24T15:29:19.010800Z

yes, they are using it!

borkdude 2021-01-24T15:29:19.011Z

this is the same problem that Clojure had. You should probably make a PR to timbre to use clojure's locking and not use monitor-enter/exit manually

ericdallo 2021-01-24T15:29:32.011200Z

i'm using latest clojure 1.10.2-rc3

ericdallo 2021-01-24T15:29:40.011400Z

oh, nice

ericdallo 2021-01-24T15:30:10.011600Z

I don't know what should I change, could you provide some materia/links?

ericdallo 2021-01-24T15:31:40.011800Z

I think I should follow this example: https://clojuredocs.org/clojure.core/locking

borkdude 2021-01-24T15:31:42.012Z

see https://github.com/clojure/clojure/commit/f5403e9c666f3281fdb880cb2c21303c273eed2d#diff-313df99869bda118262a387b322021d2cd9bea959fad3890cb78c6c37b448299 this is a very technical low level change, to ensure byte code analyzers can verify that the monitor enter and exit match up. your PR would replace monitor enter/exit to just use clojure's locking and not the current try catch

ericdallo 2021-01-24T15:32:50.012300Z

I see, I'll give a try, thanks @borkdude!

borkdude 2021-01-24T15:33:05.012500Z

you can refer to CLJ-1472 for details

1๐Ÿ‘
borkdude 2021-01-24T15:37:06.012800Z

@ericdallo timbre allows using your own appender, so for now you could copy this code, fix it and use your custom appender

borkdude 2021-01-24T15:37:16.013Z

an upstream PR is of course nicer long term

ericdallo 2021-01-24T15:37:49.013200Z

Oh, good idea, I'm coding the fix for timbre right now, but I'll proceed with a custom adapter until the PR get merged

borkdude 2021-01-24T15:38:42.013400Z

if that code is still reachable native-image might still complain, but you can patch with alter-var-root for the moment being

1๐Ÿ‘
ericdallo 2021-01-24T15:39:59.013700Z

(fn self [data]
        (let [{:keys [output_]} data]
          (let [output (force output_)]
            (locking (Object.) ; For thread safety, Ref. #251
              (try
                (with-open [^java.io.BufferedWriter w (jio/writer fname :append append?)]
                  (.write   w ^String output)
                  (.newLine w))

                (catch java.io.IOException e
                  (if (:spit-appender/retry? data)
                    (throw e) ; Unexpected error
                    (do
                      (jio/make-parents fname)
                      (self (assoc data :spit-appender/retry? true))))))))))
Does that makes sense?

ericdallo 2021-01-24T15:40:24.013900Z

I removed the monitor usages replaced by the locking defmacro

borkdude 2021-01-24T15:41:10.014400Z

should work

ericdallo 2021-01-24T15:45:31.014600Z

https://github.com/ptaoussanis/timbre/pull/334

ericdallo 2021-01-24T16:37:08.015Z

@borkdude would this code be enough to fix the issue right now? https://pastebin.com/y1L3HEPU WIth that I'm still having the same unbalanced monitors - locked objects do not match

ericdallo 2021-01-24T16:45:38.015200Z

This is the https://pastebin.com/gTB7QtB6. Not sure the alter-var-root is having effect

borkdude 2021-01-24T17:10:47.015400Z

The thing with direct linking is, is that this default appender may be used in multiple places, so alter-var-rooting it only at the source spot won't suffice if that's the case

borkdude 2021-01-24T17:11:31.015600Z

what about using a custom appender?

ericdallo 2021-01-24T17:19:53.015800Z

didn't work too, but I released a local version of my change on timbre and got other issue now, will paste the output

ericdallo 2021-01-24T17:23:16.016Z

Oh, actually it worked after adding --allow-incomplete-classpath

ericdallo 2021-01-24T17:23:28.016200Z

noice, another issue solved ๐Ÿ˜„

ericdallo 2021-01-24T17:58:00.016600Z

@borkdude I think this is the only missing issue: https://pastebin.com/JPv6vZ0s About the clojure.lang.Agent

ericdallo 2021-01-24T17:58:30.016800Z

It seems related to the threads that clojure-lsp spawns during the initialize

borkdude 2021-01-24T18:03:29.017Z

yeah, you should not do that on the top level

borkdude 2021-01-24T18:03:32.017200Z

but in the -main

ericdallo 2021-01-24T18:08:29.017400Z

Hum, but we start that on main already: https://github.com/clojure-lsp/clojure-lsp/blob/master/src/clojure_lsp/main.clj#L427

borkdude 2021-01-24T18:45:53.017700Z

you don't launch any agents at the top level?

borkdude 2021-01-24T18:46:45.017900Z

can you pinpoint who is starting this thread?

ericdallo 2021-01-24T18:57:26.018100Z

Yep, that's what I'm trying to understanding why it's not working, we just call this function run from the -main https://github.com/clojure-lsp/clojure-lsp/blob/master/src/clojure_lsp/main.clj#L431

borkdude 2021-01-24T19:04:18.018400Z

@ericdallo if you avoid the run call, do you still get the error?

1
ericdallo 2021-01-24T19:11:50.018800Z

does not work even commenting the run @borkdude ๐Ÿ˜ฎ

ericdallo 2021-01-24T19:14:47.019Z

I'll keep commenting server declaration until the erros is gone, then we ll know what line is the offending one

borkdude 2021-01-24T19:15:15.019200Z

maybe some other lib is spawning a future at the top level

borkdude 2021-01-24T19:15:30.019400Z

try eliminating some libs from the ns decl

borkdude 2021-01-24T19:15:51.019600Z

I've recently encountered this with some lib, don't remember which one it was

ericdallo 2021-01-24T19:18:01.019800Z

Yeah, probably some lsp4j class

borkdude 2021-01-24T19:19:03.020Z

you can try --initialize-at-runtime=&lt;the name of the class&gt;

ericdallo 2021-01-24T19:37:03.020200Z

Ok, found the offender require:

[nrepl.server :as nrepl.server]

ericdallo 2021-01-24T19:37:49.020400Z

weird, since we start the nrepl server here, but requiring it's enough to graal compile blow up: https://github.com/clojure-lsp/clojure-lsp/blob/master/src/clojure_lsp/main.clj#L414

borkdude 2021-01-24T19:39:43.020700Z

which library is this?

ericdallo 2021-01-24T19:39:58.020900Z

nrepl/nrepl {:mvn/version "0.8.3"}

ericdallo 2021-01-24T19:40:25.021100Z

https://github.com/nrepl/nrepl

borkdude 2021-01-24T19:41:18.021400Z

could be one of these... https://github.com/nrepl/nrepl/blob/master/src/clojure/nrepl/server.clj#L5-L15

borkdude 2021-01-24T19:41:30.021800Z

why do you have an nREPL server in LSP?

ericdallo 2021-01-24T19:41:42.022Z

Just searched graalvm and found a issue where you are mentioned ๐Ÿ˜‚ https://github.com/nrepl/nrepl/issues/181

ericdallo 2021-01-24T19:42:10.022300Z

It's for debugging AFAIK @borkdude, @snoe maybe can confirm that

borkdude 2021-01-24T19:43:05.022800Z

it sounds like this could be moved to a dev dependency

borkdude 2021-01-24T19:43:37.023Z

btw, I have an nREPL server in babashka which works with graalvm :P

borkdude 2021-01-24T19:43:41.023200Z

built from the ground up

ericdallo 2021-01-24T19:44:06.023500Z

> it sounds like this could be moved to a dev dependency I think so

borkdude 2021-01-24T19:44:38.023700Z

oh yes, now I remember!

ericdallo 2021-01-24T19:44:41.024100Z

Does babashka just require the same nrepl.server? how does that work hahaha?

borkdude 2021-01-24T19:45:00.024500Z

this completion namespace has top-level futures, very bad

borkdude 2021-01-24T19:45:17.024700Z

I made a fork of this stuff to make reply (the lein REPL) work with graalvm

borkdude 2021-01-24T19:45:34.024900Z

and I turned the futures into delays

ericdallo 2021-01-24T19:45:49.025100Z

๐Ÿ˜ฎ

ericdallo 2021-01-24T19:46:15.025300Z

do you have the code, it'd be helpful to fix on nrepl too, right?

borkdude 2021-01-24T19:49:11.025800Z

I just added you to the repo

borkdude 2021-01-24T19:49:21.026Z

I haven't open sourced the code since it's quite hacky

1๐Ÿ‘
borkdude 2021-01-24T19:49:40.026200Z

and copy/pasted the entire reply lib to hack on it

ericdallo 2021-01-24T19:51:04.026500Z

you have to many repos ๐Ÿ˜‚ that's crazy and awesome at same time ๐Ÿ˜›

borkdude 2021-01-24T19:52:09.026700Z

@ericdallo what you can maybe do is copy paste the nrepl completions namespace into your own repo

borkdude 2021-01-24T19:52:17.026900Z

and then change the futures into delays

borkdude 2021-01-24T19:52:42.027100Z

(or move nrepl to a dev dev... or some feature toggle)

ericdallo 2021-01-24T19:55:06.027300Z

moving to dev deps maybe is enough and the recommended IMO, I'll try to move to another profile so

borkdude 2021-01-24T19:55:35.027500Z

this will also yield a smaller native image

ericdallo 2021-01-24T19:56:37.027700Z

Yes! I just noticed that now it compiles but it seems that the app starts and then finish, while when running via lein run it keeps the socket open waiting for the LSP IO

ericdallo 2021-01-24T20:24:26.027900Z

@borkdude it compiles clojure-lsp perfect, is just that it just exit the app, do I need to make anything more?

ericdallo 2021-01-24T20:26:56.028100Z

I pushed what I done so far here: https://github.com/clojure-lsp/clojure-lsp/pull/267

ericdallo 2021-01-24T20:45:38.028400Z

Oh, actually, it's throwing an exception at runtime and the log is going to the file, I didn't notice it ๐Ÿ˜…

2021-01-24T20:44:40.990Z gregnix-note INFO [clojure-lsp.main:415] - Starting server...
2021-01-24T20:44:41.000Z gregnix-note ERROR [taoensso.timbre:796] - Uncaught exception on thread: main
                                              clojure_lsp.main.main                               
                                                                ...                               
                                             clojure-lsp.main/-main                  main.clj: 436
                                             clojure-lsp.main/-main                  main.clj: 438
                                               clojure-lsp.main/run                  main.clj: 418
          org.eclipse.lsp4j.launch.LSPLauncher.createServerLauncher          LSPLauncher.java:  46
                  org.eclipse.lsp4j.jsonrpc.Launcher$Builder.create             Launcher.java: 321
             org.eclipse.lsp4j.jsonrpc.Launcher$Builder.createProxy             Launcher.java: 364
org.eclipse.lsp4j.jsonrpc.services.ServiceEndpoints.toServiceObject     ServiceEndpoints.java:  41
                                                                ...                               
     com.oracle.svm.reflect.proxy.DynamicProxySupport.getProxyClass  DynamicProxySupport.java: 113
                com.oracle.svm.core.util.VMError.unsupportedFeature              VMError.java:  86
com.oracle.svm.core.jdk.UnsupportedFeatureError: Proxy class defined by interfaces [interface org.eclipse.lsp4j.services.LanguageClient, interface org.eclipse.lsp4j.jsonrpc.Endpoint] not found. Generating proxy classes at runtime is not supported. Proxy classes need to be defined at image build time by specifying the list of interfaces that they implement. To define proxy classes use -H:DynamicProxyConfigurationFiles=&lt;comma-separated-config-files&gt; and -H:DynamicProxyConfigurationResources=&lt;comma-separated-config-resources&gt; options.

ericdallo 2021-01-24T21:01:52.028700Z

Ok, after adding H:DynamicProxyConfigurationResources with those classes it seems to work ๐Ÿ˜„ , let me test it

borkdude 2021-01-24T21:41:26.028900Z

cool