Clojurians Log v2

I can help with turning lsp into a binary, but I'm not sure if that will really help anyone, except for startup time

Oh, if you could help with that PR, it'd be awesome @borkdude, me and @thiagokokada are stuck since we don't know GraalVM very well

Yeah, if someone with more experience with GraalVM could help us it would be awesome

I mostly copied some things from clj-kondo/babashka and tried to adapt it

But I really don't understand neither GraalVM nor Java too much so I got stucked afterwards

@ericdallo I chose to keep the VSCode LSP plugin as a Java project, since it's a server and Java makes more sense for long running processes imo

I see, well, I think we'll need to check how to make that happens for clojure-lsp that is made in clojure 😕

@ericdallo I mean, the JVM. Clojure on the JVM.

A JVM is better suited for high throughput due to JIT

A graalvm image is better suited for doing short-lived things

But if you have other reasons to make a native image, and you can point me to an issue, I can give some hints probably

I see, clojure-lsp uses the https://github.com/BrunoBonacci/lein-binplus to build a binary with the jar bundled but user still needs java installed, so do you think for clojure-lsp keeps as a JVM would fit better than a Graalvm native image?

Ah I see. The way I do this for IntelliJ LSP for example for clj-kondo is just distribute an uberjar

The reason I did this, is also that the user didn't have to install anything else when using VSCode

No external binaries

We also distrubute the jar in the Releases if user don't want to use the binary, for example Calva folks use the jar instead of the binary

Yes, I super like that, that's another reason we thought about using GraalVM

And other editor which use the binary version of clj-kondo usually don't use the LSP, so there the binary makes sense to me. But if you want to make it all native, sure we can do that, if all your deps cooperate. I'm not sure which deps you are using. Maybe you can do lein deps :tree or clj -Stree ?

[clj-kondo "2020.12.13-20210119.112957-43"]
   [borkdude/sci "0.2.1"]
     [borkdude/edamame "0.0.11-alpha.28"]
     [borkdude/sci.impl.reflector "0.0.1"]
   [cheshire "5.10.0"]
     [com.fasterxml.jackson.core/jackson-core "2.10.2"]
     [com.fasterxml.jackson.dataformat/jackson-dataformat-cbor "2.10.2"]
     [com.fasterxml.jackson.dataformat/jackson-dataformat-smile "2.10.2"]
     [tigris "0.1.2"]
   [com.cognitect/transit-clj "1.0.324"]
     [com.cognitect/transit-java "1.0.343"]
       [commons-codec "1.10"]
       [javax.xml.bind/jaxb-api "2.3.0"]
       [org.msgpack/msgpack "0.6.12"]
         [com.googlecode.json-simple/json-simple "1.1.1" :exclusions [[junit]]]
         [org.javassist/javassist "3.18.1-GA"]
   [io.lambdaforge/datalog-parser "0.1.7"]
   [nrepl/bencode "1.1.0"]
 [cljfmt "0.7.0" :exclusions [[rewrite-cljs]]]
   [com.googlecode.java-diff-utils/diffutils "1.3.0"]
   [org.clojure/tools.cli "1.0.194"]
   [rewrite-clj "0.6.1"]
 [clojure-complete "0.2.5" :exclusions [[org.clojure/clojure]]]
 [com.google.guava/guava "30.1-jre"]
   [com.google.code.findbugs/jsr305 "3.0.2"]
   [com.google.errorprone/error_prone_annotations "2.3.4"]
   [com.google.guava/failureaccess "1.0.1"]
   [com.google.guava/listenablefuture "9999.0-empty-to-avoid-conflict-with-guava"]
   [com.google.j2objc/j2objc-annotations "1.3"]
   [org.checkerframework/checker-qual "3.5.0"]
 [com.taoensso/tufte "2.2.0"]
   [com.taoensso/encore "3.1.0"]
     [com.taoensso/truss "1.6.0"]
 [digest "1.4.10"]
 [log4j "1.2.17" :exclusions [[javax.mail/mail] [javax.jms/jms] [com.sun.jdmk/jmxtools] [com.sun.jmx/jmxri]]]
 [medley "1.3.0"]
 [nrepl "0.8.3"]
 [org.clojure/clojure "1.10.1"]
   [org.clojure/core.specs.alpha "0.2.44"]
   [org.clojure/spec.alpha "0.2.176"]
 [org.clojure/core.async "1.3.610"]
   [org.clojure/tools.analyzer.jvm "1.1.0"]
     [org.clojure/core.memoize "1.0.236"]
       [org.clojure/core.cache "1.0.207"]
         [org.clojure/data.priority-map "1.0.0"]
     [org.clojure/tools.analyzer "1.0.0"]
     [org.ow2.asm/asm "5.2"]
 [org.clojure/data.json "1.0.0"]
 [org.clojure/tools.logging "1.1.0"]
 [org.clojure/tools.reader "1.3.4"]
 [org.eclipse.lsp4j "0.10.0" :exclusions [[org.eclipse.xtend/org.eclipse.xtend.lib]]]
   [org.eclipse.lsp4j/org.eclipse.lsp4j.generator "0.10.0"]
   [org.eclipse.lsp4j/org.eclipse.lsp4j.jsonrpc "0.10.0"]
     [com.google.code.gson/gson "2.8.2"]
 [org.eclipse.xtend/org.eclipse.xtend.lib "2.25.0.M1" :exclusions [[com.google.guava/guava]]]
   [org.eclipse.xtend/org.eclipse.xtend.lib.macro "2.25.0.M1"]
   [org.eclipse.xtext/org.eclipse.xtext.xbase.lib "2.25.0.M1"]
 [org.xerial/sqlite-jdbc "3.34.0"]
 [seancorfield/next.jdbc "1.1.613"]
   [org.clojure/java.data "1.0.78"]
 [trptcolin/versioneer "0.2.0"]

since we also have a lot of final users, like emacs, vim, vscode a native image would be useful IMO

do you use sqlite ?

Yep, for persisting our atom as sqlite in the .lsp/sqlite.db

but this can change with kondo 100% integration

and I know there are faster ways of persisting that

This is the only thing I see in here that I never got to work with graalvm. This is why I (and lispyclouds) wrote a babashka sqlite pod in go instead of graal. The way I persist information to disk in clj-kondo is through transit, no db. But there is another option: hsqldb works with graalvm and is similar to sqlite.

So if you could remove sqlite or swap it out with hsqldb, it could work.

It would be good to check the license of hsqldb though, I think LSP has to remain fully open source

Nice, thanks for the info, we have https://github.com/clojure-lsp/clojure-lsp/issues/234 with some concerns already pointed by @snoe

you are linking to the issues page - did you mean a specific issue?

oh sorry, fixed

Datalevin also doesn't work with GraalVM last time I tried it

If you want to try out and see that HSQLDB works with babashka natively: https://github.com/babashka/pod-registry/blob/master/examples/hsqldb.clj

Cool, I think would fit very well for us, we just need to check the license, I don't know too much about these things though

I did try sqlite and things like this in the very beginning of clj-kondo, since I had similar ideas as in that issue. But eventually I partitioned things basically by namespace and try to load only when necessary, minimizing I/O

Also discovering an issue with transit along the way: https://github.com/cognitect/transit-clj/issues/43

I think one of the common uses of persisted db on clojure-lsp, is the inital cache, but this may be fixed after kondo 100% integration, since we'll use clj-kondo cache

You mean the analysis output right?

yes

@snoe made some testings with that, he may know more about that

Yeah, this is exciting, since other tools can re-use this too. Nice convergence of tooling

Totally, After this migration I intend to open a PR improving the docs of the analysis, that's awesome and users need to know more about it

That's what we use from sqlite, just simple insert/query https://github.com/clojure-lsp/clojure-lsp/blob/master/src/clojure_lsp/db.clj

How big does this data tend to get, how many rows?

Maybe you could just use .edn or .json

or transit

good question, AFAIK, we only save a row, but the analysis column may be huge, since it's the kondo output

It looks weird to me that query, but I never noticed a performance issue

This is probably because you do a lot of in memory stuff and only persist when shutting down?

On master (before kondo integration), we persist the crawled jars, that should not change between lsp restart, only if user change deps which we will re-scan/persist

right

with clj-kondo integration we are saving all the analysis, but that may be wrong, since the analysis contains analysis from the external jars and from the project which will easily change

Maybe we should persist only the analysis from kondo on jars too

> But if you have other reasons to make a native image, and you can point me to an issue, I can give some hints probably Well, for me it is most an experiment, I think clojure-lsp startup does slow down the general experience of editting Clojure in Emacs (since it takes a while until it is ready), so I think GraalVM can help with this

Also, I don't think clojure-lsp is really that much CPU intensive for JIT in JVM to makes a difference. I may be wrong though

The slow down time from clojure-lsp is most related to the jars crawling, changing to GraalVM won't change that

We should gain a few secs, but not compared with minutes we take scanning huge projects

Hopefully this will change with clj-kondo analysis PR

> We should gain a few secs, but not compared with minutes we take scanning huge projects Yeah, but this is exactly what I want. A few seconds impacts a lot the feeling of a fast tooling

Try to add just 0.5 sec to your ls for example:

ls() { sleep 0.5; ls $@; }

Considering that Clojure alone takes more than half a second to start I still think it is worth to pursuit this, even if is just for a test

But well, this is a IMO

Yeah, I see your point and agree, I was just saying that for some cases, take 40 secs and take 35 secs may have no difference for most users

but I agree, every performance improvement is welcome

> We should gain a few secs, but not compared with minutes we take scanning huge projects Ah, now I understood what you said here (I thought you said GraalVM wouldn't bring improvements compared to other huge projects using it, not that clojure-lsp itself takes minutes to scan huge projects)

I still think GraalVM can helps a with this initial scanning, specially if this initial scanning happens when the VM is cold

(But only if the issue is CPU bound, if it is IO bound it will not help)

One of the reasons I picked sqlite is because it is very very good at handling all the intricacies of os io. On the kondo-pr I am also persisting the analysis to the db as it cuts startup from 20s to 5s. https://github.com/oracle/graal/issues/966 this might be an option for graal, seems like people had some success.

I just made a simple test compiling with native-image a simple sample project using the same sqlite db we use with clojure-lsp and it worked like a charm 🙂 I'll try to check why @thiagokokada branch is not working

huh... sqlite worked with graalvm?

I want to see the sample project :)

I'll push it and give you the link, one sec

I hope I did everything correct 😅

It's really working, but it gives some exceptions during the build: (but the exceptions shows up even if a simple println clojure project :man-shrugging:)

this is the sample: https://github.com/ericdallo/clojure-sample

Ah, this explains it. It's a fallback image. This is not really a native binary, but just a kind of uberjar.

Compile with --no-fallback

oh... that's sad 😕

I see, probably it'll not work so

yeah, just saw the warning log right now :man-facepalming:

Here are some standard args I always use:

args=("-H:+ReportExceptionStackTraces"
       "-H:ReflectionConfigurationFiles=reflection.json"
       "--initialize-at-build-time"
       "-H:Log=registerResource:"
       "-H:EnableURLProtocols=http,https,jar"
       "--enable-all-security-services"
       "-H:+JNI"
       "--verbose"
       "--no-fallback"
       "--no-server"
       "--report-unsupported-elements-at-runtime"
       "--native-image-info"
       "--verbose")

Ok, it gave me a lot of exceptions indeed (probably related to the reflection.json I'm not using like clj-kondo uses)

I'll step back and try to make it work without sqlite then try to add it

sorry for the false alarm @borkdude and thanks for the help

:thumbsup:

For babashka we made this: https://github.com/babashka/pod-babashka-sqlite3 It is a component which runs sqlite in a go binary. Small and fast. And it communicates using JSON back and forth, a bit like a webservice

transit actually

You could use this to workaround your issues, using the JVM pod library which also works in a GraalVM binary

but it's a bit of a workaround ;)

yeah, I see, do you think it'd work well with clojure-lsp ?

I think it would.

It supports even sending over binary blobs

I wouldn't just go and distribute this native approach to production very soon, first run some tests with power users

really nice work on that project haha 👏 I'll try to debug graalvm a little more with some more tests and follow the above post from snoe and if does not work I'll try to use pod-babashka-sqlite3

as you might run into issues with macOS users and permissions

yes, this needs to be done carefully indeed

if it works, I could test with coworkers that use MacOS on huge Nubank projects

yeah

You will need to use this lib as a JVM lib: https://github.com/babashka/pods And then do (load-pod ...) as in the README of the sqlite3 pod

I guess you aren't using deps.edn right?

@borkdude one more question 😅 those deps:https://github.com/clj-kondo/clj-kondo/blob/master/project.clj#L21-L22 they are for something clj-kondo does specific? do you think we'd need to make graalvm work on clojure-lsp too?

I am working on a PR for babashka which uses GraalVM 21 and it doesn't seem to be needed anymore.

You will need it with 20.3.0

but go with 21 I'd say

nice (I'ḿ using 20.3.0 ATM)

BTW, we are using deps.edn , not sure if it's the correct way to use though, we kind of import it from project.clj

oh then it's all fine, I was wondering if I should make a clojars release for the pods lib, but you can use it as a git lib then

Oh got it 👍

Eh I'd say it's a lein project more than deps.edn. Not sure where equivalents to lein bin and test-refresh are these days.

Well, I managed to make it work with org.xerial/sqlite-jdbc , persisting and selecting from sql from a java Class in clojure

But I'm having issues with graalvm work with next.jdbc

@ericdallo I have several working examples of next.jdbc and GraalVM, this was never the issue for me.

The driver was usually the problem

so what error are you getting?

Yeah, I saw that issue where you give some examples 😛 This is the stack: https://pastebin.com/Jsw45Rjr

oh, sorry wrong paste

https://pastebin.com/Jn3vj7LE

Can you also paste ./scripts/native-compile.sh ?

yes, I'm pushing the code right now 🙂

@ericdallo you need to use at least clojure 1.10.2+ (rc3 now)

to avoid the infamous locking error

CLJ-1472

(https://clojure.atlassian.net/browse/CLJ-1472)

Oh, good to know

https://github.com/ericdallo/clojure-sample/blob/master/scripts/native-compile.sh

Basically it uses this class which works: https://github.com/ericdallo/clojure-sample/blob/master/src-java/clojure_sample/SampleDb.java

This does works to me, what means that sql is working, right? So if next.jdbc works, basically our problems are gone, right?

Sorry for the code @borkdude it's a mess, I'm doing a lot of tests

I'm checking your examples with next.jdbc, does https://github.com/babashka/babashka-sql-pods/blob/5d04e7fce699741e1b5e8488cf0a8f70843a3e2e/script/compile#L54 works with graalvm 20.3.0?

it does yes

Ok, probably your example is using some flag that I missing, I'll try to find what is

@ericdallo What error do you get after using 1.10.2-rc3 though?

That should be done first

After that, there should basically be no problems, when your reflection config has all the things needed

I'm compiling right now with that version 😆

Oh my, I really hope 🤞

I spent all day compiling hahahah my notebook need some rest 😛

It worked!

I can't believe it 😄

This is big news! Are you using 20.3.0?

I'll clean the code and do other push, thank you for the help @borkdude with that it's proved that we can compile graalvm app using sqlite!

Actually I'm using 20.2.0 since it's the latest available for NixOS users

Yeah, I think a PR to this repo is in order: https://github.com/babashka/babashka-sql-pods/ We already support postgres, hsqldb and now oracle

But I can do that myself too using your config

So the trick was using latest clojure and the custom JNI class

custom JNI class?

Yeah, without this on classpath the sql wasn't working : https://github.com/ericdallo/clojure-sample/blob/master/src-java/clojure_sample/JNIReflectionClasses.java

I could create a fix jar like you did for clojure-reflector

WDYT?

Ah, very interesting, I've never used this before. I wonder if it works on macOS as well. I don't think this class is actually needed once you use the generated reflection json

could be wrong though

Hum, the json doesn't include org.sqlite custom reflection classes/methods

but I'll try right now

ah it only adds it to the build, but doesn't spit it out to the reflection.json?

but you are logging these classes

so you could manually add those

Hum, yeah, true indeed

It's a lot of classes, don't you think a external jar with that fix would be better?

how do I create the uberjar that your script/native-compile.sh needs?

I'm getting:

$ lein with-profiles +native-image "do" clean, uberjar
OpenJDK 64-Bit Server VM warning: forcing TieredStopAtLevel to full optimization because JVMCI is enabled
Could not find artifact rewrite-clj:rewrite-clj:jar:0.6.3-SNAPSHOT in clojars (<https://repo.clojars.org/>)

Oh sorry, this was another test of mine hahaha of another issue, you can change it to use 0.6.1 on deps.edn

you are using 1 repo for all your issues? great way to mix problems ;)

Fatal error:com.oracle.svm.core.util.VMError$HostedError: Option name "TruffleMultiThreaded" has multiple definitions: com.oracle.svm.truffle.api.SubstrateTruffleOptions.TruffleMultiThreaded and com.oracle.svm.truffle.api.SubstateTruffleOptions.TruffleMultiThreaded

hahaha yeah, I usually don't do that, but today I got things messed 😅

Hum, I didn't have this error :thinking_face:

this is probably a conflict between 20.2.0 stuff and 20.3.0 stuff

in 21 we don't need all this stuff anymore I think

but now it's running

got it

so I think when you add all the > Declaring class: org.sqlite.core.NativeDB to the reflection config (with all the methods, etc), then it should also work

Yes!

I'm so glad this works now 😄

so where are you now using sqlite, or is this only the java version?

I'll clean up this repo and externalize the JNI class to a fix jar

I'm using in the sqlite ns

clojure-sample.sqlite

this is all commented out in my checkout

Like I said: I don't think a library/jar is needed for this, you can probably add the classes to the reflection config

org.sqlite.core.NativeDB
org.sqlite.Function
org.sqlite.Function.Aggregate
org.sqlite.ProgressHandler
org.sqlite.Function.Window
org.sqlite.core.DB.ProgressObserver

Ok, I'll try then

Ah I see, you also need to have some JNI json file:

-H:JNIConfigurationFiles=/path/to/jniconfig

The difference between your class and a .json file is that your class registers these things at runtime, and a .json file is the declarative way of doing the same

you mean we need both the reflection json file with those classes and this H:JNIConfigurationFiles?

I didn't used this H:JNIConfigurationFiles , why do we need it?

Like I said: the configuration file is the declarative way of doing the same you are doing with the Java class now

both are probably fine, but that logic is often tied to a specific graalvm version whereas a .json file is not

I see but what is the difference between ReflectionConfigurationFiles and H:JNIConfigurationFiles ?

so if you can get it working with .json files, that is much better for the future

I'll try to use the json file, I just don't get it If I should add the classes to ReflectionConfigurationFiles or to the H:JNIConfigurationFiles

> JNI supports looking up classes by their names, and looking up methods and fields by their names and signatures. This requires keeping the necessary metadata for these lookups around. The native image builder must know beforehand which items will be looked up in case they might not be reachable otherwise and therefore would not be included in the native image. Moreover, the native image builder must generate call wrapper code ahead-of-time for any method that can be called via JNI. Therefore, specifying a concise list of items that need to be accessible via JNI guarantees their availability and allows for a smaller footprint.

I think all the methods that use JNI need to be also in the JNI .json file

https://www.graalvm.org/reference-manual/native-image/JNI/

I see, so I think I'll need to keep working to make it work with json files only 😄

thanks for the huge help explaining those things @borkdude

I don't know if these classes need to be in both files, or if jni config implies reflection config, but maybe try both :)

ok, I'm off now

late

good night!

thanks

I could not make it work with the jni json file 😞 This is how it looks: https://pastebin.com/MsX5Jm7q

I'm not sure if I should specify each method/constructor/field manually, or if these all* should work