Clojurians Log v2

hey, guys. I have a question when using clojure-lsp as language server. when typing code like (f x), I found the company-mode wouldn't work, but if I type f, company-mode would work. what the hell ???

Hi @steiner3044 we ATM have a issue that we can't complete the code if it's not parseable (a valid clojure code), I'm trying to fix that and I opened this issue recently: https://github.com/clj-commons/rewrite-clj/issues/87

wow, thank you for your work, I will watch this:slightly_smiling_face:

Thanks 😄

So I guess this explains an issue I had in my notes with clojure-lsp completions, (that the first time I type something like foo/ I don't see suggestions, but if I backspace the slash and type it again I do)

Something like that at least

yep, that's something I'd love to fix 😅

What conflicts do you mean? I moved to Spacemacs with Cider + lsp-mode month ago and I did not experience any issue yet. Edit: did i just find one @lvh? Namespace is not filled automatically when creating a new file in a project.

I just made a simple test compiling with native-image a simple sample project using the same sqlite db we use with clojure-lsp and it worked like a charm 🙂 I'll try to check why @thiagokokada branch is not working

huh... sqlite worked with graalvm?

I want to see the sample project :)

I'll push it and give you the link, one sec

I hope I did everything correct 😅

It's really working, but it gives some exceptions during the build: (but the exceptions shows up even if a simple println clojure project :man-shrugging:)

this is the sample: https://github.com/ericdallo/clojure-sample

Ah, this explains it. It's a fallback image. This is not really a native binary, but just a kind of uberjar.

Compile with --no-fallback

oh... that's sad 😕

I see, probably it'll not work so

yeah, just saw the warning log right now :man-facepalming:

Here are some standard args I always use:

args=("-H:+ReportExceptionStackTraces"
       "-H:ReflectionConfigurationFiles=reflection.json"
       "--initialize-at-build-time"
       "-H:Log=registerResource:"
       "-H:EnableURLProtocols=http,https,jar"
       "--enable-all-security-services"
       "-H:+JNI"
       "--verbose"
       "--no-fallback"
       "--no-server"
       "--report-unsupported-elements-at-runtime"
       "--native-image-info"
       "--verbose")

Ok, it gave me a lot of exceptions indeed (probably related to the reflection.json I'm not using like clj-kondo uses)

I'll step back and try to make it work without sqlite then try to add it

sorry for the false alarm @borkdude and thanks for the help

:thumbsup:

For babashka we made this: https://github.com/babashka/pod-babashka-sqlite3 It is a component which runs sqlite in a go binary. Small and fast. And it communicates using JSON back and forth, a bit like a webservice

transit actually

You could use this to workaround your issues, using the JVM pod library which also works in a GraalVM binary

but it's a bit of a workaround ;)

yeah, I see, do you think it'd work well with clojure-lsp ?

I think it would.

It supports even sending over binary blobs

I wouldn't just go and distribute this native approach to production very soon, first run some tests with power users

really nice work on that project haha 👏 I'll try to debug graalvm a little more with some more tests and follow the above post from snoe and if does not work I'll try to use pod-babashka-sqlite3

as you might run into issues with macOS users and permissions

yes, this needs to be done carefully indeed

if it works, I could test with coworkers that use MacOS on huge Nubank projects

yeah

You will need to use this lib as a JVM lib: https://github.com/babashka/pods And then do (load-pod ...) as in the README of the sqlite3 pod

I guess you aren't using deps.edn right?

@borkdude one more question 😅 those deps:https://github.com/clj-kondo/clj-kondo/blob/master/project.clj#L21-L22 they are for something clj-kondo does specific? do you think we'd need to make graalvm work on clojure-lsp too?

I am working on a PR for babashka which uses GraalVM 21 and it doesn't seem to be needed anymore.

You will need it with 20.3.0

but go with 21 I'd say

nice (I'ḿ using 20.3.0 ATM)

BTW, we are using deps.edn , not sure if it's the correct way to use though, we kind of import it from project.clj

oh then it's all fine, I was wondering if I should make a clojars release for the pods lib, but you can use it as a git lib then

Oh got it 👍

Eh I'd say it's a lein project more than deps.edn. Not sure where equivalents to lein bin and test-refresh are these days.

Well, I managed to make it work with org.xerial/sqlite-jdbc , persisting and selecting from sql from a java Class in clojure

But I'm having issues with graalvm work with next.jdbc

@ericdallo I have several working examples of next.jdbc and GraalVM, this was never the issue for me.

The driver was usually the problem

so what error are you getting?

Yeah, I saw that issue where you give some examples 😛 This is the stack: https://pastebin.com/Jsw45Rjr

oh, sorry wrong paste

https://pastebin.com/Jn3vj7LE

Can you also paste ./scripts/native-compile.sh ?

yes, I'm pushing the code right now 🙂

@ericdallo you need to use at least clojure 1.10.2+ (rc3 now)

to avoid the infamous locking error

CLJ-1472

(https://clojure.atlassian.net/browse/CLJ-1472)

Oh, good to know

https://github.com/ericdallo/clojure-sample/blob/master/scripts/native-compile.sh

Basically it uses this class which works: https://github.com/ericdallo/clojure-sample/blob/master/src-java/clojure_sample/SampleDb.java

This does works to me, what means that sql is working, right? So if next.jdbc works, basically our problems are gone, right?

Sorry for the code @borkdude it's a mess, I'm doing a lot of tests

I'm checking your examples with next.jdbc, does https://github.com/babashka/babashka-sql-pods/blob/5d04e7fce699741e1b5e8488cf0a8f70843a3e2e/script/compile#L54 works with graalvm 20.3.0?

it does yes

Ok, probably your example is using some flag that I missing, I'll try to find what is

@ericdallo What error do you get after using 1.10.2-rc3 though?

That should be done first

After that, there should basically be no problems, when your reflection config has all the things needed

I'm compiling right now with that version 😆

Oh my, I really hope 🤞

I spent all day compiling hahahah my notebook need some rest 😛

It worked!

I can't believe it 😄

This is big news! Are you using 20.3.0?

I'll clean the code and do other push, thank you for the help @borkdude with that it's proved that we can compile graalvm app using sqlite!

Actually I'm using 20.2.0 since it's the latest available for NixOS users

Yeah, I think a PR to this repo is in order: https://github.com/babashka/babashka-sql-pods/ We already support postgres, hsqldb and now oracle

But I can do that myself too using your config

So the trick was using latest clojure and the custom JNI class

custom JNI class?

Yeah, without this on classpath the sql wasn't working : https://github.com/ericdallo/clojure-sample/blob/master/src-java/clojure_sample/JNIReflectionClasses.java

I could create a fix jar like you did for clojure-reflector

WDYT?

Ah, very interesting, I've never used this before. I wonder if it works on macOS as well. I don't think this class is actually needed once you use the generated reflection json

could be wrong though

Hum, the json doesn't include org.sqlite custom reflection classes/methods

but I'll try right now

ah it only adds it to the build, but doesn't spit it out to the reflection.json?

but you are logging these classes

so you could manually add those

Hum, yeah, true indeed

It's a lot of classes, don't you think a external jar with that fix would be better?

how do I create the uberjar that your script/native-compile.sh needs?

I'm getting:

$ lein with-profiles +native-image "do" clean, uberjar
OpenJDK 64-Bit Server VM warning: forcing TieredStopAtLevel to full optimization because JVMCI is enabled
Could not find artifact rewrite-clj:rewrite-clj:jar:0.6.3-SNAPSHOT in clojars (<https://repo.clojars.org/>)

Oh sorry, this was another test of mine hahaha of another issue, you can change it to use 0.6.1 on deps.edn

you are using 1 repo for all your issues? great way to mix problems ;)

Fatal error:com.oracle.svm.core.util.VMError$HostedError: Option name "TruffleMultiThreaded" has multiple definitions: com.oracle.svm.truffle.api.SubstrateTruffleOptions.TruffleMultiThreaded and com.oracle.svm.truffle.api.SubstateTruffleOptions.TruffleMultiThreaded

hahaha yeah, I usually don't do that, but today I got things messed 😅

Hum, I didn't have this error :thinking_face:

this is probably a conflict between 20.2.0 stuff and 20.3.0 stuff

in 21 we don't need all this stuff anymore I think

but now it's running

got it

so I think when you add all the > Declaring class: org.sqlite.core.NativeDB to the reflection config (with all the methods, etc), then it should also work

Yes!

I'm so glad this works now 😄

so where are you now using sqlite, or is this only the java version?

I'll clean up this repo and externalize the JNI class to a fix jar

I'm using in the sqlite ns

clojure-sample.sqlite

this is all commented out in my checkout

Like I said: I don't think a library/jar is needed for this, you can probably add the classes to the reflection config

org.sqlite.core.NativeDB
org.sqlite.Function
org.sqlite.Function.Aggregate
org.sqlite.ProgressHandler
org.sqlite.Function.Window
org.sqlite.core.DB.ProgressObserver

Ok, I'll try then

Ah I see, you also need to have some JNI json file:

-H:JNIConfigurationFiles=/path/to/jniconfig

The difference between your class and a .json file is that your class registers these things at runtime, and a .json file is the declarative way of doing the same

you mean we need both the reflection json file with those classes and this H:JNIConfigurationFiles?

I didn't used this H:JNIConfigurationFiles , why do we need it?

Like I said: the configuration file is the declarative way of doing the same you are doing with the Java class now

both are probably fine, but that logic is often tied to a specific graalvm version whereas a .json file is not

I see but what is the difference between ReflectionConfigurationFiles and H:JNIConfigurationFiles ?

so if you can get it working with .json files, that is much better for the future

I'll try to use the json file, I just don't get it If I should add the classes to ReflectionConfigurationFiles or to the H:JNIConfigurationFiles

> JNI supports looking up classes by their names, and looking up methods and fields by their names and signatures. This requires keeping the necessary metadata for these lookups around. The native image builder must know beforehand which items will be looked up in case they might not be reachable otherwise and therefore would not be included in the native image. Moreover, the native image builder must generate call wrapper code ahead-of-time for any method that can be called via JNI. Therefore, specifying a concise list of items that need to be accessible via JNI guarantees their availability and allows for a smaller footprint.

I think all the methods that use JNI need to be also in the JNI .json file

https://www.graalvm.org/reference-manual/native-image/JNI/

I see, so I think I'll need to keep working to make it work with json files only 😄

thanks for the huge help explaining those things @borkdude

I don't know if these classes need to be in both files, or if jni config implies reflection config, but maybe try both :)

ok, I'm off now

late

good night!

thanks