Clojurians Log v2

I'm trying to build my project (https://git.itwont.work/samples) with graalvm and it just runs for a long time (like, half an hour) then runs out of memory

@nihilazo Instead of starting with a big project, it's often better to start with smaller parts and then add things bit by bit to see what's causing this

for reference, all my own graal projects compile within 2 minutes on CircleCI. if it takes longer, I find that suspicious

hmm, I'll try compiling a hello world project and see if that works

should I be using the jdk8 or jdk11 version of graalvm for clojure?

jvm11 works, but I've ran into some issues with it (unlikely that you will encounter them with hello world)

I'm still using jdk8

@nihilazo Are you trying to graalvm compile datahike? That's slightly more advanced than hello world I would say?

last time I tried that, it didn't work for me

oh sorry, I misread. yeah, hello world should work

https://github.com/lread/clj-graal-docs/blob/master/doc/hello-world.md

yeah, I was trying to graalvm compile datahike

which is probably a bad idea

I'll see if the hello world works

I tried that before, but that didn't work for me. that's why I ended up using transit instead of datahike in clj-kondo (I wasn't even able to make a performance benchmark because I never got that far)

if you want to have a fast CLI solution with a database you could consider babashka + the sqlite-cli: https://github.com/borkdude/babashka/blob/master/examples/sqlite.clj

I wasn't able to compile sqlite itself with graalvm either

I'm not sure I even really need a database? The queries are super simple (literally just checking if something has a tag, and checking if it doesn't) but there is potentially a reasonable amount of things being checked

actually, that said, I wonder if datahike is actually worse performance-wise than doing something like that manually

@nihilazo What I did for clj-kondo is split the data up into partitions so I can choose to only read from disk what I need. you can tailor this to your problem to make it performant

transit is compact and fast to read/write

and works with graalvm

ah, for this in every case I want to search all the data, and it's stored in a text format on disk already, I might try doing it without any DB at all and see how it goes

might even be able to do everything I was doing with datahike using a couple of filter s

but using a database seemed like the more "proper" way

@nihilazo JUXT published a post today where they plowed through some text files with babashka: https://juxt.pro/blog/posts/babashka.html

if you can make the search faster by pre-organizing it into an indexed way first, that can help

compiling the new datahike-less version results in some unsupported things, but that's progress!

also, the hello world works perfectly

I'm getting the errors from clojure.spec.gen.alpha, is it workaround time?

@nihilazo Are you loading clojure.spec yourself from anywhere?

is it the CLJ-1472 error?

https://github.com/lread/clj-graal-docs/blob/master/CLJ-1472/README.md

maybe, I'm not pulling in spec myself but I am pulling in tools.cli

actually, I don't need tools.cli, I can just remove that

tools.cli should work without this error, I've used it in a couple of projects

the only thing I'm using outside of my own code is http://clojure.java.io and clojure.string

your own code could be triggering it. are you using things like resolve?

also, it could be that you have to set the macro spec checking off

check out the options that I'm using here: https://github.com/borkdude/jet/blob/master/script/compile

ah, maybe I need the -J-Dclojure.spec.skip-macros=true line

could very well be

hmm, with that I'm getting errors due to spec as well, despite not using spec at all

btw, using http://clojure.java.io without a require is bad practice: https://git.itwont.work/samples/tree/src/samples/core.clj?h=no-datahike#n61

there's probably bad practice all over this thing tbh, I'm fairly new to clojure

I don't mean it to be offending, the reason it's "bad" is that you should not trust other tools to load http://clojure.java.io for you. If you use a linter like clj-kondo, it will warn you about such practices.

no offence taken, just warning you that most of it is probably bad

and thanks for trying to help me figure this out

might be worth building the patched spec anyway?

I'm not sure. your project doesn't seem to be doing anything weird. does it work with clojure 1.9?

seems like it

just tested with 1.9 and it works fine

when I try a graal build with 1.9, I get a different error, Error: No instances of <http://java.io|java.io>.FilePermission are allowed in the image heap as this class should be initialized at image runtime.

oh are you by any chance using graal v20?

try 19.3.1 and report this error on the github graal project. @sfyire also got it

I'm using whatever the graalvm in AUR is, I'll try 19.3.1

yeah, 20 just got out and has a couple of issues. But reporting those issues definitely helps and they will be fixed.

19.3.1 is the LTS release

https://www.graalvm.org/docs/release-notes/version-roadmap/

alright

gotta love arch linux for having the latest everything, even when the latest doesn't work

can graalvm not access environment variables? It builds perfectly with the older graalvm and older clojure but I use System/getenv and that doesn't seem to be working, seems to fail

won't read in the variable

@nihilazo It depends if you're doing that at compile time or run time. When you execute that call at compile time, the call is made too early

even bb can do this:

$ bb '(System/getenv "PWD")'
"/Users/borkdude/dre/DocSearch/app"

yeah, so the problem here is that you're doing it top-level: https://git.itwont.work/samples/tree/src/samples/core.clj?h=no-datahike#n15

that won't work, since that captures the value of that environment variable when you compile it, not when you run it

that doesn't seem like the case when I run it though

when I run it from the jar, it reacts to whatever the variable is at that point

I can change it between runs and it changes

just make it into a variable inside -main

I wonder, is your jar AOT-ed?

everything is AOT-ed

then I would even expect that not to work on the JVM

anyway, this is a problem I've had before and the solution is to make reading the environment variable a call at runtime, i.e. in the execution of -main, not top-level/before it.

ok, thanks

everything is now working on graalVM! Thank you for helping me out and answering my stupid questions

they're not stupid and you're welcome!

if you can make a tiny repro of the: > No instances of http://java.io.FilePermission are allowed in the image heap as this class should be initialized at image runtime that would be helpful for fixing it in v20

usually posting your AOT-ed uberjar + native-image build instructions is sufficient for the graalvm devs to reproduce it

so they won't have to bother with compiling clojure

I'll try that, not sure exactly what the cause was though

wasn't it just v20 jdk8 versus v19 jdk8?

it was v20 jdk11 vs v19 jdk11

it worked in v19 and not v20

but I'm not sure what the thing in the code was that caused it so I can do a tiny repro

well, that's up for them to find out I think. just saying: this works with graalvm 19.1.3 jdk 11 on macOS/linux/Windows but not on v20 .... is sufficient, provided you give them the uberjar + build arguments of native-image

the repo is here: https://github.com/oracle/graal

@nihilazo Here's an example: https://github.com/oracle/graal/issues/2136#issuecomment-587865438

ah, ok

that makes things easier

thanks