I'm trying to build my project (https://git.itwont.work/samples) with graalvm and it just runs for a long time (like, half an hour) then runs out of memory
@nihilazo Instead of starting with a big project, it's often better to start with smaller parts and then add things bit by bit to see what's causing this
for reference, all my own graal projects compile within 2 minutes on CircleCI. if it takes longer, I find that suspicious
hmm, I'll try compiling a hello world project and see if that works
should I be using the jdk8 or jdk11 version of graalvm for clojure?
jvm11 works, but I've ran into some issues with it (unlikely that you will encounter them with hello world)
I'm still using jdk8
@nihilazo Are you trying to graalvm compile datahike? That's slightly more advanced than hello world I would say?
last time I tried that, it didn't work for me
oh sorry, I misread. yeah, hello world should work
yeah, I was trying to graalvm compile datahike
which is probably a bad idea
I'll see if the hello world works
I tried that before, but that didn't work for me. that's why I ended up using transit instead of datahike in clj-kondo (I wasn't even able to make a performance benchmark because I never got that far)
if you want to have a fast CLI solution with a database you could consider babashka + the sqlite-cli: https://github.com/borkdude/babashka/blob/master/examples/sqlite.clj
I wasn't able to compile sqlite itself with graalvm either
I'm not sure I even really need a database? The queries are super simple (literally just checking if something has a tag, and checking if it doesn't) but there is potentially a reasonable amount of things being checked
actually, that said, I wonder if datahike is actually worse performance-wise than doing something like that manually
@nihilazo What I did for clj-kondo is split the data up into partitions so I can choose to only read from disk what I need. you can tailor this to your problem to make it performant
transit is compact and fast to read/write
and works with graalvm
ah, for this in every case I want to search all the data, and it's stored in a text format on disk already, I might try doing it without any DB at all and see how it goes
might even be able to do everything I was doing with datahike using a couple of
but using a database seemed like the more "proper" way
if you can make the search faster by pre-organizing it into an indexed way first, that can help
compiling the new datahike-less version results in some unsupported things, but that's progress!
also, the hello world works perfectly
I'm getting the errors from clojure.spec.gen.alpha, is it workaround time?
is it the CLJ-1472 error?
maybe, I'm not pulling in spec myself but I am pulling in tools.cli
actually, I don't need tools.cli, I can just remove that
tools.cli should work without this error, I've used it in a couple of projects
the only thing I'm using outside of my own code is http://clojure.java.io and clojure.string
your own code could be triggering it. are you using things like resolve?
also, it could be that you have to set the macro spec checking off
check out the options that I'm using here: https://github.com/borkdude/jet/blob/master/script/compile
ah, maybe I need the -J-Dclojure.spec.skip-macros=true line
could very well be
hmm, with that I'm getting errors due to spec as well, despite not using spec at all
btw, using http://clojure.java.io without a require is bad practice: https://git.itwont.work/samples/tree/src/samples/core.clj?h=no-datahike#n61
there's probably bad practice all over this thing tbh, I'm fairly new to clojure
I don't mean it to be offending, the reason it's "bad" is that you should not trust other tools to load http://clojure.java.io for you. If you use a linter like clj-kondo, it will warn you about such practices.
no offence taken, just warning you that most of it is probably bad
and thanks for trying to help me figure this out
might be worth building the patched spec anyway?
I'm not sure. your project doesn't seem to be doing anything weird. does it work with clojure 1.9?
seems like it
just tested with 1.9 and it works fine
when I try a graal build with 1.9, I get a different error,
Error: No instances of <http://java.io|java.io>.FilePermission are allowed in the image heap as this class should be initialized at image runtime.
oh are you by any chance using graal v20?
try 19.3.1 and report this error on the github graal project. @sfyire also got it
I'm using whatever the graalvm in AUR is, I'll try 19.3.1
yeah, 20 just got out and has a couple of issues. But reporting those issues definitely helps and they will be fixed.
19.3.1 is the LTS release
gotta love arch linux for having the latest everything, even when the latest doesn't work
can graalvm not access environment variables? It builds perfectly with the older graalvm and older clojure but I use System/getenv and that doesn't seem to be working, seems to fail
won't read in the variable
@nihilazo It depends if you're doing that at compile time or run time. When you execute that call at compile time, the call is made too early
even bb can do this:
so surely your custom graalvm clojure app can.
$ bb '(System/getenv "PWD")' "/Users/borkdude/dre/DocSearch/app"
yeah, so the problem here is that you're doing it top-level: https://git.itwont.work/samples/tree/src/samples/core.clj?h=no-datahike#n15
that won't work, since that captures the value of that environment variable when you compile it, not when you run it
that doesn't seem like the case when I run it though
when I run it from the jar, it reacts to whatever the variable is at that point
I can change it between runs and it changes
just make it into a variable inside -main
I wonder, is your jar AOT-ed?
everything is AOT-ed
then I would even expect that not to work on the JVM
anyway, this is a problem I've had before and the solution is to make reading the environment variable a call at runtime, i.e. in the execution of -main, not top-level/before it.
everything is now working on graalVM! Thank you for helping me out and answering my stupid questions
they're not stupid and you're welcome!
if you can make a tiny repro of the: > No instances of http://java.io.FilePermission are allowed in the image heap as this class should be initialized at image runtime that would be helpful for fixing it in v20
usually posting your AOT-ed uberjar + native-image build instructions is sufficient for the graalvm devs to reproduce it
so they won't have to bother with compiling clojure
I'll try that, not sure exactly what the cause was though
wasn't it just v20 jdk8 versus v19 jdk8?
it was v20 jdk11 vs v19 jdk11
it worked in v19 and not v20
but I'm not sure what the thing in the code was that caused it so I can do a tiny repro
well, that's up for them to find out I think. just saying: this works with graalvm 19.1.3 jdk 11 on macOS/linux/Windows but not on v20 .... is sufficient, provided you give them the uberjar + build arguments of native-image
@nihilazo Here's an example: https://github.com/oracle/graal/issues/2136#issuecomment-587865438
that makes things easier