Clojurians Log v2

I’m curious on what peoples bigger thoughts are on what needs to happen in the clojure community to make graalvm easier/simpler to target, for both libraries and applications? I’ve been doing quite a lot of graal stuff recently and it can be fiddly and at times frustrating when a native-image fails. In particular it would be good to minimise the need to do the debugging, tracing compile dance to find the combination of flags that work. I also appreciate it’s perhaps a bit early to establish solutions to this, but I wonder:

1. What best practices we might need to develop?

2. What new tools we might need?

3. How we can patch libraries to embed configuration https://www.graalvm.org/reference-manual/native-image/BuildConfiguration/#embedding-a-configuration-file . I appreciate efforts like this: https://github.com/BrunoBonacci/graalvm-clojure/ but surely there’s a better way.

4. Whether further tweaks to clojure might be ultimately be warranted (e.g. support for swapping out the dynamic classloader)

For example using tests to trace/instrument reflective calls, whilst also comparing the native-image build to the jvm build: https://github.com/borkdude/refl/pull/2#issuecomment-864849198

@rickmoynihan I think good documentation is a first step. We're trying to do that here: https://github.com/lread/clj-graal-docs Also a getting started project/tutorial: https://github.com/borkdude/jayfu (talk will be published soon).

As for reflection: I think we should promote the usage of *warn-on-reflection* more: for Clojure it's optional and more or less only a performance optimization, but for GraalVM it's not optional (or you will have to use a reflection config)

Yeah I agree documentation is a good first step… I’ve read all that, and the landscape is clearly changing again… so I guess there’s another few rounds of this.

I don't think any framework that tries to protect the user from how GraalVM works will be completely fool proof. I think it's better to educate people than try to protect them using a fancy magic framework.

Would it be worth enabling github discussions on the clj-graal-docs repo?

Yes. But I'm not an admin there. Perhaps we can move this repo to an organizations so we can add admins @lee?

Also I think we could come up with a better name. I think the other graalvm-clojure is often found as the first thing, while our documentation is more elaborate. Perhaps we could join efforts.

Yeah I don’t think a framework is an answer either (at least not yet)

On best practices; I was meaning to ask whether you thought it was better to: 1. commit hand crafted reflect configs into the repo (perhaps using a refl flow to help figure them out) 2. automate their generation / cleaning them in CI with a refl build step, where tests are instrumented and traced. 3. Combine 1 & 2, with merge config step; so you can derive most of them by tracing tests, then add any additional ones you may need with a committed file. Or do you need to do 1, because there’s a need to remove configs that refl flows can’t detect?

I'm in favor of only committing curated (not necessarily hand-crafted) reflect configs.

It helps keep the image size down in case of false positives. You should just rely on a test suite to confirm if everything is working correctly.

I haven't needed the reflection agent that much, to be honest. I usually figured it out by trial and error in the past.

For things like database drivers it's more convenient since the reflections are less obvious there (especially if the driver is closed source)

I wonder what you were using it for? Did it help you detect stuff you wouldn't have found otherwise?

Happy to move clj-graal-docs to a new home if that helps!

@lee We could consider doing so on the long term. As a first step, maybe enabling discussions would be good

enabled! https://github.com/lread/clj-graal-docs/discussions

@borkdude, am happy to transfer clj-graal-docs to your org if that helps. I started the repo but you are much more active and knowledgeable these days. Lemme know.

@lee Preferably I would see a collective effort on this and perhaps also join with @bruno.bonacci's repo, so we have one central clojure repo for all things graalvm native

Ya that'd be cool.

Perhaps we could move to clj-commons?

Exactly what I was thinking. Not a typical use case, but perhaps a good idea?

Yeah, I think it fits the "neutral" ground and we can still be the admins/maintainers of this

Perhaps @slipset and @seancorfield can provide some thoughts.

I think we would have to bring this to the #clj-commons channel

since they are not here

yup

Yeah this was essentially the sort of discussion I was wanting to have… It feels like we need some consolidation here.

where does @bruno.bonacci’s repo live in all of this? It seems to me like it’s a necessary and noble early stages attempt; but centralising like this feels ultimately the wrong approach if we can push things back upstream into the libraries.

@rickmoynihan bruno bonacci's repo is also more or less a documentation repo

which describes what libraries are compatible with graalvm and describing some specific problems

docs repo merge to central/neutral/community spot makes sense to me, but maybe there's something I don't undestand yet?

@rickmoynihan what do you mean by "push things back upstream into the libraries"?

Well I might be misunderstanding the point of the repo; but it looks to me like the point is to document how to compile your app when you depend on various clojure libraries. And that fixing it in that repo (which essentially simulates application builds) is the wrong place. Surely it would be better to fix it in the library itself so the problem doesn’t happen downstream at all. If that happened for all of the libraries mentioned in that repo, would the repo not be redundant; you’d just use library like normal and it would compile under native image without issue.

I wonder if this would serve the newcomer wanting to learn how to use clojure under graalvm native image in various ways though? Perhaps if each tip and trick pointed to, where possible, a real library that employs a described technique/nuance would be helpful.