graalvm

Discuss GraalVM related topics. Use clojure 1.10.2 or newer for all new projects. Contribute to https://github.com/clj-easy/graal-docs and https://github.com/BrunoBonacci/graalvm-clojure. GraalVM slack: https://www.graalvm.org/slack-invitation/.
2021-06-22T10:20:22.217900Z

I’m curious on what peoples bigger thoughts are on what needs to happen in the clojure community to make graalvm easier/simpler to target, for both libraries and applications? I’ve been doing quite a lot of graal stuff recently and it can be fiddly and at times frustrating when a native-image fails. In particular it would be good to minimise the need to do the debugging, tracing compile dance to find the combination of flags that work. I also appreciate it’s perhaps a bit early to establish solutions to this, but I wonder:

2021-06-22T10:20:36.218300Z

1. What best practices we might need to develop?

2021-06-22T10:20:40.218500Z

2. What new tools we might need?

2021-06-22T10:20:46.218700Z

3. How we can patch libraries to embed configuration https://www.graalvm.org/reference-manual/native-image/BuildConfiguration/#embedding-a-configuration-file . I appreciate efforts like this: https://github.com/BrunoBonacci/graalvm-clojure/ but surely there’s a better way.

2021-06-22T10:20:48.219Z

4. Whether further tweaks to clojure might be ultimately be warranted (e.g. support for swapping out the dynamic classloader)

2021-06-22T10:24:52.219700Z

For example using tests to trace/instrument reflective calls, whilst also comparing the native-image build to the jvm build: https://github.com/borkdude/refl/pull/2#issuecomment-864849198

borkdude 2021-06-22T10:24:54.220Z

@rickmoynihan I think good documentation is a first step. We're trying to do that here: https://github.com/lread/clj-graal-docs Also a getting started project/tutorial: https://github.com/borkdude/jayfu (talk will be published soon).

borkdude 2021-06-22T10:25:56.221100Z

As for reflection: I think we should promote the usage of *warn-on-reflection* more: for Clojure it's optional and more or less only a performance optimization, but for GraalVM it's not optional (or you will have to use a reflection config)

2☝️
2021-06-22T10:27:01.222200Z

Yeah I agree documentation is a good first step… I’ve read all that, and the landscape is clearly changing again… so I guess there’s another few rounds of this.

borkdude 2021-06-22T10:27:04.222500Z

I don't think any framework that tries to protect the user from how GraalVM works will be completely fool proof. I think it's better to educate people than try to protect them using a fancy magic framework.

2021-06-22T10:27:41.222600Z

Would it be worth enabling github discussions on the clj-graal-docs repo?

borkdude 2021-06-22T10:28:19.223100Z

Yes. But I'm not an admin there. Perhaps we can move this repo to an organizations so we can add admins @lee?

borkdude 2021-06-22T10:28:43.223300Z

Also I think we could come up with a better name. I think the other graalvm-clojure is often found as the first thing, while our documentation is more elaborate. Perhaps we could join efforts.

2021-06-22T10:28:49.223600Z

Yeah I don’t think a framework is an answer either (at least not yet)

2021-06-22T10:40:11.229900Z

On best practices; I was meaning to ask whether you thought it was better to: 1. commit hand crafted reflect configs into the repo (perhaps using a refl flow to help figure them out) 2. automate their generation / cleaning them in CI with a refl build step, where tests are instrumented and traced. 3. Combine 1 & 2, with merge config step; so you can derive most of them by tracing tests, then add any additional ones you may need with a committed file. Or do you need to do 1, because there’s a need to remove configs that refl flows can’t detect?

borkdude 2021-06-22T10:41:10.230600Z

I'm in favor of only committing curated (not necessarily hand-crafted) reflect configs.

borkdude 2021-06-22T10:41:42.231100Z

It helps keep the image size down in case of false positives. You should just rely on a test suite to confirm if everything is working correctly.

borkdude 2021-06-22T10:42:29.231500Z

I haven't needed the reflection agent that much, to be honest. I usually figured it out by trial and error in the past.

borkdude 2021-06-22T10:42:58.231900Z

For things like database drivers it's more convenient since the reflections are less obvious there (especially if the driver is closed source)

borkdude 2021-06-22T10:45:03.232400Z

I wonder what you were using it for? Did it help you detect stuff you wouldn't have found otherwise?

lread 2021-06-22T12:11:01.232600Z

Happy to move clj-graal-docs to a new home if that helps!

borkdude 2021-06-22T12:12:54.232800Z

@lee We could consider doing so on the long term. As a first step, maybe enabling discussions would be good

lread 2021-06-22T12:16:21.233Z

enabled! https://github.com/lread/clj-graal-docs/discussions

2🎉
lread 2021-06-22T12:37:05.233400Z

@borkdude, am happy to transfer clj-graal-docs to your org if that helps. I started the repo but you are much more active and knowledgeable these days. Lemme know.

borkdude 2021-06-22T12:38:06.233600Z

@lee Preferably I would see a collective effort on this and perhaps also join with @bruno.bonacci's repo, so we have one central clojure repo for all things graalvm native

lread 2021-06-22T12:38:53.233800Z

Ya that'd be cool.

borkdude 2021-06-22T12:38:56.234Z

Perhaps we could move to clj-commons?

lread 2021-06-22T12:39:34.234200Z

Exactly what I was thinking. Not a typical use case, but perhaps a good idea?

borkdude 2021-06-22T12:39:54.234400Z

Yeah, I think it fits the "neutral" ground and we can still be the admins/maintainers of this

borkdude 2021-06-22T12:40:13.234600Z

Perhaps @slipset and @seancorfield can provide some thoughts.

borkdude 2021-06-22T12:41:17.234800Z

I think we would have to bring this to the #clj-commons channel

borkdude 2021-06-22T12:41:19.235Z

since they are not here

lread 2021-06-22T12:41:30.235200Z

yup

2021-06-22T12:42:53.235400Z

Yeah this was essentially the sort of discussion I was wanting to have… It feels like we need some consolidation here.

2021-06-22T12:45:07.235600Z

where does @bruno.bonacci’s repo live in all of this? It seems to me like it’s a necessary and noble early stages attempt; but centralising like this feels ultimately the wrong approach if we can push things back upstream into the libraries.

borkdude 2021-06-22T12:45:51.235900Z

@rickmoynihan bruno bonacci's repo is also more or less a documentation repo

borkdude 2021-06-22T12:46:17.236100Z

which describes what libraries are compatible with graalvm and describing some specific problems

lread 2021-06-22T12:47:26.236300Z

docs repo merge to central/neutral/community spot makes sense to me, but maybe there's something I don't undestand yet?

lread 2021-06-22T12:49:32.236500Z

@rickmoynihan what do you mean by "push things back upstream into the libraries"?

ericdallo 2021-06-22T14:20:56.236700Z

We use that way on clojure-lsp (and cljfmt I think) seems a good way indeed

2021-06-22T14:46:56.236900Z

Well I might be misunderstanding the point of the repo; but it looks to me like the point is to document how to compile your app when you depend on various clojure libraries. And that fixing it in that repo (which essentially simulates application builds) is the wrong place. Surely it would be better to fix it in the library itself so the problem doesn’t happen downstream at all. If that happened for all of the libraries mentioned in that repo, would the repo not be redundant; you’d just use library like normal and it would compile under native image without issue.

lread 2021-06-22T16:36:39.237100Z

I wonder if this would serve the newcomer wanting to learn how to use clojure under graalvm native image in various ways though? Perhaps if each tip and trick pointed to, where possible, a real library that employs a described technique/nuance would be helpful.