graalvm

Discuss GraalVM related topics. Use clojure 1.10.2 or newer for all new projects. Contribute to https://github.com/clj-easy/graal-docs and https://github.com/BrunoBonacci/graalvm-clojure. GraalVM slack: https://www.graalvm.org/slack-invitation/.
alexmiller 2020-03-03T01:15:40.182200Z

so, it is important in this case because of how the body of the try gets lifted into a function, but it should not matter in 99.999% of normal cases.

alexmiller 2020-03-03T01:18:21.182400Z

what was happening before is that the locking macro expanded to a try not in the tail position, and that lifts the try body into a function (around which you can do catch/finally kinds of stuff), which means the monitorenter was in the original body and the monitorexit was in the lifted function (with the lockee as a closed over field in the function object). because these are in different methods and in a field, which gets loaded and cleared due to locals clearing, the graal analyzers can't connect that the monitorexit is attached to the monitorenter and it looks unbalanced.

alexmiller 2020-03-03T01:22:10.182600Z

the outer let in the new locking macro evaluates the lock object, that goes into the the next try block. the inner let puts it in a local, which will be not cleared due to the outer try (as it might be needed if there is a finally), the lock object then ends up on the stack for both enter and exit.

alexmiller 2020-03-03T01:23:33.182800Z

so the outer try here influences locals clearing just to make the lockee tracking easier for the graal analyzer.

alexmiller 2020-03-03T01:24:20.183Z

needless to say, this is super subtle

alexmiller 2020-03-03T01:27:00.183200Z

I think this could easily also be a do - not sure if there is any good reason to use let [] there.

alexmiller 2020-03-03T01:27:06.183400Z

could just be historical

bherrmann 2020-03-03T08:27:52.184Z

I recommend copy/pasting that explanation right above the try block.

borkdude 2020-03-03T08:35:34.184200Z

yeah, adding a comment seems good to me. maybe the linter feature will still be good because in 99.999% of the cases forgetting a catch is probably something in the category of silly errors (and the rule can be disabled)

2020-03-03T13:26:46.184600Z

this is not a good explanation

2020-03-03T13:35:34.184900Z

Here’s what the ‘locking’ problem (CLJ-1472) was with Graal and why the patch fixes it: The locking macro emits a try block. When not in tail position, try blocks get lifted into lambdas, and closed over variables (including, importantly, the lockee) become ‘fields’ of the lambda object. Thus in that situation the monitor enter/exit calls were being made on a field. Graal doesn’t trust that the value of that field will be the same in the enter and exit calls, so it balks that they might not be a balanced pair. The solution is to get the lockee onto the stack so Graal can understand that it is the same target. The problem has nothing to do with exception handling blocks or failure to emulate javac’s infinite loop trick. How the patch works: We need to get the lockee on the stack, even when lifting occurs, and use the same stack variable in try/finally. Thus we need a let around try/finally. But lifting can still happen, so that let needs to be in what’s lifted. That’s the role of the outer try. The outer try will be lifted, and the let with it, the let will put the field on the stack, Graal sees enter/exit on the same stack value and its verifier is thus happy. The inner try will always be in tail position.

15👍
borkdude 2020-03-03T13:39:26.185200Z

Thanks Rich for the detailed explanation!

alexmiller 2020-03-03T13:41:45.185700Z

yeah, like I said! ;)

borkdude 2020-03-03T13:45:04.186Z

@lee Might be good to document this in clj-graal-docs 🙂

lread 2020-03-03T13:45:59.186800Z

agreed, will do later today!

lread 2020-03-03T19:14:46.188400Z

Updated our clj-graal-docs README (and scripts) to reflect my understanding of current state of CLJ-1472 https://github.com/lread/clj-graal-docs/blob/master/CLJ-1472/README.md

alexmiller 2020-03-03T19:15:19.188800Z

patches were just applied today in clojure and spec.alpha

borkdude 2020-03-03T19:17:21.189400Z

cool!

alexmiller 2020-03-03T19:17:53.189900Z

spec.alpha 0.2.187 is available (that alone gets you past the spec issue, but not the rest)

alexmiller 2020-03-03T19:18:52.190900Z

clojure 1.10.2-master-SNAPSHOT should be available at some point, we haven't done an actual release there yet - but that includes CLJ-1472 and CLJ-2502, and depends on spec.alpha 0.2.187

4🎉
alexmiller 2020-03-03T19:21:29.191700Z

the plan is to release this stuff relatively soon, maybe with a few additional fixes

borkdude 2020-03-03T19:22:19.191900Z

that's very good news, thank you

alexmiller 2020-03-03T19:23:12.192300Z

I am still on the fence about how to think about the reflector thing

borkdude 2020-03-03T19:24:05.193300Z

I can ask about this on the graalvm github issue tracker

alexmiller 2020-03-03T19:24:43.193900Z

I expect clojure to be in this situation for a while (and then it will be replaced by some other java-version-dependent thing), so your approach of just overriding it is not an option in clojure itself

alexmiller 2020-03-03T19:25:57.195600Z

there are a variety of approaches to jvm-specific versions (classifiers, jars that support multiple runtimes, etc) but those are all pretty big changes

alexmiller 2020-03-03T19:26:22.196Z

I don't know, still thinking about it

borkdude 2020-03-03T19:26:25.196200Z

I understand. Luckily the workaround can be applied locally to final projects.

borkdude 2020-03-03T19:27:33.196900Z

btw, here's a trivial typo patch, might you be interested in getting in something low risk: https://clojure.atlassian.net/browse/CLJ-2444

alexmiller 2020-03-03T19:35:08.197200Z

more looking for high priority than trying to shove a bunch of stuff in

borkdude 2020-03-03T19:55:05.197500Z

I logged an issue about clojure.lang.Reflector + java11 here: https://github.com/oracle/graal/issues/2214

borkdude 2020-03-03T20:08:44.198400Z

Hmm, adding --report-unsupported-elements-at-runtime does make this repro work:

$ ./reflector-test
11.0.6
GraalVM 20.0.0 CE
Can access method: true
so maybe it's already solved in their v20. More testing: this also works with 19.3.0 java11. So maybe that's the way to deal with this after all.

borkdude 2020-03-03T20:28:01.199500Z

Confirmed that the new version of spec solves the 'locking' issue when the graalvm analyzer hits spec's dynaload:

{:deps {org.clojure/clojure {:mvn/version "1.10.1"}
        org.clojure/spec.alpha {:mvn/version "0.2.187"}}}
$ ./spec-test
{:major 1, :minor 10, :incremental 1, :qualifier nil}
true
🎉

borkdude 2020-03-03T20:31:11.200400Z

Thanks for dedicating time to this Alex and Rich, this makes me very happy 🙂.

wcohen 2020-03-03T21:21:43.201800Z

Are there known solutions to the <http://java.io|java.io>.FilePermission issue described at https://github.com/BrunoBonacci/graalvm-clojure/blob/master/aleph/README.md#caveats, which comes up with 20.0.0-java11? Running into it with another library.

borkdude 2020-03-03T21:24:01.202500Z

I guess use 19.3.1 java11 or 20.0.0 java 8?

borkdude 2020-03-03T21:24:11.202800Z

I see an issue about it here: https://github.com/oracle/graal/issues/2177 Not sure if that's the same problem

borkdude 2020-03-03T21:24:47.204200Z

If it is, it's maybe good to communicate you're experiencing the same problem and give some details if you have anything to add there

wcohen 2020-03-03T21:25:27.204800Z

(using different versions definitely solves the issue, just trying to figure out what's even causing it)

wcohen 2020-03-03T21:26:40.205400Z

seems like it's higher up the food chain than clojure, though

borkdude 2020-03-03T21:28:05.206500Z

might be a bug. there's also a native image / graalvm slack: https://app.slack.com/client/TN37RDLPK making a Java-only repro is certainly preferred if possible if you're going to submit an issue, but they also accept uberjars or detailed build instructions

wcohen 2020-03-03T21:28:23.206700Z

great, thanks!

wcohen 2020-03-03T22:15:10.208900Z

for posterity -- it was an issue of accessing resources and needing to assist the native-image builder with links to the resource files, which means sticking them in META-INF and utilizing (https://github.com/kumarshantanu/lein-javac-resources) when running lein-native-image

borkdude 2020-03-03T22:21:15.209800Z

@wcohen the -H:IncludeResources=... option?

wcohen 2020-03-03T22:24:11.211300Z

uberjarring the class to be native-imaged, running the uberjar with native-image-agent=config-output-dir per https://github.com/oracle/graal/blob/master/substratevm/CONFIGURE.md, then adding content from those jsons to META-INF/native-image

borkdude 2020-03-03T22:25:40.212Z

@wcohen There is an issue here https://github.com/lread/clj-graal-docs/issues/13 to write some docs about it. More than welcome if you feel like contributing.

wcohen 2020-03-03T22:26:38.212800Z

will do in the next few days!