graalvm

Discuss GraalVM related topics. Use clojure 1.10.2 or newer for all new projects. Contribute to https://github.com/clj-easy/graal-docs and https://github.com/BrunoBonacci/graalvm-clojure. GraalVM slack: https://www.graalvm.org/slack-invitation/.
Aaron Cummings 2020-03-02T13:30:08.103400Z

I have a command line program I have compiled with native-image on Windows 10 (x64). I have found that on a fresh Windows install, it won't run because of missing 'vcruntime140.dll'. If I redistribute this program, is it sufficient (and legal?) to include this dll with my distribution?

borkdude 2020-03-02T13:35:38.104100Z

@aaron383 Could it be related to the Visual C/C++ Redistributable 2010? Which is pretty much always required if you run a dynamically linked executable compiled with GraalVM on Windows I think

borkdude 2020-03-02T13:36:19.104800Z

A lot of times this is already installed because a lot of other programs need it too. The scoop maintained by @ales.najmann suggests this: https://github.com/littleli/scoop-clojure/blob/a0b245db60d824177ea60cb9616239bb875baf50/clj-kondo.json#L13

littleli 2020-03-02T13:37:16.105400Z

yes, it's quite common issue

borkdude 2020-03-02T13:37:56.106100Z

In doubt, it's best to ask in the native-image channel of the GraalVM slack (https://app.slack.com/client/TN37RDLPK/CN9KSFB40) and if you have any news, contribute it to https://github.com/lread/clj-graal-docs

Aaron Cummings 2020-03-02T13:42:06.110900Z

My users are corporate folks who don't necessarily have admin access to their laptops, so to be complete I'd need to include any dependencies. The Win10 image on my work machine does include this dll, so things "just work", but I'm concerned that this won't generally be the case.

Aaron Cummings 2020-03-02T13:42:44.112Z

I'll try the Slack you suggested; thanks for the pointer.

littleli 2020-03-02T14:12:47.114400Z

scoop in general uses user's profile, where admin rights are not necessary. But vcredist20XX are usually coming with installers, and with that it's hard to tell.

Aaron Cummings 2020-03-02T14:25:18.114800Z

I have found clarity on redistribution: https://docs.microsoft.com/en-us/visualstudio/productinfo/2017-redistribution-vs#visual-c-runtime-files

borkdude 2020-03-02T16:27:26.115900Z

@ghadi You were asking about the JDK11 client recently. I've got a fork of hato, a clj-http-inspired lib that is based on JDK11's client working here: https://github.com/borkdude/hato-native The only thing I had to do is remove some conditional requires/resolves.

borkdude 2020-03-02T16:28:14.117200Z

Compilation time of babashka went up from 2 minutes to 8 minutes though when I included it, so I'll likely not include it

1😱
ghadi 2020-03-02T17:00:31.117600Z

yeah... I'm not 100% a fan of the approach in hato

ghadi 2020-03-02T17:01:41.118300Z

compilation time 📈 because it pulls in way too many dependencies

[cheshire.core]
   [cognitect.transit]

borkdude 2020-03-02T17:13:21.118900Z

cheshire.core and cheshire.transit are not problematic in jet, this still compiles within a minute on my macbook

borkdude 2020-03-02T17:15:18.119300Z

same on circleci, about 1 minute: https://circleci.com/gh/borkdude/jet/746 (the Build binary step)

littleli 2020-03-02T18:12:21.120600Z

I suspect there is some megamorphic call site in the hato project where native image has to go through to many branches of the code.

borkdude 2020-03-02T20:12:15.121200Z

@alexmiller > The latest Graal (20) still failed on this example, but with different errors. Care to share what 20 gave? Which JDK did you use, 8?

alexmiller 2020-03-02T20:15:25.121600Z

might have been 11 - is there a separate install for 8?

alexmiller 2020-03-02T20:16:33.121800Z

ah, I see. will check there

alexmiller 2020-03-02T20:21:19.122Z

works on graal 20 java 8

alexmiller 2020-03-02T20:21:37.122100Z

here's the graal 20 java 11 failure

borkdude 2020-03-02T20:38:40.123500Z

@alexmiller GraalVM has problems with the reflective use of the method handle. There is an "easy" workaround for that if you can rely on JDK11 and forget about JDK8: https://github.com/lread/clj-graal-docs#jdk11-and-clojurelangreflector

borkdude 2020-03-02T20:40:20.124100Z

@alexmiller I think you're logging some stuff which makes this output very large. Also probably makes compilation slower.

alexmiller 2020-03-02T20:41:22.124700Z

so those are the options I'm using

borkdude 2020-03-02T20:41:55.125200Z

fwiw I try to avoid --report-unsupported-elements-at-runtime and I've never used -H:+ReportExceptionStackTraces.

borkdude 2020-03-02T20:42:24.125700Z

maybe @lee can enlighten me why this is needed 🙂

lread 2020-03-02T20:42:53.126100Z

oh hey there! 👋

lread 2020-03-02T20:43:28.126800Z

I just took our compile script @borkdude

lread 2020-03-02T20:46:20.127100Z

Extracted from here: https://github.com/lread/clj-graal-docs/blob/master/CLJ-1472/compile

lread 2020-03-02T20:47:53.128400Z

So… even though we thought excluding graal on jdk11 from reproduction steps was a good idea… maybe we should update to ensure it works.

lread 2020-03-02T20:48:18.129Z

Core team will likely verify for both jdk8 and jdk11

alexmiller 2020-03-02T20:48:41.129400Z

I added the patch around the methodhandle, fails with:

alexmiller 2020-03-02T20:48:43.129700Z

Error: com.oracle.svm.hosted.substitute.DeletedElementException: Unsupported method java.lang.ClassLoader.defineClass1(ClassLoader, String, byte[], int, int, ProtectionDomain, String) is reachable
To diagnose the issue, you can add the option --report-unsupported-elements-at-runtime. The unsupported element is then reported at run time when it is accessed the first time.
Detailed message:
Trace:
	at parsing java.lang.ClassLoader.defineClass(ClassLoader.java:1016)
Call path from entry point to java.lang.ClassLoader.defineClass(String, byte[], int, int, ProtectionDomain):
	at java.lang.ClassLoader.defineClass(ClassLoader.java:1014)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:877)
	at clojure.lang.DynamicClassLoader.defineClass(DynamicClassLoader.java:46)
	at clojure.core$get_proxy_class.invokeStatic(core_proxy.clj:288)
	at clojure.core$get_proxy_class.doInvoke(core_proxy.clj:276)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at spec.main(Unknown Source)
	at com.oracle.svm.core.JavaMainWrapper.runCore(JavaMainWrapper.java:151)
	at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:186)
	at com.oracle.svm.core.code.IsolateEnterStub.JavaMainWrapper_run_5087f5482cc9a6abc971913ece43acb471d2631b(generated:0)

lread 2020-03-02T20:50:38.130800Z

I verified steps against graal 19.3.1 - iirc that’s the LTS version, right @borkdude?

alexmiller 2020-03-02T20:50:38.130900Z

well, I re-ran with --report-unsupported-elements-at-runtime and it worked

alexmiller 2020-03-02T20:50:54.131100Z

so that's weird

lread 2020-03-02T20:54:55.131300Z

There are so many GraalVM options, it is hard to keep track, you seem to be using one of these for clj-kondo https://github.com/borkdude/clj-kondo/blob/63414903822d523c5bc154f7cf75035c287c5879/script/compile#L16

ghadi 2020-03-02T20:56:50.132300Z

it's saying that get-proxy-class calls ClassLoader.defineClass, which it does

alexmiller 2020-03-02T20:57:47.133100Z

if I add the flag --report-unsupported-elements-at-runtime, it reports nothing, and produces a working executable

ghadi 2020-03-02T20:57:51.133300Z

yup

alexmiller 2020-03-02T20:57:54.133500Z

¯\(ツ)

ghadi 2020-03-02T20:58:00.133800Z

that makes sense to me

borkdude 2020-03-02T20:58:09.134200Z

ah, I am ok.

ghadi 2020-03-02T20:58:27.134800Z

surprised that graal jdk8 doesn't say the same

lread 2020-03-02T20:59:10.135200Z

I frankly don’t know how/when/where/why we started using --report-unsupported-elements-at-runtime for clj-graal-docs

borkdude 2020-03-02T21:03:17.137Z

fwiw, here is a working jdk11 project example: https://github.com/borkdude/sci/tree/jdk11 the compile script is in script/compile. it uses a patched clojure.lang.Reflector in src-java.

lread 2020-03-02T21:03:44.137500Z

So, I’m wondering under what graalvm env core team should verify that this is fixed.

alexmiller 2020-03-02T21:04:18.138600Z

I don't see the CLJ-1472 problem in any env, so I consider it verified from that perspective

alexmiller 2020-03-02T21:04:55.139500Z

works on graal 19 java 8, graal 20 java 8, graal 20 java 11 with --report-unsupported-elements-at-runtime, and see different problems under graal 20 java 11 w/o that flag

borkdude 2020-03-02T21:05:00.139700Z

it doesn't use --report-unsupported-elements-at-runtime, maybe because I'm using some java opts: https://github.com/borkdude/sci/blob/ebb7b03230dc3a0808606482eb118613c2283e7b/project.clj#L29-L30 not sure ¯\(ツ)

ghadi 2020-03-02T21:06:26.141200Z

@alexmiller did you get that get-proxy-class error with direct-linking enabled?

alexmiller 2020-03-02T21:06:49.141700Z

direct linking enabled when compiling what?

alexmiller 2020-03-02T21:07:00.142Z

clojure? spec? example code?

ghadi 2020-03-02T21:07:31.142700Z

all of those

alexmiller 2020-03-02T21:07:32.142800Z

the way I'm testing clojure should be direct linked, the others not

borkdude 2020-03-02T21:07:51.143500Z

I'm compiling the uberjar with direct linking, just for performance

ghadi 2020-03-02T21:08:03.144Z

I'm just looking for a differential on why that error is there.

borkdude 2020-03-02T21:08:18.144300Z

and the spec macro setting is to avoid the locking issue

borkdude 2020-03-02T21:08:32.144700Z

without that I've had it pop up more often

borkdude 2020-03-02T21:09:18.145400Z

anyway, I think the focus is on CLJ-1472 and if that issue is solved with your patch, I think that is nice progress. other issues should be addressed another time probably

alexmiller 2020-03-02T21:09:39.145800Z

are there other issues to address in clojure 1.11 timeframe?

borkdude 2020-03-02T21:10:08.146300Z

The only issue I know of is the one with clojure.lang.Reflector on JDK11

lread 2020-03-02T21:13:15.147100Z

We never did figure out pprint did we?

borkdude 2020-03-02T21:13:28.147300Z

it's relatively easy to address. I wouldn't know how to fix that one properly without making separate releases for separate JDK versions, to avoid the runtime check on Java versions (this was about clojure.lang.Reflector)

lread 2020-03-02T21:17:20.149700Z

Not sure if this is slated for 1.11? https://clojure.atlassian.net/browse/CLJ-2502

lread 2020-03-02T21:17:43.150100Z

oh yeah it is

borkdude 2020-03-02T21:17:46.150400Z

ah right, I run with a patched version of that too, forgot about it

lread 2020-03-02T21:17:56.150600Z

fix versions: 1.11

alexmiller 2020-03-02T21:18:01.150800Z

yes

ghadi 2020-03-02T21:18:27.151300Z

do we have a bug for the "Reflector thing"? needs clarity

alexmiller 2020-03-02T21:20:01.151800Z

no jira that I know of, but I'm not sure that it's something we should address in clojure itself

lread 2020-03-02T21:21:51.153100Z

here’s what @borkdude logged on pprint on clj-graal-docs: https://github.com/lread/clj-graal-docs/issues/19 - we closed because it was stale

lread 2020-03-02T21:24:31.154500Z

perhaps at least logging pprint issue as a symptom in jira would be helpful?

alexmiller 2020-03-02T21:25:29.154900Z

it's not helpful unless there's more information than that

lread 2020-03-02T21:26:07.155100Z

ok

lread 2020-03-02T21:27:36.155800Z

if I find the time to diagnose why pprint does not work, I will log

borkdude 2020-03-02T21:33:08.156600Z

> do we have a bug for the "Reflector thing"? needs clarity The JDK11 support for GraalVM is relatively new, I think most people are still using JDK8, that's why it probably never showed up. If you want, I can make an issue for it.

borkdude 2020-03-02T21:42:34.157300Z

@alexmiller Following the exact same repro from lread but with this code:

(ns spec-test.core
  (:gen-class)
  (:require [clojure.pprint :as pprint]))

(defn -main [& _args]
  (println *clojure-version*)
  (pprint/pprint (range 100)))
also triggers the monitor mismatch error.

borkdude 2020-03-02T21:43:28.157900Z

Removing the references to pprint will produce a working binary with 1.10.1

alexmiller 2020-03-02T21:44:57.158100Z

related to locking?

borkdude 2020-03-02T21:45:01.158300Z

yes

alexmiller 2020-03-02T21:49:33.158600Z

with --report-unsupported-elements-at-runtime, works with the patch

borkdude 2020-03-02T21:55:44.158900Z

@alexmiller confirmed!

$ ./spec-test
{:major 1, :minor 10, :incremental 1, :qualifier patch_38bafca9_clj_1472_5}
(0
 1
 2
 3
 4

ghadi 2020-03-02T21:56:31.159400Z

cool, but I need an explanation!

ghadi 2020-03-02T21:56:38.159600Z

pprint doesn't use locking AFAIK

borkdude 2020-03-02T21:56:58.160Z

@ghadi this might be triggered by the Clojure compiler invoking spec on macro syntax checking?

borkdude 2020-03-02T21:57:22.160700Z

and that uses locking in dynaload

ghadi 2020-03-02T21:57:22.160800Z

what macro?

ghadi 2020-03-02T21:58:12.161300Z

if that were the case, it would be any form affected

borkdude 2020-03-02T21:59:12.162400Z

I don't know, but somehow that code is reachable by the analyzer. I don't know the details. I've been able to get around this using "-Dclojure.spec.skip-macros=true" which is why I thought that would be somehow related to it.

borkdude 2020-03-02T22:02:17.162900Z

@ghadi Maybe this option could help explain it: https://github.com/lread/clj-graal-docs/blob/master/README.md#report-what-is-being-analyzed

borkdude 2020-03-02T22:02:23.163200Z

Note that the output can get quite large.

ghadi 2020-03-02T22:03:00.163700Z

well -- in any case clj-1472 takes care of it

ghadi 2020-03-02T22:03:14.164100Z

I don't the ability to make that report at the moment

ghadi 2020-03-02T22:03:38.164700Z

but I am idly curious still 🙂 . I don't like "spooky" / "magic"

borkdude 2020-03-02T22:04:42.165200Z

btw @lee - that script... pure awesome, so easy to test out a clj-1472 patch 🙂

1😊
borkdude 2020-03-02T22:23:02.167600Z

I'm able to compile clj-kondo with graalvm 20 jdk11 and 1472 patch 5. I consider clj-kondo to be a non-trivial example

borkdude 2020-03-02T22:29:45.169600Z

@alexmiller The spec repro example works for me in GraalVM 20 java11:

$ ./spec-test
{:major 1, :minor 10, :incremental 1, :qualifier patch_38bafca9_clj_1472_5}
11.0.6
GraalVM 20.0.0 CE
true

borkdude 2020-03-02T22:30:13.169800Z

This is the program:

(ns spec-test.core
  (:gen-class)
  (:require [clojure.spec.alpha :as s]))

(s/def ::g int?)

(defn -main [& _args]
  (println *clojure-version*)
  (println (System/getProperty "java.version"))
  (println (System/getProperty "java.vm.version"))
  (println (s/valid? ::g 1)))

alexmiller 2020-03-02T22:37:41.170700Z

it works for me with --report-unsupported-elements-at-runtime, but not without it

borkdude 2020-03-02T22:39:27.171200Z

thanks

alexmiller 2020-03-02T22:41:33.171500Z

rich ok'ed clj-2502 too

2
alexmiller 2020-03-02T22:42:05.172400Z

and we've vetted the clj-1472 patch on a much larger test bed with datomic too

borkdude 2020-03-02T22:47:34.173100Z

I'm happy clj-1472 is working out, but I have no idea why it worked 🙂 I did read the explanation and tried understanding it. Why is there a try inside a try, is the outer try still doing anything?

alexmiller 2020-03-02T22:51:42.173700Z

well, it helps if you're looking at the bytecode :)

borkdude 2020-03-02T22:53:07.174700Z

> Why is there a try inside a try, is the outer try still doing anything? Remember that we talked recently about there being no legit use case for a try without a catch / finally for a linter feature? 🙂 I wonder if this matter for the bytecode?

bherrmann 2020-03-03T08:27:52.184Z

I recommend copy/pasting that explanation right above the try block.

borkdude 2020-03-03T08:35:34.184200Z

yeah, adding a comment seems good to me. maybe the linter feature will still be good because in 99.999% of the cases forgetting a catch is probably something in the category of silly errors (and the rule can be disabled)

2020-03-03T13:26:46.184600Z

this is not a good explanation

alexmiller 2020-03-02T22:53:41.175500Z

it might actually matter here for the scope of the exception table, I will look into it a little more

alexmiller 2020-03-02T22:54:13.175900Z

or maybe it could actually just be a do

alexmiller 2020-03-02T22:54:43.176300Z

this is like v5 of this patch and it was doing something in the prior version

borkdude 2020-03-02T22:56:42.176900Z

I remember seeing an empty let (let [] ...) somewhere in core once

borkdude 2020-03-02T23:00:39.178600Z

maybe it's there for a similar reason

borkdude 2020-03-02T23:18:41.180500Z

I have a repro for the clojure.lang.Reflector issue on JDK11:

(ns spec-test.core
  (:gen-class)
  (:import [clojure.lang Reflector])
  (:require [<http://clojure.java.io|clojure.java.io> :as io]))

(defn -main [&amp; _args]
  (println *clojure-version*)
  (println (System/getProperty "java.version"))
  (println (System/getProperty "java.vm.version"))
  (println (Reflector/invokeInstanceMethod (io/file ".") "exists" (object-array []))))
reflection.json:
[{
  "name" : "<http://java.io|java.io>.File",
  "allPublicMethods" : true,
  "allPublicFields" : true,
  "allPublicConstructors" : true
}]
With 20.0.0 java8:
$ ./spec-test
{:major 1, :minor 10, :incremental 1, :qualifier patch_38bafca9_clj_1472_5}
1.8.0_242
GraalVM 20.0.0 CE
true
With 20.0.0 java11:
$ ./spec-test
{:major 1, :minor 10, :incremental 1, :qualifier patch_38bafca9_clj_1472_5}
11.0.6
GraalVM 20.0.0 CE
Exception in thread "main" com.oracle.svm.core.jdk.UnsupportedFeatureError: Invoke with MethodHandle argument could not be reduced to at most a single call or single field access. The method handle must be a compile time constant, e.g., be loaded from a `static final` field. Method that contains the method handle invocation: java.lang.invoke.Invokers$Holder.invoke_MT(Object, Object, Object, Object)
	at com.oracle.svm.core.util.VMError.unsupportedFeature(VMError.java:101)
	at clojure.lang.Reflector.canAccess(Reflector.java:49)

borkdude 2020-03-02T23:20:00.181200Z

So I've been able to work around that like here: https://github.com/lread/clj-graal-docs#jdk11-and-clojurelangreflector

borkdude 2020-03-02T23:20:14.181600Z

Let me know if you want an issue for it.