Clojurians Log v2

I have a command line program I have compiled with native-image on Windows 10 (x64). I have found that on a fresh Windows install, it won't run because of missing 'vcruntime140.dll'. If I redistribute this program, is it sufficient (and legal?) to include this dll with my distribution?

@aaron383 Could it be related to the Visual C/C++ Redistributable 2010? Which is pretty much always required if you run a dynamically linked executable compiled with GraalVM on Windows I think

A lot of times this is already installed because a lot of other programs need it too. The scoop maintained by @ales.najmann suggests this: https://github.com/littleli/scoop-clojure/blob/a0b245db60d824177ea60cb9616239bb875baf50/clj-kondo.json#L13

yes, it's quite common issue

In doubt, it's best to ask in the native-image channel of the GraalVM slack (https://app.slack.com/client/TN37RDLPK/CN9KSFB40) and if you have any news, contribute it to https://github.com/lread/clj-graal-docs

My users are corporate folks who don't necessarily have admin access to their laptops, so to be complete I'd need to include any dependencies. The Win10 image on my work machine does include this dll, so things "just work", but I'm concerned that this won't generally be the case.

I'll try the Slack you suggested; thanks for the pointer.

scoop in general uses user's profile, where admin rights are not necessary. But vcredist20XX are usually coming with installers, and with that it's hard to tell.

I have found clarity on redistribution: https://docs.microsoft.com/en-us/visualstudio/productinfo/2017-redistribution-vs#visual-c-runtime-files

@ghadi You were asking about the JDK11 client recently. I've got a fork of hato, a clj-http-inspired lib that is based on JDK11's client working here: https://github.com/borkdude/hato-native The only thing I had to do is remove some conditional requires/resolves.

Compilation time of babashka went up from 2 minutes to 8 minutes though when I included it, so I'll likely not include it

yeah... I'm not 100% a fan of the approach in hato

compilation time 📈 because it pulls in way too many dependencies

[cheshire.core]
   [cognitect.transit]

cheshire.core and cheshire.transit are not problematic in jet, this still compiles within a minute on my macbook

same on circleci, about 1 minute: https://circleci.com/gh/borkdude/jet/746 (the Build binary step)

I suspect there is some megamorphic call site in the hato project where native image has to go through to many branches of the code.

@alexmiller > The latest Graal (20) still failed on this example, but with different errors. Care to share what 20 gave? Which JDK did you use, 8?

might have been 11 - is there a separate install for 8?

ah, I see. will check there

works on graal 20 java 8

here's the graal 20 java 11 failure

@alexmiller GraalVM has problems with the reflective use of the method handle. There is an "easy" workaround for that if you can rely on JDK11 and forget about JDK8: https://github.com/lread/clj-graal-docs#jdk11-and-clojurelangreflector

@alexmiller I think you're logging some stuff which makes this output very large. Also probably makes compilation slower.

I was following https://github.com/lread/clj-graal-docs/blob/master/CLJ-1472/steps-to-reproduce.md

so those are the options I'm using

fwiw I try to avoid --report-unsupported-elements-at-runtime and I've never used -H:+ReportExceptionStackTraces.

maybe @lee can enlighten me why this is needed 🙂

oh hey there! 👋

I just took our compile script @borkdude

Extracted from here: https://github.com/lread/clj-graal-docs/blob/master/CLJ-1472/compile

So… even though we thought excluding graal on jdk11 from reproduction steps was a good idea… maybe we should update to ensure it works.

Core team will likely verify for both jdk8 and jdk11

I added the patch around the methodhandle, fails with:

Error: com.oracle.svm.hosted.substitute.DeletedElementException: Unsupported method java.lang.ClassLoader.defineClass1(ClassLoader, String, byte[], int, int, ProtectionDomain, String) is reachable
To diagnose the issue, you can add the option --report-unsupported-elements-at-runtime. The unsupported element is then reported at run time when it is accessed the first time.
Detailed message:
Trace:
	at parsing java.lang.ClassLoader.defineClass(ClassLoader.java:1016)
Call path from entry point to java.lang.ClassLoader.defineClass(String, byte[], int, int, ProtectionDomain):
	at java.lang.ClassLoader.defineClass(ClassLoader.java:1014)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:877)
	at clojure.lang.DynamicClassLoader.defineClass(DynamicClassLoader.java:46)
	at clojure.core$get_proxy_class.invokeStatic(core_proxy.clj:288)
	at clojure.core$get_proxy_class.doInvoke(core_proxy.clj:276)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at spec.main(Unknown Source)
	at com.oracle.svm.core.JavaMainWrapper.runCore(JavaMainWrapper.java:151)
	at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:186)
	at com.oracle.svm.core.code.IsolateEnterStub.JavaMainWrapper_run_5087f5482cc9a6abc971913ece43acb471d2631b(generated:0)

I verified steps against graal 19.3.1 - iirc that’s the LTS version, right @borkdude?

well, I re-ran with --report-unsupported-elements-at-runtime and it worked

so that's weird

There are so many GraalVM options, it is hard to keep track, you seem to be using one of these for clj-kondo https://github.com/borkdude/clj-kondo/blob/63414903822d523c5bc154f7cf75035c287c5879/script/compile#L16

it's saying that get-proxy-class calls ClassLoader.defineClass, which it does

if I add the flag --report-unsupported-elements-at-runtime, it reports nothing, and produces a working executable

yup

¯\(ツ)/¯

that makes sense to me

ah, I am ok.

surprised that graal jdk8 doesn't say the same

I frankly don’t know how/when/where/why we started using --report-unsupported-elements-at-runtime for clj-graal-docs

fwiw, here is a working jdk11 project example: https://github.com/borkdude/sci/tree/jdk11 the compile script is in script/compile. it uses a patched clojure.lang.Reflector in src-java.

So, I’m wondering under what graalvm env core team should verify that this is fixed.

I don't see the CLJ-1472 problem in any env, so I consider it verified from that perspective

works on graal 19 java 8, graal 20 java 8, graal 20 java 11 with --report-unsupported-elements-at-runtime, and see different problems under graal 20 java 11 w/o that flag

it doesn't use --report-unsupported-elements-at-runtime, maybe because I'm using some java opts: https://github.com/borkdude/sci/blob/ebb7b03230dc3a0808606482eb118613c2283e7b/project.clj#L29-L30 not sure ¯\(ツ)/¯

@alexmiller did you get that get-proxy-class error with direct-linking enabled?

direct linking enabled when compiling what?

clojure? spec? example code?

all of those

the way I'm testing clojure should be direct linked, the others not

I'm compiling the uberjar with direct linking, just for performance

I'm just looking for a differential on why that error is there.

and the spec macro setting is to avoid the locking issue

without that I've had it pop up more often

anyway, I think the focus is on CLJ-1472 and if that issue is solved with your patch, I think that is nice progress. other issues should be addressed another time probably

are there other issues to address in clojure 1.11 timeframe?

The only issue I know of is the one with clojure.lang.Reflector on JDK11

We never did figure out pprint did we?

it's relatively easy to address. I wouldn't know how to fix that one properly without making separate releases for separate JDK versions, to avoid the runtime check on Java versions (this was about clojure.lang.Reflector)

Not sure if this is slated for 1.11? https://clojure.atlassian.net/browse/CLJ-2502

oh yeah it is

ah right, I run with a patched version of that too, forgot about it

fix versions: 1.11

yes

do we have a bug for the "Reflector thing"? needs clarity

no jira that I know of, but I'm not sure that it's something we should address in clojure itself

here’s what @borkdude logged on pprint on clj-graal-docs: https://github.com/lread/clj-graal-docs/issues/19 - we closed because it was stale

perhaps at least logging pprint issue as a symptom in jira would be helpful?

it's not helpful unless there's more information than that

ok

if I find the time to diagnose why pprint does not work, I will log

> do we have a bug for the "Reflector thing"? needs clarity The JDK11 support for GraalVM is relatively new, I think most people are still using JDK8, that's why it probably never showed up. If you want, I can make an issue for it.

@alexmiller Following the exact same repro from lread but with this code:

(ns spec-test.core
  (:gen-class)
  (:require [clojure.pprint :as pprint]))

(defn -main [& _args]
  (println *clojure-version*)
  (pprint/pprint (range 100)))

Removing the references to pprint will produce a working binary with 1.10.1

related to locking?

yes

with --report-unsupported-elements-at-runtime, works with the patch

@alexmiller confirmed!

$ ./spec-test
{:major 1, :minor 10, :incremental 1, :qualifier patch_38bafca9_clj_1472_5}
(0
 1
 2
 3
 4

cool, but I need an explanation!

pprint doesn't use locking AFAIK

@ghadi this might be triggered by the Clojure compiler invoking spec on macro syntax checking?

and that uses locking in dynaload

what macro?

if that were the case, it would be any form affected

I don't know, but somehow that code is reachable by the analyzer. I don't know the details. I've been able to get around this using "-Dclojure.spec.skip-macros=true" which is why I thought that would be somehow related to it.

@ghadi Maybe this option could help explain it: https://github.com/lread/clj-graal-docs/blob/master/README.md#report-what-is-being-analyzed

Note that the output can get quite large.

well -- in any case clj-1472 takes care of it

I don't the ability to make that report at the moment

but I am idly curious still 🙂 . I don't like "spooky" / "magic"

btw @lee - that script... pure awesome, so easy to test out a clj-1472 patch 🙂

I'm able to compile clj-kondo with graalvm 20 jdk11 and 1472 patch 5. I consider clj-kondo to be a non-trivial example

@alexmiller The spec repro example works for me in GraalVM 20 java11:

$ ./spec-test
{:major 1, :minor 10, :incremental 1, :qualifier patch_38bafca9_clj_1472_5}
11.0.6
GraalVM 20.0.0 CE
true

This is the program:

(ns spec-test.core
  (:gen-class)
  (:require [clojure.spec.alpha :as s]))

(s/def ::g int?)

(defn -main [& _args]
  (println *clojure-version*)
  (println (System/getProperty "java.version"))
  (println (System/getProperty "java.vm.version"))
  (println (s/valid? ::g 1)))

it works for me with --report-unsupported-elements-at-runtime, but not without it

thanks

rich ok'ed clj-2502 too

and we've vetted the clj-1472 patch on a much larger test bed with datomic too

I'm happy clj-1472 is working out, but I have no idea why it worked 🙂 I did read the explanation and tried understanding it. Why is there a try inside a try, is the outer try still doing anything?

well, it helps if you're looking at the bytecode :)

> Why is there a try inside a try, is the outer try still doing anything? Remember that we talked recently about there being no legit use case for a try without a catch / finally for a linter feature? 🙂 I wonder if this matter for the bytecode?

it might actually matter here for the scope of the exception table, I will look into it a little more

or maybe it could actually just be a do

this is like v5 of this patch and it was doing something in the prior version

I remember seeing an empty let (let [] ...) somewhere in core once

https://github.com/clojure/clojure/blob/653b8465845a78ef7543e0a250078eea2d56b659/src/clj/clojure/core.clj#L1966

maybe it's there for a similar reason

I have a repro for the clojure.lang.Reflector issue on JDK11:

(ns spec-test.core
  (:gen-class)
  (:import [clojure.lang Reflector])
  (:require [<http://clojure.java.io|clojure.java.io> :as io]))

(defn -main [&amp; _args]
  (println *clojure-version*)
  (println (System/getProperty "java.version"))
  (println (System/getProperty "java.vm.version"))
  (println (Reflector/invokeInstanceMethod (io/file ".") "exists" (object-array []))))

[{
  "name" : "<http://java.io|java.io>.File",
  "allPublicMethods" : true,
  "allPublicFields" : true,
  "allPublicConstructors" : true
}]

$ ./spec-test
{:major 1, :minor 10, :incremental 1, :qualifier patch_38bafca9_clj_1472_5}
1.8.0_242
GraalVM 20.0.0 CE
true

$ ./spec-test
{:major 1, :minor 10, :incremental 1, :qualifier patch_38bafca9_clj_1472_5}
11.0.6
GraalVM 20.0.0 CE
Exception in thread "main" com.oracle.svm.core.jdk.UnsupportedFeatureError: Invoke with MethodHandle argument could not be reduced to at most a single call or single field access. The method handle must be a compile time constant, e.g., be loaded from a `static final` field. Method that contains the method handle invocation: java.lang.invoke.Invokers$Holder.invoke_MT(Object, Object, Object, Object)
	at com.oracle.svm.core.util.VMError.unsupportedFeature(VMError.java:101)
	at clojure.lang.Reflector.canAccess(Reflector.java:49)

So I've been able to work around that like here: https://github.com/lread/clj-graal-docs#jdk11-and-clojurelangreflector

Let me know if you want an issue for it.