graalvm

Discuss GraalVM related topics. Use clojure 1.10.2 or newer for all new projects. Contribute to https://github.com/clj-easy/graal-docs and https://github.com/BrunoBonacci/graalvm-clojure. GraalVM slack: https://www.graalvm.org/slack-invitation/.
Saikyun 2020-06-15T07:01:44.395500Z

thanks πŸ™‚

kwrooijen 2020-06-15T12:23:40.396300Z

Is it possible to create tab-completion with a graalvm compiled project? Like the video below? If so, can someone point me at the right direction?

kwrooijen 2020-06-15T12:25:17.397Z

Or is that more of an OS think that you need to package?

alexmiller 2020-06-15T12:42:03.397600Z

It’s os/shell like https://iridakos.com/programming/2018/03/01/bash-programmable-completion-tutorial

kwrooijen 2020-06-15T12:43:24.398Z

ah perfect, no idea why that was so hard for me to find haha

kwrooijen 2020-06-15T12:43:33.398200Z

Thanks

Johannes F. Knauf 2020-06-15T13:06:38.399400Z

@kevin.van.rooijen https://github.com/l3nz/cli-matic/issues/39 might be of interest to you as well. If you want to go further into the direction of generating the completion from an abstract option declaration, I can recommend https://github.com/spf13/cobra/blob/master/bash_completions.go as reference.

naomarik 2020-06-15T13:16:56.401300Z

Added clj-http to my project and getting this error:

Error: No instances of javax.net.ssl.SSLContext are allowed in the image heap as this class should be initialized at image runtime. Object has been initialized without the native-image initialization instrumentation and the stack trace can't be tracked.
Tried setting this flag --initialize-at-run-time=<http://javax.net|javax.net>.ssl.SSLContext and other things but can't seem to get anything going.

naomarik 2020-06-15T13:29:30.401600Z

I can use clj-http-lite but I'm just being stubborn at this point.

kwrooijen 2020-06-15T13:49:33.401700Z

Thanks, this is really helpful!

kwrooijen 2020-06-15T13:51:02.401900Z

Seems like the graalvm-clojure project struck out of luck as well https://github.com/BrunoBonacci/graalvm-clojure/tree/master/http-kit#notes-on-http-kit-client

kwrooijen 2020-06-15T13:51:50.402200Z

Assuming you're trying to use the client?

borkdude 2020-06-15T13:54:59.402800Z

@naomarik isnt't this just these flags?

"-H:EnableURLProtocols=http,https"
       "--enable-all-security-services"

borkdude 2020-06-15T13:55:51.403Z

apparently not: https://github.com/oracle/graal/issues/1074

naomarik 2020-06-15T13:56:03.403400Z

yeah was about to post that https://github.com/oracle/graal/issues/1074

naomarik 2020-06-15T13:56:32.404100Z

I'm playing around with attaching a debugger but I have no idea what I'm doing... seeing if I can get a breakpoint somewhere

borkdude 2020-06-15T13:56:52.404500Z

if you can figure that one out, it would be great. clj-http-lite has been the escape hatch for it for a while

borkdude 2020-06-15T13:57:14.405200Z

which works quite well also btw

kwrooijen 2020-06-15T13:57:20.405400Z

erm. sorry. this is http-kit, not clj-http

naomarik 2020-06-15T13:57:46.406100Z

yeah it works for me, but I tend to get stubborn with issues and it's hard to not dig myself in a hole until I get things working the way I want

naomarik 2020-06-15T14:52:21.408200Z

Not sure how to do this, in clojure top level require seems to initialize classes which cascades all the way down to the java SSL stuff and trying to wrap http client in a function to dynamically require gets it compiled out of graalvm.

borkdude 2020-06-15T14:55:39.408800Z

@naomarik you can't dynamically require something in a graalvm image, at runtime. it has to required at compile time

naomarik 2020-06-15T14:56:15.409200Z

yeah, don't have enough knowledge to deal with this

borkdude 2020-06-15T14:56:36.409700Z

it boils down to this: eval does not work in graalvm native-image

naomarik 2020-06-15T15:16:27.411Z

I hacked clj-http wrapping the problematic code in a delay and it works, passes all tests too.

borkdude 2020-06-15T15:17:22.411500Z

cool!

naomarik 2020-06-15T15:17:25.411700Z

The classes were in a def so always being executed.

borkdude 2020-06-15T15:17:49.411900Z

do the tests pass with the native binary?

naomarik 2020-06-15T15:18:13.412300Z

I can compile it into my graalvm and it works, is that what you mean?

naomarik 2020-06-15T15:18:27.412800Z

lein test passes and my standalone graalvm doesn't complain about anything

borkdude 2020-06-15T15:18:39.413100Z

I mean, can you do everything that's done in the units tests, with your graalvm binary? that remains to be seen right?

naomarik 2020-06-15T15:19:22.414Z

not sure how to run all the unit tests in a graalvm binary

borkdude 2020-06-15T15:19:36.414300Z

@naomarik one way of testing this is to bind the clj-http functions into babashka and then run the tests using babashka

borkdude 2020-06-15T15:20:28.414700Z

I also test other libraries like this such as clj-commons/clj-yaml

naomarik 2020-06-15T15:21:05.415300Z

Not sure how to do that ;/ you have an example?

naomarik 2020-06-15T15:22:17.415600Z

I can throw my project up on a fork if you want to try it

borkdude 2020-06-15T15:23:47.416500Z

sure: 1) create one of those and hook it up in babashka/main.clj: https://github.com/borkdude/babashka/blob/master/feature-yaml/babashka/impl/yaml.clj 2) copy the library tests to https://github.com/borkdude/babashka/tree/master/test-resources/lib_tests/clj_yaml 3) run all tests within the bb binary: https://github.com/borkdude/babashka/blob/master/script/lib_tests/clj_yaml_test

naomarik 2020-06-15T15:27:03.416700Z

Here's a patch: https://pastebin.com/raw/d5CCRK9z

naomarik 2020-06-15T15:27:49.417100Z

Will try that

borkdude 2020-06-15T15:30:10.417600Z

@naomarik let me know if you get stuck. this is very interesting

borkdude 2020-06-15T15:30:39.418100Z

here are some hints how to get started. most often people don't clone recursively: https://github.com/borkdude/babashka/blob/master/doc/dev.md

naomarik 2020-06-15T16:53:17.424600Z

Hey @borkdude, I copied https://github.com/dakrone/clj-http/blob/3.x/test/clj_http/test/core_test.clj into test-resources/lib_tests like like clj-yaml.core-test , I edited babashka.impl.clj-http and main.clj get rid of each error as it came up and got around here:

➜  lib_tests git:(master) βœ— ./clj_http_test
java.lang.Exception: Could not require ring.adapter.jetty. [at /home/naomarik/babashka/test-resources/lib_tests/clj_http/core_test.clj, line 1, column 1]
Error encountered performing task 'run' with profile(s): 'test'
Suppressed exit
➜  lib_tests git:(master) βœ— ./clj_http_test
java.lang.Exception: Unable to resolve classname: java.net.SocketTimeoutException [at /home/naomarik/babashka/test-resources/lib_tests/clj_http/core_test.clj, line 1, column 1]
Error encountered performing task 'run' with profile(s): 'test'
Suppressed exit
before giving up, there's an enormous amount of imports and not sure whether I was putting the right code in the impl file.

borkdude 2020-06-15T17:02:30.425900Z

@naomarik I see: the clj-http tests run a server to test clj-http. this requires a bit of a different approach. the server needs to be started in a JVM, and then the clj-http test code needs to run in babashka itself

naomarik 2020-06-15T17:05:12.426700Z

yeah seems like a long story πŸ™‚ can't spend time on it anymore unfortunately, but running around babashka source, man you've done a lot of work!

borkdude 2020-06-15T17:06:28.427700Z

@naomarik you can try to do a couple of request just outside the tests though: bb -e "(require '[clj-http ...] (client/get ...)", etc

naomarik 2020-06-15T17:11:20.427900Z

yup! actually i did that and it works

borkdude 2020-06-15T17:16:33.428100Z

@naomarik do you have a link to your fork?

borkdude 2020-06-15T17:18:46.429Z

might be worth doing an issue + PR against the original repo

borkdude 2020-06-15T17:39:03.430Z

@naomarik I also tried it locally myself now with your fork.

$ ./bb "(require '[clj-http.client :as client]) (:status (client/get \"<https://www.google.com>\"))"
java.lang.ClassNotFoundException: org.apache.commons.logging.impl.LogFactoryImpl [at line 1, column 50]
adds about 40MBs to the bb binary (100 mb in total) and eats a lot more ram (if I give it 8GB, it will eat 8GB)

ghadi 2020-06-15T17:40:00.430600Z

I would categorically seek to avoid adding clj-http.client to a native image

ghadi 2020-06-15T17:40:06.430900Z

it's not that good of an http library

ghadi 2020-06-15T17:40:13.431100Z

though many use it

ghadi 2020-06-15T17:40:35.431600Z

and it's certainly not designed for native-image (many many deps)

ghadi 2020-06-15T17:41:27.432600Z

40MB is a lot

borkdude 2020-06-15T17:41:43.433Z

yep

borkdude 2020-06-15T17:42:07.433400Z

saw a similar thing with hato. clj-http-lite might be a nice middle ground for native-image CLIs

borkdude 2020-06-15T17:42:28.433900Z

or if shelling out to curl is fine, then https://github.com/borkdude/babashka.curl is probably the lightest solution

borkdude 2020-06-15T17:43:18.434700Z

it's still cool that naomarik got it to compile though, kudos

naomarik 2020-06-15T18:20:23.435400Z

oh strange, for my project clj-http compiles to 66MB, clj-http-lite is 57MB

naomarik 2020-06-15T18:24:08.436300Z

Without either it's 56MB, so clj-http adds 10MB for me.

naomarik 2020-06-15T18:27:26.436700Z

Using GRAALVM_VERSION="20.1.0" , openjdk version "11.0.7" 2020-04-14 under WSL2 ubuntu

borkdude 2020-06-15T18:27:39.437Z

that's weird. I'm on mac here

borkdude 2020-06-15T18:28:23.437400Z

These are binaries built on CI:

[8:02 PM] [linux-static - dakrone-clj-http@bfffbd9400c04e3c345f2251b4ab8a4fa68a7d19]:Β <https://10052-201467090-gh.circle-artifacts.com/0/release/babashka-0.1.3-SNAPSHOT-linux-static-amd64.zip>
[8:02 PM] [linux-static - dakrone-clj-http@bfffbd9400c04e3c345f2251b4ab8a4fa68a7d19] binary size: 105772312
[8:03 PM] [linux - dakrone-clj-http@bfffbd9400c04e3c345f2251b4ab8a4fa68a7d19]:Β <https://10049-201467090-gh.circle-artifacts.com/0/release/babashka-0.1.3-SNAPSHOT-linux-amd64.zip>
[8:03 PM] [linux - dakrone-clj-http@bfffbd9400c04e3c345f2251b4ab8a4fa68a7d19] binary size: 110654720
[8:05 PM] [macos - dakrone-clj-http@bfffbd9400c04e3c345f2251b4ab8a4fa68a7d19]:Β <https://10050-201467090-gh.circle-artifacts.com/0/release/babashka-0.1.3-SNAPSHOT-macos-amd64.zip>
[8:05 PM] [macos - dakrone-clj-http@bfffbd9400c04e3c345f2251b4ab8a4fa68a7d19] binary size: 103977664

borkdude 2020-06-15T18:28:27.437600Z

all over 100MB

borkdude 2020-06-15T18:28:51.437800Z

without clj-http ~60MB

borkdude 2020-06-15T18:29:07.438100Z

maybe you did some other things apart from the patch @naomarik?

naomarik 2020-06-15T18:29:22.438400Z

no nothing, are you using the latest graalvm?

naomarik 2020-06-15T18:29:53.438600Z

i believe the graalvm I'm using is java 11 as well

naomarik 2020-06-15T18:30:10.438800Z

not sure if that makes a difference

naomarik 2020-06-15T18:58:14.439900Z

http-kit works btw on latest dev version

borkdude 2020-06-15T19:08:41.440500Z

I'm using the same version you are using. Maybe you already had a lot of overlapping deps with clj-http and my project didn't