I'm trying to do the hello-world tutorial on clj-graal-docs on windows and hitting issues. I have my folder structure and the main.clj and deps.edn. It compiles when I do
Then it says verify it works on the JVM with
clojure -M -e "(compile 'hello-world.main)"
When I run that java command I get an error.
$ java -cp $(clojure -Spath):classes hello_world.main Hello world!
The output to the classes folder it asked me to create was this: I'm not familiar enough with the CLI tools to know if that command to compile is wrong, or I'm doing something else stupid.
Error: Could not find or load main class :classes Caused by: java.lang.ClassNotFoundException: :classes
ah ok, doh!
I do have some windows examples
let me check
@qmstuart For Windows (and in general now) I tend to first build an uberjar and then feed that uberjar to graalvm. This also takes care of "too long" classpath issues on Windows
You can find an example here: https://github.com/borkdude/carve/blob/master/script/compile https://github.com/borkdude/carve/blob/master/script/compile.bat https://github.com/borkdude/carve/blob/master/deps.edn
^ @lee this is what I was getting at with the Windows classpath / uberjar problem, it's one of the two issues
So I'm compiling a command-line app using native-image, and the resulting binary weighs in at a whopping 94 MB. I ran native-image with
-H:DashboardDump=dashboard -H:+DashboardCode to try to tell where all this weight is coming from (`-H:+DashboardAll` crashes with a message about deadlock), but looking at the result at https://www.graalvm.org/docs/tools/dashboard the total size adds up to only 37 MB. Any idea what's going on here?
@wombawomba There are a few things which can bloat the binary size in clojure projects. I have seen this happen mostly with runtime requires/requiring-resolve usages
Usually I find out about this by starting small and then adding more stuff, to find out what triggers it
@borkdude any ideas on how to start if I want to try something like that? Currently my build command is
native-image --allow-incomplete-classpath --enable-http --enable-https --initialize-at-build-time --initialize-at-run-time=org.jline.terminal.impl.jansi.win.JansiWinSysTerminal,org.jline.terminal.impl.jna.win.JnaWinSysTerminal --no-fallback --no-server --report-unsupported-elements-at-runtime --verbose (the
--inititalize-at-run-time arg seem to be required for https://github.com/jline/jline3)
The first start is just print hello world from your main.
And then add one function to it at a time
@borkdude your video on graalvm native hello world is good and helpful. Thanks for making these little videos 🙂1🙏
thanks, I think I can work this into a tip
or we can add a Windows-friendly hello world in there
Where can I specify x86 v x64 ?
Error: Native-image building on Windows currently only supports target architecture: AMD64 (x86 unsupported) Error: To prevent native-toolchain checking provide command-line option -H:-CheckToolchain com.oracle.svm.core.util.UserError$UserException: Native-image building on Windows currently only supports target architecture: AMD64 (x86 unsupported)
or maybe this a consequence of teh VS I have installed, and therefore the cl.exe ?
OK, you need to use the x64 developer command prompt to run graal-vm
it works, i have a native windows.exe 😄
thanks for all the help 👍
What kind of file sizes do you folks see for binaries created with native-image? I always end up with sizes between 75-90mb (for clj+jetty+postgres+jdbc).
This 75-90mb is from uberjars around 10mb 🙂
@thegeez Hello world is around 10mb. With sci (interpreter) not much more. Adding postgres to babashka adds around 3mb, not much. But I think the incremental size depends on what you were already using before.
Babashka itself is now 70mb on macos. I curate every lib that goes in to make sure the binary size doesn't blow up
There are a few things that can trigger binary size to blow up, such as runtime requires/resolve
Thanks, then it might be worth it to investigate getting lower sizes for my case. Thus far I always assumed 60mb as the lowest bar
clojure.pprint has some usage of
find-var inside of it. I patched that to make the vars known at at compile time. This took care of a couple of dozen mbs.
There is also https://github.com/borkdude/dynaload which gives better binary size than the clojure.spec dynaload
@thegeez babashka has httpkit built in. this only added 1.5mb to the binary including the http client
But adding the full ring stack will probably bloat stuff again due to some require at runtime somewhere
This has already been patched by someone who wanted to add ring to bb
Alright I guess I'll need to do a day of trail-and-error compilation to try to get down to those numbers. Too bad each native-image compilation takes around 6 mins on my machine
Yes, this can be a painful process...
I'm genuinely curios about the concern around binaries size - if it's was 0.5GB I'd be worried, but 90MB feels like nothing nowadays. It kinda makes sense for bb in some way, then again
node_modules in one of our apps is 650MB....
It is not a problem for my use-case, which is aws lambda, the zip size is 20mb. But if there's any further improvement possible I'll gladly take it. Because deploy is compilation (~6min) -> upload 20mb
@thegeez Yeah, maybe not worry about this. 90mb sounds fine for an "app". clojure-lsp is also this size. Thinking about it though: both projects use next.jdbc. After upgrading recently I saw the binary size of babashka pods grow to 50mb, so it may be explained from this lib usage as well. Anyway, I agree with @lukaszkorecki: if this is something you use in private, then it's no big deal, if it's used by the masses, maybe more, but maybe still not a big deal.
I'm already happy that I don't need to change to use for instance ClojureScript on nodejs to use lambda, so far native-image is great already.
But it would also be awesome if I could email around the final binaries 🙂
If you're using amazon, put it in a zipfile on s3? ;)
Ah, I was not aware of the lambda file limit - we deploy Clojure on JVM in lambda and the uberjar size is <10MB so we're well below that. That said, the newly released Lambda + container images effectively eradicates that issue.
It's zipfile on s3 right now, that won't change even with a smaller size
I don't think thegeez said that aws lambda has a binary size limit?
It's nice to have, rather than must have. Which is good enough already
@thegeez What is the use case of a webapp on aws lambda? afaik you don't need a webserver to process lambda invocations right?
It's API Gateway -> Lambda bootstrap custom runtime. The runtime handles http calls containing lambda events from the API Gateway which gets http calls from browsers
I think all lambda request handler libraries/frameworks expose a web handler to receive requests, but perhaps there are other ways
I’m working on a custom Lambda runtime at Cognitect, interested in your use cases
I prototyped bb+Lambda+container image, doesn't get easier than this: https://gist.github.com/lukaszkorecki/a1fe27bf08f9b98e9def9da4bcb3264e
no need for custom runtimes etc
My use case is web-app with cljs front-end for real-time collab over API Gateway websockets, web app is API Gateway HTTP api to lambda talking to Postgres rds database. This is a prototype atm
So the webserver is short-lived, as soon as the person disconnects, the lambda can go to sleep again, let's say?
A single lambda can handle consecutive reqs, but may disappear at any time (in my setup; there's no full webserver handling concurrent requests)
right, so if there are no requests for a long time, you don't pay amazon. that's nice for low traffic websites/apps
I saw a tweet that describes serverless as "pay per request" architecture, that fits I think
and also for very high traffic, since it scales automatically =)
My goal is more lowest devops/most convenient devops required. I don't do well managing hardware
You're still managing hardware though ;)
Thanks for sharing your setup.
Yes serverless is a lie 🙂
Thanks for sharing your numbers
FYI, we moved to AWS now at Doctor Evidence, no more manual hardware fixes ;)
Correction to what I said above, there's no webserver in the native-image. It's browser -req-> API Gateway <-req- native app. So the native app on lambda only needs an http client.
Then the image could be fairly small (depending on what else it does). Which client are you using? Clj-http is known to bloat the binary. Http-kit is much better there
Does the image still do the db stuff?
http through clj-http-lite, the jdbc.next+postgres is still there
yes, but don't use a wrapper like hato, it has even worse bloat due to all sorts of runtime requires
Perhaps https://github.com/exoscale/telex fairs better, since it's really minimal
Underneath your lambda never runs a web server - it GETs event data via a private metadata endpoint and POSTs results to it. So in Lambda runtime/SDK there usually is some http client included.
has anyone tried to compile a project depending on https://github.com/igrishaev/etaoin to a native-executable?
clj-http giving me some issues
This is a better example of the usage: https://github.com/babashka/pod-registry/blob/master/examples/etaoin.clj I should really update the README
@jarvinenemil I think we'll have to update it with etaoin proper as well
it's 100 commits or so behind I see
I mean etaoin-graal. I will update it now
I figured, cool, thanks!
Really awesome that you had forked etaoin and made a graal-vm compatible version. I was just about to do the same
@jarvinenemil Published upgraded version as: borkdude/etaoin-graal 0.4.2-deb92aaa21c59c52521f42c0177bbce9ac10a0e2
Now I'll update the version at pod-babashka-etaoin and I'll await your PR1👍
Heads up if anyone is using
The --enable-all-security-services option is now deprecated and it will be removed in a future release.
It will be deprecated in favor of automatically detecting by static analysis