Clojurians Log v2

I'm trying to do the hello-world tutorial on clj-graal-docs on windows and hitting issues. I have my folder structure and the main.clj and deps.edn. It compiles when I do

clojure -M -e "(compile 'hello-world.main)"

$ java -cp $(clojure -Spath):classes hello_world.main
Hello world!

Error: Could not find or load main class :classes
Caused by: java.lang.ClassNotFoundException: :classes

@qmstuart $(clojure -Spath) is bash syntax

ah ok, doh!

I do have some windows examples

let me check

@qmstuart For Windows (and in general now) I tend to first build an uberjar and then feed that uberjar to graalvm. This also takes care of "too long" classpath issues on Windows

You can find an example here: https://github.com/borkdude/carve/blob/master/script/compile https://github.com/borkdude/carve/blob/master/script/compile.bat https://github.com/borkdude/carve/blob/master/deps.edn

thank you!

^ @lee this is what I was getting at with the Windows classpath / uberjar problem, it's one of the two issues

So I'm compiling a command-line app using native-image, and the resulting binary weighs in at a whopping 94 MB. I ran native-image with -H:DashboardDump=dashboard -H:+DashboardCode to try to tell where all this weight is coming from (`-H:+DashboardAll` crashes with a message about deadlock), but looking at the result at https://www.graalvm.org/docs/tools/dashboard the total size adds up to only 37 MB. Any idea what's going on here?

@wombawomba There are a few things which can bloat the binary size in clojure projects. I have seen this happen mostly with runtime requires/requiring-resolve usages

Usually I find out about this by starting small and then adding more stuff, to find out what triggers it

@borkdude any ideas on how to start if I want to try something like that? Currently my build command is native-image --allow-incomplete-classpath --enable-http --enable-https --initialize-at-build-time --initialize-at-run-time=org.jline.terminal.impl.jansi.win.JansiWinSysTerminal,org.jline.terminal.impl.jna.win.JnaWinSysTerminal --no-fallback --no-server --report-unsupported-elements-at-runtime --verbose (the --inititalize-at-run-time arg seem to be required for https://github.com/jline/jline3)

The first start is just print hello world from your main.

And then add one function to it at a time

@borkdude your video on graalvm native hello world is good and helpful. Thanks for making these little videos 🙂

alright

thanks, I think I can work this into a tip

or we can add a Windows-friendly hello world in there

that too

Error: Native-image building on Windows currently only supports target architecture: AMD64 (x86 unsupported)
Error: To prevent native-toolchain checking provide command-line option -H:-CheckToolchain
com.oracle.svm.core.util.UserError$UserException: Native-image building on Windows currently only supports target architecture: AMD64 (x86 unsupported)

or maybe this a consequence of teh VS I have installed, and therefore the cl.exe ?

OK, you need to use the x64 developer command prompt to run graal-vm

ls

it works, i have a native windows.exe 😄

thanks for all the help 👍

yay!

What kind of file sizes do you folks see for binaries created with native-image? I always end up with sizes between 75-90mb (for clj+jetty+postgres+jdbc).

Not bad!

This 75-90mb is from uberjars around 10mb 🙂

@thegeez Hello world is around 10mb. With sci (interpreter) not much more. Adding postgres to babashka adds around 3mb, not much. But I think the incremental size depends on what you were already using before.

Babashka itself is now 70mb on macos. I curate every lib that goes in to make sure the binary size doesn't blow up

There are a few things that can trigger binary size to blow up, such as runtime requires/resolve

Thanks, then it might be worth it to investigate getting lower sizes for my case. Thus far I always assumed 60mb as the lowest bar

E.g. clojure.pprint has some usage of find-var inside of it. I patched that to make the vars known at at compile time. This took care of a couple of dozen mbs.

There is also https://github.com/borkdude/dynaload which gives better binary size than the clojure.spec dynaload

@thegeez babashka has httpkit built in. this only added 1.5mb to the binary including the http client

But adding the full ring stack will probably bloat stuff again due to some require at runtime somewhere

This has already been patched by someone who wanted to add ring to bb

Alright I guess I'll need to do a day of trail-and-error compilation to try to get down to those numbers. Too bad each native-image compilation takes around 6 mins on my machine

Yes, this can be a painful process...

I'm genuinely curios about the concern around binaries size - if it's was 0.5GB I'd be worried, but 90MB feels like nothing nowadays. It kinda makes sense for bb in some way, then again node_modules in one of our apps is 650MB....

It is not a problem for my use-case, which is aws lambda, the zip size is 20mb. But if there's any further improvement possible I'll gladly take it. Because deploy is compilation (~6min) -> upload 20mb

@thegeez Yeah, maybe not worry about this. 90mb sounds fine for an "app". clojure-lsp is also this size. Thinking about it though: both projects use next.jdbc. After upgrading recently I saw the binary size of babashka pods grow to 50mb, so it may be explained from this lib usage as well. Anyway, I agree with @lukaszkorecki: if this is something you use in private, then it's no big deal, if it's used by the masses, maybe more, but maybe still not a big deal.

I'm already happy that I don't need to change to use for instance ClojureScript on nodejs to use lambda, so far native-image is great already.

But it would also be awesome if I could email around the final binaries 🙂

If you're using amazon, put it in a zipfile on s3? ;)

Ah, I was not aware of the lambda file limit - we deploy Clojure on JVM in lambda and the uberjar size is <10MB so we're well below that. That said, the newly released Lambda + container images effectively eradicates that issue.

It's zipfile on s3 right now, that won't change even with a smaller size

I don't think thegeez said that aws lambda has a binary size limit?

It's nice to have, rather than must have. Which is good enough already

@thegeez What is the use case of a webapp on aws lambda? afaik you don't need a webserver to process lambda invocations right?

It's API Gateway -> Lambda bootstrap custom runtime. The runtime handles http calls containing lambda events from the API Gateway which gets http calls from browsers

I think all lambda request handler libraries/frameworks expose a web handler to receive requests, but perhaps there are other ways

I’m working on a custom Lambda runtime at Cognitect, interested in your use cases

I prototyped bb+Lambda+container image, doesn't get easier than this: https://gist.github.com/lukaszkorecki/a1fe27bf08f9b98e9def9da4bcb3264e

no need for custom runtimes etc

My use case is web-app with cljs front-end for real-time collab over API Gateway websockets, web app is API Gateway HTTP api to lambda talking to Postgres rds database. This is a prototype atm

So the webserver is short-lived, as soon as the person disconnects, the lambda can go to sleep again, let's say?

A single lambda can handle consecutive reqs, but may disappear at any time (in my setup; there's no full webserver handling concurrent requests)

right, so if there are no requests for a long time, you don't pay amazon. that's nice for low traffic websites/apps

I saw a tweet that describes serverless as "pay per request" architecture, that fits I think

and also for very high traffic, since it scales automatically =)

My goal is more lowest devops/most convenient devops required. I don't do well managing hardware

You're still managing hardware though ;)

Thanks for sharing your setup.

Yes serverless is a lie 🙂

Thanks for sharing your numbers

FYI, we moved to AWS now at Doctor Evidence, no more manual hardware fixes ;)

Nice!

Correction to what I said above, there's no webserver in the native-image. It's browser -req-> API Gateway <-req- native app. So the native app on lambda only needs an http client.

Then the image could be fairly small (depending on what else it does). Which client are you using? Clj-http is known to bloat the binary. Http-kit is much better there

Does the image still do the db stuff?

http through clj-http-lite, the jdbc.next+postgres is still there

http://java.net.http FTW

yes, but don't use a wrapper like hato, it has even worse bloat due to all sorts of runtime requires

Perhaps https://github.com/exoscale/telex fairs better, since it's really minimal

Underneath your lambda never runs a web server - it GETs event data via a private metadata endpoint and POSTs results to it. So in Lambda runtime/SDK there usually is some http client included.

has anyone tried to compile a project depending on https://github.com/igrishaev/etaoin to a native-executable?

clj-http giving me some issues

@jarvinenemil yes: https://github.com/babashka/pod-babashka-etaoin using that you can use etaoin from babashka

This is a better example of the usage: https://github.com/babashka/pod-registry/blob/master/examples/etaoin.clj I should really update the README

@jarvinenemil I'm looking back now. I made a fork of etaoin: https://github.com/borkdude/etaoin-graal and published it to clojars: borkdude/etaoin-graal "0.3.7-alpha.1"

@borkdude I tried https://github.com/babashka/pod-babashka-etaoin but I will have to make pull-request filling in a few functions that I need (such as etaoin.api/get-element-attr . Working on it

@jarvinenemil I think we'll have to update it with etaoin proper as well

it's 100 commits or so behind I see

ah

I mean etaoin-graal. I will update it now

I figured, cool, thanks!

Really awesome that you had forked etaoin and made a graal-vm compatible version. I was just about to do the same

@jarvinenemil Published upgraded version as: borkdude/etaoin-graal 0.4.2-deb92aaa21c59c52521f42c0177bbce9ac10a0e2

https://clojars.org/borkdude/etaoin-graal

Now I'll update the version at pod-babashka-etaoin and I'll await your PR

Heads up if anyone is using --enable-all-security-services ,

The --enable-all-security-services option is now deprecated and it will be removed in a future release.

It will be deprecated in favor of automatically detecting by static analysis