Clojurians Log v2

tzzh/aws is compiled using Go and that usually works on any platform since it doesn't depend on libc++ like graalvm binaries

You can load pods from anywhere on the file system, you just can provide a file path

On nixOS you likely have to install libc++ alongside the binary to make it work. You can see this using the ldd command

there is likely a nixOS package for babashka or clj-kondo too, this should work similarly

> On nixOS you likely have to install libc++ alongside the binary to make it work. You can see this using the ldd command @borkdude NixOS doesn't have the libraries in the common location (.e.g.: they're on /nix/store, not in /usr/lib), so even when they're "installed" most programs can't find them

> there is likely a nixOS package for babashka or clj-kondo too, this should work similarly Actually, the static linux binaries from Babashka works well in NixOS, and I think if AWS pod were compiled statically it would works as well

@ericdallo Can you shine more light on this, since you are also using nixOS and bb?

The static binary doesn't work on every system, the use case for it is mostly alpine/busybox docker images

Well, I helped @ericdallo with NixOS issues too

> The static binary doesn't work on every system, the use case for it is mostly alpine/busybox docker images Huh... Curious, can you give an example where it doesn't :thinking_face: ?

AFAIK static libs should work with any linux distribution including NixOS, also only static bb works fine on NixOS last time I tested it

@thiagokokada I don't have a specific example, but it comes up every now and then when people try the static linux binary on their linux system and run into problems with it. My default question is: did you try the static binary... try the dynamic one instead. It has come up a handful of times. @rahul080327 knows more about the trade-offs between static and dynamic here

the issue is that every lib should be static, for example, even if compile clojure-lsp as static, it still fails since sqlite lib is not static

Having said this, it's no problem to provide a static binary for the aws pod

Yeah, I know about this issue @ericdallo, but in this case Babashka looks like a fully static library

yes, it looks indeed

It doesn't seem to try to load anything dynamically neither

@borkdude > You can load pods from anywhere on the file system, you just can provide a file path Yeah, I know about it, but I am thinking about the best solution for having some custom pods without needing to commit them to repository, for example

If we could have custom repositories, this would be a breeze

@thiagokokada Please make a PR to https://github.com/babashka/pod-babashka-aws. It needs an environment variable in the circleci config to say whether it should include the static flag or not. And then the script/compile script should pick up on that environment variable. I apply this in clj-kondo/clj-kondo and babashka/babashka as well, so you could use those as examples. After that there will be a static pod, but currently there is no way to retrieve a static binary from the registry pod so you will have to load it using a relative or absolute path yourself.

But I can find other solutions if this isn't desirable

I haven't thought about custom repositories but that doesn't sound too hard

> After that there will be a static pod, but currently there is no way to retrieve a static binary from the registry pod so you will have to load it using a relative or absolute path yourself. Yeah, this is also another reason why I think custom repositories would be interesting

I could for example point the repositories to the static version of the pods

Maybe the base URL of the repository can be tweaked using an environment variable, or maybe we should support a list of base-urls which will be tried in order? Have to think about. AFK for groceries now :)

I was thinking maybe allowing pods/load-pods receive an extra argument for options (probably a map or something), or maybe have the URL in a using (def ^:dynamic ...) var that could be overwrite with binding (I would prefer to avoid environment variables though) Having a list of URLs instead of only one would be useful too

Yeah, that makes sense

https://github.com/babashka/pod-babashka-aws/pull/35 :reviewplease:

> Huh... Curious, can you give an example where it doesn't @thiagokokada we have seen issues with the static binary having issues when its being run in an environment providing a different/newer versions of ABI. The linux kernel verion being different also causes it to segfault. The binary produced by the static graalvm compilation is liked for Linux 2.6.32 and whenever we see it being run in a higher kernel version it segfaults at various places. Most notably when doing network IO. I'm not really sure what exactly is the root cause, haven't gotten around to explore that deep.

its mostly something to do with the linking done by the graal native image. Go binaries which are mostly static seem to this better. Also my hunch is around the musl versions linked to the graal binary. since Go neither uses musl nor glibc, its able to produce more portable static binaries. My very uneducated guess would be the version of musl used to by graal to build native images fails when the kernel is different. Specially when the OS is glibc/dynamic linking based like Debian etc and not musl based like Alpine

@ben.sless have a look here if you wanna know the issues we have faced with static binary, if you have some ideas, they are most welcome! 🙂

> musl I don't want to condemn it but you may be SOL. A static built with musl on glibc systems even starting up is a miracle

yeah 😕 its something we need to address upstream with the GraalVM devs somehow.

Go binaries are built completely differently, even its concept of assembly is different

exactly!

The only difference between normal bb is that it has been built with the static flag of GraalVM native image. I don’t think it has anything specific to musl, but perhaps read the docs

@borkdude we do need to specify --libc=musl right?

No

We are just using the static flag, which I can’t type on my phone

The other variations were introduced much later in GraalVM

right, the docs on Graal static says

To build a static native image, use:

native-image --static --libc=musl [other arguments] Class

but in order to have a static binary on linux we definitely need to have musl linkage

i guess they could be adding it by default with --static

@thiagokokada Could you perhaps try the same that bsless tried in the main channel, on nixOs with the static bb image? (shelling out)

Perhaps there is some issue around ProcessBuilder in the static image on some OS/kernels?

Ben's versions:

OS: Pop 20.10 groovy
Kernel: x86_64 Linux 5.11.0-7614-generic

This seems to work on alpine:

$ docker run --rm -i babashka/babashka:0.4.0-alpine <<< '(:out (babashka.process/sh "ls"))'
Babashka v0.4.0 REPL.
Use :repl/quit or :repl/exit to quit the REPL.
Clojure rocks, Bash reaches.

user=> "bin\ndev\netc\nhome\nlib\nlib64\nmedia\nmnt\nopt\nproc\nroot\nrun\nsbin\nsrv\nsys\ntmp\nusr\nvar\n"

@borkdude whats the kernel version here?

/ # uname -r
4.19.121-linuxkit

@thiagokokada could you try https://329-325868827-gh.circle-artifacts.com/0/release/pod-babashka-aws-0.0.5-linux-amd64-static.zip ? this is the freshly brewed musl linked binary!

@rahul080327 Just did a static build from Babashka too: https://github.com/babashka/babashka/pull/827 My approach is different since I build musl from scratch, but I will try your binary

yeah i managed to shave off the really long musl compilation

musl-tools is installed via apt and seems to work fine

also @borkdude and i were thinking is musl works fine, we can make that the static one

@rahul080327 Yeah, it works :thumbsup:. But keep in mind I never had issues with other static binaries from Babashka, so maybe I am not the best person to test this.

Yay! well we havent compiled any other clojure pods statically and i think using musl should be much more stable

> musl-tools is installed via apt and seems to work fine As far I understand, there is some disadvantages of using musl-tools - It is an out-dated version of musl (in my branch it uses an 1.2.0 that is also out-dated, ideally we should use musl 1.2.2 since it has a much better malloc implementation; but building from scratch is easier to update musl) - Also, I am not sure that copying libz.a from Debian as is isn't going to cause some problems.

But I am fine either way, if it works the advantage of your approach is not building musl (that takes quite a long time)

yeah so i build the libz after building with musl-tools, that should be okay? i copied the libz build from the link you sent

Yeah, I think building libz with musl-tools should be a better approach to avoid issues

Should avoid e.g.: ABI incompatibilities

yeah so explicitly copied that one instead of installing. i agree on trying on using a newer musl-tools, building it from source is really long 😕 maybe we can get around it with caching?

BTW, bb statically-build with musl is slightly smaller too:

ls -lah bb
-rwxr-xr-x 1 thiagoko users 80M mai  8 13:15 bb
ls -lah bb
-rwxr-xr-x 1 thiagoko users 76M mai  9 14:45 bb

> maybe we can get around it with caching? Yeah, I think caching would be the way to go if we want to build musl from source

do you reckon there could be a way to install the newer musl-tools? maybe using the debian-testing mirrors?

> BTW, bb statically-build with musl is slightly smaller too this is awesome!

> do you reckon there could be a way to install the newer musl-tools? maybe using the debian-testing mirrors? I think using musl-tools from testing in stable is risk for the same reason that using glibc for static linking is risk (there is a chance for glibc version mismatch)

> Yay! well we havent compiled any other clojure pods statically We have, the sql pods are also available as static compiled

If it is available in debian backports it would be safer, but even debian-testing has a chance of being out-dated

> I think using musl-tools from testing in stable is risk for the same reason that using glibc for static linking is risk (there is a chance for glibc version mismatch) Unless musl-tools is build statically, them it should be safe to use it

@rahul080327 https://packages.debian.org/unstable/musl-tools In unstable musl-tools is up-to-date ☝️

yeah so we can add the apt sources and just pull musl tools from it?

I'm fine with any approach, but maybe we should document this somewhere, possibly in the https://github.com/lread/clj-graal-docs repo (once the correct approach is figured out)

> yeah so we can add the apt sources and just pull musl tools from it? Yeah, I think we can try Just avoid installing anything unnecessary from unstable (otherwise we may break something else)

I think the best way is to add unstable with low priority, and explicitly install musl-tools from unstable

yeah thats what im thinking

> I think the best way is to add unstable with low priority, and explicitly install musl-tools from unstable http://jaqque.sbih.org/kplug/apt-pinning.html

With musl 1.2.2 the binaries are almost the same as glibc in size:

ls -lah bb
-rwxr-xr-x 1 thiagoko users 79M mai  9 15:44 bb

pushed https://github.com/babashka/pod-babashka-aws/commit/e683491e8a7bd635c1af09bf0550a5098983aa88

test are passing @borkdude!

if youre okay with this approach @thiagokokada what say we stick to this?

LGTM

@rahul080327 Maybe you can add a few comments to the setup-musl script to explain why this approach was chosen?

sure, should i add it as a comment in the file itself?

yeah, I think that's the best place

just a few comments at the start

on it

pushed

@borkdude i think its good to merge?

also maybe one last check with https://346-325868827-gh.circle-artifacts.com/0/release/pod-babashka-aws-0.0.5-linux-amd64-static.zip @thiagokokada?

Seems to works fine :thumbsup:

@rahul080327 Do you have a PR I should merge?

the musl branch, should i raise a PR?

raising one now

https://github.com/babashka/pod-babashka-aws/pull/37

merged

🙏 !

🎉