Clojurians Log v2

Hi babashkackerz ! On the README, I read "Interop with commonly used classes (System, File, java.time., java.nio.)", so apart from these Java classes we can't use any other like org.freedesktop.dbus.DBusInterface for example, right?

That is correct, unless it is explicitly added in the babashka binary

I was wondering if we could do without Python for this kind of script for example. http://blog.muhuk.com/2015/09/09/displaying_clojure_test_results_in_taskbar.html

#!/usr/bin/env python

import dbus
import sys


def notify(summary, body='', app_name='', app_icon='',
        timeout=5000, actions=[], hints=[], replaces_id=0):
    _bus_name = 'org.freedesktop.Notifications'
    _object_path = '/org/freedesktop/Notifications'
    _interface_name = _bus_name

    session_bus = dbus.SessionBus()
    obj = session_bus.get_object(_bus_name, _object_path)
    interface = dbus.Interface(obj, _interface_name)
    interface.Notify(app_name, replaces_id, app_icon,
            summary, body, actions, hints, timeout)


if __name__ == '__main__':
    notify(*sys.argv[1:])

Thank you for confirmation @borkdude

@admin055 The ways to interact with classes not inside babashka are currently: - shelling out to another CLI that can do it - via a pod that is designed for this use case (which is also kind of a CLI but nicer integration code-wise)

@admin055 So if this is a frequent use case, you could invest in building a pod for it

Yes pods are excellent feature, I had toying a little bit with pod-jaydeesimon-jsoup and it was nice ! I also plan to take some time soon to install the Graalvm toolchains and attempt to contribute with pods etc.

Maybe this also works: http://www.kaizou.org/2014/06/dbus-command-line.html

@borkdude Yes I will. Thanks the CLI solution could a simplified first approach, thanks for the tip.

I want to simplify installing pods so you can put some config in a babashka.edn and the pod will be automatically downloaded and resolved by babashka.pods/load-pod (or maybe (babashka.pods/require 'foo.bar) since a pod management system could also list which namespaces belong to which pods). I hope that this will improve the adoption of pods. I already made something like this: https://github.com/borkdude/glam, but I think I will make something specific for pods in babashka to download the pods into a cache, etc.

Yeah, great idea! 👍

@borkdude docker build --target BASE -t bb-builder . on the latest master fails for me during RUN ./script/uberjar due to > Retrieving borkdude/sci.impl.reflector/0.0.1-java11/sci.impl.reflector-0.0.1-java11.jar from clojars > OpenJDK 64-Bit Server VM warning: forcing TieredStopAtLevel to full optimization because JVMCI is enabled > Syntax error (ClassNotFoundException) compiling at (/tmp/form-init12305127242809915869.clj:1:74). > org.yaml.snakeyaml.error.YAMLException > ... > Error encountered performing task 'do' with profile(s): 'base,system,user,provided,dev,transit,reflection' Building outside docker seems OK. Any idea why / how to fix or troubleshoot? Is BABASHKA_FEATURE_YAML actually required?

This seems an error related to the feature flag. It will only try to load the yaml class if BABASHKA_FEATURE_YAML=true, or at least, it should.

I should mentioned I changed Dockerfile, setting all BABASHKA_FEATURE_* to false

@holyjak You can use BABASHKA_LEAN to accomplish the same thing. But you should set the flag for both the uberjar and compilation step

OK, thanks! BTW I have noticed https://github.com/borkdude/babashka/blob/master/script/compile#L73 includes most features but datascript and lanterna., Is that intentional or a mistake?

lanterna and datascript aren't enabled by default, so they are already set to false

Btw, the Github action JVM tests aren't working anymore, while everything on CircleCI works flawlessly. My experience with Github actions so far has been that it can be pretty flakey... If someone knows a solution, I'm open to suggestions.

But the same is true for JDBC and postgres yet they are on the list 🙂

That's probably because postgres and JDBC were part of babashka during one release, but I removed it in the next release

What if I want to build bb as lean as possible but with some selected features (namely csv, jdbc, oracle)? Then I cannot use LEAN. I could set all the other default features to false but then I will run into the problem ☝️ that yaml somehow is required anyway... But it is no big deal, I suppose they do not add that much to the size anyway.

@holyjak That is a bug which I cannot reproduce locally

We could make LEAN take into account variables that are set explicitly

I'm open for suggestions and PRs in this area

I'm also considering moving to the clojure CLI for compilation at one point, but that takes some refactoring to aliases for features

I added

ENV BABASHKA_FEATURE_CORE_ASYNC=false
ENV BABASHKA_FEATURE_CSV=false
ENV BABASHKA_FEATURE_JAVA_NIO=false
ENV BABASHKA_FEATURE_JAVA_TIME=false
ENV BABAHSKA_FEATURE_TRANSIT=false
ENV BABASHKA_FEATURE_XML=false
ENV BABASHKA_FEATURE_YAML=false
ENV BABASHKA_FEATURE_HTTPKIT_CLIENT=false
ENV BABASHKA_FEATURE_HTTPKIT_SERVER=false
ENV BABASHKA_FEATURE_JDBC=false
ENV BABASHKA_FEATURE_POSTGRESQL=false
ENV BABASHKA_FEATURE_HSQLDB=false
ENV BABASHKA_FEATURE_ORACLEDB=false
ENV BABASHKA_FEATURE_DATASCRIPT=false
ENV BABASHKA_FEATURE_LANTERNA=false

let me try that locally

export BABASHKA_FEATURE_CORE_ASYNC=false
export BABASHKA_FEATURE_CSV=false
export BABASHKA_FEATURE_JAVA_NIO=false
export BABASHKA_FEATURE_JAVA_TIME=false
export BABAHSKA_FEATURE_TRANSIT=false
export BABASHKA_FEATURE_XML=false
export BABASHKA_FEATURE_YAML=false
export BABASHKA_FEATURE_HTTPKIT_CLIENT=false
export BABASHKA_FEATURE_HTTPKIT_SERVER=false
export BABASHKA_FEATURE_JDBC=false
export BABASHKA_FEATURE_POSTGRESQL=false
export BABASHKA_FEATURE_HSQLDB=false
export BABASHKA_FEATURE_ORACLEDB=false
export BABASHKA_FEATURE_DATASCRIPT=false
export BABASHKA_FEATURE_LANTERNA=false

script/uberjar && script/compile

Can you maybe print the value of BABASHKA_FEATURE_YAML in the script

to debug

ok, nevermind then. Thanks for trying! I will get back if it turns out to be a problem again.

😎 https://twitter.com/borkdude/status/1326858985094471681

Any tips for troubleshooting when compile inside docker fails with

...
[bb:58]     (clinit):  20,613.49 ms,  2.03 GB
[bb:58]   (typeflow): 223,789.23 ms,  2.03 GB
[bb:58]    (objects): 936,796.65 ms,  2.03 GB
[bb:58]   (features):  21,151.56 ms,  2.03 GB
[bb:58]     analysis: 1,262,818.93 ms,  2.03 GB
[bb:58]     universe:  71,729.57 ms,  2.03 GB
Error: Image build request failed with exit status 137
com.oracle.svm.driver.NativeImage$NativeImageError: Image build request failed with exit status 137
	at com.oracle.svm.driver.NativeImage.showError(NativeImage.java:1558)
	at com.oracle.svm.driver.NativeImage.build(NativeImage.java:1308)
	at com.oracle.svm.driver.NativeImage.performBuild(NativeImage.java:1269)
	at com.oracle.svm.driver.NativeImage.main(NativeImage.java:1228)
	at com.oracle.svm.driver.NativeImage$JDK9Plus.main(NativeImage.java:1740)
The command '/bin/sh -c ./script/compile' returned a non-zero code: 137

This is an out of memory error. Bump BABASHKA_XMX by setting it to -J-Xmx8g or whatever amount you have available. Could also be a Docker setting

thanks a lot!!!

I have now run with --build-arg BABASHKA_XMX=-J-Xmx8g and still getting the error 😢

@holyjak Could it be that your docker doesn't have enough memory?

What os are you on?

Could be. OSXm 16GB mem in total.

Go to preferences and then resources

in your Docker Desktop

According to Activity Monitor Docker uses under 4GB so it would seem to be the problem

Note that you should not set BABASHKA_XMX higher than the total available memory minus some overhead, else you will get the same error

so if you set Docker memory to let's say 5GB, set BABASHKA_XMX to 4500m max

thanks a lot!

I have made some progress with -Xmx8575549440 : > Exception in thread "native-image pid watcher" java.lang.OutOfMemoryError: GC overhead limit exceeded 😢 I suspect that too much docker memory was on swap and thus too slow.

To how much have you set your docker memory

10GB

How much is 8 575.54944. Isn't that 8 megabytes?

Maybe you should supply a unit

Also, how are you providing BABASHKA_XMX? Can you post the entire Dockerfile setup?

You can also peek at the babashka docker build on circleci to get some ideas

this was printed in the stdout, I did run it with 8g

aha

Can you try to compile without the Oracle driver, to see if that is the culprit?

I was running docker build --build-arg BABASHKA_XMX=-J-Xmx3g and in the Dockerfile I had ARG BABASHKA_XMX="-J-Xmx8g" I will wait until the osx build of bb finishes, which takes forever, then try again with Docker max 5g and build max 4500m, making sure Docker has enough memory available not to be swapped out

3g is too little

Somewhere between 4 and 5 gb is probably the minimal requirement

Any luck?

@borkdude I have managed to replicate the missing YAML error. When you run uberjar from https://github.com/borkdude/babashka/pull/641 with

BABASHKA_FEATURE_CSV=true
BABASHKA_FEATURE_JDBC=true
BABASHKA_FEATURE_ORACLEDB=true
BABASHKA_LEAN=true
BABASHKA_FEATURE_YAML=true

BTW the loooong compile times and OOM errors seem to have been caused by me adding a few more functions and namespaces from next.jdbc.

@holyjak Is 641 good for merge?

It would be worth checking into which function(s) cause this. Sometimes it's related to a run-time resolve. I have forked my share of code to make it well suited for GraalVM

Thanks for looking into it. I was just seeing the PR

yes

yes, I plan to try to find out which function(s) cause this

fwiw, the babashka pods compile in minutes on CircleCI. I'm not sure if those have an up to date next.jdbc version

I compared the next.jdbc version in the pods to babashka, it's the same

I upgraded it from.. 10 to... 13 AND added more stuff from it. Running experiments...

The functions in the babashka-sql-pods should work fine, I have tested those with GraalVM. So maybe you could start with those, to confirm you get the same results. See the binary build here: https://app.circleci.com/pipelines/github/babashka/babashka-sql-pods/83/workflows/c154eb40-ad4f-4c05-8c8f-07be42b4895e/jobs/251 A little over 1 minute.

Keep me posted :)

FYI building latest master with only ..JDBC=true takes 451s, JDBC+Oracle takes 581s, with no extra features 421s, LEAN fails with ClassNotFoundException YAMLException

This is in Docker right?

no, on my 4y old maskin