babashka

https://github.com/babashka/babashka. Also see #sci, #nbb and #babashka-circleci-builds .
steveb8n 2021-01-22T05:13:17.011100Z

cheers for the tip!

Marco Pas 2021-01-22T07:39:53.013400Z

I am a bit lost on how one should write and execute unit tests for bb scripts. Checked the bb book but could not find any pointers on how to start doing this. Quick remark that this question is coming from a clojure newbie 🙂 Is there something like $ bb run-tests ?

Marco Pas 2021-01-22T07:42:04.014Z

Would be more then happy to write some howto that maybe can be added to the book if this is usefull by the way 🙂

borkdude 2021-01-22T07:51:10.014700Z

Alternatively you can use a hard coded path in load pod

borkdude 2021-01-22T07:51:27.015300Z

And download the pod yourself in Dockerfile

borkdude 2021-01-22T07:52:57.017100Z

@marco.pasopas go to the book and Ctrl-f on clojure.test. It is in there somewhere

borkdude 2021-01-22T08:31:57.017500Z

@marco.pasopas https://book.babashka.org/#_running_tests

Marco Pas 2021-01-22T08:33:55.018100Z

Found the recipe indeed trying to understand what it is doing 🙂

Marco Pas 2021-01-22T08:37:00.018600Z

Fighting name-spaces and folders 🙂

Marco Pas 2021-01-22T08:37:03.018800Z

But i will get there 🙂

borkdude 2021-01-22T08:56:06.019300Z

@marco.pasopas I now updated that part of the book, hopefully it's clearer

borkdude 2021-01-22T10:32:05.020700Z

pod-babashka-aws 0.0.5 released: fixes an issue with nil in the aws-api response which prevented using it with the lambda API. Now it works.

Adrian Smith 2021-01-22T10:32:38.021Z

Will espresso for Graalvm allow Babashka to run more Clojure programs?

borkdude 2021-01-22T10:33:16.021200Z

Maybe, possibly, I don't know yet :)

👍 1
Marco Pas 2021-01-22T12:08:37.024500Z

@borkdude Thanks this surely helps.. But i am not there yet 🙂 Fighting the concepts of namespaces and filename is guess.. Having the following simple test project..

.
├── run-test.sh
└── test
    └── util_test.clj
# util_test.clj
(ns util-test)

(deftest testme
  (is (= true)))

(deftest testme2
  (is (= 8 (+ 1 2))))
# run-test.sh
#!/usr/bin/env bash
bb -cp "src:test:resources" \§
   -e "(require '[clojure.test :as t] '[util])
       (let [{:keys [:fail :error]} (t/run-tests 'util)]
         (System/exit (+ fail error)))
Still it shows up with running 0 tests running

borkdude 2021-01-22T12:09:30.024900Z

@marco.pasopas '[util] should be 'util-test

borkdude 2021-01-22T12:10:05.025200Z

also in run-tests

borkdude 2021-01-22T12:10:47.025700Z

that invocation should throw with "Could not find namespace util" or something similar

borkdude 2021-01-22T12:12:05.026Z

The name of your namespace is not util, it is util-test

👍 1
Marco Pas 2021-01-22T12:16:05.026400Z

Guess i need to include something more..

borkdude 2021-01-22T12:16:34.026900Z

:refer [deftest]

borkdude 2021-01-22T12:16:42.027100Z

The example has (t/deftest ...)

borkdude 2021-01-22T12:17:42.027800Z

I mean, in util you need to require clojure.test as well and :refer [deftest] if you want to use it unqualified

👍 1
borkdude 2021-01-22T12:21:59.028300Z

(ns util (:require [clojure.test :as t :refer [deftest is]]))

Jakub Holý 2021-01-22T13:39:55.029900Z

Hello! I have read https://github.com/babashka/pods but did not find an explanation of how bb finds the pod. Does it use https://github.com/babashka/pod-registry to find the source gh repo and then downloads whatever is needed? And what if I want to package the pod with my app (aws lambda, actually)? Thanks!

borkdude 2021-01-22T13:41:31.031500Z

@holyjak You can load pods using a qualified symbol, like org.babashka/aws and a version "0.0.5". In that case it will download it from the pod-registry by looking up a manifest. When calling load-pod with a string or vector of strings, the pod is looked up on the local file system (either using the PATH, or using an absolute path)

borkdude 2021-01-22T13:42:02.032100Z

If you want to take pods along into a docker image, probably the best is to download them manually inside the docker image and use an absolute or relative path

borkdude 2021-01-22T13:42:40.032600Z

But you can also just copy the ~/.babashka directory where the downloaded pods are living, that should also work

borkdude 2021-01-22T13:43:36.033300Z

@holyjak I noticed you were using clj-http-lite from source, this isn't necessary anymore, now that there is babashka.curl and org.httpkit as built-in clients

👍 1
borkdude 2021-01-22T13:43:50.033600Z

But it still works

Jakub Holý 2021-01-22T14:06:43.033900Z

Thanks a lot!

Jakub Holý 2021-01-22T14:13:15.034500Z

@borkdude I guess it would be nice to update the "About" of https://github.com/babashka/babashka to point to https://book.babashka.org/ 🙏

borkdude 2021-01-22T14:14:05.034900Z

done

❤️ 1
borkdude 2021-01-22T14:14:24.035200Z

We will perhaps have a nice landing page on http://babashka.org sometime soon.

Jakub Holý 2021-01-22T15:14:17.036200Z

FYI I will be looking into adding Oracle support to the sql pod

borkdude 2021-01-22T15:15:42.036400Z

nice!

Jakub Holý 2021-01-22T15:19:18.037100Z

@borkdude I have just noticed https://github.com/babashka/babashka/pull/642 is still open. What would you like me to do with it?

borkdude 2021-01-22T15:25:13.037700Z

Up to you, you can close it or finish it. Beware of Oracle Licensing when distributing pods

borkdude 2021-01-22T15:26:42.038200Z

@holyjak When adding stuff, I would recommend to take very small steps, so we can monitor the growth of the binary

👍 1
borkdude 2021-01-22T15:27:13.038800Z

Some things trigger excessive growth (usually runtime resolve or require), so taking it one var and test at a time is probably the best

Jakub Holý 2021-01-22T15:28:03.039Z

What is missing for it to be finished? I believe the licensing is nowadays ok, i.e. you can re-distribute, but I will check it again and link to some official docs.

borkdude 2021-01-22T15:30:25.039300Z

I would prefer to add these vars to the pod first, so we can write real tests to verify if they work correctly and to see what impact they have on the size. See other message in channel

👍 1
borkdude 2021-01-22T15:30:45.039500Z

If they are verified to work in the pod, they can be added to the feature flag as well

Jakub Holý 2021-01-22T15:31:50.039800Z

ok!

Jakub Holý 2021-01-22T15:32:21.040Z

I will first work on adding oracle to the pod then come back to extend the next-jdbc support

borkdude 2021-01-22T15:33:05.040200Z

good plan!

Jakub Holý 2021-01-22T16:39:04.041500Z

@borkdude here is my first attempt https://github.com/babashka/babashka-sql-pods/pull/15 However it fails to compile, with > Error: Incompatible change of initialization policy for java.lang.Math$RandomNumberGeneratorHolder: trying to change RUN_TIME from the command line to RERUN for random number generator > com.oracle.svm.core.util.UserError$UserException: Incompatible change of initialization policy for java.lang.Math$RandomNumberGeneratorHolder: trying to change RUN_TIME from the command line to RERUN for random number generator Any tips how to trouubleshoot & fix that? 🙏

borkdude 2021-01-22T16:40:40.042Z

@holyjak Yes, with GraalVM 20.3.0 this line should be removed

borkdude 2021-01-22T16:40:53.042300Z

Feel free to also update GraalVM in circleci / appveyor to 20.3.0

Jakub Holý 2021-01-22T16:42:04.043100Z

I have 20.3:

🐟  $GRAALVM_HOME/bin/native-image --version
GraalVM Version 20.3.0 (Java Version 11.0.9+10-jvmci-20.3-b06)
yet I still am getting this error 😭

borkdude 2021-01-22T16:42:26.043400Z

> Yes, with GraalVM 20.3.0 this line should be removed I mean, you should actually delete that line

Jakub Holý 2021-01-22T16:42:50.043800Z

ah, this one

--initialize-at-run-time=java.lang.Math\$RandomNumberGeneratorHolder \
in compile ?

borkdude 2021-01-22T16:43:16.044400Z

yes

borkdude 2021-01-22T16:43:21.044700Z

and compile.bat

Jakub Holý 2021-01-22T16:43:43.045200Z

I guess I should also update generate_circleci.clj to use 20.3.0?

borkdude 2021-01-22T16:43:49.045400Z

yes

borkdude 2021-01-22T16:43:53.045600Z

and appveyor.yaml

borkdude 2021-01-22T16:43:56.045900Z

🙏

borkdude 2021-01-22T16:44:08.046100Z

you can do it as a separate PR

👍 1
Jakub Holý 2021-01-22T16:44:11.046200Z

I don't see this file

borkdude 2021-01-22T16:44:31.046500Z

oh sorry, this is not part of this repo. there is no Windows version yet, ok

borkdude 2021-01-22T16:45:06.047Z

@holyjak let's take this to the dev channel, I just invited you there

👍 1
borkdude 2021-01-22T19:02:24.047700Z

Welcome @tgg

😄 1
👋 1
seancorfield 2021-01-22T22:05:47.049Z

Does babashka work with next.jdbc? I thought it did but I just got an issue opened that next.jdbc doesn't compile with GraalVM: https://github.com/seancorfield/next-jdbc/issues/156

borkdude 2021-01-22T22:11:27.050200Z

@seancorfield We have a feature flag for next.jdbc and also some sql "pods" which are optional binaries which bb can download to get more features. These are all compiled with GraalVM native-image, but probably not with the most recent versions of next.jdbc.

borkdude 2021-01-22T22:13:07.050900Z

@seancorfield The issue isn't very descriptive of what goes wrong, but we can update our next.jdbc to see if it still works, in the sql pods

borkdude 2021-01-22T22:15:11.051200Z

we were using "1.1.610", now trying upgrade to 613

Jakub Holý 2021-01-23T12:01:06.064300Z

Also notice that the sql pod exposes only a subset of next-jdbc. My experience was that when I tried to add the .prepare ns, the compile time and CPU use exploded until my PC could not take it anymore. So it is possible that not all parts of next-jdbc can work in GraalVM (yet)

seancorfield 2021-01-23T22:39:54.089100Z

I'm happy to review PRs that folks send in about GraalVM compatibility but it's very low priority for me since I have zero interest in native-compiled Clojure right now.

borkdude 2021-01-23T22:41:29.089300Z

@seancorfield The most important thing for GraalVM users is avoiding reflection warnings, so as long as you have (set! *warn-on-reflection* true) everywhere, and you can avoid using require, resolve and eval (or variations) during runtime (top-level is ok), this will cover 99% of the problems

seancorfield 2021-01-23T22:44:05.089500Z

I think I have (set! *warn-on-reflection* true) everywhere... Ah, I don't have it in next.jdbc.default-options or next.jdbc.plan. Will add it there (and fix any reflection warnings).

seancorfield 2021-01-23T22:52:50.089700Z

Added. No reflection warnings. A couple of places have requiring-resolve (for whether camel-snake-kebab is on the classpath or not). There's one place in next.jdbc.prepare that refers to a symbol from next.jdbc.result-set (which already requires next.jdbc.prepare) so that's probably what @holyjak ran into...

seancorfield 2021-01-23T22:54:23.089900Z

The CSK stuff is all handled in macros so if you do not have CSK on your classpath, there's no dynamic require in the runtime (but it does mean you can't create a GraalVM native image if you try to use CSK).

seancorfield 2021-01-23T22:57:47.090100Z

That prepare one is nasty since it's trying to avoid a circular dependency. I'll open an issue for that.

borkdude 2021-01-23T23:05:34.090300Z

Ah that makes a lot of sense, yes, these things can make native-images 30-50 mb larger and also analysis takes a lot longer. I sometimes work around circular dependencies by creating a volatile in another namespace and vreset-ing the defined function to that volatile, so I can use it from other places. A bit hacky, but it works

seancorfield 2021-01-23T23:23:12.090500Z

Interesting workaround. In my case, prepare/execute-batch! is really in the wrong place. It should have been in result-set instead. But I could use your volatile approach to fix it -- although there might still be paths through people's code to next.jdbc.prepare that don't go through next.jdbc or next.jdbc.result-set which would be the two obvious candidates to include that vreset...

borkdude 2021-01-22T22:29:47.051300Z

@seancorfield Just upgraded to 613 and everything works correctly: https://github.com/babashka/babashka-sql-pods

seancorfield 2021-01-22T23:10:12.051700Z

Thanks!

seancorfield 2021-01-22T23:13:51.051900Z

I really appreciate you providing all that information in the GH issue!