babashka

https://github.com/babashka/babashka. Also see #sci, #nbb and #babashka-circleci-builds .
grzm 2021-05-18T00:58:35.002900Z

I’m working with the pod.babashka.aws pod and it’s very noisy with logging statements (via clojure.tools.logging). Is there a way to silence these logs? If this were straight up clojure, I’d do something along the lines of -Dclojure.tools.logging.factory=clojure.tools.logging.impl/disabled-logger-factory. I see I can set Java system properties that are read by babashka using -D, but I suspect they’re not being passed to the pod? The more likely case is I’m just Doing it Wrong™.

borkdude 2021-05-18T08:17:08.003800Z

@grzm There is a workaround for this. You can start the pod using the filename instead of the qualified symbol and then pass "-Dclojure.tools.logging.factory=clojure.tools.logging.impl/disabled-logger-factory" as an argument to it

borkdude 2021-05-18T08:17:45.004200Z

I think we could support this better by allowing to pass :command-line-args or so via the options

borkdude 2021-05-18T08:53:21.005100Z

I'm not sure if this even works, since the command line property may already have been read at compile time (top level) so it would be good to try this out first.

2021-05-18T12:33:07.006200Z

correct me if this thing already exists, but I think it would be nice to have somewhere in the docs a link to a comparison/translation between "standard" bash scripts and the equivalent Babashka scripts. Would be maybe a good way to onboard people who know bash but don't really know clojure much, if they can see how to do things they already know

👍 2
1
borkdude 2021-05-18T13:01:19.007Z

There is currently an experimental setup possible for getting clojure-lsp support in babashka scripts: https://github.com/babashka/babashka/issues/733#issue-811939787 Feel free to give it a try and suggest improvements.

🎉 3
borkdude 2021-05-18T13:05:27.007200Z

@ericdallo I wonder how clojure-lsp behaves if you have both a deps.edn and a bb.edn in the same project, which is pretty common

ericdallo 2021-05-18T13:06:19.007500Z

it will the one specified from user I think

borkdude 2021-05-18T13:06:41.007700Z

but what if you want lsp support in both src/myclojure.clj and script/mybb.clj ?

ericdallo 2021-05-18T13:08:28.007900Z

Oh, it has the capability to support both but I found a bug where it uses only the specified by users instead of merging with the default

borkdude 2021-05-18T13:08:43.008100Z

which might make sense in some cases

ericdallo 2021-05-18T13:08:45.008300Z

so user would need to have two project-specs until this bug is fixed

borkdude 2021-05-18T13:08:58.008500Z

ok

ericdallo 2021-05-18T13:09:20.008700Z

ATM it is (or (get settings :project-specs) default-project-specs)

borkdude 2021-05-18T13:09:42.008900Z

I think that makes sense, it gives you the power to override lsp

borkdude 2021-05-18T13:10:11.009100Z

but now lsp will use the bb classpath for clojure sources too?

borkdude 2021-05-18T13:10:34.009400Z

which may not be a big issue

ericdallo 2021-05-18T13:10:52.009600Z

no, it will use only if finds the path configured by project-path

ericdallo 2021-05-18T13:10:59.009800Z

in this case bb.edn

borkdude 2021-05-18T13:11:31.010Z

yeah ok, but if I added one for deps.edn as well

borkdude 2021-05-18T13:11:45.010200Z

then the classpaths would be concatenated?

ericdallo 2021-05-18T13:11:55.010400Z

yes

borkdude 2021-05-18T13:12:20.010600Z

that's probably fine, unless people use different versions of libraries in their clojure projects than what babashka uses

ericdallo 2021-05-18T13:12:24.010800Z

it will use the same classpath for both script and src folder if that's what you are asking

borkdude 2021-05-18T13:12:37.011Z

and this is why the script may be a better solution since it gives you the flexibility to change something

ericdallo 2021-05-18T13:13:07.011200Z

Hum, I still think it's too much manual work but it's a good start 😅

ericdallo 2021-05-18T13:13:25.011400Z

what users could do that makes sense is have a project root different for the scripts

borkdude 2021-05-18T13:13:27.011600Z

> it will use the same classpath for both script and src folder if that's what you are asking yes, so it will use the babashka.fs library from bb itself to give LSP support in a Clojure project too, which may use an entirely different babashka.fs version

👍 1
ericdallo 2021-05-18T13:14:06.011900Z

users could have different project-roots on lsp-mode one for scripts and another for the src

ericdallo 2021-05-18T13:14:19.012100Z

and different .lsp/config.edn with different project-specs

ericdallo 2021-05-18T13:14:33.012300Z

but yeah, it looks too hard to configure IMO for the end user

borkdude 2021-05-18T13:17:56.012500Z

probably in practice it should be fine

👍 1
borkdude 2021-05-18T13:18:07.012700Z

let's first see what users come across

borkdude 2021-05-18T13:35:22.013Z

Btw. babashka also has an nREPL server which gives auto-completion for built-in libraries. Can this be used by LSP somehow

ericdallo 2021-05-18T13:36:32.013200Z

No, LSP don't uses/connect to any repl

ericdallo 2021-05-18T13:36:54.013400Z

this is a frequent question that people think that LSP should use the repl like cider or something, but this is not part of the LSP spec

borkdude 2021-05-18T13:37:59.013600Z

that clojure-lsp uses clj-kondo behind the scenes is also not relevant to the LSP spec. similarly it could use an nREPL server to get extra info from

borkdude 2021-05-18T13:41:42.013900Z

but people might as well use CIDER with the bb nrepl-server then, which people already do

ericdallo 2021-05-18T13:42:29.014100Z

Yeah, it could be added but there are already libs that do that very well, not sure it worth, but it's and idea that could be explored

borkdude 2021-05-18T13:44:02.014300Z

@ericdallo Like: clojure-lsp could offer some hook where people can insert a script that fetches the necessary information for clojure-lsp to do something. e.g. it could be a babashka script that spits out some EDN. where it gets this EDN from is up to the user.

borkdude 2021-05-18T13:44:26.014500Z

One such script could just make an nREPL request and clojure-lsp can be completely agnostic about this

ericdallo 2021-05-18T13:45:19.014700Z

yeah, looks cool, but is not clear to me when clojure-lsp would do the request, what would do with the result

borkdude 2021-05-18T13:45:51.014900Z

think of it as an alternative way of getting clj-kondo analysis information, from some other source than clj-kondo

ericdallo 2021-05-18T13:46:07.015100Z

hum, I think cider does that right?

ericdallo 2021-05-18T13:46:32.015300Z

looks kind of huge parse the NREPL response and handle as analysis

2021-05-18T13:46:48.015500Z

not sure what's the best way to do that, maybe just a cookbook style, with two columns for each example and an explanation before/after the two implementations

borkdude 2021-05-18T13:46:59.015800Z

well, the script would just give you the processed data, the script would then be responsible for the nREPL stuff

borkdude 2021-05-18T13:47:08.016Z

and clojure-lsp can be agnostic about nREPL

borkdude 2021-05-18T13:47:27.016200Z

the script could provide it in the same format as clj-kondo analysis

borkdude 2021-05-18T13:48:43.016400Z

so instead of saying: give me a classpath to analyze, you could say: give me the analysis

ericdallo 2021-05-18T13:49:10.016600Z

I see, it could work, but it looks too work for user, isn't ? I don't see too many users using that feature

borkdude 2021-05-18T13:49:49.016800Z

the complexity would be contained in the script, which can be offered as a library. users use complex libraries all the time without understanding what it does internally

borkdude 2021-05-18T13:55:25.017100Z

just a thought, you're probably right that this isn't used much

ericdallo 2021-05-18T13:59:20.017300Z

Yeah, but it looks to me a nice extension for clojure-lsp, giving this optional power to users, thanks for the idea 🙂 Maybe in the future I could do some POC

grzm 2021-05-18T14:03:19.017500Z

Cheers! There might be some issues with the using the disabled-logger-factory this way in particular (see https://clojurians.slack.com/archives/C03S1KBA2/p1621346387005200), but the general idea likely still holds.

2021-05-18T14:13:41.017800Z

added it https://github.com/babashka/babashka/issues/852 to not forget

richiardiandrea 2021-05-18T18:02:06.019700Z

Hi there, noticed the following - I was expecting a map in output:

ls | bb -i -O '{:report true :paths (vec *input*)}'
[:report true]
[:paths ["bin" "README.md" "resources" "scripts" "setup" "src" "target" "test" "tests.edn"]]
Should I file a bug or it is the expected behavior?

Jakub Zika 2021-05-18T18:31:49.021800Z

Hi there, I am trying to uberjar my script clojure -e "(require 'postgryoshka.core) (compile 'postgryoshka.core)" but I am getting Could not locate babashka/fs__init.class, babashka/fs.clj or babashka/fs.cljc on classpath. Uberscript and running bb script works. What should I do to create *.jar with all the dependencies?

borkdude 2021-05-18T19:53:26.021900Z

This is a bug. There is a predicate that determines if multiple objects should be printed and currently this is coll?, but obviously that is wrong. Probably sequential is better

👍 1
borkdude 2021-05-18T19:53:35.022100Z

Issue welcome.

borkdude 2021-05-18T19:54:16.022400Z

To create an uberjar with dependencies for babashka itself, you should only have to do:

bb uberjar foo.jar

borkdude 2021-05-18T19:54:51.022600Z

I'm not sure why you are trying to do clojure -e etc

2021-05-18T21:14:19.023900Z

Hi, how to source some of the existing bash script file and run specific bash function from that file in babashka?

borkdude 2021-05-18T21:17:18.024300Z

@grzegorzrynkowski_clo babashka doesn't interpret bash, it is a Clojure scripting environment

borkdude 2021-05-18T21:17:57.025Z

if you want to run bash, use bash by shelling out to it using babashka.process (or if you are using tasks: shell)

borkdude 2021-05-18T21:19:30.025600Z

e.g. you can do (shell "bash -c 'source foo && foo_fn'")

2021-05-18T21:27:12.025700Z

makes sense, thanks a lot 🙇🙏

Jakub Zika 2021-05-18T21:30:04.026600Z

I used an example from babashks book ... To exclude these dependencies, you can use the following :classpath-overrides in your deps.edn...

Jakub Zika 2021-05-18T21:30:34.026900Z

`$ rm foo.jar $ bb -cp $(clojure -A:remove-clojure -Spath) uberjar foo.jar -m foo $ bb foo.jar :hello $ ls -lh foo.jar -rw-r--r-- 1 borkdude staff 871B Aug 19 17:07 foo.jar`

borkdude 2021-05-18T21:31:54.027200Z

This was written before bb.edn was there. You can now just put your deps inside bb.edn and run bb uberjar foo.jar

👍 1
borkdude 2021-05-18T21:32:16.027400Z

I will update the book with this info.

👍 1
richiardiandrea 2021-05-18T22:00:36.027700Z

I have fixed it quickly, sorry I might have broken the tests - will try to improve the PR later with more time at hand https://github.com/babashka/babashka/pull/853

kokada 2021-05-18T22:07:25.028600Z

I was trying babashka.curl on Windows (just for curiosity, not that I care) and got the following error:

user=&gt; (curl/get  "<http://www.google.com|www.google.com>")
clojure.lang.ExceptionInfo: curl: option --compressed: the installed libcurl version doesn't support this
curl: try 'curl --help' for more information
[at &lt;repl&gt;:1:1

borkdude 2021-05-18T22:08:21.029Z

hmmm

borkdude 2021-05-18T22:10:12.030800Z

I wonder if it's possible to probe curl if it supports this option, so we can cache that for the next call

kokada 2021-05-18T22:11:17.031500Z

This works, but it be good to detected it instead:

(curl/get "<http://www.google.com|www.google.com>" {:compressed false})

kokada 2021-05-18T22:12:53.033Z

> I wonder if it's possible to probe curl if it supports this option, so we can cache that for the next call Maybe with --version ?

curl.exe --version                               
curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL Release-Date: 2017-11-14, security patched: 2019-11-05
Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL  

borkdude 2021-05-18T22:13:37.033600Z

I get this on macOS:

curl 7.64.1 (x86_64-apple-darwin19.0) libcurl/7.64.1 (SecureTransport) LibreSSL/2.8.3 zlib/1.2.11 nghttp2/1.39.2
Release-Date: 2019-03-27
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS GSS-API HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL UnixSockets

kokada 2021-05-18T22:13:58.034Z

I think it is the libz feature. On NixOS:

curl --version
curl 7.74.0 (x86_64-pc-linux-gnu) libcurl/7.74.0 OpenSSL/1.1.1k zlib/1.2.11 libssh2/1.9.0 nghttp2/1.41.0
Release-Date: 2020-12-09
Protocols: dict file ftp ftps gopher http https imap imaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS GSS-API HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz NTLM NTLM_WB SPNEGO SSL TLS-SRP UnixSocke

borkdude 2021-05-18T22:16:45.034400Z

Another interesting thread: https://bugzilla.mozilla.org/show_bug.cgi?id=1605540

borkdude 2021-05-18T22:17:07.034900Z

ok, I'll make an issue for this tomorrow if you don't beat me to it :)

kokada 2021-05-18T22:22:04.036300Z

> Another interesting thread: https://bugzilla.mozilla.org/show_bug.cgi?id=1605540 So it seems it needs to check both for libz and brotli :thinking_face: > ok, I'll make an issue for this tomorrow if you don't beat me to it 🙂 I don't care too much about Windows support, but I can make an issue

👍 1
🙏 1
borkdude 2021-05-19T12:31:33.053500Z

I have made a check in a delay so it checks for this only once during the duration of a bb program, but it adds 10ms or so when doing the first request.. Perhaps the result could be cached in a file, but this leads to problems like: what if the system updates the curl binary.

borkdude 2021-05-19T12:32:38.053900Z

on linux systems this time is probably lower

borkdude 2021-05-19T12:33:22.054100Z

perhaps it's worth it though

borkdude 2021-05-19T12:33:32.054300Z

and usually requests take anywhere between 50-100 ms anyway

borkdude 2021-05-19T12:34:45.054500Z

perhaps catching the exception and then retry also a way to handle it

borkdude 2021-05-19T12:35:30.054700Z

or perhaps only applying this check on Windows

borkdude 2021-05-19T13:29:34.054900Z

ironically, I have Windows tests in place but apparently appveyor has a curl installed that does support this

borkdude 2021-05-19T13:43:14.055100Z

hmm, maybe I should just add some docs for Windows 10 that they should either upgrade curl or use :compressed false

borkdude 2021-05-19T13:43:30.055300Z

and not bother trying to work around some edge case in windows

borkdude 2021-05-19T13:45:26.055500Z

### Compression

From babashka 0.2.4 onwards, this library will call `curl` with `--compressed`
by default. To opt out, pass `:compressed false` in the options.  On Windows 10
the default installed curl does not support this option. You can either upgrade
curl or perform all requests using `:compressed false`.

kokada 2021-05-19T19:27:42.063600Z

Well, seems fine to me While the behavior right now is not really portable, at least it is consistent and the error is clear. Checking for features could lead to too much magic where people that expected e.g. that the compression worked would suddenly have a surprise when their scripts doesn't have compression on Windows

borkdude 2021-05-19T19:29:29.063800Z

Right. I am stalking the people at MSFT now. https://twitter.com/borkdude/status/1395090669098700800

littleli 2021-05-18T22:52:20.039900Z

@borkdude How complete is the included implementation of agents in bb? I can create new instance of an agent. I can send and send-off, but not send-via and if I put agent into failed state, I cannot recover with restart-agent . I cannot tell if it's a feature or a 🐛

borkdude 2021-05-19T07:57:13.045500Z

@ales.najmann These functions should be in babashka/impl/clojure/core.clj in core-extras, I think it's just a matter of adding them

littleli 2021-05-19T07:58:05.045700Z

Should I create an issue on GH?

littleli 2021-05-19T07:59:27.045900Z

My concern is to have full support for agents if possible.

littleli 2021-05-19T08:00:41.046100Z

or not having them at all, which would be a loss in my opinion, as they are great worker pattern.

borkdude 2021-05-19T08:03:29.046300Z

@ales.najmann I have created a branch here: https://github.com/babashka/babashka/tree/more-agents Please inspect the commit to see if I have caught them all

borkdude 2021-05-19T08:03:41.046500Z

and then you can go to appveyor to test it (if you run on Windows)

borkdude 2021-05-19T08:03:48.046700Z

or CircleCI for macOS and linux

borkdude 2021-05-19T08:03:54.046900Z

or get it from #babashka-circleci-builds

borkdude 2021-05-19T08:05:10.047100Z

I would also be interested in hearing about your pattern :)

littleli 2021-05-19T08:07:04.047300Z

I have no pattern 🙂 it's just that agents have nice properties.

borkdude 2021-05-19T08:07:24.047500Z

you said: great worker pattern

borkdude 2021-05-19T08:07:30.047700Z

so I was curious how you are using them

borkdude 2021-05-19T08:07:54.047900Z

Perhaps I could have used them to implement parallel tasks, I see await is a lot simpler than what I did with core.async ;)

littleli 2021-05-19T08:08:54.048100Z

to be honest, I was just learning agents and I was curious how to use them with bb... so sorry, I but don't have anything specific 😄

borkdude 2021-05-19T08:09:06.048300Z

no worries, I think it's good to add support for it

borkdude 2021-05-19T08:16:14.048500Z

the build is done. https://ci.appveyor.com/project/borkdude/babashka/build/artifacts

borkdude 2021-05-19T08:16:18.048700Z

please test :)

borkdude 2021-05-19T08:18:31.048900Z

hmm, I need to patch these functions for binding conveyance as well.

littleli 2021-05-19T08:18:36.049100Z

there are more functions to add...

await-for
agent-error
error-handler
error-mode
get-validator
shutdown-agents

add-watch # ?
remove-watch # ?

borkdude 2021-05-19T08:18:39.049300Z

I'll make an issue for that

borkdude 2021-05-19T08:19:44.049500Z

I did add agent-error

borkdude 2021-05-19T08:19:48.049700Z

what is handler-error?

borkdude 2021-05-19T08:20:01.049900Z

can you use the full var names?

littleli 2021-05-19T08:20:53.050200Z

it's error-handler... sorry

borkdude 2021-05-19T08:22:26.050500Z

add-watch and remove-watch are already there

👍 1
borkdude 2021-05-19T08:22:48.050800Z

Added these now:

+   'await-for (copy-core-var await-for)
+   'error-handler (copy-core-var error-handler)
+   'error-mode (copy-core-var error-mode)
+   'get-validator (copy-core-var get-validator)

littleli 2021-05-19T08:23:00.051Z

I listed functions from this page: https://clojure.org/reference/agents#_related_functions

borkdude 2021-05-19T08:23:44.051200Z

ok, I have them all now

borkdude 2021-05-19T08:24:16.051400Z

there is a small patch needed for binding conveyance. when you have a dynamic var binding, the started agent thread also needs to see it. but this should be doable

borkdude 2021-05-19T08:24:41.051600Z

for now you can test the build when it's ready

littleli 2021-05-19T08:25:41.051800Z

🙏 great

littleli 2021-05-19T08:28:08.052Z

there is also agent-errors, but is marked as deprecated in clojure, and I don't think it should be included.

borkdude 2021-05-19T08:28:35.052200Z

ok

borkdude 2021-05-19T09:38:18.052400Z

So the problem I'm fixing right now:

$ clojure -M:babashka/dev -e '(def ^:dynamic *foo* 1) (def a (agent nil)) (binding [*foo* 2] (send-off a (fn [_] *foo*))) @a'
1
This should be 2

borkdude 2021-05-19T09:38:48.052600Z

Luckily this is relatively easy

borkdude 2021-05-19T09:45:57.052800Z

Fix pushed

littleli 2021-05-19T10:16:24.053Z

Nice PR. I'm surprised it makes sense to my untrained eyes.

borkdude 2021-05-19T12:01:06.053200Z

ok, I merged it now