Clojurians Log v2

@jeroenvandijk Found a way to automatically update the path after install:

function glam-install {
    export PATH="$(clojure -M:glam --install $@):$PATH"
}

(and renamed the repo, since it conflicted with the C++ package manager: https://github.com/borkdude/glam)

@borkdude nice!

Glam is also a nice find 🙂

Hello @borkdude, I wrote some small tools with Babashka an really like it. So thanks for the work you put into. I currently create a more complex project and would like to add some tests. But putting the script into a Clojure folder structure brings up errors when running it. Also I don’t know how to run tests, especially with pods. Do you have an example where I can look at?

@danielgrosse When putting it in a Clojure folder structure you will need to add src and test to the classpath: -cp src:test

@danielgrosse Here is a snippet that shows how you can run tests: https://github.com/borkdude/babashka#running-tests

And thanks.

I’m trying to write a Pod in babashka using pod-babashka-hsqldb as an example (https://gist.github.com/jeroenvandijk/55dc9005035bc506493ca279d43b2236). Something goes wrong in the communication while loading the pod, but I don’t see it how it is different from the example. This is the error

bb <http://pod-usage.bb|pod-usage.bb>
{format edn, namespaces [{name (quote calculate), vars [{name (quote +)}]}], opts {shutdown {}}}
[message {id #object[[B 0x167d0535 [B@10c7ae618], op #object[[B 0x10cb3ebd [B@10c7aec00]}]
{:id2 ab473001-1b2b-482f-abd8-c9b2a7f3a619, :op2 :describe}
----- Error --------------------------------------------------------------------
Type:     java.lang.ClassCastException
Message:  clojure.lang.PersistentVector cannot be cast to byte[]
Location: /Users/jeroen/Projects/Github/bb-pod/pod-usage.bb:3:1

----- Context ------------------------------------------------------------------
1: (require '[babashka.pods :as pods])
2:
3: (pods/load-pod ["bb" "<http://pod.bb|pod.bb>"])
   ^--- clojure.lang.PersistentVector cannot be cast to byte[]
4:
5: (println "pod loaded")
6:
7: (require '[calculate])
8:

----- Stack trace --------------------------------------------------------------
user - /Users/jeroen/Projects/Github/bb-pod/pod-usage.bb:3:1

:EOF

Not sure, I've got a meeting coming up, so I'll answer later today.

o wait i see it i think nope

Thanks

@jeroenvandijk it often helps to write to stderr in the pod in case of an error

Ok it was a quoting issue in describe map , now it works 🙂

😎

I’ve updated the gist. Will make something more interesting later I hope 🙂

I’ve released v0.0.2 of https://github.com/tzzh/pod-tzzh-aws with improved pagination, better code generation and more services (happy to include more if anybody needs)

:thumbsup:

@thomas.ormezzano btw, I checked:

borkdude@DESKTOP-JN2UNTV ~ $ /mnt/c/Windows/System32/WindowsPowerShell/v1.0/powershell.exe

Try the new cross-platform PowerShell <https://aka.ms/pscore6>

PS Microsoft.PowerShell.Core\FileSystem::\\wsl$\Ubuntu-20.04\home\borkdude&gt; clojure
Clojure 1.10.1
user=&gt; (System/getProperty "os.arch")
"amd64"
;; Linux:
borkdude@DESKTOP-JN2UNTV ~ $ clojure
Clojure 1.10.1
user=&gt; (System/getProperty "os.arch")
"amd64"
                                                                                                                                                                                                         ;; MacBook

$ clj
Clojure 1.10.1
user=&gt; (System/getProperty "os.arch")
"x86_64"

so on macOS or maybe depending on which Java version you are on, os.arch responds with x86_64 or amd64 hmm

ah ok they are the same though right ? I just copied what had been auto generated in the binaries I created tbh

yeah, maybe the package manager should just deal with that

I'll fix that, thanks

borkdude@MBP2019 ~ $ glam install tzzh/pod-tzzh-aws -g
borkdude@MBP2019 ~ $ pod-tzzh-aws

nice 🙂

Here is a proof of concept for loading arbitrary Clojure libraries via Pods https://github.com/jeroenvandijk/pod-babashka-clj/blob/master/examples/aws.bb.clj

It needs some more work around keeping track of the started jvm processes, but the basics work

why not directly go with Clojure in this case, would be the obvious question?

True, i’m thinking you could have a process running in the background that can deal with the things that Babashka can’t. So if you use AWS a lot it only needs to start once

yeah. can also use the nREPL client for that

yeah maybe that would be an alternative. This is using the prepl so same idea i guess

cool idea

it’s a combination of pods and this idea https://github.com/jeroenvandijk/clojure-scripting

:thumbsup:

Writing a pod in babashka itself was easier than i had expected

Have you already tested the tzzh aws pod, since you are interested in AWS stuff?

You can now install it using glam :)

No I haven’t. My main concern is not having to learn new libraries 😅 I think the tzzh aws pod is inventing a new API

i’ll check it out now before i say something stupid

I think a similar approach could work for Clojure. Just generate all the code at compile time instead of using reflection/dynamic requires

(at least, that's how I think the cognitect lib works?)

yeah I agree. pretty sure the aws-api can be made compatible with Graalvm

it just requires some work

the tzzh pod looks quite easy to use though.

although I don't know the Java AWS API, I can't imagine why they would re-invent a different API in Go or Python

it's all going to the same endpoints eventually I guess

no i mean the clojure api is not the same

the backend is definitely the same

so I mean the api of pod-tzzh-aws is not the same as cognitect aws-api. So i would have to learn both

cannot reuse my code

And personally I don’t want to debug go code if things go wrong

But that’s just me. it’s nice that pod-tzz-aws exists

fair

I think the idea to use a small fast starting script to invoke something in a larger stateful context is similar to Datomic Ions with AWS Lambda + Datomic running on other machines

so you could have one Clojure REPL running on an EC2 instance permanently

and babashka in AWS lambda sending commands to its REPL

yeah i think you are right, pretty similar

It could also be useful if stateful means loading a really big library that would otherwise impact startup time

I’m not sure if the prepl should be used for concurrent commands, but the concept could be implemented via other protocols as well

nREPL supports this well via ids

sounds like its more suited for something like this then

Small example: https://github.com/borkdude/babashka#communicating-with-an-nrepl-server

A more full example can be seen in the tests of babashka.nrepl

The babashka pod protocol is more or less the same as nREPL, with a few twists

cool, i haven’t really dived into that protocol yet. Not sure what the tradeoffs are compared with the prepl protocol

is this a way to generically use any clojure/java library?

(I am looking for a quick way to run the Mongo or the RabbitMQ clients in babashka 😄)

Oh I see what you did there, kinda cool good job!

Yeah the idea is that anything that tools.deps can load, can be accessed via this pod

Currently there is no proper management of the backend process and the port is hardcoded right now, but this can be fixed

And maybe some error scenarios are hard to debug because of the limited communication between the pod and the babashka process right now. Probably needs another layer to get this right

seems like this could be a good candidate for being embedded in babashka.. pods are nice for interop with other langs but if you already have a clojure libs it can be easier

Yeah maybe. Pods give us a nice way to experiment with this idea for now at least. If you want to try with Mongo and RabbitMQ I can help out if you run into problems

doesn't mongo support an http api btw?

what would be equivalent in babashka to a common pattern $ cat file.txt | sort | uniq -c ?

one version:

bb '(-> (slurp "file.txt") (str/split #"\n") sort distinct)'

you are too quick 🙂 I can only add clojure.string/split-lines over str/split

ah, good one

I like to let bb to the input, so I would prefer this:

cat file.txt | bb -i '(-> *input* sort distinct)'

seems a bit cleaner

I think he needs frequencies , no?

yep, that uniq -c is the tricky peace

ah, true, replace distinct with frequencies

cat file.txt | bb -i -O '(-> *input* sort frequencies)'

to make the number show up first:

cat file.txt | bb -i -O '(-> *input* sort frequencies (->> (map reverse)))'

absolute closest to the original:

cat file.txt | bb -i -o '(-> *input* sort frequencies (->> (map reverse) (map (partial str/join " "))))'

ok, I'll stop now, bb is just too fun

i'm getting an error trying to use drewr/postal with bb . is this something that should work or is it not supported? if the latter is there another way to send an email? i don't need anything fancy, it's just for error notifications. smtp or sendmail would be fine

----- Error --------------------------------------------------------------------
Type:     java.lang.Exception
Message:  Unable to resolve classname: java.text.SimpleDateFormat
Location: postal/date.clj:24:1

----- Context ------------------------------------------------------------------
20: ;; WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
21: ;; FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
22: ;; OTHER DEALINGS IN THE SOFTWARE.
23:
24: (ns postal.date
    ^--- Unable to resolve classname: java.text.SimpleDateFormat
25:   (:import [java.util Date]
26:            [java.text SimpleDateFormat ParsePosition]))
27:
28: (defn make-date
29:   ([tmpl s]

I think it does

actually no, https://stackoverflow.com/questions/7386740/does-mongodb-have-a-native-rest-interface

Nicer formatting:

cat README.md | bb -io '(->> *input* sort frequencies (map (fn [[l n]] (format "%3s %s" n l))))'

@lucian303 Not yet supported.

@lucian303 I did in fact had one branch with postal in it once: https://github.com/borkdude/babashka/issues/366, but I ran into a JDK 11 bug

gotcha. thanks. are there any alternative libraries that you know of that might work? @borkdude

Probably not out of the box I think. The only alternative now is probably shelling out to sendmail or alike or using an HTTP mail service like sendgrid

gotcha. thanks. i'm probably just going to shell out then

Come to think of it, I think this can be perfectly implemented as a mailer pod

since an email is just plain data

e.g. drewr/postal could be compiled with GraalVM java 8 and then be used from babashka

it's a bit of work, but not too hard since there's already plenty of examples around

more info: https://github.com/babashka/babashka.pods https://github.com/borkdude/babashka/blob/master/doc/libraries.md#pods

thank you! i've added couple of things and end up with

tshark -r /var/tmp/DNS.pcap00.gz -T fields -e dns.qry.name dns.qry.type==1 | bb -io '(->> *input* sort frequencies clojure.set/map-invert (into (sorted-map)) (map (fn [[k v]] (format "%6s %s" k v))))'

:thumbsup: clojure power!

The fix for postal in java 9+ is super easy https://github.com/drewr/postal/issues/107

Ah nice. We can re-consider adding drewr/postal using that approach or implementing it as a pod first (depending on the extra size, how many people need this, etc)

Made an issue here: https://github.com/borkdude/babashka/issues/599 Feel free to leave feedback

btw, this is another lib in the SMPT space: https://github.com/toyokumo/tarayo

wow

be careful with map-invert, it'll collapse ties

nice. i'll try tarayo as i'm a bit short on time

(clojure.set/map-invert {:foo 1 :bar 1}) -> {1 :bar}

I don't think that will work with bb out of the box though

bb doesn't have the javax.mail.Transport etc classes currently

yeah, it won't