Clojurians Log v2

Hi. I have a requirement to periodically push data into kafka topic in batches. If its a one time run, i could have directly run it in with bb script only construct the data. But since i need to run on a periodic basis, i need to push to kafka through bb script. But i keep running into dependency problems with every clojure jar i use. The java import

(import '[org.apache.kafka.clients.producer KafkaProducer ProducerRecord])

is this intended behavior?

{:tasks {repro {:doc (str "This tasks repro is: " "showing a doc string")
                :task (shell "ls")}}}

❯ bb tasks
The following tasks are available:

repro (str "This tasks repro is: " "showing a doc string")

I'm trying to build a doc string from some default values, but it interprets the value of :doc literally

Currently the docstring isn't evaluated, except those on functions, if you refer to them with :task foo.bar/baz

I mean, then the docstring is picked from the function

I see

We could change that, this is just currently how it works

it's a bit difficult to decide how to evaluate those string args though

yeah, it would come in handy, but looking at most commandline tools that support 'actions', the overview doesn't give a lot of detail, and only does so when you would bb repro --help

because it may refer to things depending on previous tasks etc

ok, no worries. I can work with this, was just wondering if this was intended behavior

thanks for babashka tasks by the way, it's super awesome for writing dev tooling

I’m looking at this libary to do consistent hashing https://github.com/replikativ/hasch. The lib has a platform ns: (:import java.io.ByteArrayOutputStream java.nio.ByteBuffer java.security.MessageDigest) . Are these dependencies under consideration?

Or, any more bash-like idea for consistent hashing? I’m simply trying to cache which rows/objects/maps have been already processed in a data pipeline.

I like hasch b/c it’s very thorough.

I think these classes might already be in bb

digging into the code, the lib uses protocols as well: that’s the larger hurdle, right?

protocols are also supported in bb

oh! great

just try out the library, you'll likely run into something else ;)

indeed.

If I hunt in here: https://github.com/babashka/babashka/blob/master/src/babashka/impl/classes.clj and don’t e.g. ByteArrayOutputStream then it’s not in bb, yet. Correct?

ah, but it is there. https://github.com/babashka/babashka/blob/d90f4e7bacb0af902764f66be4ba0255528d795c/src/babashka/impl/classes.clj#L93 Well, thanks for your help anyway!

Let's talk in a thread.

borkdude@MBP2019 /tmp $ export BABASHKA_CLASSPATH=$(clojure -Spath -Sdeps '{:deps {io.replikativ/hasch {:mvn/version "0.3.7"}}}')
borkdude@MBP2019 /tmp $ bb -e "(use 'hasch.core)"
----- Error --------------------------------------------------------------------
Type:     clojure.lang.ExceptionInfo
Message:  Unable to resolve classname: java.io.FileInputStream
Location: hasch/platform.clj:162:24
Phase:    analysis

----- Context ------------------------------------------------------------------
158:   java.io.File
159:   (-coerce [f md-create-fn write-handlers]
160:     (let [^MessageDigest md (md-create-fn)
161:           len (.length f)]
162:       (with-open [fis (java.io.FileInputStream. f)]
                            ^--- Unable to resolve classname: java.io.FileInputStream
163:         (encode (:binary magics)
164:                 ;; support default split-size behaviour transparently
165:                 (if (< len split-size)
166:                   (let [ba (with-open [out (java.io.ByteArrayOutputStream.)]
167:                              (<http://clojure.java.io/copy|clojure.java.io/copy> fis out)

I think we can just add that class.

Is there a way for me to quickly add, and iterate through all the dependencies?

yeah, you can clone babashka and invoke it using clojure -M:babashka/dev ... when you make an alias for it, that's how I do it

ok

After adding that class, it appears to work

$ clojure -M:babashka/dev -e "(use 'hasch.core) (uuid5 (edn-hash \"hello world\"))"
#uuid "1227fe0a-471b-5329-88db-875fb82737a8"

great!

I will commit to master. New binary will appear in #babashka-circleci-builds in a few minutes

wow

@bocaj ok, it's there now and it seems to work.

$ bb -e "(use 'hasch.core) (edn-hash [\"hello world\" {:a 3.14} #{42} '(if true nil \f)])"
(245 139 93 212 32 95 155 217 230 42 204 224 210 124 22 156 241 230 65 199 21 108 160 143 225 185 228 16 141 66 80 96 35 189 202 198 252 99 12 214 23 106 43 113 138 129 16 131 7 110 102 1 55 15 4 148 118 187 201 28 144 90 192 21)

Thanks, again! Do you do windows builds each release?

@bocaj yes: https://ci.appveyor.com/project/borkdude/babashka/build/artifacts

wow, again

So glad this is all in the open, I have lots to learn.

@bocaj To be clear, you can just download the binary from appveyor and test it out, right now

You don't have to wait for a release and you can do some testing pre-release if you want

Yes, I assumed that. It’s working on my mac.

I have a windows server, so I’ll check that later. Safe to assume windows binary always works out?

Just to be sure, I'll add hasch's unit tests to babashka's CI later this week. Those tests also run on Windows CI

but usually there aren't any Windows-specific problems

I added all tests to CI now. They are all passing, except around hashing records like (defrecord Foo []). This is because records are implemented differently in bb.