babashka

https://github.com/babashka/babashka. Also see #sci, #nbb and #babashka-circleci-builds .
Ben Sless 2021-05-24T08:18:40.110100Z

Couple of use cases I've been thinking about, wondering if tasks can be used to implement them: Specifying which tasks can be ran in parallel with each other and which can't Something similar to github actions strategy matrix Does it make sense or should I provide more details?

borkdude 2021-05-24T08:33:52.111400Z

@ben.sless How it currently works: you can either provide --parallel on the command line or (run 'task {:parallel true}) programmatically. What this means: all children will be ran in parallel (in so far the dependency tree allows it).

Ben Sless 2021-05-24T08:41:10.111800Z

So there's no fine-grained support in that resolution

Ben Sless 2021-05-24T08:41:30.112200Z

Regarding an expanding test matrix, is it possible to model something like that?

Ben Sless 2021-05-24T08:42:01.112500Z

including overrides and and exclusions

borkdude 2021-05-24T08:44:48.113400Z

@ben.sless I've modeled it after use cases that I deemed common and useful. I wasn't aware of such a test matrix thingie, but feel free to tell more.

borkdude 2021-05-24T08:45:20.114Z

It's basically the same as make -j4 where you run steps in parallel with max threads = 4

Ben Sless 2021-05-24T08:48:20.116300Z

I created a project for stress testing different servers under different configurations. Some servers aren't compatible with Java 8 Different Java versions can have different GC algorithms Different servers can handle different loads so it makes no sense to stress them all at the same rate This leads to a very large set of options and special cases and I'm looking for the best way to model it

Ben Sless 2021-05-24T08:48:51.116700Z

Right now I just have some terrible imperative script https://github.com/bsless/stress-server/blob/master/run.clj

borkdude 2021-05-24T08:51:09.118800Z

Imperative scripts aren't necessarily bad, it's usually what shell scripting is for. As for executing such a thing with tasks, you could probably get a long way with just iteration and using run and binding *command-line-args* to pass command line args down to tasks. Or you could start new instances of bb using shell by providing env vars with :extra-env

Ben Sless 2021-05-24T08:52:52.119900Z

Imperative scripts aren't bad, but I would prefer to describe it as much as I can as data and not code. Might end up reaching for dynamic variables instead, they're the in-process equivalent of an environment anyway

borkdude 2021-05-24T08:53:39.120300Z

When I want to run such a thing in CircleCI or whatever, I usually generate the yaml using a script

borkdude 2021-05-24T08:53:50.120600Z

and don't bother learning their matrix DSL at all :)

borkdude 2021-05-24T08:54:07.120900Z

but if the use case is commons enough, we could consider it later on

Ben Sless 2021-05-24T08:58:57.122800Z

It might be covered by (doseq [option options] (binding ,,, (run ,,)) However, even an alias for that would be extremely nice because the loses the whole mechanism of specifying dependencies

borkdude 2021-05-24T09:12:18.123300Z

@ben.sless > because the loses the whole mechanism of specifying dependencies Not sure if I understood what you were saying there

Ben Sless 2021-05-24T09:14:45.125100Z

I would have preferred specifying the dependency order of the parameters which need to be scanned for example: • jdk version • GC algorithms • server library as task dependencies i.e. profile -> (depends on) server -> gc -> jdk

Ben Sless 2021-05-24T09:15:01.125600Z

where each level of traversal up the tree will expand to the possible options

borkdude 2021-05-24T09:16:22.126400Z

that still works if you invoke run and bind to different values of those options

Ben Sless 2021-05-24T09:16:22.126500Z

Although theoretically the order doesn't matter in this part so it could be tasks aren't the best model

Ben Sless 2021-05-24T09:18:35.126600Z

I'll try playing around with it, see if it's sufficient and if I can provide overrides

Ben Sless 2021-05-24T09:44:47.128700Z

I'm sorry, I'm still confused by this Let's say I have this rough layout for tasks' dependencies and their components

{:init (do (defn print-args []
              (prn (:name (current-task))
                   *command-line-args*)))
  :enter (print-args)
  init {:task {}}
  jdk {:depends [init]
       :init (def opts [8 11 15])}
  server {:depends [jdk]
          :init (def opts '[httpkit aleph])}
  profile {:depends [server]
           :task (println "profiling" *command-line-args*)}
  }
how can I take advantage of the dependencies resolution order and not just have to resort to run everythere?

Ben Sless 2021-05-24T09:45:37.129100Z

I can always do this:

(doseq [opt opts]
  (binding [*command-line-args* (assoc *command-line-args* k opt)]
    (run task)))
But doesn't it defeat the purpose?

borkdude 2021-05-24T09:46:47.130100Z

I think in jdk you would use *jdk* so only one. And then in some parent task you would bind *jdk* to the row of values while invoking the dependency tree

borkdude 2021-05-24T09:47:13.130300Z

same for server

borkdude 2021-05-24T09:47:30.130500Z

let me try to make an example

Ben Sless 2021-05-24T09:47:51.130900Z

Please do, It's not clicking for me at all

Ben Sless 2021-05-24T09:48:20.131400Z

Very possible I'm trying to square the circle with babashka here

borkdude 2021-05-24T10:00:38.132400Z

@ben.sless I found a couple of edge cases (nicer word for bugs ;)), these should be fixed, but here's the idea with workaround for those issues:

{:tasks {:init (do
                 (ns my-ns)
                 (def ^:dynamic *jdk* nil)
                 ;; workaround
                 (alter-meta! (var *jdk*) assoc :dynamic true)
                 (def ^:dynamic *server* nil)
                 (alter-meta! (var *server*) assoc :dynamic true))
         :enter (println "Task:" (:name (babashka.tasks/current-task)))
         jdk (println "JDK:" *jdk*)
         server {:depends [jdk]
                 :task (println "Server:" *server*)}
         run-all (doseq [jdk [8 11 15]
                         server [:foo :bar]]
                   (binding [*jdk* jdk
                             *server* server]
                     (babashka.tasks/run 'server)))}}
bb run-all
Task: run-all
Task: jdk
JDK: 8
Task: server
Server: :foo
Task: jdk
JDK: 8
Task: server
Server: :bar
Task: jdk
JDK: 11
...

borkdude 2021-05-24T10:01:27.133200Z

The issues: - dynamic vars aren't dynamic (due to loss of metadata during processing). Workaround: alter-meta!. - each run runs in a random namespace, but all tasks should run in the same namespace. Workaround: set namespace manually and use fully qualified symbols for task built-ins. Both issues should be fixed.

borkdude 2021-05-24T10:12:23.133700Z

Made an issue here: https://github.com/babashka/babashka/issues/865

borkdude 2021-05-24T10:17:22.134200Z

but does the idea make sense now @ben.sless (aside from the inconvenient issues)?

borkdude 2021-05-24T10:18:38.134900Z

So run-all sets the environment and the rest of the tasks just work as if there is one value at a time

Ben Sless 2021-05-24T11:20:44.135500Z

I'll get some food in me and write a coherent response

Ben Sless 2021-05-24T12:18:15.136400Z

ok, this makes sense, it's just the implementation I was hoping to avoid Ideally, I wouldn't want to spell out the iteration and binding manually

borkdude 2021-05-24T12:23:18.136600Z

right

borkdude 2021-05-24T12:24:14.136900Z

although (aside from the issues) it doesn't seem too much boilerplate

Ben Sless 2021-05-24T12:26:36.137700Z

It isn't, it just expands the more options are involved. I'm trying to figure out the correct idiom

borkdude 2021-05-24T12:27:14.138Z

you could write a small macro in :init which expands these options ;)

borkdude 2021-05-24T12:28:50.138700Z

although I'm not sure if the .edn syntax can withstand the macro syntax. you could put it in a file on your classpath instead and then require it

Ben Sless 2021-05-24T12:29:41.139500Z

There are two pieces which are orthogonal - one is the ask ordering which is handled correctly by tasks, one is creating the combination of options

borkdude 2021-05-24T12:29:43.139700Z

it could probably be just a function too, since with-bindings* is a normal function

Ben Sless 2021-05-24T12:36:34.142900Z

This is an idea, and I'm not saying it should be integrated into the task running, but what do you think about the following enhancement to the syntax: Tasks take an additional key, :matrix. Besides *command-line-arguments* add another global variable, *matrix* which starts off as an empty map Every task which has a :matrix will run in the context of a doseq over the parameters where *matrix* is bound to (assoc *matrix* task-name param)

Ben Sless 2021-05-24T12:41:57.143100Z

Roughly

{server {:depends [jdk]
         :matrix [:a :b :c]}
 jdk {:matrix [8 11 15]
      :task (setup-jdk (*matrix* jdk))}
 run {:depends [server]}}

borkdude 2021-05-24T12:44:46.143700Z

This needs more hammock time, so I would prefer if this can be done in "user" space first. I was trying this:

(def matrix [{:jdk 11 :server "foo"} {:jdk 8 :server "bar"}])

(defn var-name [k]
  (symbol (str "*" (name k) "*")))

(doseq [k (keys (first matrix))]
  (intern *ns* (with-meta (var-name k)
                 {:dynamic true})))

(defn run-matrix []
  (doseq [row matrix]
    (let [ks (keys row)
          vars (map (fn [k]
                      (resolve (var-name k))) ks)
          vals (vals row)]
      (with-bindings* (zipmap vars vals)
        (fn []
          (println (map deref vars)))))))

(run-matrix)

borkdude 2021-05-24T12:45:08.144100Z

but somehow I get: Can't dynamically bind non-dynamic var: user/*jdk* (in normal Clojure)

borkdude 2021-05-24T12:47:12.144300Z

user=> (intern *ns* (with-meta '*foo* {:dynamic true}))
#'user/*foo*
user=> (binding [*foo* 2])
Execution error (IllegalStateException) at user/eval233 (REPL:1).
Can't dynamically bind non-dynamic var: user/*foo*
:thinking_face:

Ben Sless 2021-05-24T13:12:49.145800Z

run-matrix loses the notion of task dependency order 😞 Also, why not just use a single dynamic variable *matrix* then work in its context?

borkdude 2021-05-24T13:13:35.146500Z

run-matrix was just an out of context demo, not related to tasks. you can use run within the body of run-matrix.

borkdude 2021-05-24T13:14:10.147600Z

"Why not just" assumes that this is a trivial thing to quickly add, which in my opinion it isn't, I have to think about it more for a while and see how often this comes up

Ben Sless 2021-05-24T13:14:42.147800Z

I meant in the demo, not in babashka's guts, sorry if that wasn't clear

Ben Sless 2021-05-24T13:14:48.148Z

seems easier than interning vars on the fly

Ben Sless 2021-05-24T13:15:05.148200Z

not trying to push 🙂

borkdude 2021-05-24T13:15:38.148400Z

yeah, that can work too :)

borkdude 2021-05-24T13:17:01.148600Z

that probably simplifies things a bit

borkdude 2021-05-24T13:18:46.148800Z

so in user space I guess you could read from the *matrix* your task's entries and run n times

borkdude 2021-05-24T13:18:49.149Z

or so?

borkdude 2021-05-24T13:19:04.149200Z

that won't work either I think

borkdude 2021-05-24T13:19:33.149400Z

you need some kind of doseq like construction on the top level

Ben Sless 2021-05-24T13:20:14.149600Z

yes. My hope was for a means to make that doseq implicit and still maintain the task ordering without having to run them "manually"

borkdude 2021-05-24T13:23:09.149800Z

well, you only have to run the top-level one manually, but it's the bindings you would have to manage

Ben Sless 2021-05-24T15:08:05.150200Z

I may have figured out a way to hack it, just need to break free of a recursive loop

Ben Sless 2021-05-24T15:14:03.150400Z

Alright, got it, I'll send it over when I get WiFi on my machine

Ben Sless 2021-05-24T15:38:13.151500Z

This sort-of works:

{:min-bb-version "0.4.0"
 :tasks
 {:init (do
          (ns user
            (:require
             [babashka.tasks :refer :all]))
          (def ^:dynamic *matrix* {})
          (alter-meta! (var *matrix*) assoc :dynamic true)
          (defn $ [k] (get *matrix* k))
          (defn on-enter []
            (prn (:name (current-task)) *matrix*))
          (defn enter-the-matrix
            []
            (let [{:keys [name matrix]} (current-task)]
              (when matrix
                (when-not ($ name)
                  (doseq [e matrix]
                    (binding [*matrix* (assoc *matrix* name e)]
                      (run name))))))))
  :enter (do (on-enter) (enter-the-matrix))
  a {:depends [b]
     :matrix [+ -]}
  b {:depends [c x]
     :matrix [:a :b]}
  x {:matrix ["foo" "bar"]}
  c {:matrix [1 2 3]}}}

borkdude 2021-05-24T15:39:56.151700Z

that's pretty cool :)

Ben Sless 2021-05-24T15:40:25.152100Z

Yeah, but it's incorrect 🙃

Ben Sless 2021-05-24T15:40:46.152600Z

Which is evident by the printed output

2021-05-24T16:11:47.159900Z

I need a little help on my workflow. I'm using babashka with vim-iced. I open my script and connect with :IcedInstantConnect babashka using nrepl. In my main script I want to leverage some util fns I have in utils.clj which looks like this:

#!/usr/bin/env bb
(ns utils)

(def bar "barrrr")
Now in my main script I require it like so:
(ns my-script
  (:require [babashka.fs :as fs]
            [clojure.java.shell :as shell :refer [sh]]
            [clojure.string :as str]
            [utils :as ut]))
So, immediately after connect, in my my-script buffer, I use :IcedRequire and my stdout buffer shows me:
;;
;; Iced Buffer
;;
clojure.lang.ExceptionInfo: Could not find namespace: utils.
Now, I can switch to the utils.clj buffer and use :IcedRequire there, switch back to the main script buffer, and then use :IcedRequire there successfully. I'm just not sure this is the correct workflow. Could anyone offer any pointers as to how to do the above properly? As a follow-up, how do I then deal with changes to either the main script or the utils file? Do I have to reconnect vim-iced to a new babashka buffer and repeat the dance in order for it to see my latest changes?

borkdude 2021-05-24T16:12:38.160300Z

@randumbo Do you use bb.edn perhaps? Which version are you on?

borkdude 2021-05-24T16:13:57.161200Z

You need to set the classpath so babashka knows where it can find utils.clj

borkdude 2021-05-24T16:14:07.161500Z

And the easiest way to do this now is using bb.edn :paths

2021-05-24T16:14:54.162200Z

@borkdude I have a ~/bb.edn but it only has:

{:tasks
 {foo (shell "echo foo!!!")}}
I'm on babashka v0.4.0.

borkdude 2021-05-24T16:15:09.162500Z

and where is your utils.clj file?

2021-05-24T16:15:20.162800Z

Same directory as my script that I am calling.

borkdude 2021-05-24T16:15:38.163200Z

ok, then add to bb.edn: paths ["."]

borkdude 2021-05-24T16:16:13.163700Z

it's a bit unusual to do this, normally you use a subdirectory like src

2021-05-24T16:17:06.164400Z

And that particular bb.edn should be in the same directory as the script and utils file?

borkdude 2021-05-24T16:17:33.164600Z

yes

borkdude 2021-05-24T16:17:52.165100Z

similar to project.clj and deps.edn

borkdude 2021-05-24T16:18:45.166300Z

if you don't like that, you can also set the classpath manually using babashka.classpath/add-classpath

borkdude 2021-05-24T16:18:58.166800Z

or load the file using load-file instead of require

2021-05-24T16:19:07.167Z

The reason I don't have a src/ or something for this is because I'm just writing a small script in my repos folder to do a bit of housekeeping. I suppose I could move it to a repos/bb_scripts/ subfolder or to a different parent altogether.

borkdude 2021-05-24T16:19:54.167400Z

the takeaway: if you want to use require with your own scripts, make sure the classpath is set

2021-05-24T16:20:44.168500Z

Yep, got that. Given that I do that, are changes to those files then automatically picked up or do I have to do another step to re-eval their contents?

borkdude 2021-05-24T16:21:42.168900Z

if you want to re-eval you can use (require 'foo :reload)

2021-05-24T16:22:23.169400Z

Thanks very much. That should cover it.