Clojurians Log v2

Couple of use cases I've been thinking about, wondering if tasks can be used to implement them: Specifying which tasks can be ran in parallel with each other and which can't Something similar to github actions strategy matrix Does it make sense or should I provide more details?

@ben.sless How it currently works: you can either provide --parallel on the command line or (run 'task {:parallel true}) programmatically. What this means: all children will be ran in parallel (in so far the dependency tree allows it).

So there's no fine-grained support in that resolution

Regarding an expanding test matrix, is it possible to model something like that?

including overrides and and exclusions

@ben.sless I've modeled it after use cases that I deemed common and useful. I wasn't aware of such a test matrix thingie, but feel free to tell more.

It's basically the same as make -j4 where you run steps in parallel with max threads = 4

I created a project for stress testing different servers under different configurations. Some servers aren't compatible with Java 8 Different Java versions can have different GC algorithms Different servers can handle different loads so it makes no sense to stress them all at the same rate This leads to a very large set of options and special cases and I'm looking for the best way to model it

Right now I just have some terrible imperative script https://github.com/bsless/stress-server/blob/master/run.clj

Imperative scripts aren't necessarily bad, it's usually what shell scripting is for. As for executing such a thing with tasks, you could probably get a long way with just iteration and using run and binding *command-line-args* to pass command line args down to tasks. Or you could start new instances of bb using shell by providing env vars with :extra-env

Imperative scripts aren't bad, but I would prefer to describe it as much as I can as data and not code. Might end up reaching for dynamic variables instead, they're the in-process equivalent of an environment anyway

When I want to run such a thing in CircleCI or whatever, I usually generate the yaml using a script

and don't bother learning their matrix DSL at all :)

but if the use case is commons enough, we could consider it later on

It might be covered by (doseq [option options] (binding ,,, (run ,,)) However, even an alias for that would be extremely nice because the loses the whole mechanism of specifying dependencies

@ben.sless > because the loses the whole mechanism of specifying dependencies Not sure if I understood what you were saying there

I would have preferred specifying the dependency order of the parameters which need to be scanned for example: • jdk version • GC algorithms • server library as task dependencies i.e. profile -> (depends on) server -> gc -> jdk

where each level of traversal up the tree will expand to the possible options

that still works if you invoke run and bind to different values of those options

Although theoretically the order doesn't matter in this part so it could be tasks aren't the best model

I'll try playing around with it, see if it's sufficient and if I can provide overrides

I'm sorry, I'm still confused by this Let's say I have this rough layout for tasks' dependencies and their components

{:init (do (defn print-args []
              (prn (:name (current-task))
                   *command-line-args*)))
  :enter (print-args)
  init {:task {}}
  jdk {:depends [init]
       :init (def opts [8 11 15])}
  server {:depends [jdk]
          :init (def opts '[httpkit aleph])}
  profile {:depends [server]
           :task (println "profiling" *command-line-args*)}
  }

I can always do this:

(doseq [opt opts]
  (binding [*command-line-args* (assoc *command-line-args* k opt)]
    (run task)))

I think in jdk you would use *jdk* so only one. And then in some parent task you would bind *jdk* to the row of values while invoking the dependency tree

same for server

let me try to make an example

Please do, It's not clicking for me at all

Very possible I'm trying to square the circle with babashka here

@ben.sless I found a couple of edge cases (nicer word for bugs ;)), these should be fixed, but here's the idea with workaround for those issues:

{:tasks {:init (do
                 (ns my-ns)
                 (def ^:dynamic *jdk* nil)
                 ;; workaround
                 (alter-meta! (var *jdk*) assoc :dynamic true)
                 (def ^:dynamic *server* nil)
                 (alter-meta! (var *server*) assoc :dynamic true))
         :enter (println "Task:" (:name (babashka.tasks/current-task)))
         jdk (println "JDK:" *jdk*)
         server {:depends [jdk]
                 :task (println "Server:" *server*)}
         run-all (doseq [jdk [8 11 15]
                         server [:foo :bar]]
                   (binding [*jdk* jdk
                             *server* server]
                     (babashka.tasks/run 'server)))}}

bb run-all
Task: run-all
Task: jdk
JDK: 8
Task: server
Server: :foo
Task: jdk
JDK: 8
Task: server
Server: :bar
Task: jdk
JDK: 11
...

The issues: - dynamic vars aren't dynamic (due to loss of metadata during processing). Workaround: alter-meta!. - each run runs in a random namespace, but all tasks should run in the same namespace. Workaround: set namespace manually and use fully qualified symbols for task built-ins. Both issues should be fixed.

Made an issue here: https://github.com/babashka/babashka/issues/865

but does the idea make sense now @ben.sless (aside from the inconvenient issues)?

So run-all sets the environment and the rest of the tasks just work as if there is one value at a time

I'll get some food in me and write a coherent response

ok, this makes sense, it's just the implementation I was hoping to avoid Ideally, I wouldn't want to spell out the iteration and binding manually

right

although (aside from the issues) it doesn't seem too much boilerplate

It isn't, it just expands the more options are involved. I'm trying to figure out the correct idiom

you could write a small macro in :init which expands these options ;)

although I'm not sure if the .edn syntax can withstand the macro syntax. you could put it in a file on your classpath instead and then require it

There are two pieces which are orthogonal - one is the ask ordering which is handled correctly by tasks, one is creating the combination of options

it could probably be just a function too, since with-bindings* is a normal function

This is an idea, and I'm not saying it should be integrated into the task running, but what do you think about the following enhancement to the syntax: Tasks take an additional key, :matrix. Besides *command-line-arguments* add another global variable, *matrix* which starts off as an empty map Every task which has a :matrix will run in the context of a doseq over the parameters where *matrix* is bound to (assoc *matrix* task-name param)

Roughly

{server {:depends [jdk]
         :matrix [:a :b :c]}
 jdk {:matrix [8 11 15]
      :task (setup-jdk (*matrix* jdk))}
 run {:depends [server]}}

This needs more hammock time, so I would prefer if this can be done in "user" space first. I was trying this:

(def matrix [{:jdk 11 :server "foo"} {:jdk 8 :server "bar"}])

(defn var-name [k]
  (symbol (str "*" (name k) "*")))

(doseq [k (keys (first matrix))]
  (intern *ns* (with-meta (var-name k)
                 {:dynamic true})))

(defn run-matrix []
  (doseq [row matrix]
    (let [ks (keys row)
          vars (map (fn [k]
                      (resolve (var-name k))) ks)
          vals (vals row)]
      (with-bindings* (zipmap vars vals)
        (fn []
          (println (map deref vars)))))))

(run-matrix)

but somehow I get: Can't dynamically bind non-dynamic var: user/*jdk* (in normal Clojure)

user=> (intern *ns* (with-meta '*foo* {:dynamic true}))
#'user/*foo*
user=> (binding [*foo* 2])
Execution error (IllegalStateException) at user/eval233 (REPL:1).
Can't dynamically bind non-dynamic var: user/*foo*

run-matrix loses the notion of task dependency order 😞 Also, why not just use a single dynamic variable *matrix* then work in its context?

run-matrix was just an out of context demo, not related to tasks. you can use run within the body of run-matrix.

"Why not just" assumes that this is a trivial thing to quickly add, which in my opinion it isn't, I have to think about it more for a while and see how often this comes up

I meant in the demo, not in babashka's guts, sorry if that wasn't clear

seems easier than interning vars on the fly

not trying to push 🙂

yeah, that can work too :)

that probably simplifies things a bit

so in user space I guess you could read from the *matrix* your task's entries and run n times

or so?

that won't work either I think

you need some kind of doseq like construction on the top level

yes. My hope was for a means to make that doseq implicit and still maintain the task ordering without having to run them "manually"

well, you only have to run the top-level one manually, but it's the bindings you would have to manage

I may have figured out a way to hack it, just need to break free of a recursive loop

Alright, got it, I'll send it over when I get WiFi on my machine

This sort-of works:

{:min-bb-version "0.4.0"
 :tasks
 {:init (do
          (ns user
            (:require
             [babashka.tasks :refer :all]))
          (def ^:dynamic *matrix* {})
          (alter-meta! (var *matrix*) assoc :dynamic true)
          (defn $ [k] (get *matrix* k))
          (defn on-enter []
            (prn (:name (current-task)) *matrix*))
          (defn enter-the-matrix
            []
            (let [{:keys [name matrix]} (current-task)]
              (when matrix
                (when-not ($ name)
                  (doseq [e matrix]
                    (binding [*matrix* (assoc *matrix* name e)]
                      (run name))))))))
  :enter (do (on-enter) (enter-the-matrix))
  a {:depends [b]
     :matrix [+ -]}
  b {:depends [c x]
     :matrix [:a :b]}
  x {:matrix ["foo" "bar"]}
  c {:matrix [1 2 3]}}}

that's pretty cool :)

Yeah, but it's incorrect 🙃

Which is evident by the printed output

I need a little help on my workflow. I'm using babashka with vim-iced. I open my script and connect with :IcedInstantConnect babashka using nrepl. In my main script I want to leverage some util fns I have in utils.clj which looks like this:

#!/usr/bin/env bb
(ns utils)

(def bar "barrrr")

(ns my-script
  (:require [babashka.fs :as fs]
            [clojure.java.shell :as shell :refer [sh]]
            [clojure.string :as str]
            [utils :as ut]))

;;
;; Iced Buffer
;;
clojure.lang.ExceptionInfo: Could not find namespace: utils.

@randumbo Do you use bb.edn perhaps? Which version are you on?

You need to set the classpath so babashka knows where it can find utils.clj

And the easiest way to do this now is using bb.edn :paths

@borkdude I have a ~/bb.edn but it only has:

{:tasks
 {foo (shell "echo foo!!!")}}

and where is your utils.clj file?

Same directory as my script that I am calling.

ok, then add to bb.edn: paths ["."]

it's a bit unusual to do this, normally you use a subdirectory like src

And that particular bb.edn should be in the same directory as the script and utils file?

yes

similar to project.clj and deps.edn

if you don't like that, you can also set the classpath manually using babashka.classpath/add-classpath

or load the file using load-file instead of require

The reason I don't have a src/ or something for this is because I'm just writing a small script in my repos folder to do a bit of housekeeping. I suppose I could move it to a repos/bb_scripts/ subfolder or to a different parent altogether.

the takeaway: if you want to use require with your own scripts, make sure the classpath is set

Yep, got that. Given that I do that, are changes to those files then automatically picked up or do I have to do another step to re-eval their contents?

if you want to re-eval you can use (require 'foo :reload)

Thanks very much. That should cover it.