Clojurians Log v2

About parallel :depends . what about

:depends ["sequential" ... ]
or
:depends #{"parallel" ..}

@maxp Interesting, I like the idea. What if you want to execute p1 sequentially and then p2 and p3 in parallel? [p1 #{p2 p3}]] ?

maybe that case is getting too complex and one should just make another step in between

:depends #{"parent" ...} implies that the task understands the internal details of parent and whether it can be parallelized. Having a task return future or value leaves that decision in the task. The latter kind of orchestration looks similar to the approach in pedestal interceptors and pathom resolvers: leave the details of sync/async to the the task itself, describe a tree of dependencies separately, and then a higher-level orchestrator is just responsible for calling each of them when it is appropriate (and blocking when necessary).

I figure, if it the orchestration gets any more complicated than that, maybe it's time to revisit a configuration ala onyx.

@pithyless I think for something like make / bb.edn it's the other way around, at least in some cases. At work I now have this:

{:tasks {dev-cljs {:doc "Runs front-end compilation"
                   :task (clojure "-M:frontend:cljs/dev")}
         dev-less {:doc "Compiles less"
                   :task (clojure "-M:frontend:less/dev")}
         dev-backend {:task (clojure (str "-A:backend:backend/dev:" platform-alias)
                                     "-X" "dre.standalone/start")}
         -dev {:depends [dev-cljs dev-less dev-backend]}
         dev {:doc "Runs app in dev mode. Compiles cljs, less and runs JVM app in parallel."
              :task (shell "bb run --parallel -dev")}}}

I think make runs anything in parallel if you use -j4 (at most 4 jobs in parallel).

Wild idea and not properly thought through; if tasks are properly isolated, parallel could be the default, “sequential” could be a debugging option?

I think sequential has its place too perhaps. When you run everything in parallel, the output may look more garbled, etc. So having some way to control it might be nice

Some people may depend on side effects happening in p1 first, etc

Although then p2 should depend on p1

And good question - what means 'parallel' in this case. some special task does fork/join?

btw, for my mind 'depends' does not mean 'execute in strict particular order'

A nice example provided by @grazfather is this one: https://gist.github.com/borkdude/356669d9b829fe5a891e44f920105f26 When you run it with bb coffeep it takes 11 seconds. When you run it with bb run --parallel coffeep it takes 6 seconds, because bb runs some of the dependencies in parallel.

@maxp I think that's reasonable: if you say #{p1 p2} you don't care about the order, so bb can (and will by default?) optimize this by running them side by side. And wait for all of them to finish before continuing.

What about just :depends [p1 p2] and the rule is always: all these tasks are dependencies that may by run in any-order and in parallel. Then you can supply a -j0 (sequential, any order) to -j4 to (in parallel, any order). If semantics are important (either ordering or using return values from one task in another) then those semantics will be explicit in the code - ie. don't use depends but write out the code in the :task ?

@pithyless I think the nice thing about #{p1 p2} is that you have finer control over which things you want to run in parallel. E.g. for my work tasks I have:

clean (do (shell "bash -c 'rm -rf frontend/public/js/app.js'")
                   (shell "bash -c 'rm -rf frontend/public/js/app.out'")
                   (shell "bash -c 'rm -rf frontend/public/css/main*'"))
         dev:clean {:depends [clean dev]}

and I think you will use #{p1 p2} more sparingly, only for deps that take a few seconds or more probably

yeah, I understand; I actually do like the #{p1 p2} and [p1 p2] distinction; I'm just worried we may go down a sugar-syntax rabbit hole. (1) Do we support something like [p1 #{p2 p3} p4]? (2) Will the semantics be obvious to me when I look at a script in a month, without looking up the babashka docs? :)

@pithyless for [p1 #{p2 p3} p4] I intend not to support it. if you want this, then make it so: p0: depends [p1 p2 p3] p2: depends #{p4 p5}

@borkdude BTW, how does your dev example work in practice? are those 3 processes that startup and block? Do they garble the output or do they not print to stdout?

@pithyless those three processes just run in the background and the dev task waits for them to finish, which is never, or until I hit ctrl-c. I have a script for this:

#!/usr/bin/env bb

(ns dev)

(require '[babashka.process :refer [destroy-tree process]])

(def opts {:out :inherit
           :err :inherit
           :shutdown destroy-tree})

(defn cljs []
  (process ["clojure" "-Sforce" "-M:frontend:cljs/dev"] opts))

(defn less []
  (process ["clojure" "-Sforce" "-M:frontend:less/dev"] opts))

(defn clojure []
  (process ["clojure" "-Sforce"
            "-A:backend:backend/dev"
            "-X" "dre.standalone/start"]
           (assoc opts :in :inherit)))

(cljs)
(less)
(-> @(clojure) :exit (System/exit))

and yes, the output may become garbled, but in practice this doesn't bother me that much, doesn't happen that often

so the above script effectively becomes https://clojurians.slack.com/archives/CLX41ASCS/p1618301439082200

it's similar to starting processes in bash with &

Btw, I think it might be nice to release a stable version of bb tasks at ClojureD in June perhaps. You can vote for the talk topic here: https://twitter.com/borkdude/status/1381897272024305665

https://twitter.com/borkdude/status/1381943766756569091

Click the link to see info about an informal meetup after the talk organized by Nikolay. It will be a kind of Russian Clojure meetup on Zoom. Join the Telegram channel for more info.

Pasted my bb.edn from work here: https://gist.github.com/borkdude/35bc0a20bd4c112dec2c5645f67250e3

does it make more sense to have an :env block or something outside of :tasks, even, that is used to set vars?

e.g. for`platform-alias`

and i imagine that soon people will want to override these vars

@grazfather I thought about this, e.g :vars ... but what benefit would this have over :init?

an invoker could provide these

it would only be more restricting I thing, while introducing semantic problems, like ordering, etc

e.g. bb run dev-backend platform-alias=Solaris would avoid running the thing that normally sets platform-alias

I would say, just use an env var for that. BACKEND=foo bb dev and (def backend (or (System/getenv "BACKEND" ...))

Again, I am just taking make semantics… I don’t think it’s something we really need to worry about

ok, yeah I agree

i think for tasks you should write a ‘recipe book’ about how to get things from make in BB. you showed the perfect example of one

did you change your mind about :init vs just a ‘recipe’` for init being a dependency?

@wilkerlucio convinced me that we should leave :init in it ;)

can I read the reasoning somewhere?

found it

the main reason is that it can get more verbose

sure, but you lose the ability to put ‘expensive’ init, since it runs for all tasks

you can put an expensive init still in its own task

is 1 more word in each task really more verbose?

you can have both options

it is when you have to change from

(my-task "foo")

{:depends [init]
 :task (my-task "foo")}

yep

yeah I think I agree

btw, I also introduced :requires so you can require expensive namespaces in tasks, so not for every task

on master

i saw that in your work example

and all implicit aliases go away, e.g. no automatic fs

I think I like that too. I prefer less magic

more explicit

so if you want to use fs in multiple tasks you can do {:tasks {:requires ([babashka.fs :as fs])}}

or just for one:

{:tasks {foo {:requires ([babashka.fs :as fs])}}}

(built-in namespaces aren't expensive to require tho, interpreted ones can be)

nice

I think I'll also include the idea from @maxp about :depends #{p1 p2}, it comes in quite handy for my work bb.edn

to automatically parallelize?

yes

I think it’s better if that were not supported. Tasks should be parallelizable by default by virtue that they should list all their dependencies, anything not listed, then, must be able to run at the same time. if you need them to run in a certain order, then you define the tasks that way If you want dev to clean and then build you don’t say dev depends on clean and build, you say that dev depends on build which depends on clean

but build doesn't always depend on clean

I also want to be able to run build without clean

so having :depends [clean dev] with order semantics does cover that use while :depends #{p1 p2} means: don't care about the order

but can’t you just do bb run clean dev?

that doesn't work since clean can have command line args

how do you provide args to a task?

bb foo arg1 arg2

ok, so you can’t kick off multiple tasks at all

bb clean && bb dev

haha i mean within one invocation obviously

leiningen has this:

bb do clean foo bar, dev foo bar

so it uses the comma as a special delimiter in do

I kinda like that: bb do clean, dev

ah interesting

we could also have:

:pre [clean]
:post [foobar]

but yeah, it’s worth trying out the set dependencies thing

but :depends [clean] also kind of means :pre

yeah, supporting both seems it would get confusing quickly

:depends [foo x], if foo depends on x then x is executed first anyway, so it can't always guarantee the order, but it will try to do it if possible

feel free to convince me otherwise, but for me personally this is convenient. if it's really a bad idea I'd also like to know

you can also do:

clean:dev (do (shell "bb clean") (shell "bb dev"))

What would be your recommended approach for something like this?

{:task (do (future (shell "bb watch-cljs")) (shell "bb run"))}

That's one I use inside a docker container that starts shadowcljs and the main server.

@duck This would be solved if we implemented the :depends #{watch-cljs run} but now you can do bb run --parallel the-task, this will run the deps in parallel

not sure if this will stay in if we also support #{foo bar}

yeah, I still need to grab the parallel updates

so this is what I do myself right now: https://gist.github.com/borkdude/35bc0a20bd4c112dec2c5645f67250e3#file-bb-edn-L10-L18

hello, just figured a cool snippet to copy things to clipboard (macos only) using babashka.process and wanted to share with you here 🙂

(proc/process "pbcopy" {:in "any string to put on clipboard"})