Clojurians Log v2

is it looking like :when won’t be a thing?

tasks are looking pretty damn good to me

@grazfather Thanks. I had trouble figuring out what the behavior of :when in dependent tasks should be. E.g. invoke foo which depends on bar which depends on baz. But :when in bar returns false, then what?

@grazfather I think you can have a creative workaround like this:

{:tasks {clojars-user-name
         (when-not (System/getenv "CLOJARS_USER_NAME")
           (println "Needs clojars username")
           (System/exit 1))
         deploy {:depends [clojars-user-name]
                 :task (println "Deploying...")}}}

@borkdude at first I was skeptical, but I managed to convert some code to the new tasks format:

{:init
  (do
    (ns my.tasks
      (:require
       [proj.repo.task.version :as task.version]
       [proj.repo.ednopt :as ednopt]))

    (def default-args
      {:min-version/clojure  "1.10.3.814"
       :min-version/babashka "0.3.3"})

    (def args (merge default-args
                     (ednopt/parse *command-line-args*))))

  check-versions
  {:doc "Check runtime versions are supported"
   :task (task.version/check-versions args)}}

(3) is kind of rambling, I'm just wondering if the current notions of task dependencies and guards in Makefile/Justfile/etc are just a poor-man's approach to idempotent functions (and maybe we can leverage clojure idioms more here)

@pithyless co-incidentally, I just posted an example of :depends that replaces the need for :when possibly, around the same millisecond you posted your long message

I'll take a look at your other comments later today. Thanks for the feedback

@pithyless 1+2, cool. 3 :when doesn't exist anymore. :depends is there to topologically sort tasks, so when two tasks depend on a third task and you depend on these two tasks, the third task is only executed once, which is a common feature in build systems.

ad 3. Yeah, that makes sense, without :when there is no question about the semantics of how it interacts with :depends.

right that was the reason I removed it (`:when`)

you can just short-circuit using System/exit if needed

I'm going to try to build up something more substantial with the existing bb tasks. For me the missing feature was an easy way to parameterize tasks, and I think I found it via combining the existing :init with that previous idea I talked about inspired by passing EDN kw-args on the command line.

Thanks for all the work @borkdude! 🙂

Thanks for trying and looking forward to more feedback

Yep I was using when/when-not instead,no big deal :)

If foo depends on bar which depends on baz, then if bar’s :when says not to run, then it implies that baz is done as well.

I don’t think it’s very critical that you have this feature, but if you do, then I would liteirally just copy the behaviour from make. I was using a makefile I wrote myself to compare my bb task

https://twitter.com/borkdude/status/1381248261034545153

does Make have something like when?

only for non-phony targets, it uses the time stamp of the target vs any of its dependencies

a: b c

a {:when (and (modified-since a b) (modified-since a c)}

but

a: b
b: c

i might have gotten the order of the args for modified-since wrong 🙂

one example I had in mind was the above:

{:tasks {a {:depends [b]} {b {:when (System/getenv "CLOJARS_USER_NAME)}}}

I think it’s pretty safe to not rush :when and let :depend ( which is more important) to shake out any issues first

let's say that b is deploying something based on that env var and a depends on the thing having been deployed

oh, Make isn’t smart enough for that, the way you’d do it in make is you could put the env var ITSELF as an uinbuildable dependency

so a and b depend on env and env doesn’t set anything, but just fails if the end var is not set

right. we could encode some special return value like :tasks/invalid which would yield the entire dependency tree invalid, for example.

that makes me wonder: Do you capture failure from tasks and propagate then?

exactly 🙂

again I stress that we should not make up new semantics as best we can

right now this is done using System/exit. So if a shell command doesn't return with 0 the entire process quits with that exit code

that might be the least surprising way indeed

If you have any topics for a talk at a virtual Clojure conf, feel free to respond on Twitter or here (in a thread): https://twitter.com/borkdude/status/1380950518567145473

yep that is perfect to me

nice. hidden tasks!

yep. - will make it hidden, or :private true.

You can still call it from the command line if you want, it just won't show up in the tasks list

beautiful

I am still pretty slow writing lisps and edn but I think I will soon use this to write all my ‘test targets’. a lot of then depend on certain docker containers to be running, and some need to run inside the container. We do a lot with makefiles but it gets hacky do determine if a container is running and if you’re in a container

great! keep me posted as we have still room to improve and break things if we need to

Yep I will let you know if I hit any snags

posted without comment 😈

bash
$ time make coffee
measuring beans
grinding beans
pouring water
heating water
getting filter
getting mug
brewing coffee

real    0m15.121s
user    0m0.016s
sys     0m0.023s
$ time make -j4 coffee
pouring water
measuring beans
getting mug
getting filter
grinding beans
heating water
brewing coffee

real    0m9.039s
user    0m0.016s
sys     0m0.024s

Can you post a comment anyway? What am I looking at? What's -j4 ?

parallelizes (4 threads)

zomg, yeah we could do that

the topo sort can find independent chains

see how it’s 15s vs 9 seconds?

yeah

first first heats water then grinds beans. the second does both at the same time. I use it as an example always to explain makefiles 🙂

in tasks you can have futures or core.async and do this yourself as well

yeah I am going to play with it right now. I was just brewing coffee and literally thought “might as well start the kettle while I grind” 😉

keep the Java flowing

so I am not actually sure how/if you could share futures between tasks, I just made this dummy task that acts like a parallelized make coffee but it doesn’t properly specify dependencies

(defn measure-beans []
 (println "measuring beans")
 (Thread/sleep 1000)) 

(defn grind-beans []
  @(future (measure-beans))
  (println "grinding beans")
  (Thread/sleep 2000))

(defn pour-water []
 (println "pouring water")
 (Thread/sleep 1000)) 

(defn heat-water []
  @(future (pour-water))
  (println "heating water")
  (Thread/sleep 2000))

(defn get-filter []
  (println "getting filter")
  (Thread/sleep 1000))

(defn get-mug []
  (println "getting-mug")
  (Thread/sleep 1000))

(defn make-coffee []
 (let [grounds (future (grind-beans))
       hot-water (future (heat-water))
       filter (future (get-filter))
       mug (future (get-mug))]
   (println "brewing coffee")
   @grounds
   @hot-water
   @filter
   @mug
   (Thread/sleep 3000)))

(defn- main []
 (make-coffee)) 

(main)

@(future (pour-water))

How does make parallelization work when one of the tasks returns with a non-zero exit code?

Yeah I know about the @(future thing 🙂 it’s just there so that I show that i am doing it in another thread, not real code

if anything in the dependency chain fails the failures percolate upward

play with it 🙂

.PHONY: a b c

coffee: grounds hot_water filter mug
	@echo "brewing coffee"
	@sleep 3

grounds: beans
	@echo "grinding beans"
	@sleep 2
	exit 1

hot_water: water
	@echo "heating water"
	@sleep 2

filter:
	@echo "getting filter"
	@sleep 1

mug:
	@echo "getting mug"
	@sleep 1

water:
	@echo "pouring water"
	@sleep 1

beans:
	@echo "measuring beans"
	@sleep 1

I am not sure if/how Make stops ‘parallel’ tasks. I know it doesn’t interrupt them

but say that a depends on b and c, and b depends on b2 and c on c2, if b2 and c2 are running and c2 fails, I am pretty sure that make will stop b from running (after b2)

but also I think that that’s less important

make: *** [beans] Error 1
make: *** Waiting for unfinished jobs....

how tasks are currently implemented, bb would just quit while the others are still running (because it has no concept of parallel jobs yet)

makes sense

I'm trying to use Cursive as IDE to write bb scripts. What library should I include to get babashka.deps resolved?

(require '[babashka.deps :as deps])
Execution error (FileNotFoundException) at tasks/eval17985 (REPL:1).

unfortunately that lib isn't available from the JVM as this is very bb specific and not generally applicable

maybe we could have a special (quit! 1) or so function that would delay the real exit while other tasks are still running

(set-exit-code! 1) something in this vain

yeah, it makes sense that we could have an implicit check before starting a task to see if the whole task tree has been abandoned. I think, actually, that the user shouldn’t have to see/use this. Just implicitly set the global fair when a task fails. would you support kicking off more than one task at a time? Then this failure signal would have to be local to that task chain

I guess throwing an exception would be the better, more supported alternative

Yeah I am thinking abstractly, not really how it would be implemented (I don’t really know Java)

I am just thinking about the semantics and about whether the ‘ui’ would have people set the error and check it.. I think that they should not. It definitely can be considered ugly/less pure but it makes sense to me that if A is kicked off and b2 fails while c2 is running, c2 can finish (`Waiting for jobs to finish`) but c never starts since it implicitly checks if the job (which is A) is still running before starting

@borkdude, thanks. I'm trying to rewrite my app and lib templates using babashka only. I want to throw away just or make utilities from templates.

yeah, I'm thinking how people can signal the flow to stop and which exit code to use, you could do (throw (ex-info "" {:babashka/exit 1})) which would get caught by the consumer of your future and this exit could would then be handled by babashka

this is already how futures work, once you deref them and an exception occurred, then the exception is re-thrown

so for the above you could have:

(def b2 (future ....))
(def c2 (future ....))
(def b (future (do @b ....)))
(def c (future (do @c2 ...)))
(def a (do @b @c ...))

we could make this work relatively easy by letting people just write future around their task and handle the rest automatically. this would give some control about which tasks you want to run in a future and which you don't

sounds cool :)

Hi all. Is there possibility to have the autocomplete on the BB repl? I am running it on the Mac OS shell as rlwrap bb (I might have missed it in the previous discussions).

@dennisa not with the terminal REPL, but with the nrepl server you will get more functionality like that

Great, thanks will give it a try

@grazfather On master. Binary in #babashka-circleci-builds for trying.

you’re a machine!

Just looked at the commit, very cool. I am learning a lot

I love parallelized tasks!

$ time ./bb coffee
measuring beans
grinding beans
pouring water
heating water
getting filter
getting mug
brewing coffee

real    0m11.294s
user    0m0.023s
sys     0m0.042s

$ time ./bb coffeep
measuring beans
grinding beans
pouring water
heating water
getting filter
getting mug
brewing coffee

real    0m5.052s
user    0m0.018s
sys     0m0.018s

Any chance that java.nio.charset.Charset will be supported in the future? I was hoping to use https://github.com/helins/binf.cljc via babashka. That class is the current blocker. I have a custom format that I decode via that library and given I am supporting clojurescript too its nice to call something thats got the bit wrangling for both clojure/script sorted already. It would be great if a little babashka script could just call on my existing regular library clojure/script code to do the decoding rather than having to write a separate path.

to reproduce:

(require '[babashka.deps :as deps]) (deps/add-deps '{:deps {io.helins/binf {:mvn/version "1.0.0-beta1"}}}) (require '[helins.binf :as binf])

@slack1003 Please make an issue and I'll investigate

k will do

https://github.com/babashka/babashka/issues/784

Thanks!

Implemented the `bb run --parallel <task>` option now so tasks run in parallel when they can.

More info: https://github.com/babashka/babashka/discussions/779#discussioncomment-597354