Clojurians Log v2

To complete the example :-): (data or {}).get("domain", {}).get("name", "bar") That being said these days I end up with a get-in function in python code.

Also note that pmap will very likely run more than 2+cpus tasks at the same time due to chunking: https://github.com/jumarko/clojure-experiments/blob/master/src/clojure_experiments/experiments.clj#L556-L576

the above is a nice addition. I still prefer clojure to python by quite a bit, but python isn't so bad

Hi everyone! 😉 Folks, Node Congress has CFP open. Please submit if have something to say. CFP is open until January 10 Join here — http://bit.ly/CFPNode

same here! ie. python isn't bad but I prefer clojure

Is https://github.com/MastodonC/kixi.stats the most used statistics library for Clojure nowadays?

I am trying to figure out what states a certain state of a state machine can transition to. I am now dealing with data that looks like this:

{:com.fulcrologic.fulcro.ui-state-machines/handler #unknown function (env)
 {return com.fulcrologic.fulcro.ui_state_machines.activate
  (env, new cljs.core.Keyword ("state", "button1", "state/button1", 936328262))}}

it seems you are dealing with an opaque function object and not data?

Do you know of any documentation on this I can use?

documentation on what exactly?

I think I get what you meant now. Thanks for this. I probably need to call the function to figure out the return value

What is the idiomatic way these days to implement a CLI in a project? I'm new to deps.edn - does it allow specifying a default main namespace, like the equivalent of :main in project.clj? I'm envisioning that a user could git-clone the repo and invoke some minimal clojure command to pass control to main method of intended namespace (which actually implements the CLI using tools.cli and provides usage info when invoked with no args, as per usual).

see -M (main) or -X (arbitrary fn) here: https://clojure.org/guides/deps_and_cli#_using_a_main https://clojure.org/reference/deps_and_cli

it looks a bit like javascript?? if you have a javascript function, you can call .toString on it to get it's source and parse that string? is that what you mean?

Thanks. I'm still working my way through all that documentation. So in my example a user would have to know to invoke

clojure -M -m cli

clojure -M

I suppose it's a moot question, once the project were packaged up as a proper release, presumably a jar file with a start script.

in an alias: :main-opts ["-m" "my.ns.with.main" "arg1" "arg2"]

then: clj -M:thealias

@scottbale If your CLI only uses clojure.core and tools.cli and some other commonly used libs, you could also consider babashka, since that is built with this usage in mind.

Another option (for regular JVM Clojure) is just writing a script which you can invoke directly, not with a main. Like here: https://gist.github.com/borkdude/e6f0b12f9352f3375e5f3277d2aba6c9

But typically (with deps.edn) you would write an alias with pre-defined main args like explained above.

Thanks to you both, this is very helpful and exactly what I was wondering about: how is this typically done.

Yes, sometimes, but now it's just a design choice, not a limitation of the paradigm. Key is just a function implemented with:

(defn key
  "Returns the key of the map entry."
  [map-entry]
  (-key map-entry))

I wasn't specifically singling out Python, more OO vs Functional.

My point being, what if you wanted a .get that can handle None or any other type, maybe vector, etc.

In OO, all types would need to agree to share a .get interface, and provide an implementation for it

But also, in this particular case, ya I do find Python's handling of None on .get less then ideal. Think Clojure's handling is much nicer specifically because I think the above is a common source of bug.

And not withstanding, I found this example because it was in our case 😅

Don't think so

Hum, actually it does seem popular. This one is as well: https://generateme.github.io/fastmath/fastmath.stats.html

Can't say which one is most popular though. I'd say both are production ready if that's what you're worried about

I guess it depends who the user you are targeting is

If a Clojure dev, then use an alias. Then they don't even need to git clone or anything

They just add the alias to their deps.edn user config, and now they can use it

@jumar I don't think you're correct here. The parallelization level is restricted by the thread pool it uses, chunking won't change that.

I think the difference is fastmath uses Java implementations under the hood, while kick.stat is fully implemented in Clojure using transducers.

the parallelization is controlled by the lag between the launch of new futures and the deref, it uses future which is an expanding unlimited pool

chunking changes the behavior of (map #(future (f %)) coll) which is what actually creates the threads

so the answer is weird and complicated (another reason I don't like pmap) - chunking causes futures to be launched a chunk at a time, if the input is chunked, otherwise the number of futures in flight is controlled by the lag between future generation and future realization (which is done via the blocking deref)

(defn pmap
  "Like map, except f is applied in parallel. Semi-lazy in that the
  parallel computation stays ahead of the consumption, but doesn't
  realize the entire result unless required. Only useful for
  computationally intensive functions where the time of f dominates
  the coordination overhead."
  {:added "1.0"
   :static true}
  ([f coll]
   (let [n (+ 2 (.. Runtime getRuntime availableProcessors))
         rets (map #(future (f %)) coll)
         step (fn step [[x & xs :as vs] fs]
                (lazy-seq
                 (if-let [s (seq fs)]
                   (cons (deref x) (step xs (rest s)))
                   (map deref vs))))]
     (step rets (drop n rets))))
  ([f coll & colls]
   (let [step (fn step [cs]
                (lazy-seq
                 (let [ss (map seq cs)]
                   (when (every? identity ss)
                     (cons (map first ss) (step (map rest ss)))))))]
     (pmap #(apply f %) (step (cons coll colls))))))

the (drop n rets) creates the lag between creation of new futures and blocking deref to wait on them

breaking a common piece of advice to not mix lazy calculation with procedural side effects

Oh ya, my bad, I was thinking of agent send

I actually never deep dived the impl of pmap, hum..

Doesn't the implementation of step here unchunks?

;; changes to this atom will reported via println

(def snitch (atom 0))

(add-watch snitch :logging
           (fn [_ _ old-value new-value]
             (print (str "total goes from " old-value " to " new-value "\n"))))

(defn exercise
  [coll]
  (doall
   (pmap (fn [x]
           (swap! snitch inc)
           (print (str "processing: " x "\n"))
           (swap! snitch dec)
           @snitch)
         coll)))

user=> (exercise (range 10))
total goes from 3 to 4
total goes from 4 to 5
total goes from 2 to 3
total goes from 1 to 2
total goes from 0 to 1
processing: 0
processing: 4
processing: 2
processing: 3
processing: 1
total goes from 5 to 4
total goes from 4 to 3
total goes from 1 to 0
total goes from 2 to 1
total goes from 3 to 2
total goes from 0 to 1
total goes from 1 to 2
processing: 6
processing: 7
total goes from 2 to 3
total goes from 3 to 4
total goes from 5 to 4
total goes from 4 to 5
processing: 8
total goes from 4 to 3
processing: 9
processing: 5
total goes from 3 to 2
total goes from 2 to 1
total goes from 1 to 0
(0 0 0 0 0 0 3 2 0 0)

Cool

@didibus I am not good enough with lazy-seqs to read the pmap code and know whether it unchunks, so I'm working empirically

Haha, no one is 😛

yeah, here's my version of exercise that captures the max parallelism:

(defn exercise
  [coll]
  (let [biggest (atom 0)]
    (dorun
     (pmap (fn [x]
             (swap! snitch inc)
             (swap! biggest max @snitch)
             (print (str "processing: " x "\n"))
             (swap! snitch dec)
             @snitch)
           coll))
    @biggest))

lmk if that's flawed, but to my eye that will accurately tell you the max futures spawned concurrently by pmap

(nb range is chunked, which is why I'm using it here)

Hum. Ya, looking at the code, its kind of hard to get a full picture. I think the branch of if-let that uses cons will unchunk, but the other branch would not. And the drop n will also trigger the first chunk.

all the retries on that poor little atom make the output with bigger inputs absurd

or maybe that's caused by the printing contention...

Might be better to use a sempahore? I think a lock instead of atom's retry maybe would make this more clear?

(the reason all the prints call str is because otherwise the parts of the prints overlap in the output

hmm

Oh, no I don't think that's what I meant. Whatever the thing that is a locking counter is called

Then again, hum... What if you changed the impl of pmap so that inside the future it incremented and decremented the counter before and after running f ?

that would be the same behavior, with more work to achieve it

hum..

I rewrote to an agent (doesn't retry), the prints are now in intelligible order, the answer is still high (33, 37, 38, 39, 36 ...)

max value in theory is 42 (32 chunk size + 8 processors + 2)

Ya, so that matches my interpretation of the code

The first branch I think unchunks, but the drop is what triggers the first chunk

So instead of getting n parallelization, you get size of first chunk

+n

+n hum..

(when you overlap the next chunk)

Oh boy, that's one confusing little function haha. It does seem like, it was written pre-chunking though, so I guess chunking just wasn't taken into account. Hum, I wonder if that explains why I see poor performance improvements from it in practice, like with chunking, the thread overhead is way too high for parallelization

it launches chunk-size futures, but iterates by nproc+2 delay between reader of input and reader of future values, if your input is big enough to have multiple chunks you can have more than chunk size in flight

that could be - I consider it more like "an example of what you could do to parallelize a specific problem" that happened to make it into the codebase, and it doesn't match most people's problems

reducers are more general, but I haven't used them in anger and haven't seen much usage of them in the wild

Ya, I think having to require their namespace and the fact that only fold is still useful now that we have transducers makes them kind of DOA

I've also built a CLI with https://github.com/l3nz/cli-matic (features on top of tools.cli ) and I had a good experience.

docopt is also an option: https://github.com/nubank/docopt.clj (also works in babashka as a lib)