To complete the example :-):
(data or {}).get("domain", {}).get("name", "bar")
That being said these days I end up with a get-in
function in python code.
Also note that pmap will very likely run more than 2+cpus tasks at the same time due to chunking: https://github.com/jumarko/clojure-experiments/blob/master/src/clojure_experiments/experiments.clj#L556-L576
the above is a nice addition. I still prefer clojure to python by quite a bit, but python isn't so bad
Hi everyone! 😉 Folks, Node Congress has CFP open. Please submit if have something to say. CFP is open until January 10 Join here — http://bit.ly/CFPNode
same here! ie. python isn't bad but I prefer clojure
Is https://github.com/MastodonC/kixi.stats the most used statistics library for Clojure nowadays?
I am trying to figure out what states a certain state of a state machine can transition to. I am now dealing with data that looks like this:
{:com.fulcrologic.fulcro.ui-state-machines/handler #unknown function (env)
{return com.fulcrologic.fulcro.ui_state_machines.activate
(env, new cljs.core.Keyword ("state", "button1", "state/button1", 936328262))}}
I need to extract the "state/button1" from this data. Does anyone have tips on how to do that?it seems you are dealing with an opaque function object and not data?
It's CLJS
Do you know of any documentation on this I can use?
documentation on what exactly?
I think I get what you meant now. Thanks for this. I probably need to call the function to figure out the return value
What is the idiomatic way these days to implement a CLI in a project? I'm new to deps.edn
- does it allow specifying a default main namespace, like the equivalent of :main
in project.clj
? I'm envisioning that a user could git-clone the repo and invoke some minimal clojure
command to pass control to main method of intended namespace (which actually implements the CLI using tools.cli
and provides usage info when invoked with no args, as per usual).
@scottbale You may pick my template for app using cli https://github.com/redstarssystems/app-template
1This template adapted for IDEA. Use make for control.
see -M (main) or -X (arbitrary fn) here: https://clojure.org/guides/deps_and_cli#_using_a_main https://clojure.org/reference/deps_and_cli
it looks a bit like javascript?? if you have a javascript function, you can call .toString
on it to get it's source and parse that string? is that what you mean?
Thanks. I'm still working my way through all that documentation. So in my example a user would have to know to invoke
clojure -M -m cli
at a minimum. My question is, can the cli
namespace somehow be specified in deps.edn
? Is there some even more minimal clojure
command like
clojure -M
or like a way to have project specific usage resulting from clojure -?
. Or is it just not designed|intended for that?I suppose it's a moot question, once the project were packaged up as a proper release, presumably a jar file with a start script.
in an alias: :main-opts ["-m" "my.ns.with.main" "arg1" "arg2"]
then: clj -M:thealias
@scottbale If your CLI only uses clojure.core and tools.cli and some other commonly used libs, you could also consider babashka, since that is built with this usage in mind.
1👍Another option (for regular JVM Clojure) is just writing a script which you can invoke directly, not with a main. Like here: https://gist.github.com/borkdude/e6f0b12f9352f3375e5f3277d2aba6c9
But typically (with deps.edn) you would write an alias with pre-defined main args like explained above.
Thanks to you both, this is very helpful and exactly what I was wondering about: how is this typically done.
Yes, sometimes, but now it's just a design choice, not a limitation of the paradigm. Key is just a function implemented with:
(defn key
"Returns the key of the map entry."
[map-entry]
(-key map-entry))
If it wanted, it could handle nil in any way.I wasn't specifically singling out Python, more OO vs Functional.
My point being, what if you wanted a .get that can handle None or any other type, maybe vector, etc.
In OO, all types would need to agree to share a .get interface, and provide an implementation for it
But also, in this particular case, ya I do find Python's handling of None on .get less then ideal. Think Clojure's handling is much nicer specifically because I think the above is a common source of bug.
And not withstanding, I found this example because it was in our case 😅
Don't think so
Hum, actually it does seem popular. This one is as well: https://generateme.github.io/fastmath/fastmath.stats.html
Can't say which one is most popular though. I'd say both are production ready if that's what you're worried about
I guess it depends who the user you are targeting is
If a Clojure dev, then use an alias. Then they don't even need to git clone or anything
They just add the alias to their deps.edn user config, and now they can use it
1👍1@jumar I don't think you're correct here. The parallelization level is restricted by the thread pool it uses, chunking won't change that.
I think the difference is fastmath uses Java implementations under the hood, while kick.stat is fully implemented in Clojure using transducers.
the parallelization is controlled by the lag between the launch of new futures and the deref, it uses future which is an expanding unlimited pool
chunking changes the behavior of (map #(future (f %)) coll)
which is what actually creates the threads
so the answer is weird and complicated (another reason I don't like pmap) - chunking causes futures to be launched a chunk at a time, if the input is chunked, otherwise the number of futures in flight is controlled by the lag between future generation and future realization (which is done via the blocking deref
)
(defn pmap
"Like map, except f is applied in parallel. Semi-lazy in that the
parallel computation stays ahead of the consumption, but doesn't
realize the entire result unless required. Only useful for
computationally intensive functions where the time of f dominates
the coordination overhead."
{:added "1.0"
:static true}
([f coll]
(let [n (+ 2 (.. Runtime getRuntime availableProcessors))
rets (map #(future (f %)) coll)
step (fn step [[x & xs :as vs] fs]
(lazy-seq
(if-let [s (seq fs)]
(cons (deref x) (step xs (rest s)))
(map deref vs))))]
(step rets (drop n rets))))
([f coll & colls]
(let [step (fn step [cs]
(lazy-seq
(let [ss (map seq cs)]
(when (every? identity ss)
(cons (map first ss) (step (map rest ss)))))))]
(pmap #(apply f %) (step (cons coll colls))))))
the (drop n rets)
creates the lag between creation of new futures and blocking deref to wait on them
breaking a common piece of advice to not mix lazy calculation with procedural side effects
Oh ya, my bad, I was thinking of agent send
I actually never deep dived the impl of pmap, hum..
Doesn't the implementation of step here unchunks?
;; changes to this atom will reported via println
(def snitch (atom 0))
(add-watch snitch :logging
(fn [_ _ old-value new-value]
(print (str "total goes from " old-value " to " new-value "\n"))))
(defn exercise
[coll]
(doall
(pmap (fn [x]
(swap! snitch inc)
(print (str "processing: " x "\n"))
(swap! snitch dec)
@snitch)
coll)))
user=> (exercise (range 10))
total goes from 3 to 4
total goes from 4 to 5
total goes from 2 to 3
total goes from 1 to 2
total goes from 0 to 1
processing: 0
processing: 4
processing: 2
processing: 3
processing: 1
total goes from 5 to 4
total goes from 4 to 3
total goes from 1 to 0
total goes from 2 to 1
total goes from 3 to 2
total goes from 0 to 1
total goes from 1 to 2
processing: 6
processing: 7
total goes from 2 to 3
total goes from 3 to 4
total goes from 5 to 4
total goes from 4 to 5
processing: 8
total goes from 4 to 3
processing: 9
processing: 5
total goes from 3 to 2
total goes from 2 to 1
total goes from 1 to 0
(0 0 0 0 0 0 3 2 0 0)
max parallelism here is 5 - I'm going to try a version where I capture the max and exercise it more aggressivelyCool
@didibus I am not good enough with lazy-seqs to read the pmap code and know whether it unchunks, so I'm working empirically
Haha, no one is 😛
yeah, here's my version of exercise that captures the max parallelism:
(defn exercise
[coll]
(let [biggest (atom 0)]
(dorun
(pmap (fn [x]
(swap! snitch inc)
(swap! biggest max @snitch)
(print (str "processing: " x "\n"))
(swap! snitch dec)
@snitch)
coll))
@biggest))
(exercise (range 1000))
prints a lot more than I'm going to paste here, and returns 19lmk if that's flawed, but to my eye that will accurately tell you the max futures spawned concurrently by pmap
(nb range is chunked, which is why I'm using it here)
Hum. Ya, looking at the code, its kind of hard to get a full picture. I think the branch of if-let that uses cons will unchunk, but the other branch would not. And the drop n will also trigger the first chunk.
all the retries on that poor little atom make the output with bigger inputs absurd
or maybe that's caused by the printing contention...
Might be better to use a sempahore? I think a lock instead of atom's retry maybe would make this more clear?
(the reason all the prints call str
is because otherwise the parts of the prints overlap in the output
hmm
Oh, no I don't think that's what I meant. Whatever the thing that is a locking counter is called
Then again, hum... What if you changed the impl of pmap so that inside the future it incremented and decremented the counter before and after running f ?
that would be the same behavior, with more work to achieve it
hum..
I rewrote to an agent (doesn't retry), the prints are now in intelligible order, the answer is still high (33, 37, 38, 39, 36 ...)
max value in theory is 42 (32 chunk size + 8 processors + 2)
Ya, so that matches my interpretation of the code
The first branch I think unchunks, but the drop is what triggers the first chunk
So instead of getting n parallelization, you get size of first chunk
+n
+n hum..
(when you overlap the next chunk)
Oh boy, that's one confusing little function haha. It does seem like, it was written pre-chunking though, so I guess chunking just wasn't taken into account. Hum, I wonder if that explains why I see poor performance improvements from it in practice, like with chunking, the thread overhead is way too high for parallelization
it launches chunk-size futures, but iterates by nproc+2 delay between reader of input and reader of future values, if your input is big enough to have multiple chunks you can have more than chunk size in flight
that could be - I consider it more like "an example of what you could do to parallelize a specific problem" that happened to make it into the codebase, and it doesn't match most people's problems
reducers are more general, but I haven't used them in anger and haven't seen much usage of them in the wild
Ya, I think having to require their namespace and the fact that only fold is still useful now that we have transducers makes them kind of DOA
I've also built a CLI with https://github.com/l3nz/cli-matic (features on top of tools.cli
) and I had a good experience.
docopt is also an option: https://github.com/nubank/docopt.clj (also works in babashka as a lib)
1👀1✔️