Clojurians Log v2

Hmm, that might be it. But what would that mean? Like observing a half-set value? What would that even be?

Maybe because of this: https://docs.oracle.com/javase/tutorial/essential/concurrency/memconsist.html

I.e. same reason why we mark variables as volatile.

Exactly, you need something that orders both reads and writes respective of each other, otherwise the jvm can do things like read the array index once and cache it in a register, and just say your writes all happened after the read

@jumar locking emits a memory fencing instruction prevents operation reordering and that makes sure your CPU caches are synced. One thread might update a value in L1 cache (which are per-core) then another thread on another core might read the same value in it’s own L1 cache. Typically memory fence causes the changes to get pushed to L3 cache which isn’t per core. writing a volatile does the same, so generally for scalar values (int, long, writing a reference), volatile is sufficient

Ah right, so the read lock is there only to provide a fresh value - otherwised it could get cached; I think it's unlikely to happen here (I increment the array values in 100 concurrent threads, then read them all afterwards), maybe because the cache coherence protocol will actually fetch the proper value when it's modified by the aset operation (even when there's no lock in aget) I definitely couldn't find any consistency issue when removing the aget lock and testing it (https://github.com/jumarko/clojure-experiments/blob/master/src/clojure_experiments/books/joy_of_clojure/ch10_mutation_and_concurrency.clj#L286-L290)

Such bugs are notoriously difficult to test for. Sometimes you may catch them with such tests, but there is no guarantee you will

Yeah, based on my understanding of JMM and memory consistency properties (https://docs.oracle.com/en/java/javase/15/docs/api/java.base/java/util/concurrent/package-summary.html#MemoryVisibility) they do the right thing in the book; in particular: > Actions prior to "releasing" synchronizer methods such as Lock.unlock, Semaphore.release, and CountDownLatch.countDown happen-before actions subsequent to a successful "acquiring" method such as Lock.lock, Semaphore.acquire, Condition.await, and CountDownLatch.await on the same synchronizer object in another thread. Here are some good resources dealing with more details regarding the notion of volatile et al meaning "flush to main memory" (which was the impression I got from reading some Java book a decade ago but found much later that this is likely false when reading about the MESI cache coherence protocol): • https://stackoverflow.com/questions/1850270/memory-effects-of-synchronization-in-java • https://mechanical-sympathy.blogspot.com/2013/02/cpu-cache-flushing-fallacy.html • https://stackoverflow.com/questions/42746793/does-a-memory-barrier-ensure-that-the-cache-coherence-has-been-completed/42750844#42750844

@jumar locks or not, there's a race condition here because the sequence can be constructed before the threads are done mutating. Look at the result of (-> (make-safe-array Integer/TYPE 8) (doto pummel) seq)

Oh yeah, you're right. I think they basically rely on the reader waiting until the threads are done (which is quick for a human experimenting in the REPL 🙂 ). ... in which case, I think, the read lock basically doesn't matter at all but would be the right thing to do for an operation happening immediately after a previous aset, right?

there's many possible reasons why you could see the latest value without explicit synchronization, but in general physical time is not something you should rely on

Yes, sometimes, but now it's just a design choice, not a limitation of the paradigm. Key is just a function implemented with:

(defn key
  "Returns the key of the map entry."
  [map-entry]
  (-key map-entry))

the above is a nice addition. I still prefer clojure to python by quite a bit, but python isn't so bad

same here! ie. python isn't bad but I prefer clojure

I wasn't specifically singling out Python, more OO vs Functional.

My point being, what if you wanted a .get that can handle None or any other type, maybe vector, etc.

In OO, all types would need to agree to share a .get interface, and provide an implementation for it

But also, in this particular case, ya I do find Python's handling of None on .get less then ideal. Think Clojure's handling is much nicer specifically because I think the above is a common source of bug.

And not withstanding, I found this example because it was in our case 😅

I think we understood and agreed with your point, but we didn't think that the comparison was fair. In practice (at least on python codebases I worked on) that python code would look like: get-in(data, ('domain', 'name'), 'bar') or get-in(data, 'domain.name', 'bar') which doesn't compare that unfavourably to (get-in data ["domain", "name"], "bar") as your initial example.

It's possible, no one on our team is really a pro at Python, more like learned at university or picked it up here and there. This code is in a script file part of our infra, so it also doesn't get the same level of code review scrutiny and all. I can't seem to find get-in though? Is that from a popular library?

If so, I think it demonstrates my point pretty well, and I'd be curious to look at the implementation. My guess is get-in is a function that people create for this very problem. Instead of adding a method to Dictionaries and None, if people have found the need to change get from a method to a function, that would be a good example of what I'm talking about. In Python, you could argue that you want a null error to be thrown, maybe you prefer the fail fast, and if you didn't explicitly handle null, maybe you consider a null appearing a bug that you'd want to know about. So that can be a design choice, what do you do with data being None? And while I like that Clojure has get handle nil by default, I don't want to say that throwing a null error if get encounters a null is necessarily worse or bad. But, in OOP, you actually can't do anything about it if you did want to handle this case the way Clojure does. That's because of how methods work versus functions. If the type is wrong, the methods won't exist. All you can do is add the method to more and more types, but even then, there's always a chance a type shows up that doesn't have the method, and you get an error again. That's one of the Functional advantages in my opinion. Which you could also do in Python, since it has Functions, you could make get a function and do this.

@jumar I don't think you're correct here. The parallelization level is restricted by the thread pool it uses, chunking won't change that.

the parallelization is controlled by the lag between the launch of new futures and the deref, it uses future which is an expanding unlimited pool

chunking changes the behavior of (map #(future (f %)) coll) which is what actually creates the threads

so the answer is weird and complicated (another reason I don't like pmap) - chunking causes futures to be launched a chunk at a time, if the input is chunked, otherwise the number of futures in flight is controlled by the lag between future generation and future realization (which is done via the blocking deref)

(defn pmap
  "Like map, except f is applied in parallel. Semi-lazy in that the
  parallel computation stays ahead of the consumption, but doesn't
  realize the entire result unless required. Only useful for
  computationally intensive functions where the time of f dominates
  the coordination overhead."
  {:added "1.0"
   :static true}
  ([f coll]
   (let [n (+ 2 (.. Runtime getRuntime availableProcessors))
         rets (map #(future (f %)) coll)
         step (fn step [[x & xs :as vs] fs]
                (lazy-seq
                 (if-let [s (seq fs)]
                   (cons (deref x) (step xs (rest s)))
                   (map deref vs))))]
     (step rets (drop n rets))))
  ([f coll & colls]
   (let [step (fn step [cs]
                (lazy-seq
                 (let [ss (map seq cs)]
                   (when (every? identity ss)
                     (cons (map first ss) (step (map rest ss)))))))]
     (pmap #(apply f %) (step (cons coll colls))))))

the (drop n rets) creates the lag between creation of new futures and blocking deref to wait on them

breaking a common piece of advice to not mix lazy calculation with procedural side effects

Oh ya, my bad, I was thinking of agent send

I actually never deep dived the impl of pmap, hum..

Doesn't the implementation of step here unchunks?

;; changes to this atom will reported via println

(def snitch (atom 0))

(add-watch snitch :logging
           (fn [_ _ old-value new-value]
             (print (str "total goes from " old-value " to " new-value "\n"))))

(defn exercise
  [coll]
  (doall
   (pmap (fn [x]
           (swap! snitch inc)
           (print (str "processing: " x "\n"))
           (swap! snitch dec)
           @snitch)
         coll)))

user=> (exercise (range 10))
total goes from 3 to 4
total goes from 4 to 5
total goes from 2 to 3
total goes from 1 to 2
total goes from 0 to 1
processing: 0
processing: 4
processing: 2
processing: 3
processing: 1
total goes from 5 to 4
total goes from 4 to 3
total goes from 1 to 0
total goes from 2 to 1
total goes from 3 to 2
total goes from 0 to 1
total goes from 1 to 2
processing: 6
processing: 7
total goes from 2 to 3
total goes from 3 to 4
total goes from 5 to 4
total goes from 4 to 5
processing: 8
total goes from 4 to 3
processing: 9
processing: 5
total goes from 3 to 2
total goes from 2 to 1
total goes from 1 to 0
(0 0 0 0 0 0 3 2 0 0)

Cool

@didibus I am not good enough with lazy-seqs to read the pmap code and know whether it unchunks, so I'm working empirically

Haha, no one is 😛

yeah, here's my version of exercise that captures the max parallelism:

(defn exercise
  [coll]
  (let [biggest (atom 0)]
    (dorun
     (pmap (fn [x]
             (swap! snitch inc)
             (swap! biggest max @snitch)
             (print (str "processing: " x "\n"))
             (swap! snitch dec)
             @snitch)
           coll))
    @biggest))

lmk if that's flawed, but to my eye that will accurately tell you the max futures spawned concurrently by pmap

(nb range is chunked, which is why I'm using it here)

Hum. Ya, looking at the code, its kind of hard to get a full picture. I think the branch of if-let that uses cons will unchunk, but the other branch would not. And the drop n will also trigger the first chunk.

all the retries on that poor little atom make the output with bigger inputs absurd

or maybe that's caused by the printing contention...

Might be better to use a sempahore? I think a lock instead of atom's retry maybe would make this more clear?

(the reason all the prints call str is because otherwise the parts of the prints overlap in the output

hmm

Oh, no I don't think that's what I meant. Whatever the thing that is a locking counter is called

Then again, hum... What if you changed the impl of pmap so that inside the future it incremented and decremented the counter before and after running f ?

that would be the same behavior, with more work to achieve it

hum..

I rewrote to an agent (doesn't retry), the prints are now in intelligible order, the answer is still high (33, 37, 38, 39, 36 ...)

max value in theory is 42 (32 chunk size + 8 processors + 2)

Ya, so that matches my interpretation of the code

The first branch I think unchunks, but the drop is what triggers the first chunk

So instead of getting n parallelization, you get size of first chunk

+n

+n hum..

(when you overlap the next chunk)

Oh boy, that's one confusing little function haha. It does seem like, it was written pre-chunking though, so I guess chunking just wasn't taken into account. Hum, I wonder if that explains why I see poor performance improvements from it in practice, like with chunking, the thread overhead is way too high for parallelization

it launches chunk-size futures, but iterates by nproc+2 delay between reader of input and reader of future values, if your input is big enough to have multiple chunks you can have more than chunk size in flight

that could be - I consider it more like "an example of what you could do to parallelize a specific problem" that happened to make it into the codebase, and it doesn't match most people's problems

reducers are more general, but I haven't used them in anger and haven't seen much usage of them in the wild

Ya, I think having to require their namespace and the fact that only fold is still useful now that we have transducers makes them kind of DOA

Well, maybe this chunking behavior is actually a blessing in disguise? Now it means using this re-chunk function:

(defn re-chunk [n xs]
  (lazy-seq
   (when-let [s (seq (take n xs))]
     (let [cb (chunk-buffer n)]
       (doseq [x s] (chunk-append cb x))
       (chunk-cons (chunk cb) (re-chunk n (drop n xs)))))))

(dorun (pmap (fn[_] (Thread/sleep 100)) (re-chunk 1 (range 1000)))) Will give you ~2+cores (dorun (pmap (fn[_] (Thread/sleep 100)) (re-chunk 100 (range 1000)))) Will give you ~100

Not sure what to think about this. It probably just be nice if pmap was re-written to unchunk and take the number of cores+2 or an optional n.

I have the same feeling and that’s why I created map-throttled in the repo; but it’s for a very specific use case. In most cases it’s better to use Executors or claypoole