Clojurians Log v2

purely for learning purpose why does

(require '[clojure.core.async :refer [go >!! close! <!! >! <! thread]])

(def c (chan))

(do
  (go
    (do
      (doseq [i (range 1000000)]
        (>! c i))
      (close! c)))

  (go
    (time
     (loop [i (<! c)]
       (when i
         (recur (<! c)))))))

(require '[clojure.core.async :refer [go >!! close! <!! >! <! thread]])

(def c (chan))

(do
  (thread
    (do
      (doseq [i (range 1000000)]
        (>!! c i))
      (close! c)))

  (thread
    (time
     (loop [i (<!! c)]
       (when i
         (recur (<!! c)))))))

for the go example, it could theoretically just bounce back and forth between pushing and pulling on the same thread.

ya that’s what i thought too but the thread ids were surprisingly different

Interesting. not sure how to investigate further without using a profiler like https://github.com/clojure-goes-fast/clj-async-profiler

at a low level a process / thread context switch is much more expensive than a goroutine state machine context switch

for example the saving of state needs to save all state when doing an OS context switch, while go reuses most of its state and just switches out state machines to pick up different blocks. even if multiple thread ids are being used, it's a fixed number of them with cooperative context switching, and it never spends time in OS allocated work slices waiting on a signal like thread can.

Rather a newbie-flavoured question here re: combining core.async with Stuart Sierra’s Component machinery. Which would be better style? (i) Have one of the component modules own all the channels in the system, and do (chan) and (close!) on them all, with other components referring to them when firing up their (go) blocks (ii) Assuming a pipeline of consume-produce components, have each one create and close the channel it sends to only (which it can be assumed to “own”)?

I would not do that

I think you got in as I accidentally posted … I went back and edited.

yes, I would do i

i is making a global singleton

Hmm, (ii) feels better structured to me, but (i) feels simpler and more reliable.

Bonus question: given that components are firing up their own go threads/coroutines, should they also be responsible for shutting them down? Or should the go blocks be required to exit on closed input channels, so that close! everywhere brings them down? I’m veering towards the latter: close the channels to shut down the go blocks.

i is only more reliable if you don't understand that component does things synchronously, so if you have asynchronous tasks (via go blocks, threads, threadpools, whatever) you need to bridge that divide

components must be responsible for shutting them down

your stop function shouldn't return until all the async tasks you have started have exited (it is not enough to signal them to stop)

(Aside: I’m in CLJS so it’s all coroutined.) The only way to ensure a shutdown then is to make every go block hang on an alt! and have explicit shutdown channels. That feels like a lot of machinery to achieve something which feels like it should be simpler.

Unless I’m missing an obvious pattern to do that.

that is not entirely correct

Am happy to be enlightened…!

it is often easier to use an explicit shutdown channel, but you can make all your go blocks check for reading nil from the input channel

oh, in cljs you are screwed anyway

Yes - they do - so a closed channel will always bring down a go-block consumer.

you can't bridge the async -> sync divide

so in clj the component's stop will close the input, which will signal the go block to exit, and then do a blocking take from the channel returned when that go block is started, to ensure that after the component is stopped the go loop has exited

(if you just close the channel, your component may return from stop while the go loop is processing a back log of messages in the channel)

of course you can't do blocking takes in cljs

I was about to say…!

There’s no way to block on a channel from within the “main thread” in component, so I guess all I can do is arrange for go blocks to stop on their next read-from-closed, and be content with that.

the "correct" thing would be to write your own version of component where the lifecycle protocol is itself asynchronous

😐

Sometimes something can be too correct!

OK, so it feels like I have to live with some decoupling here - I can shut down a component, and make it shut channels, but have to code the go blocks so that they obligingly shut themselves down when the thread jumps to them.

Am happy to go with a singleton channel “owner” which mints fresh ones on start and closes them all on stop. I don’t think that leads to any deadlocks or orphaned go contexts.

Thanks very much for the discussion - it’s been illuminating.

"orphaned go contexts" - IIRC go blocks are stored on channels, if they aren't running or waiting on a channel they go out of scope and get gc'd

I believe so - they’re just parked contexts. But I think I read something recently that hinted that they came from a pool. As I think about it, that doesn’t make sense to me - I don’t see any reason why they’d need to be limited.

threads are pooled, the blocks are "unlimited"

Sure - though in CLJS I don’t have threads and am just jumping around inside a coroutine state machine, so I guess there are no limits of any kind.

well "only one thread ever exists" is a bit of a limit 😄

True - but once you get over that, the sky’s the limit.