Clojurians Log v2

So I watched this talk by Michael Sperber the other day on Concurrent ML: https://youtu.be/pf4VbP5q3P0?t=1573 Where he essentially argues that CML is one of the best/forgotten abstractions for concurrent programming (summarising). Towards the end of the talk he tries to draw some comparisons to other approaches, one of which is core.async; where he makes some criticisms of it; that I don’t quite understand. Can anyone offer some wisdom on what the benefit of CML over core.async is, and what exactly is Sperber saying that core.async got wrong?? My interpretation (summary) of what he’s saying seems to be is:

1. Having combinators on read actions but not on any of other actions… What would the other actions be? Presumably putting stuff on channels, what else? What would such combinators look like? Also isn’t alts! a combinator on putting too?!

2. core async complects lightweight concurrency (threads/fibres) with message based concurrency

3. A vague assertion that core.async looks like it might work but in practice it kind of doesn’t

4. It gets backpressure wrong in a way I don’t quite follow… I think he’s saying that there are 3 buffers (a queue of threads trying to send, a buffer on the channel, and a queue on threads trying to receive). What is he arguing here, that there should just be one buffer on every rendezvous not three?!

Sperber seems to be trying to say something here, but I don’t really follow it… Does anyone have any experience with CML, and why it might on some axis at least be superior?

4. he's talking about pending take queue and pending put queue I guess, but there are ways to affect back-pressure he's not mentioning like the ret val of various flavors of put/takes and derivatives (offer, poll etc). You have way to know when flow control has to happen and act on it yourself in some cases, arguably it's not simple but then again it's better to have the knobs than not

core.async is quite low level, maybe that's the complain. There are many ways to shot one-self in the foot and the docs (or lack, thereof) do not help. But overall I do not agree personally with his point. Not sure about the superiority of one approach vs the other, I think it's also quite situational

like actors vs csp, depends on caracteristics of the platform (vm), problem to solve and so on. No silver bullets

I feel like core.async often feels like half of a solution, and it doesn't feel like it's really under very active development. I use it all the time, but it often feels like you have to be quite careful, and there is a lack (in my opinion) of non-trivial examples of its use, especially things showing a realistically robust lifecycle.

I quite like it in current shape, modulo the few bugs. I don't know of a super easy to use solution to these kind of problems personally, erlang is full of traps too for instance and I suspect the same is true for cml and others

I like it quite a lot as well, to be honest. I think if we were able to make progress on some long-standing bugs, and if we just cleaned up or closed old tickets that were never going to be addressed, it would help the impression a bit. Like, tickets like this exist, even though the last time I discussed it with someone they thought it might not even be possible: https://clojure.atlassian.net/projects/ASYNC/issues/ASYNC-27?filter=allissues&orderby=priority%20DESC&keyword=line%20numbers

But from reading that ticket, you might have the impression that it should already be working in CLJ!

I mean, I'm still shocked that logical operators don't short-circuit in go blocks, and it is months since I tripped over that one.

it needs love for sure.

but I'd rather have spec2 first, so I won't complain too much about it 🙂

given the current bus factor

I mean, it's a volunteer project, so it isn't like we can really complain about it anyway. I'm just talking about wishes.

I'll be interested to see where spec gets to eventually, but last time I checked it was very functiony and not very data-y.

I should read up on spec2.

it's debatable. spec2 has a "data" layer, then "forms" might be off-putting but technically it's data 🙂, ultimately there's a predicate at the leaf no matter the approach

but who knows what it's going to look like in the end. Apparently it's still in flux, we ll see

Yeah. I have a quite a bit of faith in their design process. Just need more patience I suppose.

a good read about flow control (it's erlang flavored but a good read) https://ferd.ca/handling-overload.html

Thanks - I will give it a look.

Yeah, I don’t have any particular complaints about core.async, but I only use it in a few places where it’s been great. I did get into Erlang about 15 years ago; but eventually grew to dislike it and the actor model. I definitely like that clojure has multiple idiomatic approaches to concurrency as I’ve previously also felt there’s no one true way to do it. However the CMLers seem to really believe it gets a lot right that other models don’t, and I’m curious what that is and what it might look like… It sounds like it might be a bit like the blub paradox… I suppose I’ll need to buy the book to find out…

WAT?! Really?!

(clojure.core.async/<!! (clojure.core.async/go (or nil (println "hello") 10)))
hello
10

(clojure.core.async/<!! (clojure.core.async/go (or 20 (println "hello") 10)))
20

looks like they do

it's a bug in cljs only iirc

ah… not the end of the world

Yes, my apologies for not being specific enough. Most of the libraries I write at my company where we use core.async end up being used in both clj and cljs, so I tend to muddy these two together. But I don't think the bug is any less severe for only affecting one target platform. You could argue that it's worse because it's a significant behavioral change between the two.

Sure — It’s probably a painful one to debug; but relying on short circuiting to prevent effects isn’t that common, and once you know you can work around easily enough at least.

this is https://clojure.atlassian.net/projects/ASYNC/issues/ASYNC-91 btw

my impression of the main difference between the CML model(I've never actually used CML, but have read papers an played with implementing the api in clojure) and core.async's more CSP style api is that CML's notion of events make first class and explicit things that are kind implicit in core.async. And making events first class allows for all kinds of interesting usage (events as capabilities, composition of events, etc), and makes it possible to extend and add behaviors to existing events using guards, nacks, and wrap

guards and nacks are particularly interesting for integrating with external services and allow you do things like create some kind of event that represents reading a message from rabbitmq, and if someone synchronizes on that event it goes out to fetch a messages, but if the synchronization was part of an alt (select) and that event isn't chosen, automatically nack the fetched message

Thanks, this seems to tally with what he was saying earlier in the talk. I did find a CML implementation on top of core.async… I’ll need to take a look at it: https://github.com/polytypic/poc.cml

first class events make the CML api much more powerful because it sort adds a loop to the graph of opertions

in core.async with a channel, you can >!, <!, or alt, you get some value out

in CML when you take or put you get an event, and select takes events and returns an event

Interesting…

it sure is

So the other thing he says is core.async complects fibres with message based concurrency… I mean sure, I can see that… But to implement lightweight threads; don’t you have to complect them with a communication mechanism?! Presumably CML complects in a similar way too; is it just that its event based mechanism is considered superior?

I dunno, I haven't watched the video, but yes cml is all kinds of complected, it is a whole other dialect of ml with a language runtime written to support the cml api

Sure, though it actually seems most CML implementations aren’t actually in ML… e.g. guile has one, racket, *haskell etc

those languages/runtimes all have green threads independent of their cml implementation I believe

totally, but if you’re implementing green threads (like core.async had to) don’t you also need to implement a way to communicate with them? Put another way what might core.async style green threads look like if decomplected from channels?!

coroutines

of course

thanks for connecting those dots for me

Are there any other implementations of co-routines in clojure that you’re aware of?

the go macro in core.async is more or less writeing the code to make delimited continuations work, the go macro itself establishes a reset, and each channel op is a yield

https://github.com/positronic-solutions/pulley.cps https://github.com/swannodette/delimc

I am sure there are others, I've got one not published and half done

and about 10% of an http2 server written that uses a continuation monad for similar effect

a continuation monad is kind of a neat way to get continuations without needing a complicated macro like core.async, but it does make your code look really weird, and if anything makes the stacktraces even worse

Yeah roll on project loom 🙂

continuation monads are a neat trick, that’s for sure — thanks for the other pointers too 🙇