Clojurians Log v2

What happens if I create too many gos?

a queue somewhere has too many callbacks in it I guess

Will an exception get thrown?

I wonder

How many gos can one create before these unknown consequences happen?

it's probably a heap issue, or maybe there's a bound on the queue

someone else here probably has a real answer, for now I'm experimenting

(dotimes [n 5000000]
  (async/go (async/>! c n)))

java.lang.AssertionError: Assert failed: No more than 1024 pending puts are allowed on a single channel. Consider using a windowed buffer.
                          (< (.size puts) impl/MAX-QUEUE-SIZE)

user=> (def a (atom 0))
#'user/a
(cmd)user=> (dotimes [_ 500000] (>/go (>/timeout 10000) (swap! a inc)))
nil
(ins)user=> @a
500000

adding another 0 to the end of that number made it take 10x as long, but still no error

oh I wasn't taking from the timeout!

that just changes wait time though

oh cool, with enough of those timeouts I was able to get the state machine to start crawling and my CPU is heating up :D

anyway, I bet @ghadi would be much more informative on this subject

experimentally with enough parked go blocks the throughput gets really low and bursty

Any idea if any of the go blocks gets dropped?

the numbers in the atom move slower but everything still looks correct

but that's not real proof of anything, it just means my toy test worked without making core.async break

I think the assertion error you got was because everything was trying to use c

or too many things using some queue that my example doesn't use, clearly

I am seeing something in the cljs implementation of core.async that looks like a serious bug. It appears to me that using a promise-chan will break non-deterministic choice in cljs. Like, if I have an alts! choosing between a timeout and a fulfilled promise-chan, it appears to never choose the timeout chan at all.

Something along the lines of:

(go
  (let [p (promise-chan)
        t (timeout 1000)]
    (>! p {})
    (go-loop []
      (let [[val port] (alts! [p t])]
        (if (= port p)
          (do (println "loop") (recur))
          (println "huh it timed out"))))))

From looking at the implementation, I don't understand why this would be, since the promise chan is just implemented in terms of a special buffer. We ran into this when porting some code from clj into cljs and started seeing some confusing behaviors (basically infinite loops). I'd be happy to be mistaken here but I'm not sure this is user error.

I believe that is a consequence of js being single threaded

I don't understand why, since it has an alts! in the middle, and it's rewritten in terms of being a state machine. This isn't just like a normal tight loop, is it ?

in order for the timeout to fire the control of the js thread needs to be yielded

I mean, to be clear, if you replace the promise chan with a separate go-loop that infinitely puts onto a chan, you don't get this same behavior.

What I am saying is, doesn't alts! yield it?

basically no

it can yield it

but if it can commit to one of the channels without yielding I believe it does so

That seems like it breaks the entire idea of non-deterministic choice, from a CSP perspective, doesn't it ?

It seems like it makes it a challenge to write safe code unless you know the provenance and type of every channel you might be alts!-ing on ...

I am not defending it

(To be clear, I appreciate your help and insight.)

but js is a tricky runtime, and I hate it, and as far as I can tell everyone who writes clojurescript(using core.async or otherwise) is basically teeting on a razors edge of fast behavior for the optimal case and falling off some weird cliff for other cases

but at the same time, if you replaced your random number generator with 5 for non-deterministic choice, I am not sure csp says that is "wrong"

Yeah. It feels pretty dangerous, to be honest. It sucks having code that runs stably for months in the clj version seemingly to 'randomly' hang on the cljs version for mysterious reasons that are hard to understand.

I guess it depends on whether we think non-determinism is allowed to mean 'determinism'.

In any case, it seems to me that if I just stop using promise-chans (or anything else where the buffer might indefinitely supply results), and just use a manually-written go-loop equivalent, the problem won't occur. I hope!

Hey @hiredman, do you happen to know the code well enough to give me a reference to the place where it will choose not to yield? I'd love to read it (although my understanding of the core.async internals is poor at best).

In any case, thanks for the help !

I am actually not sure if it is entirely due to the single threaded nature of js

alt! is non-deterministicly choose the first available, and in your case unless the timeout has expired, a fulfilled promise will always be chosen

as for not yielding, the state machine callback is passed to ioc-alts!, which calls do-alts! and if do-alts! doesn't return nil, ioc-alts! returns :recur which continues the state machine loop

I am less familiar with the cljs side of things though

My understanding is to some degree informed by https://wingolog.org/archives/2017/06/29/a-new-concurrent-ml which isn't about core.async per se, but has some analogous parts. it describes channel operations as either happening in the optimistic case or the pessimistic case. The core.async code that corresponds to the optimistic case results in no yield to the js thread.

Yeah, it may be that my understanding is a bit broken as well, because I hate the idea that I have to know what is on the other end of chans (or what types of chans they are) to have an idea of how my code is likely to behave.

Especially given that chans have no easy mechanism to know their identities or types or anything like that (ie, they do not have names and probably do not implement IMeta). It seems fundamentally more fragile than it needs to be.

I guess? it still isn't clear to me if you are passing an expired timeout in or not, if you are passing in an unexpired timeout, and a promise-chan that has been delivered to, the fact that it picks the promise-chan every time is 100% everything working as expected and as designed, and if you expect otherwise then you don't understand the kind of choice alts! makes

alt chooses the first thing that it can(the first operation that can complete), and if it can choose multiple things it picks one at random

the choosing at random is only done to break a tie, and if one of the things is a take from a timeout channel that isn't expired yet, and the other is an immediately doable take from another channel, there is no tie to break