Clojurians Log v2

Yeah, I'm probably going to be building something that uses a/map to implement something that has the semantics of mix, because you can't exactly add inputs to a map, although you can create another map that is composed of submaps

I am trying to a/put! all incoming messages from an external API onto a chan. Ususlly consumers have no issues keeping up with producers but occasionally there will be huge bursts of 10k messages from a single producer. I cannot discard any of these messages. What are the usual strategies for dealing with this? Just make my buffer size large enough to account for the largest burst? Implement some sort of logic to group the messages prior to the put!?

the typical amount of messages I'm dealing with is less than 10 per second, and the 10k bursts are rare but predictable, so it doesn't feel quite right to just set the buffer size to 10k, but I could be wrong

I would just do that and have a big buffer to account for the bursts, but > Implement some sort of logic to group the messages prior to the put!? depending on how you are implementing the producer. You mentioned is an external API, are you requesting from it? Or your program has the API and each request gets somehow feeded into core.async?

I am requesting from the API, and it answers asynchronously

How do you receive the answers?

on a socket. So I just wait for things on the socket and put them on a chan

I should also note that this is an attempt at a library, so I'm hesitant to make the decision for end users regarding buffer size when only certain types of requests require a large buffer for responses. So I guess the best way around this is to allow the users to just pass the whole chan they require themselves with whatever buffer requirements they need

I just wasn't sure if there was some fancy buffer strategy for these type of bursty situations

Not that I am aware of, I can think of partitions, debounces, and thinks like that, but are more complex than just a big buffer. I’d go with accepting chan as parameter and let the user choose

ok great, thanks!

wait.... why doesn't this work? when are transducers on channels applied?

(go (>! (chan 1 (map (fnil identity ::NOTHING))) nil))

that returns a "can't put nil on a channel" error

but it seems like it should definitely not do that

nils are not valid channel values

you're putting nil on the channel

transducers are applied after you put it on the channel and before you take it off

with some intentional hand-waving about whether it's the producer or consumer thread (as both can occur)

after I put it on the channel and before I take it off? So, does that mean its applied as at the beginning of the put take operation?

oh huh

how about in cljs?

no threads

Same deal.

same semantics

less hand-waving :)

A user of a channel knows not that there is a transducer

hahaha, so timing wise, does it occur during the take operation?

it happens during put

but after the put

so its part of the put operation, and it happens after its already on the channel

Sorta yeah. The transducer might be a filter that avoids calling the internal put

it does not

So it can't happen during take

as I just verified

I was using it to avoid calling the internal put, but that does not work

?

if that was so, this would work (go (>! (chan 1 (map (fnil identity ::NOTHING))) nil))

err

wait

hahaha

sorry

i meant this:

(go (>! (chan 1 (filter some?)) nil))

you can't >! or put! nil, fundamentally

no matter what the channel arg is

it's an invariant of the system

right, but if the filter occurred before the put operation, then it would never reach that invariant

my expectation is that it simply would never reach out to the transducing process (the put operation)

https://github.com/clojure/core.async/blob/master/src/main/clojure/clojure/core/async/impl/channels.clj#L291 the transducer transforms the internal buffer add operation

https://github.com/clojure/core.async/blob/master/src/main/clojure/clojure/core/async/impl/channels.clj#L67-L70 values are checked at the door

ahh, gotcha

hmm. I mean I guess you could just actually use the put operation as the terminal step in a transducer, then...

you could do a lot of things 🙂

hahaha, that you could! thanks for the clarification, its really helpful