Clojurians Log v2

morning

Good Morning!

Morning

mawning

Moin moin

Do any folks here have much experience with core.async? I've dropped a query into #core-async about something that sounds sensible to me, but might just be utterly insane because I've not used it much before and haven't properly grokked something. Could do with eyes on, and I suspect most of the rest of the channel is currently in a timezone where they're asleep 😉

had a quick look from a general async perspective @carr0t (i don't use core.async so much, but use manifold-streams lots in similar ways) and it looks generally sane. the only thing i would observe is that you probably want an agent rather than an atom of channel registrations - since atoms will retry swap! actions under load

Atomically swaps the value of atom to be:
(apply f current-value-of-atom args). Note that f may be called
multiple times, and thus should be free of side effects.  Returns
the value that was swapped in.

for similar cases we use a slightly different architecture - api handler async sends an event onto a kafka topic and async subscribes to a gnatsd topic for response, separate kafka-streams app consumes event in kafka's synchronous way, when finished puts message on a gnatsd topic, which wakes the api handler to continue - but it adds another architectural component, which is probably worth avoiding if you don't have a need for it (we did have a need - we drive websockets for large numbers of users, and kafka doesn't play with large numbers of topics)

Does that not run the risk of 2 assoc calls (for different keys) running in parallel against the agent and one of them getting lost in the final version? I mean, if we fail to assoc because state has changed or similar we have to retry before we can continue anyway

but, as long as your atom is just a registration of channels with no side effect it's fine

I've not used agents before, so I'm wary of using them 'wrong' 😉

I guess the validator could check that the key exists?

agent serializes the actions

so there will never be two calls running in parallel - they will always run one after the other... but i'm overthinking it - if all your swap! is doing is assoc or dissoc of a chan with no side-effects then it's fine

i'm just a bit scarred by a very non-obvious bug i had which turned out to be because a side-effecting swap! fn was getting repeatedly executed 😬 - but the issue was the side-effect, not the atom

Yeah, I guess there's not much difference between repeated swap! assoc calls until it works, and send assoc followed by await.

yeah, the differences only become apparent when there's a side effect. if you are already async, you don't need to await a send though - you put your continuation action (resolve a promise, send to a chan etc) at the end of your send action, so you get guaranteed serialization of the state change without any blocking

having never really used the STM stuff, serializing side-effecting state updates has been the only time i've ever used an agent in anger

Potentially I don't even need core.async, as @cursork has pointed out. The respond-fn when using ring-jetty-adapter with :async? true already has the context to correctly match up the request. So I could just put the respond-fn for a given request directly into the atom, and have them all executed by the consumer thread. I don't know how much they'd block on sending the response to a servlet over a buffer mind you, but it should be minimal

And probably basically the same as the channel

oh, cool - that's better than introducing a channel

We've ended up going back on that after some more talking it through 😉 The channel gives us a nice mechanism for sending a 504 response to the client and cleaning up expired requests from the atom when a timeout is hit, which putting the entire respond-fn into the atom and responding via the consumer thread doesn't. We could still clean up when we eventually got the message response (assuming a major error downstream didn't mean we never got one), but we wouldn't be able to nicely send the client a 504 if that was a long time after the HTTP timeout

how are you doing your async @carr0t?

What do you mean?

as in are you using raw callbacks, manifold, core.async etc

Current plan is core.async

if you are still at the planning stage (and implementing on the JVM) then for promise-like interactions (i.e. things expected to return single values, rather than streams of values), mpenet/auspex and funcool/promesa are both also worth a look (i'm using ztellman/manifold heavily, but i hesitate to recommend that anymore because it's abandonware)

I hpoe manifold gets adopted as some point

it has nice constructs

I just bought ztellman's book actually and really liking it so far

i agree @dharrigan - promises/deferreds and streams work really nicely together. it has shortcomings too though, mainly around streams - most obviously that there is no error-state for streams (core.async has the same issue) so you end up having to wrap all the manifold.streams fns with versions which interpret errors on the stream

yeah, it could be worked upon to make it better, it would be a shame for it to just stagnate.

Hi! I've been lurking here every now and then. I have been waking up at 5am every day to learn Clojure on my own, hoping that one day it would pay off. Today it has: I've signed my first contract for a role as Clojure developer! 🎉