Clojurians Log v2
core-async
What’s the best way to use clj-http concurrently? I know it supports async out of the box, but it’s callback based and felt a bit clunky. Are there clj-http wrappers that integrates it with core.async API i.e. channels? I feel like it should be quite simple too to write my own wrapper, but I’m new to async clojure so any pointers are welcomed!
You can do a sort of mix of the put! and >!! approaches which is similar to how pipeline-async handles the async function - allocate a channel of size 1 and put! in it, then synchronize on it in whichever way you want
(defn request
[req]
(let [ch (chan 1)]
(http/request
req
(fn [resp]
(put! ch resp)
(close! ch)))
ch))
My point is that people seem to reach to core.async without a good reason.
I often find that using plain future or something like claypoole lib is more than enough (simpler, easier, more straightforward)
@noisesmith
> also re: futures, you do not need that at all, clj-http already manages a threadpool it uses for its requests
Do you mean the connection thread pool? How that helps in not blocking the client until the request is done without using something like future ?
Perhaps you meant the :async / :async? option? https://github.com/dakrone/clj-http#async-http-request
(but it seems those provide only callback style APIs)
> people seem to reach to core.async without a good reason. That is true
@jumar I misunderstood you, and yes, I meant the async option (which would be impossible without the thread pool)
I articulated it poorly, but my thought was that you are already using a thread pool for the requests, you don't need thread allocation if you are using that lib, you just need management of the results
I’m looking to achieve something like JS Promise.all, I have a usecase with hundreds of requests that needs to be fired, and it’s proving to be the bottleneck right now. My idea was that core.async would have such facilities. I might be wrong but where else in clojure should I look?
for promise-like things (i.e. single values rather than streams) i generally find promise interfaces easier to use - i've often used funcool/promesa , which is a nice wrapper around java's CompletableFuture (and js/promises in cljs-land) - https://funcool.github.io/promesa/latest/promesa.core.html#var-all
mpent/auspex is another option based on CompletableFuture, but i haven't yet used that
thanks promesa seems to be what I’m looking for
https://github.com/exoscale/telex works with auspex out of the box fyi
it's quite easy to parallelise work if that's what you're after
auspex is quite similar to promesa, promesa attempts to work a bit everywhere (jvm, cljs), auspex is conceptually simpler but under the hood it's using the same java machinery on the jvm
(auspex is also designed to make porting manifold code very easy, the api is nearly 1-1)
in my experience using put! inside the callback to a clj-http call is all it takes to integrate seamlessly with core.async
or perhaps >!! instead, depending on whether you'd rather error on heavy load vs. blocking and waiting it out
also re: futures, you do not need that at all, clj-http already manages a threadpool it uses for its requests
(and core.async/thread integrates better with go blocks when you do need to spawn threads, since it returns a channel you can park on)