I have a small problem I want to tackle with clojure.async. I have an API I need to get resources from, but the results are paginated and I can only get the next page after receiving a url from the previous page. I want to get each page, then merge the results with the last page into a map.
(defn get-data-async
[req]
(let [in-c (async/chan)
out-c (async/chan)]
;; Blocking HTTP gets are run in a thread, putting results into in-c as they come in
(async/thread
(loop [current-req req]
(if (nil? current-req)
(async/close! in-c)
(let [result (get-result current-req)]
(async/>!! in-c result)
(recur (next-req result))))))
;; Process data as it arrives into a single map; result is put into out-c
(async/go-loop [result (async/<! in-c)
collector nil]
(if (nil? result)
(async/>! out-c collector)
(recur (async/<! in-c) (merge-result collector result))))
out-c))
Was hoping I could get input on how to make this better. Appreciate any help!Why sending :stop instead of closing the channel, which is the standard way to signal end? While reading <!! If nil you know you should send the collected result.
@holyjak Thanks, fixing my code to close channel instead
Take unfold from https://clojure.atlassian.net/browse/CLJ-1906 rewrite it to produce items on a channel instead of a seq and to treat the generating function as returning a channel
thanks for sharing this!