core-async

chrisulloa 2019-12-26T18:03:13.060400Z

I have a small problem I want to tackle with clojure.async. I have an API I need to get resources from, but the results are paginated and I can only get the next page after receiving a url from the previous page. I want to get each page, then merge the results with the last page into a map.

(defn get-data-async
  [req]
  (let [in-c (async/chan)
        out-c (async/chan)]
    ;; Blocking HTTP gets are run in a thread, putting results into in-c as they come in
    (async/thread
     (loop [current-req req]
       (if (nil? current-req)
         (async/close! in-c)
         (let [result (get-result current-req)]
           (async/>!! in-c result)
           (recur (next-req result))))))
    ;; Process data as it arrives into a single map; result is put into out-c
    (async/go-loop [result (async/<! in-c)
                    collector nil]
      (if (nil? result)
        (async/>! out-c collector)
        (recur (async/<! in-c) (merge-result collector result))))
    out-c))
Was hoping I could get input on how to make this better. Appreciate any help!

Jakub Holý 2019-12-27T17:53:18.065Z

Why sending :stop instead of closing the channel, which is the standard way to signal end? While reading <!! If nil you know you should send the collected result.

👍 1
chrisulloa 2019-12-27T18:29:52.065400Z

@holyjak Thanks, fixing my code to close channel instead

2019-12-26T18:29:02.062900Z

Take unfold from https://clojure.atlassian.net/browse/CLJ-1906 rewrite it to produce items on a channel instead of a seq and to treat the generating function as returning a channel

🆒 1
chrisulloa 2019-12-26T18:30:12.063100Z

thanks for sharing this!