Clojurians Log v2

We are considering adding a new function to Clojure called iteration, and it would be very useful for consuming paginated APIs like Amazon's

(defn log-groups
  "gets all CloudWatch Log Groups"
  [client]
  (->> (iteration
        (fn [token]
          (aws/invoke client (cond-> {:op :DescribeLogGroups}
                               token (assoc :request {:nextToken token}))))
        :kf :nextToken
        :vf :logGroups)
       (into [] cat)))

https://clojure.atlassian.net/browse/CLJ-2555

(still WIP, reserve the right to not do it, ymmv, etc.)

Nice. First thought is how do I handle anomalies? Sometimes (most times) I'd like to essentially have a reduced if an anomaly occurs, returning the anomaly.

the iteration above will terminate on anomalies (because :kf won't return anything from an anomaly) -- the (into [] cat) is swallowing it though

you take out :vf or pass in a smarter :vf

then could check for the anomaly on the outside

(into [] (halt-when anomaly? retf) (iteration....))

halt-when will return the failing input when the retf isn't given

Hmm, I see. Clever. Looking at the pagination code we are using, the only other thing we keep track of is page-count. In some cases we use that for logging when certain page counts are hit for long running iterations & other cases for capping pages followed. Seems like you can do that here by returning a different data structure from :kf. I think this would cover our use case.

probably better to pass in a smarter :vf (for the anomaly case)

The only other comment is I almost always prefer apis where the function takes its options in a map instead of the variadic approach. Easer to programmatically build up when needed. Was there a reason you chose the variadic args?

I suspect that this isn't going to be a case where calls are programmatically built up, but I'll take that feedback back.

my suggestion for halt-when isn't useful for (into []) but is for transduce

user=> (transduce (halt-when odd?) (completing conj! persistent!) (transient []) [2 2 2 3])
3
user=> (into [] (halt-when odd?) [2 2 2 3])
Execution error (ClassCastException) at user/eval197 (REPL:1).
class java.lang.Long cannot be cast to class clojure.lang.ITransientCollection (java.lang.Long is in module java.base of loader 'bootstrap'; clojure.lang.ITransientCollection is in unnamed module of loader 'app')

That’s far less clean looking

if into was redefined slightly it would be possible

It does seem like into should support halt-when.

(defn into'
  ([to xform from]
   (let [add-meta #(with-meta % (meta to))]
     (if (instance? clojure.lang.IEditableCollection to)
       (transduce xform (completing conj! (comp add-meta persistent!)) (transient to) from)
       (transduce xform (completing conj add-meta) to from)))))

redefined like that ^ it supports it

user=> (into' [] (halt-when #{:anomaly} (fn [& args] (zipmap [:result :failing-input] args))) [1 2 3 :anomaly 4])
{:result [1 2 3], :failing-input :anomaly}

Any idea if there’s a reason into wasn’t written to support halt-when from the start?

if that problem is valid, it's an oversight or bug