aws

http://status.aws.amazon.com/ https://www.expeditedssl.com/aws-in-plain-english
ghadi 2020-02-01T19:18:59.017800Z

We are considering adding a new function to Clojure called iteration, and it would be very useful for consuming paginated APIs like Amazon's

(defn log-groups
  "gets all CloudWatch Log Groups"
  [client]
  (->> (iteration
        (fn [token]
          (aws/invoke client (cond-> {:op :DescribeLogGroups}
                               token (assoc :request {:nextToken token}))))
        :kf :nextToken
        :vf :logGroups)
       (into [] cat)))
^ here's how it could work with Cognitect Labs' aws-api

đź‘Ť 3
ghadi 2020-02-01T19:19:28.018100Z

https://clojure.atlassian.net/browse/CLJ-2555

ghadi 2020-02-01T19:20:44.019100Z

(still WIP, reserve the right to not do it, ymmv, etc.)

kenny 2020-02-01T19:44:31.020500Z

Nice. First thought is how do I handle anomalies? Sometimes (most times) I'd like to essentially have a reduced if an anomaly occurs, returning the anomaly.

ghadi 2020-02-01T19:48:42.021400Z

the iteration above will terminate on anomalies (because :kf won't return anything from an anomaly) -- the (into [] cat) is swallowing it though

ghadi 2020-02-01T19:49:02.021800Z

you take out :vf or pass in a smarter :vf

ghadi 2020-02-01T19:49:26.022300Z

then could check for the anomaly on the outside

ghadi 2020-02-01T19:50:19.023300Z

(into [] (halt-when anomaly? retf) (iteration....))

ghadi 2020-02-01T19:56:50.025100Z

halt-when will return the failing input when the retf isn't given

kenny 2020-02-01T20:04:12.027700Z

Hmm, I see. Clever. Looking at the pagination code we are using, the only other thing we keep track of is page-count. In some cases we use that for logging when certain page counts are hit for long running iterations & other cases for capping pages followed. Seems like you can do that here by returning a different data structure from :kf. I think this would cover our use case.

ghadi 2020-02-01T20:04:13.027800Z

probably better to pass in a smarter :vf (for the anomaly case)

kenny 2020-02-01T20:06:29.029600Z

The only other comment is I almost always prefer apis where the function takes its options in a map instead of the variadic approach. Easer to programmatically build up when needed. Was there a reason you chose the variadic args?

ghadi 2020-02-01T20:09:08.030700Z

I suspect that this isn't going to be a case where calls are programmatically built up, but I'll take that feedback back.

ghadi 2020-02-01T20:10:26.031500Z

my suggestion for halt-when isn't useful for (into []) but is for transduce

ghadi 2020-02-01T20:11:05.031800Z

user=> (transduce (halt-when odd?) (completing conj! persistent!) (transient []) [2 2 2 3])
3
user=> (into [] (halt-when odd?) [2 2 2 3])
Execution error (ClassCastException) at user/eval197 (REPL:1).
class java.lang.Long cannot be cast to class clojure.lang.ITransientCollection (java.lang.Long is in module java.base of loader 'bootstrap'; clojure.lang.ITransientCollection is in unnamed module of loader 'app')

kenny 2020-02-01T20:30:57.032700Z

That’s far less clean looking

ghadi 2020-02-01T20:31:35.033100Z

if into was redefined slightly it would be possible

kenny 2020-02-01T20:32:52.033600Z

It does seem like into should support halt-when.

ghadi 2020-02-01T20:35:23.033800Z

(defn into'
  ([to xform from]
   (let [add-meta #(with-meta % (meta to))]
     (if (instance? clojure.lang.IEditableCollection to)
       (transduce xform (completing conj! (comp add-meta persistent!)) (transient to) from)
       (transduce xform (completing conj add-meta) to from)))))

ghadi 2020-02-01T20:35:32.034200Z

redefined like that ^ it supports it

ghadi 2020-02-01T20:35:56.034600Z

user=> (into' [] (halt-when #{:anomaly} (fn [& args] (zipmap [:result :failing-input] args))) [1 2 3 :anomaly 4])
{:result [1 2 3], :failing-input :anomaly}

kenny 2020-02-01T20:38:31.035600Z

Any idea if there’s a reason into wasn’t written to support halt-when from the start?

ghadi 2020-02-01T20:39:16.036300Z

if that problem is valid, it's an oversight or bug