Clojurians Log v2

is "Throttling" represented as an http status code, or only within [:ErrorResponse :Error :Code]) ?@ghadi You asked the above. Throttling is only represented via that path. Here's a simple example:

(def c (aws-api/client {:api :workspaces}))
(def results
  (vec (pmap (fn [_]
               (aws-api/invoke c {:op :DescribeWorkspaceDirectories}))
             (range 20))))

(first (filter anom/anomaly? results))
=>
{:__type "ThrottlingException", :message "Rate exceeded", :cognitect.anomalies/category :cognitect.anomalies/incorrect}

(get-in (meta *1) [:http-response :status])
=> 400

do you have the complete response headers?

Yes

{"x-amz-date" "20200309T215544Z",
 "x-amz-target" "WorkspacesService.DescribeWorkspaceDirectories",
 "content-type" "application/x-amz-json-1.1",
 "host" "<http://workspaces.us-west-2.amazonaws.com|workspaces.us-west-2.amazonaws.com>",
 "authorization" "..."}

that's the response?

seems like the request

Headers

Oops, yeah

{"x-amzn-requestid" "59bbfe02-e3b0-49fa-a2cb-2027a236531a",
 "connection" "close",
 "content-length" "58",
 "date" "Mon, 09 Mar 2020 21:55:44 GMT",
 "content-type" "application/x-amz-json-1.1"}

We also have to use this for :retriable? for the apis we use.

(def api->retriable-fn
  "Map of AWS API to a fn that will be called to see if the request should be retried
  based on the HTTP response. This is needed because various AWS APIs indicate
  failures in non-standard ways."
  {:workspaces #(= "ThrottlingException" (:__type %))
   :pricing    #(= "ThrottlingException" (:__type %))
   :monitoring #(= "Throttling" (get-in % [:ErrorResponse :Error :Code]))})

what else is in the json response?

I'm trying to see if there is something generic we can detect and return a :busy anomaly

The body is this: {:type "ThrottlingException", :message "Rate exceeded", :cognitect.anomalies/category :cognitect.anomalies/incorrect}

From my experience, I don't think there is 😞

what about the raw response JSON?

Just slurping the :body?

yeah, should be available

(slurp (:body (:http-response (meta (first (filter anom/anomaly? results))))))
=> "{\"__type\":\"ThrottlingException\",\"message\":\"Rate exceeded\"}"

Thanks AWS™

Yep... It's frustrating. The above retry works but it results in incorrect anomalies getting returned. In the above example, it returns incorrect when it should be busy.

should really send back a status 429

Agreed. This happens in several of their apis though.

yeah

@dchelimsky do we read :errors from the descriptors?

Not yet. We looked at it as a means of normalizing error handling, but there is nothing normalizable (every service does its own thing).

Also, I ran into a few cases in which the descriptors themselves listed error shapes in operation maps, but those weren't present in the shapes map.

@ghadi @kenny am I missing some value we'd get from reading them anyway?

just need to be able to dispatch on the error shape name

is that always in __type?

we can't generically understand the errors without a lookup table

That's only a thing in some json services.

There is no spec for this @ghadi

No consistency across services.

right.

The error shape name is in the operation, which is already part of the model we use.

error shape name is useful enough

I think that should be in (ops client)

(don't have a repl up for this at the moment)

the ThrottlingException @kenny encountered isn't even in the workspaces service descriptor :)

That's what I'm talking about 😕

Can't exactly knock AWS for this. Nobody said "hey, use these descriptors to build your own SDKs" 😉

They should return standard http status codes though

https://docs.aws.amazon.com/workspaces/latest/api/CommonErrors.html

it doesn't say s* about how those errors are encoded on the wire

Can't break promises you don't make!

This all came up recently, btw: https://github.com/cognitect-labs/aws-api/issues/123

Having the ability to manually transform response bodies before the :retriable? call would help us in this situation. The transformed response would also need to be returned to the caller. I realize it's not a general solution, but it's sounding like there is not one.

https://github.com/cognitect-labs/aws-api/issues/134

Please feel free to make suggestions there in comments.

On a totally unrelated note, @ghadi you may recall I brought up the issue of an AWS API returning an invalid next-token. After a month or so, I finally got it again! Unfortunately the code to log the request-id had a typo in the key to lookup so I didn't get a request-id from the failure 😞 The next-token looks totally standard for this API. The system retrying the same call ~30s later and it worked. Quite odd behavior. Wonder if AWS has some sort of race condition.

{:__type "InvalidParameterValuesException",
 :message "Invalid NextToken provided.",
 :cognitect.anomalies/category :cognitect.anomalies/incorrect,
 :next-token "af286f1d-36ac-4fd2-bd5a-fa79fbb72f4b",
 :page-count 4}

I am determined to get AWS to fix this issue. Hopefully next time this error occurs I'll have a request id for them to debug this!