Clojurians Log v2

Sometimes aws-api will return an anomaly with an incorrect category (e.g., incorrect or fault) when the actual category should be busy. This is on AWS for returning the wrong status code. We would, however, like to change the categories of these anomalies to the correct category. For example, the CloudWatch API will indicate a throttling error like so #(= "Throttling" (get-in % [:ErrorResponse :Error :Code])). Is there any thinking on the best way to do this? It would seem like the best way is to create our own invoke function and ensure every caller uses that instead of cognitect.aws.client.api/invoke.

When trying to use s3 :SelectObjectContent I get an error about failing to parse xml

(aws/invoke s3
            {:op :SelectObjectContent
             :request
             {:Bucket bucket
              :Key my-key
              :Expression "select * from S3Object s"
              :ExpressionType "SQL"
              :InputSerialization {:CSV {:RecordDelimiter "\r\n"}}
              :OutputSerialization {:JSON {:Type "LINES"}}}})

;; =>

{:cognitect.anomalies/category :cognitect.anomalies/fault,
 :cognitect.aws.client/throwable #error {
 :cause "ParseError at [row,col]:[1,1]\nMessage: Content is not allowed in prolog."
 :via
 [{:type javax.xml.stream.XMLStreamException
   :message "ParseError at [row,col]:[1,1]\nMessage: Content is not allowed in prolog."
   :at [<http://com.sun.org|com.sun.org>.apache.xerces.internal.impl.XMLStreamReaderImpl next "XMLStreamReaderImpl.java" 652]}]
 :trace
 [[<http://com.sun.org|com.sun.org>.apache.xerces.internal.impl.XMLStreamReaderImpl next "XMLStreamReaderImpl.java" 652]
  [clojure.data.xml.jvm.parse$pull_seq$fn__29940 invoke "parse.clj" 78]
  [clojure.lang.LazySeq sval "LazySeq.java" 42]
  [clojure.lang.LazySeq seq "LazySeq.java" 51]
  [clojure.lang.RT seq "RT.java" 535]
  [clojure.core$seq__5402 invokeStatic "core.clj" 137]
  [clojure.core$seq__5402 invoke "core.clj" 137]
  [clojure.data.xml.tree$seq_tree$fn__29773 invoke "tree.clj" 39]
  [clojure.lang.LazySeq sval "LazySeq.java" 42]
  [clojure.lang.LazySeq seq "LazySeq.java" 51]
  [clojure.lang.LazySeq first "LazySeq.java" 73]
  [clojure.lang.RT first "RT.java" 692]
  [clojure.core$first__5384 invokeStatic "core.clj" 55]
  [clojure.core$ffirst__5394 invokeStatic "core.clj" 103]
  [clojure.core$ffirst__5394 invoke "core.clj" 103]
  [clojure.data.xml.tree$event_tree invokeStatic "tree.clj" 70]
  [clojure.data.xml.tree$event_tree invoke "tree.clj" 66]
  [clojure.data.xml$parse invokeStatic "xml.clj" 109]
  [clojure.data.xml$parse doInvoke "xml.clj" 84]
  [clojure.lang.RestFn invoke "RestFn.java" 486]
  [cognitect.aws.util$xml_read invokeStatic "util.clj" 148]
  [cognitect.aws.util$xml_read invoke "util.clj" 145]
  [cognitect.aws.shape$xml_parse invokeStatic "shape.clj" 217]
  [cognitect.aws.shape$xml_parse invoke "shape.clj" 214]
  [cognitect.aws.protocols.rest$parse_body invokeStatic "rest.clj" 243]
  [cognitect.aws.protocols.rest$parse_body invoke "rest.clj" 235]
  [cognitect.aws.protocols.rest$parse_http_response invokeStatic "rest.clj" 260]
  [cognitect.aws.protocols.rest$parse_http_response invoke "rest.clj" 249]
  [cognitect.aws.protocols.rest_xml$eval32111$fn__32112 invoke "rest_xml.clj" 23]
  [clojure.lang.MultiFn invoke "MultiFn.java" 239]
  [cognitect.aws.client$handle_http_response invokeStatic "client.clj" 49]
  [cognitect.aws.client$handle_http_response invoke "client.clj" 44]
  [cognitect.aws.client$send_request$fn__31513$state_machine__20194__auto____31540$fn__31543 invoke "client.clj" 112]
  [cognitect.aws.client$send_request$fn__31513$state_machine__20194__auto____31540 invoke "client.clj" 108]
  [clojure.core.async.impl.ioc_macros$run_state_machine invokeStatic "ioc_macros.clj" 973]
  [clojure.core.async.impl.ioc_macros$run_state_machine invoke "ioc_macros.clj" 972]
  [clojure.core.async.impl.ioc_macros$run_state_machine_wrapped invokeStatic "ioc_macros.clj" 977]
  [clojure.core.async.impl.ioc_macros$run_state_machine_wrapped invoke "ioc_macros.clj" 975]
  [clojure.core.async.impl.ioc_macros$take_BANG_$fn__20212 invoke "ioc_macros.clj" 986]
  [clojure.core.async.impl.channels.ManyToManyChannel$fn__15077$fn__15078 invoke "channels.clj" 95]
  [clojure.lang.AFn run "AFn.java" 22]
  [java.util.concurrent.ThreadPoolExecutor runWorker "ThreadPoolExecutor.java" 1128]
  [java.util.concurrent.ThreadPoolExecutor$Worker run "ThreadPoolExecutor.java" 628]
  [clojure.core.async.impl.concurrent$counted_thread_factory$reify__14946$fn__14947 invoke "concurrent.clj" 29]
  [clojure.lang.AFn run "AFn.java" 22]
  [java.lang.Thread run "Thread.java" 830]]}}

I'm guessing that because the aws-api is generic it assuming this endpoint returns xml when it does not. Is there anyway to make it not try to parse the body as xml?

@jimmy capture the (meta response)

should have a response body that might be helpful

(but obviously it's CSV)

can you file an issue? In the meantime, maybe try re-reading the response inputstream body correctly in userspace

as a workaround

Thanks for the work around. I will definitely file an issue about it

Any info you want me to capture other than what I've provided?

@kenny same request for you -- is "Throttling" represented as an http status code, or only within [:ErrorResponse :Error :Code]) ?

@jimmy any setup code necessary to make an object that is readable in this way

Before you file an issue - please check your dependencies.

this one isn't probably our favorite data.xml dep issue --

no?

but knowing the deps will be useful

no, he's asking for a special response handling from S3:

:InputSerialization {:CSV {:RecordDelimiter "\r\n"}}
              :OutputSerialization {:JSON {:Type "LINES"}}}})

dunno if that conflicts with the client's reading of the service descriptor

It does

https://docs.aws.amazon.com/AmazonS3/latest/API/RESTSelectObjectAppendix.html

looks like this is a Two-fer Tuesday

Type is not part of the JSON map under OutputSerialization

True, but changing that does not make a difference.

Are you using lein or tools.deps or other?

tools.deps

what version clojure/data.xml is in clj -Spath

OK - never mind. I see I'm barking up the wrong tree.

It is 0.2.0-alpha6. But yeah, if you look at the response body it isn't just straight forward xml. I'll file an issue that given a bucket you should be able to test it and recreate it. Sadly the body is complicated enough, probably not worth me trying to do the work around right now.

the response format is..... artisanal

Or custom, in which case you could customize it. But how do you artisanize it?

https://github.com/aws/aws-sdk-java-v2/blob/master/services/s3/src/main/resources/codegen-resources/service-2.json#L7710-L7735

:eventstream true

fun

Made the issue https://github.com/cognitect-labs/aws-api/issues/132 Thanks for the help @ghadi