Clojurians Log v2

> even their own api suggests that Unfortunately, their machine-readable API descriptor does not, so chunked uploads are a special case that we have to handle explicitly. It's on the list, but not alone. We'll get there sooner or later.

@dchelimsky chunked uploads don't exist, but what was being referred to is Multipart upload, which we do support already. It's non-multipart, streamed upload (with the funky singing) that we don't support yet.

By "funky singing", do you mean singing poorly or with a groove?

😉

lol

need my coffee to start working 🙂

s/sing/sign

Oh, now I get it! 😄

:CreateMultipartUpload -*> :UploadPart -> :CompleteMultipartUpload

> which we do support already To be clear - "we do support" it because the aws service descriptor describes it.

i.e. we're not doing anything special to do that

anyone using the cognitec aws-api would have an idea what this means?

{:cognitect.anomalies/category :cognitect.anomalies/fault,
 :cognitect.aws.client/throwable #error {
 :cause "Property null is not supported"
 :via
 [{:type java.lang.IllegalArgumentException
   :message "Property null is not supported"
   :at [com.sun.xml.internal.stream.XMLInputFactoryImpl setProperty "XMLInputFactoryImpl.java" 246]}]
 :trace
 [[com.sun.xml.internal.stream.XMLInputFactoryImpl setProperty "XMLInputFactoryImpl.java" 246]
  [clojure.data.xml$new_xml_input_factory invokeStatic "xml.clj" 323]
  [clojure.data.xml$new_xml_input_factory invoke "xml.clj" 319]
  [clojure.data.xml$source_seq invokeStatic "xml.clj" 332]
  [clojure.data.xml$source_seq doInvoke "xml.clj" 326]
  [clojure.lang.RestFn applyTo "RestFn.java" 139]
  [clojure.core$apply invokeStatic "core.clj" 667]
  [clojure.core$apply invoke "core.clj" 660]
  [clojure.data.xml$parse invokeStatic "xml.clj" 346]
  [clojure.data.xml$parse doInvoke "xml.clj" 340]
  [clojure.lang.RestFn invoke "RestFn.java" 486]
  [cognitect.aws.util$xml_read invokeStatic "util.clj" 137]
  [cognitect.aws.util$xml_read invoke "util.clj" 134]
  [cognitect.aws.shape$xml_parse invokeStatic "shape.clj" 217]
  [cognitect.aws.shape$xml_parse invoke "shape.clj" 214]
  [cognitect.aws.protocols.rest$parse_body invokeStatic "rest.clj" 247]
  [cognitect.aws.protocols.rest$parse_body invoke "rest.clj" 235]
  [cognitect.aws.protocols.rest$parse_http_response invokeStatic "rest.clj" 260]
  [cognitect.aws.protocols.rest$parse_http_response invoke "rest.clj" 249]
  [cognitect.aws.protocols.rest_xml$eval16877$fn__16878 invoke "rest_xml.clj" 23]
  [clojure.lang.MultiFn invoke "MultiFn.java" 239]
  [cognitect.aws.client$handle_http_response invokeStatic "client.clj" 48]
  [cognitect.aws.client$handle_http_response invoke "client.clj" 42]
  [cognitect.aws.client$send_request_STAR_$fn__15799 invoke "client.clj" 82]
  [clojure.core.async.impl.channels.ManyToManyChannel$fn__4659$fn__4660 invoke "channels.clj" 95]
  [clojure.lang.AFn run "AFn.java" 22]
  [java.util.concurrent.ThreadPoolExecutor runWorker "ThreadPoolExecutor.java" 1128]
  [java.util.concurrent.ThreadPoolExecutor$Worker run "ThreadPoolExecutor.java" 628]
  [java.lang.Thread run "Thread.java" 835]]}}

I'm trying to connect to a local minio server, and the config I copied exactly the same code from another project that actually works

the config passed to aws/client is identical as well, as the libraries versions

if I upgrade the aws-api libraries it actually hangs forever instead and not even show this error

the config is just something like

{:api :s3,
 :region :eu-west-1,
 :endpoint-override
 {:protocol "http", :hostname "localhost", :port 9000},
 :credentials-provider
 #object[cognitect.aws.credentials$basic_credentials_provider$reify__15734 0x695418f5 "cognitect.aws.credentials$basic_credentials_provider$reify__15734@695418f5"]}

the weird thing is that not all operations fail, I can for example create a bucket

> if I upgrade the aws-api libraries it actually hangs forever instead and not even show this error Are you using aws-api 0.8.437?

> the weird thing is that not all operations fail, I can for example create a bucket which op is causing that error?

ah yes ListBuckets is failing, and yeah it hangs with the latest version, but blow up with with "0.8.408"

I got the other ops to work though so I think it's all fine

"the latest version" - you mean 0.8.437?

yes sorry

Thanks

they both don't work though, in different ways, but the same thing works on another project, so something funny in this project, just would be curious to know what that error could mean

The error means that aws-api can't parse it's getting back from the server

What do you get (in the repl) if you do this:

(with-redefs [cognitect.aws.protocols.rest/parse-body (fn [_ body _] (println (cognitect.aws.util/bbuf->str body)))]
  (aws/invoke s3 {:op :ListBuckets}))

<?xml version="1.0" encoding="UTF-8"?>
<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4</ID><DisplayName></DisplayName></Owner><Buckets><Bucket><Name>hello</Name><CreationDate>2020-02-17T15:45:02.969Z</CreationDate></Bucket><Bucket><Name>test-bucket</Name><CreationDate>2020-02-17T16:11:46.874Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>

there is some content

I think the problem is the empty DisplayName.

interestingly now all my tests work locally, but it doesn't work on CI (just doesn't fetch an object after creating it), and trying again from the repl I get again that same problem

where should the DisplayName come from btw?

the test is just

(defn- create-test-bucket
  []
  (aws/invoke (s3-client)
              {:op :CreateBucket
               :request {:Bucket (config/value [:s3 :bucket])}}))

(deftest list-files-test
  (testing "Uploading and fetching a file works"
    (create-test-bucket)
    (sut/put-key "resources/sample.txt" "sample.txt")
    (let [tmp-path (sut/fetch-key "sample.txt")]
      (is (some? tmp-path))
      (is (= (slurp tmp-path) (slurp "resources/sample.txt"))))))

I need to roll off here - wanna submit a github issue w/ all this info to keep it on the radar?

https://github.com/cognitect-labs/aws-api/issues

yeah if I can reproduce it enough, but one stupid question, I use (s3-client) to create the client more dynamically, maybe should I just have a global s3 as in the examples?

it should not matter right?

I don't think that should matter for this issue.

uhm ok it turns out the problem on CI was just other AWS_* env variables shadowing the ones I was trying to set

that makes sense I guess but I should have ideally got some unauthorized error right?