Clojurians Log v2

Why would :Body be nil on a S3 GetObject request?

(aws-api/invoke
    s3-client
    {:op      :GetObject
     :request {:Bucket billing-s3-bucket
               :Key    "hourly-cur/hourly-cur/20200201-20200301/fc247861-fb77-460c-9f91/hourly-cur-7.csv.gz"}})
=>
{:LastModified #inst"2020-02-04T09:04:14.000-00:00",
 :ETag "\"d41d8cd98f00b204e9800998ecf8427e\"",
 :Metadata {},
 :ServerSideEncryption "AES256",
 :ContentLength 0,
 :ContentType "application/octet-stream",
 :AcceptRanges "bytes",
 :Body nil}

@kenny you can have keys that have no associated content. That :ContentLength is 0 suggests that is the case, in which case s3 sends back no body. You can see this if you eval (:http-response (meta *1)) (assuming your last exp was the invoke call you posted).

Ah yes, that was it. Thanks.

I believe there is an issue with S3 returning a key that looks like this // . If you create a CUR with no prefix set, AWS will store the CURs under the path //<report name>/<date range>/<assembly id>/<report item>. If I call :ListObjectsV2 on this bucket, I get a list of objects returned but the :Key is not correct. For example, one key returned looks like this "/test/20190501-20190601/7c0795d3-9b7e-438b-bb5d-de3ead37baad/test-1.csv.gz" when its actual key should be "//test/20190501-20190601/7c0795c3-9b7e-437b-bb5d-de3ead37baad/test-1.csv.gz" . If I call :GetObject on the key returned by aws-api, I get a NoSuchKey anomaly. If I try using the actual key with the //, I get a forbidden anomaly:

{:Error {:HostIdAttrs {},
         :StringToSignBytes "...",
         :CanonicalRequestBytes "...",
         :CanonicalRequestAttrs {},
         :Message "The request signature we calculated does not match the signature you provided. Check your key and signing method.",
         :StringToSign "AWS4-HMAC-SHA256
                        20200213T160537Z
                        20200213/us-west-2/s3/aws4_request
                        ...",
         :CodeAttrs {},
         :RequestIdAttrs {},
         :SignatureProvidedAttrs {},
         :HostId "...",
         :StringToSignBytesAttrs {},
         :MessageAttrs {},
         :RequestId "E92073AA791F0EB6",
         :Code "SignatureDoesNotMatch",
         :SignatureProvided "...",
         :AWSAccessKeyIdAttrs {},
         :CanonicalRequest "GET
                            /compute-cost-reports//test/20190501-20190601/7c0795c3-9b7e-437b-bb5d-de3ead37baad/test-1.csv.gz
                            
                            host:<http://s3.us-west-2.amazonaws.com|s3.us-west-2.amazonaws.com>
                            x-amz-content-sha256:...
                            x-amz-date:20200213T160537Z
                            
                            host;x-amz-content-sha256;x-amz-date
                            ...",
         :CanonicalRequestBytesAttrs {},
         :StringToSignAttrs {},
         :AWSAccessKeyId "..."},
 :ErrorAttrs {},
 :cognitect.anomalies/category :cognitect.anomalies/forbidden}

What happens if you do the same with the aws cli?

Listing:

aws s3 ls <s3://my-bucket>
                           PRE /
2020-02-12 19:22:09          4 aws-programmatic-access-test-object

aws s3 cp <s3://my-bucket//test/20190501-20190601/test-Manifest.json> .
download: <s3://my-bucket//test/20190501-20190601/test-Manifest.json> to ./test-Manifest.json

It appears to work. The file is downloaded on my computer.

This is what it looks like in the console. It's definitely a strange case. However, it is the default for creating a CUR without a prefix so I'd expect this to work.

@kenny would you kindly submit another issue?

what is a CUR?

@kenny

cost and usage report

GetObjects shouldn't have leading slashes in the keys

It appears to work :man-shrugging::skin-tone-2: Apparently the aws billing team thinks it's okay. It certainly seems very odd to me.

i thought you're posting because it doesn't appear to work

Note the aws cli is baroque & does some unspecified transforms of cmdline arguments, and is not a canonical guide

aws-api does not support keys with a path like this //test/20190501-20190601/test-Manifest.json. The aws billing team stores CURs under that path if you don't specify a prefix.

there's no such thing as that path AFAIK

There is. Create a CUR with no prefix.

I think aws cli eats the //

please create a CUR and then aws s3 ls the bucket

I think I did that above?

the listing above says aws-programmatic-access-test-object

is that the one?

Yes. There's 2 keys there. One that is / and one that is aws-programmatic-access-test-object

I see

https://github.com/cognitect-labs/aws-api/issues/126

thanks, I added to it

This is what it looks like in the S3 console folders list. The first time I saw this, I didn't even think it was a folder.

https://github.com/cognitect-labs/aws-api/blob/5cf8f6c3a86392fa74423906bfe19685875f781f/src/cognitect/aws/signers.clj#L54

storing stuff with the / prefix is not a great idea 🙂

Lovely

not sure if that's related, but it's suspect

Curious what the potential solution is if that's the problem. Not use http://java.net.URI?

I'm sure that was chosen for a reason. I'm pretty sure I added the (str/replace #"//+" "/") when I was refactoring some things, but didn't question the choice of URI at the time.

More context:

(-> uri
                         (str/replace #"//+" "/") ; (URI.) throws Exception on '//'.
                         (str/replace #"\s" "%20"); (URI.) throws Exception on space.
                         (URI.)
                         (.normalize)
                         (.getPath)
                         (uri-encode "/"))

So it looks like we're using URI to avoid having to do the work done in .normalize

it might be a red herring

or URI might be doing unwanted things

FWIW, it sounds like // is a legal path https://stackoverflow.com/questions/20523318/is-a-url-with-in-the-path-section-valid

"So: yes, it is valid, no, don't use it." 🙂

user=&gt; (.normalize (<http://java.net|java.net>.URI/create "<https://ghadi.net//fooo>"))
#object[<http://java.net|java.net>.URI 0x54e81b21 "<https://ghadi.net/fooo>"]

You spelled foo wrong

my b

From https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html:

In exception to this, you do not normalize URI paths for requests to Amazon S3. For example, if you have a bucket with an object named my-object//example//photo.user, use that path. Normalizing the path to my-object/example/photo.user will cause the request to fail. For more information, see Task 1: Create a Canonical Request in the Amazon Simple Storage Service API Reference. 

We don't make that exception.

yup

We do need to do it everywhere else, however.