Clojurians Log v2

@mruzekw you may interested in https://github.com/stediinc/cdk-clj.

Trying to do S3 listing from within of an aws lambda fails for me with aws/api and aws/s3

{:cognitect.anomalies/category :cognitect.anomalies/fault, :cognitect.anomalies/message java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;, :cognitect.http-client/throwable #error {
 :cause java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;
 :via
 [{:type java.lang.NoSuchMethodError
   :message java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;
   :at [cognitect.http_client$empty_bbuf invokeStatic http_client.clj 37]}]
 :trace
 [[cognitect.http_client$empty_bbuf invokeStatic http_client.clj 37]
  [cognitect.http_client$empty_bbuf invoke http_client.clj 34]
  [cognitect.http_client$on_content$fn__10264 invoke http_client.clj 140]
  [clojure.core$update invokeStatic core.clj 6196]
  [clojure.core$update invoke core.clj 6188]
  [cognitect.http_client$on_content invokeStatic http_client.clj 139]
  [cognitect.http_client$on_content invoke http_client.clj 136]
  [clojure.lang.Atom swap Atom.java 51]

;; code starting like

(let [s3-listing (aws/invoke s3 {:op :ListObjectsV2 :request
                                   {:Bucket bucket-name
                                    :Prefix files-prefix}})
        s3-files (map :Key (:Contents s3-listing))]
    (println s3-listing)

Any advice is welcome, some parts of the internet suggest a jdk version issue, aws lambda is running java8

which JDK?

and you are aot'ing

and if so with what jdk

Will report a little later

I was compiling with leiningen on java11 openjdk, and Amazon is running java8 openjdk for java lambdas

Once I convinced leiningen to run with java8 for uberjar build the problem seemed to go away

yeah, that's what I would have recommended

there have been some changes in the jdk that affect the compiled method invocation here. I don't remember the details but it is pretty subtle

The aws-api readme says: > specs which are generated from the source descriptions There's no obvious documentation on how to use these. How do I access these generated specs?

Ah found it. You're supposed to use response-spec-key.

How come enum values are not enumerated in the generated specs?

The actual AWS API reference docs seem to have access to enum values.

(s/def :cognitect.aws.kinesis/ConsumerStatus #{"DELETING" "CREATING" "ACTIVE"})

@kenny can you post stuff you tried, whether you were successful or not?

Not all of them are like that:

(s/def :cognitect.aws.workspaces/WorkspaceState string?)

> Valid Values: PENDING | AVAILABLE | IMPAIRED | UNHEALTHY | REBOOTING | STARTING | REBUILDING | RESTORING | MAINTENANCE | ADMIN_MAINTENANCE | TERMINATING | TERMINATED | SUSPENDED | UPDATING | STOPPING | STOPPED | ERROR

looking -- what's the mvn coordinate you're using?

com.cognitect.aws/workspaces {:mvn/version "758.2.549.0"}

thx

Just tried the latest 770.2.568.0 and same thing.

Same thing for (s/def :cognitect.aws.workspaces/ConnectionState string?)

https://docs.aws.amazon.com/workspaces/latest/api/API_WorkspaceConnectionStatus.html?shortFooter=true

investigating

The generators for these specs also don't work.

(gen/generate (s/gen :cognitect.aws.workspaces.Workspace/DirectoryId))
Execution error (ExceptionInfo) at clojure.test.check.generators/fn (generators.cljc:435).
Couldn't satisfy such-that predicate after 100 tries.

regex specs is a known problem that @dchelimsky is working on

not all regexes are invertible, even if we use a library like test.chuck

i'm looking into why (s/def :cognitect.aws.workspaces/ConnectionState string?) is not an enum

True. Seems like test.chuck may be good enough for most/all of the aws cases though?

you would think but @dchelimsky tested it and it's about 50/50 from what I recall

some of the regexes AWS has are truly bizarre

Not surprised haha

test.check doesn't work on the DirectoryId regex. Oh well.

@kenny it turns out that the enum excludes values that the api actually responds to

we found that to be the case on some APIs

Geez. Does the enum def match the api doc reference?

which enum def?

The one you looked at

ConnectionState

i didn't look at a specific one, I just found out the history for the change

it probably matches in 95% of the cases, if I were to guess

How do you know it doesn't match?

matching meaning the API documentation (non-canonical) and the service descriptor (non-canonical)

the truth is what the API accepts and returns

which might not be in sync with descriptors or documentation

these specs are mechanically generated from json descriptors

The only was you'd know "the truth" is by executing API calls, right?

yeah

So these specs have been manually overridden?

If I were to guess what happened: someone tried to hit an API with the library's validation turned on, and a value was rejected because it wasn't in the enum spec

enums are closed specs

(value rejected by the library's validation)

not by the service itself

Interesting. A bit surprised someone else is using Clojure & WorkSpaces haha. So then you guys must be storing which specs do not match and overriding with the "open" enum -- string?.

i never said anything about Workspaces

it's a general issue we hit somewhere else, that made us re-examine how we emit specs

So all enums are open?

are enums in AWS specced as string? ?

But this one isn't? https://clojurians.slack.com/archives/C09N0H1RB/p1573839150185200

yeah that might be a stale artifact, I'll confirm on a more recent mvn

So all AWS enums are supposed to be specced as string??

(s/def :cognitect.aws.kinesis/ConsumerStatus string?)

on latest ^

I understand. It's kinda a tricky situation. I'm planning to have functions that operate over certain AWS properties. Having generators generate relevant values is important. Seems like the enum values should be worked into the generated generators.

understood

If the gens for things like ConsumerStatus always generate a random string it will not hit all the code paths, making the gen test bad. For now I can do something like this for all the enum specs:

(s/def :workspace-info/State
  (s/with-gen
    :cognitect.aws.workspaces/WorkspaceState
    #(gen/one-of [(s/gen #{"PENDING"
                           "AVAILABLE"
                           "IMPAIRED"
                           "UNHEALTHY"
                           "REBOOTING"
                           "STARTING"
                           "REBUILDING"
                           "RESTORING"
                           "MAINTENANCE"
                           "ADMIN_MAINTENANCE"
                           "TERMINATING"
                           "TERMINATED"
                           "SUSPENDED"
                           "UPDATING"
                           "STOPPING"
                           "STOPPED"
                           "ERROR"})
                  (s/gen string?)])))

Also would be great to have a note in the readme or, even better, next to the enum specs noting why it is specced as an open enum.

can you assist us by capturing a github issue with details?

the concerns are documentation/discovery of enums, and associated gen of enum specs

Sure

thanks!

https://github.com/cognitect-labs/aws-api/issues/104

@kenny side note: you'll get better feedback (s/describe and s/explain) if you use the s/spec macro with :gen than the s/with-gen function.

Weird. That doesn't seem like the intended use for s/spec.