Clojurians Log v2

so we'd have something similar to -rest-json-call for query string protocol?

if not the same time but parameterized

yes

{“name” #{},
 “http” #{“method” “requestUri”},
 “input” #{“shape”},
 “output” #{“resultWrapper” “shape”},
 “errors” #{}}

and here are the keys that I found for the query protocol

errors contains some shapes representing errors

so really simplified compared to rest-json

so my goal is to translate each shape (`map` list, atomic ones) into query-string or form-params

if I have time, I could take a look at grabbing docs from separate files, maybe on some evening

if you can check also if everything is fine for the LATEST thing

yup

depends if our little one goes to sleep easily 🙂

food for thought http://www.serverlesscomputing.org/wosc2/presentations/s2-wosc-slides.pdf

>>> Takeaways • The future is granular, interactive and massively parallel. • Many applications can benefit from this “Laptop Extension” model. • Better platforms are needed to be built to support “bursty” massively-parallel jobs.

nodding on the last bullet

it’s too easy to just throw users at a database and then wonder why stuff doesn’t work performantly

In the next few weeks, it’s open-source, no deadline 🙂

Could you tell more?

@cgrand is it a conférence you attended ?

nope some slides I came across

ok

you are working on distributed computing ?

I really don’t understand how you can split an encoding job between 5000' thread

on conferences, https://www.eventbrite.com/e/clojutre-goes-helsinki-2018-tickets-45337528769?aff=eand

at my current project, we have this 12 node Redshift cluster

I have a feeling that I'm slowly morphing into a dbadmin

I have very good feedbacks on ClojuTRE

I have to go this year

I think 2014 was my first ClojuTRE, has been really good event every time 🙂

The attribute stuff is so uncool

https://docs.aws.amazon.com/sns/latest/api/API_SetPlatformApplicationAttributes.html

Similar things are to watch out when using spark, but haven’t used a real distributed cluster. Would like to see a Spark cluster used with powderkeg 🙂

question, should we create different (spec/conformer) based on the protocol to format the request or use only specs globally and rework the input

e.g :

[{:topic-arn "2w",
   :label "h",
   :awsaccount-id ["8L" "" "" "" "ff"],
   :action-name ["1" "m" "n" "ne" "" "GR" "" "56" "6"]}
  {"TopicArn" "2w",
   "Label" "h",
   "AWSAccountId" ["8L" "" "" "" "ff"],
   "ActionName" ["1" "m" "n" "ne" "" "GR" "" "56" "6"]}]

this exercise on :portkey.aws.sns.-2010-03-31/add-permission-input returns vectors for the list “type”

If I read the doc well, this translate to something like that =>

<http://sns.us-east-1.amazonaws.com/>
        ?TopicArn=arn%3Aaws%3Asns%3Aus-east-1%3A123456789012%3AMy-Test
        &amp;ActionName.member.1=Publish &amp;ActionName.member.2=GetTopicAttributes
        &amp;Label=NewPermission &amp;AWSAccountId.member.1=987654321000
        &amp;AWSAccountId.member.2=876543210000 &amp;Action=AddPermission &amp;SignatureVersion=2
        &amp;SignatureMethod=HmacSHA256 &amp;Timestamp=2010-03-31T12%3A00%3A00.000Z
        &amp;AWSAccessKeyId=(AWS Access Key ID)
        &amp;Signature=k%2FAU%2FKp13pjndwJ7rr1sZszy6MZMlOhRBCHx1ZaZFiw%3D 

with ActionName.member.1=Publish &ActionName.member.2=GetTopicAttributes as values

kind of the same pattern for maps

so structure, map and list are the 3 compound types

but we can exclude structure as it’s a global wrapper

I might be off but spec.tools has a way of changing a conformer: https://github.com/metosin/spec-tools/blob/master/README.md#spec-coercion

I wrote something like that =>

(defn conformed-input->query-protocol-input
    "Given a conformed input, transform the input to be compliant to
  the `query` protocol of AWS."
    [input]
    (into {} (for [[k v] input]
               (cond
                 (string? v) [k v]
                 (vector? v) (into {} (map-indexed (fn [i v'] [(str k ".member." i) v'])) v)
                 (map? v) (into {}
                                (comp
                                 (x/for [[k' v'] %]
                                   [[(str k ".entry.") k'] [(str k ".entry.") v']])
                                 (map-indexed (fn [i [[k1 v1] [k2 v2]]]
                                                [[(str k1 i ".key") v1] [(str k2 i ".value") v2]]))
                                 cat)
                                v)))))

I am not sure it’s the best way but it works

My plan was/is to decouple spec and transformation. (Wip in my old wip branch)

you mean not use spec conformers ?

what we know is that description files are the same for the 5 protocols

so specs are working that way

but output/conforming has little differences

by the way, same goes for the output specs

does it recall you something @cgrand =>

(concat (map #(str "ser-" %) inputs) (map #(str "req<-" %) input-roots)
                     (map #(str "deser-" %) outputs) (map #(str "resp->" %) output-roots))

https://github.com/portkey-cloud/aws-clj-sdk/commit/046b92cd34812c385754dab8ead8d325409f67d1#diff-0b329dfd4d7adb3e2cec38b224f876fdR263

the shape-seq makes me really nervous

https://github.com/portkey-cloud/aws-clj-sdk/commit/046b92cd34812c385754dab8ead8d325409f67d1#diff-0b329dfd4d7adb3e2cec38b224f876fdR18

😂

@cgrand when you have the time, it would be awesome of you can explain to me the overall archetecture that you were looking for

from there I can try to draw some stuff and then go back to your code