Clojurians Log v2

hi! Does anyone know if there is a programmatic way to discover what AMIs are available in the marketplace? e.g. the identifier of all deep learning AMIs?

@aaelony https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeImages.html ?

@dottedmag that looks interesting. I will check it out. Thank-you

hmmm. not sure this is working well just yet, but something along these lines:

(defn find-AMIs
  "Search the `Name` field for a matching string in all available AMIs."
  ;; <https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeImages.html>
  [regex-string]
  (let [re (re-pattern (str ".*" regex-string ".*"))
        ami-maps (:Images (aws/invoke ec2-client {:op :DescribeImages
                                                  :request {:Filters [{:Public true}]
                                                            ;;:DryRun true
                                                            }}))
        ami-names (-&gt;&gt; ami-maps
                      (mapv :Name)
                      (filterv #(not (nil? %))))
        ami-name-matches (-&gt;&gt; ami-names
                              (filterv #(re-matches re)))
        ]
    (vec ami-name-matches)
    ))

the idea is that you could then seek out deeplearning AMI’s with (find-AMIs “deeplearning”) or so

you can also use their filter syntax, this one filters on the Description field but you can do it with Name as well: aws ec2 describe-images --owners amazon --filters 'Name=description,Values=*deep learning*' 'Name=state,Values=available' --output json - it should be easy enough to do something like that with the function you have there

I saw that but I didn’t have time to figure it out just yet. If you have a sense for it, please lmk

also, I believe the AMIs would need “deep*learning” since deeplearning strings are common

Something like: {:Filters [{:State "available"} {:Description (str "*" regex-string "*")}]}

yeah, you can do that too

I’ll give that a try

this helps explain what you can do: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Filtering.html#Filtering_Resources_CLI

it's a case-sensitive search, so you might still need to fetch all of them and run them through a filter as you've done, just depends on your requirements

I recall now, I tried what you suggest and it doesn’t return any results

That’s why I opted to just use clojure’s filterv

but I think it was an error on my part

give me a few minutes, I think I can get it working

interesting. I ran one without the filter and one with the filter. Both return a count of 84214.

I think there are roughly 116 or so deeplearning AMIs

…which is what the filterv version returns

well feel free to ignore that suggestion if it doesn't work! I kind of want to know why now though, but there's nothing incorrect about the filter solution certainly, unless you're worried about API limits or conservation of network resources

I think I must be missing it, but I don’t see Cloudwatch among the releases for the cognitect aws client api. https://github.com/cognitect-labs/aws-api/blob/master/latest-releases.edn

Something that would correspond to https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/index.html or https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/cloudwatch/AmazonCloudWatchClient.html

Might be best to file a github issue so this doesn’t get lost - I don’t know off the top of my head

https://github.com/cognitect-labs/aws-api/issues/32

@gws, I don’t doubt there is a way to make the filter part work in the request. I just haven’t found it yet and the clojure side filter does in fact work. At some point, I hope to be able to devote time to figuring out the request side filter but can’t at the moment (and the workaround suffices)