Clojurians Log v2

does anyone have an example of downloading a binary file from S3? I’ve attempted to use cognitect, amazonica, and clj-aws-s3, and all download a file, but with different MD5s than aws-cli

what I’m doing seems pretty simple:

(let [resp (aws/invoke s3-client {:op :GetObject :request {:Bucket bucket :Key filename}})]
    (io/copy (:Body resp) (io/output-stream (io/file filename)))
    resp)

hrm, the pure java version works:

(let [java-client (com.amazonaws.services.s3.AmazonS3ClientBuilder/defaultClient)]
    (with-open [obj (.getObject java-client (com.amazonaws.services.s3.model.GetObjectRequest. bucket key))]
      (java.nio.file.Files/copy (.getObjectContent obj) (.toPath (io/file dest-path)) (into-array java.nio.file.CopyOption [java.nio.file.StandardCopyOption/REPLACE_EXISTING]))))

different MD5s from different clients is suspicious

$ md5 datomic-pro-*.zip
MD5 (datomic-pro-0.9.6045-amazonica.zip) = 9b1c544ce7b68a6d4bcc7477c501cf6a
MD5 (datomic-pro-0.9.6045-aws-cli.zip) = 50d60403c5cf505ac12f31a7081995b5
MD5 (datomic-pro-0.9.6045-clj-s3.zip) = 9b1c544ce7b68a6d4bcc7477c501cf6a
MD5 (datomic-pro-0.9.6045-cognitect.zip) = 3d95d706e81ea53363b2c3f29b64ec74
MD5 (datomic-pro-0.9.6045-java.zip) = 50d60403c5cf505ac12f31a7081995b5

aws-cli and java are correct. Unsure why the others are wrong

try it without the io/output-stream around the io/file

are the file sizes different?

yes

copy doesn't close streams that are passed in to it if I recall

-rw-r--r--   1 arohner  staff  333382279 Apr  3 10:57 datomic-pro-0.9.6045-amazonica.zip
-rw-r--r--   1 arohner  staff  333390244 Apr  3 10:12 datomic-pro-0.9.6045-aws-cli.zip
-rw-r--r--   1 arohner  staff  333382279 Apr  3 11:12 datomic-pro-0.9.6045-clj-s3.zip
-rw-r--r--   1 arohner  staff  333389824 Apr  3 11:29 datomic-pro-0.9.6045-cognitect.zip
-rw-r--r--   1 arohner  staff  333390244 Apr  3 12:33 datomic-pro-0.9.6045-java.zip

<http://clojure.java.io/copy|clojure.java.io/copy>
([input output &amp; opts])
  Copies input to output.  Returns nil or throws IOException.
  Input may be an InputStream, Reader, File, byte[], char[], or String.
  Output may be an OutputStream, Writer, or File.

  Options are key/value pairs and may be one of

    :buffer-size  buffer size to use, default is 1024.
    :encoding     encoding to use if converting between
                  byte and char streams.

  Does not close any streams except those it opens itself
  (on a File).

yeah, that was it, thanks