Clojurians Log v2

hello guys, I am trying to use the s3-pod inside a bb app.. but I am having trouble to figure out what is the expected format for my :Body object in a PutObject operation. I tried to pass the same format as the aws-api from cognitect expects and got this error.

29: (defn run [opts]
30:   (let [{:keys [filepath partner content dag-name]} opts]
31:     (s3/put-object {:Bucket "captalys-analytics"
        ^--- Cannot JSON encode object of class: class java.io.BufferedInputStream: java.io.BufferedInputStream@7fa4620f4f20
32:                     :Key "testando"
33:                     :Body (io/input-stream (io/file filepath))
34:                     :Tagging "dagName=OK"})))

If I try to pass bytes (as is specified in the go sdk) I then ge tthe following error

29: (defn run [opts]
30:   (let [{:keys [filepath partner content dag-name]} opts]
31:     (s3/put-object {:Bucket "captalys-analytics"
        ^--- json: cannot unmarshal string into Go struct field PutObjectInput.Body of type io.ReadSeeker
32:                     :Key "testando"
33:                     :Body (.getBytes "okokk")
34:                     :Tagging "dagName=OK"})))

Does this work in the standard REPL? If not you may want to ask in #aws

when using the aws-api from cognitect it works without a problem. The issue is the interface between clj and golang sdk managed by https://github.com/tzzh/pod-tzzh-aws

@thomas.ormezzano ^

@iagwanderson Have you tried passing in a string? The arguments to a pod are serialized via EDN or JSON, so there should be special handling of bytes in a pod (e.g. serialize to base64 under the hood in bb and deserialize to bytes at the pod side). This can be implemented so the user doesn't even notice, but it does need attention. Maybe @thomas.ormezzano can think about this?

ah yes good point I am afraid bytes arguments are not supported atm I need to look into this

also I noticed there are a bunch of helpers functions in the s3manager module that could be helpful for uploading/downloading https://docs.aws.amazon.com/sdk-for-go/api/service/s3/s3manager/ - I mostly use the s3 fns for listing/deleting files so I haven’t looked into this as much but going to have a look when I have a bit of time

@thomas.ormezzano Should be doable I think using a client side wrapper which coerces the bytes into base64 (or something similar which can go through json)

@thomas.ormezzano Maybe pods should start supporting transit. If there's a transit library for Go which supports bytes, then we could just use that

@thomas.ormezzano Alternatively you could invent your own representation of bytes, like: {"tzzh/bytes" "base64string"} so if you encounter an arg like that on the pod side, deal with it You could even choose to just use bencode as the byte representation, since bencode supports sending just bytes I think

This is what I do with database connection objects and terminal objects in pods too. Right now I'm working on a lantera pod: https://github.com/babashka/pod-babashka-lanterna

ok thanks I will take a look and see what’s easier :thumbsup:

Found a Go transit lib here: https://github.com/russolsen/transit If you can verify that it's possible to serialize bytes using this lib, I'm willing to add support for transit to babashka.pods

hello guys, thanks for the comments. I tried to pass a plan string but got the same error

29: (defn run [opts]
30:   (let [{:keys [filepath partner content dag-name]} opts]
31:     (s3/put-object {:Bucket "captalys-analytics"
        ^--- json: cannot unmarshal string into Go struct field PutObjectInput.Body of type io.ReadSeeker
32:                     :Key "testando"
33:                     :Body "new string!"
34:                     :Tagging "dagName=OK"})))

ok, there's not much you can do then right now. see the above.

yeah you won’t be able to get this to work at the moment as the underlying go type is

Body io.ReadSeeker `type:"blob"

I’ll try to find have a look later today/this week if I have time

meanwhile I will see if I can support transit in pods. would be nice either way

ok, got it!

How do I check if var is bound in bb?

Babashka v0.2.2 REPL.
Use :repl/quit or :repl/exit to quit the REPL.
Clojure rocks, Bash reaches.

user=> (def a "a")
#'user/a
user=> (bound? #'a)
java.lang.ClassCastException: sci.impl.vars.SciVar cannot be cast to clojure.lang.Var [at <repl>:2:1]

@i.slack Ah, that needs fixing. For now you can maybe use:

user=> (find-var 'user/a)
#'user/a

or resolve / ns-resolve

made an issue here: https://github.com/borkdude/sci/issues/430

I was trying to see if *file* is bound to distinguish from clj run

is there a better way to know that it is running in bb?

you can check (System/getProperty "babashka.version")

ahh, thank you! that would be much cleaner.

For authoring pods, and this example for error handling:

{"id" "1d17f8fe-4f70-48bf-b6a9-dc004e52d056"
 "ex-message" "Illegal input"
 "ex-data" "{\"input\": 10}
 "status" "[\"done\", \"error\"]"}

yes, error signals whether we should raise an error and done signals that it's the last message for this id

oh ok, cool

Meanwhile: tetris works in pod-babashka-lanterna :)

For replies from the pod where the messages send a few messages for a request id without being done, how are the messages accumulated/handled on the pod client?

@isak you can check the source: https://github.com/babashka/babashka.pods/blob/master/src/babashka/pods/impl.clj

@isak the README is also helpful maybe: https://github.com/babashka/babashka.pods#introduction

typically for async functions you have multiple messages coming in

:thinking_face: I've been reading that document, but I couldn't see that it was specified.

Looked at the source, but not obvious what will happen without knowing how the functions in that namespace will be called. I guess I'll just experiment, will be faster

Makes sense if only async calls are involved with the :success, :error and :done callbacks, just not as obvious for the normal calls

@isak Note that normal functions can also send multiple "out" messages which will be sent to stdout. Also they can send the value and status in separate messages.

This is more or less the same as with the nREPL protocol

Ok makes sense. But only 1 message can deliver the value when not calling the async way, correct? @borkdude

Correct

Ok, then I'm no longer confused, thanks

In a C# pod I'm trying to make, I'm getting an error from babashka when sending this message from the pod:

async Task SendResult(string id, object result) {
    var resp = new BDictionary {
        ["id"] = new BString(id),
        ["value"] = new BString("[]"),
        ["status"] = new BString(JsonSerializer.Serialize(new string[] { "done"}))
    };
    await _writer.WriteAsync(resp.EncodeAsBytes());
}

#error {
 :cause "java.lang.Byte cannot be cast to byte[]"
 :via
 [{:type java.lang.ClassCastException
   :message "java.lang.Byte cannot be cast to byte[]"
   :at [babashka.pods.impl$bytes__GT_string invokeStatic "impl.clj" 22]}]
 :trace
 [[babashka.pods.impl$bytes__GT_string invokeStatic "impl.clj" 22]
  [babashka.pods.impl$bytes__GT_string invoke "impl.clj" 22]
  [clojure.core$comp$fn__5811 invoke "core.clj" 2571]
  [clojure.core$map$fn__5870 invoke "core.clj" 2757]
  [clojure.lang.LazySeq sval "LazySeq.java" 42]
  [clojure.lang.LazySeq seq "LazySeq.java" 51]
  [clojure.lang.RT seq "RT.java" 535]
  [clojure.core$seq__5405 invokeStatic "core.clj" 137]
  [clojure.core$reduce1 invokeStatic "core.clj" 930]
  [clojure.core$set invokeStatic "core.clj" 4115]
  [babashka.pods.impl$processor invokeStatic "impl.clj" 108]
  [babashka.pods.sci$fn__15492$fn__15525 invoke "sci.clj" 67]
  [sci.impl.vars$binding_conveyor_fn$fn__7722 invoke "vars.cljc" 144]
  [clojure.core$binding_conveyor_fn$fn__5758 invoke "core.clj" 2032]
  [clojure.lang.AFn call "AFn.java" 18]
  [java.util.concurrent.FutureTask run "FutureTask.java" 264]
  [java.util.concurrent.ThreadPoolExecutor runWorker "ThreadPoolExecutor.java" 1128]
  [java.util.concurrent.ThreadPoolExecutor$Worker run "ThreadPoolExecutor.java" 628]

This is on windows. (The describe response looks to be correct already, because I'm able to invoke.)