Clojurians Log v2

Hello, I have a question. I am trying to set up a server that can receive and process multipart messages. My attempt is with aleph with yada resources. But I have difficulties in processing the manifold.stream in the response. Do you have some tips and tricks?

This is the code I have:

(yada/listener
    ["/some-route"
     (yada/resource {:methods
                     {:post {:consumes "multipart/form-data"
                             :produces "application/json"
                             :response (fn [ctx]
                                         (let [boundary (last (clojure.string/split (get-in ctx [:request :headers "content-type"]) #"="))]
                                           (try   (->> (get-in ctx [:request :body])
                                                       (yada.multipart/parse-multipart boundary 10000 1000)
                                                       (manifold.stream/transform (xf-bytes->content))
                                                       manifold.stream/stream->seq)
                                                  (catch Exception e
                                                    (println (.getMessage e)))))
                                         (clojure.data.json/write-str {:success true}))}}})]
    {:port        1234
     :raw-stream? true
     :executor    (flow/utilization-executor 0.9 64)})

what do you want to do with the stream @eelke? note that stream->seq should probably come with a warning - it's a blocking function

Hey @mccraigmccraig, thanks for the response. I would like to process what comes through the stream. I can be that the client sends large multipart requests, containing tens of images or videos, and the server needs to handle this

@(d/chain (get-in ctx [:request :body])
           #(manifold.stream/map bs/to-byte-array %)
           #(manifold.stream/reduce conj [] %)
           bs/to-string)
                                                                      

This does result in a string but it does not always contain everything that was send, I think

ha, i'm recalling a very bad evening spent debugging...

question - do you have {:raw-stream? true} specified in your aleph.http/start-server opts ?

as per: https://github.com/juxt/yada#troubleshooting-faq

if you don't, flaky multipart parsing ensues

ahah, I can imagine. I have it as an option indeed

you can see it in the map I sent with the listener

doh, yeah

I feel like I am close

i've not been using the stream-based multipart interface - what does yada.multipart/parse-multipart create a stream of ?

Is there something else you use for multipart?

we've been using the PartConsumer interface ... but iirc that's because that came first and we've been using it since the beginning - i would have preferred a stream-based interface

i'd have a look around for an example of what the canonical yada resource should do now... PartConsumer wasn't the easiest to get working

ah ok, I'll keep trying