Clojurians Log v2

How do I approach network programming in Clojure as beginner?

With care? :troll:

Seriously speaking, you need to be more specific than that

Trying to implement a server to host multimedia like images, videos. I'm a beginner.

Ok. And apparently you are planning to use Aleph. Well, the easiest solution is to just return an InputStream object to your media file to the browser

This allows streaming the file from filesystem to the browser without you having to first load it into server memory

Which would be disastrous especially with videos and multiple concurrent users

Plus would make it way slower to start the download for the client since server would have to first load the whole file. By returning a stream client will start to receive the file immediately

Now when implementing a true video media server then you'd have to take into account also jumping around in the video, this is more about video codecs and formats and how to deal with those

But basically the basic idea is the same there: you find the correct position in the video file, then create an InputStream to that file starting at that specific point in time and send that to the client

But it gets more complex since you cannot just start streaming from a random byte in a video file: in most codecs there is a concept called "keyframe" which is a point in time where you can rewind into and start streaming from there

Basically a time segment between two keyframes is a "full video file" so normal video file is actually a sequence of these

And this is the actual video part of the file, audio is a story on itself

But anyway, you get the point. This is a way too complex topic to cover in detail here but the very basics of it is very simple when it comes to your original question

With images however it is very simple, client always needs the whole image. Just send it in whole to the client as a stream

Remember to set proper Content-Type header so browser knows what kind of data it is it is receiving

All this has nothing to do with Aleph itself really although Aleph can be used as a server for this

Hope this helps

Thanks alot 🙂

Hello, I have a TarArchiveInputStream object and I want to make it available as a download over http. I tried the response on the handler like this

{:status  200
 :headers {"Content-Type"        "archive/tar"
           "Content-Disposition" "inline; filename=download.tar"}
 :body    (get-stream)}

You need to provide also Content-Length header

Aleph cannot deduce that automatically from stream (in order to do that it would first have to read the whole stream which would make the whole stream pretty pointless)

Thus you need to retrieve that information from somewhere else,

If it is in filesystem, retrieve that data from filesystem

if it is in database, record the content-length to database when you store it as a blob

Also if your file is in the filesystem you can just pass a File object to aleph and it understands what you want to do

Aleph will get the file size from filesystem and will send the FIle as a stream to the client

However, if you are calculating the tar file on fly from something else then you need to solve your problem separately

@ahmed1hsn Please, check this example I’ve implemented recently: static files server on top of Aleph https://github.com/kachayev/nasus

Other than that… the question is really about details: number of clients, size of files and frequency of changes, geo distribution, goals/plans, hardware/network in terms of bandwidth etc etc etc

If you need to implement backend for video- or sound streaming…. that’s an entirely different dimension and different problems

@niklas.collin in fact… it’s better not to provide content-length header. In such a case the chance that it would work correctly is actually higher 🙂 If body is represented as an InputStream, Aleph should invoke send-streaming-body that automatically sets “Transfer-Encoding: chunked” header when content length is not provided manually https://github.com/ztellman/aleph/blob/626bec56dea84ddc5930f8bd9852fc1a236e23f7/src/aleph/http/core.clj#L260 (usually you’re not able to tell the size of the stream upfront anyways)

@rahul080327 Do you have any debug information about packets sent? (either from browser or wireshark or similar) So we can check raw request/responses?

@kachayev So to give some context im building a clojure system over a lib i created. It exposes a path in a docker container as a TarArchiveInputStream. https://github.com/lispyclouds/clj-docker-client/blob/master/src/clj_docker_client/core.clj#L318 This is beacuse the Spotify docker lib im using does it. I want to expose this over HTTP so that a client can download the stuff from the container

This is how i call my lib:

(defn get-stream-of
  "Fetches the TarStream of a path from a container."
  [path id]
  (docker/stream-path e/conn id path))

This is my aleph handler:

(defn stream-artifact
  [name group number artifact]
  (d/let-flow [pipeline (u/name-of name group)
               id       (a/artifact-container-of pipeline number)
               path     (a/get-artifact-path-of pipeline artifact)
               stream   (a/get-stream-of path id)]
    (if (f/failed? stream)
      (res/bad-request {:message "No such artifact."})
      {:status  200
       :headers {"Content-Type"        "archive/tar"
                 "Content-Disposition" (format "inline; filename=%s.tar"
                                               artifact)}
       :body    stream})))

im able to list all files in if i do it in the REPL. just that i cant send it over http

If you want the browser to start downloading, you need attachment not inline, right?

tried that too

inline means that you want this to be shown on the page, which is impossible. Yeah, I known this is not the root of the problem, just noticed

Just tried a curl, totally empty response

Could you please show the result of curl -vvv ...?

$ curl -vvv <http://localhost:7777/api/pipeline/test/test/1/artifact/source>
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 7777 (#0)
&gt; GET /api/pipeline/test/test/1/artifact/source HTTP/1.1
&gt; Host: localhost:7777
&gt; User-Agent: curl/7.54.0
&gt; Accept: */*
&gt;
&lt; HTTP/1.1 200 OK
&lt; Content-Type: archive/tar
&lt; Content-Disposition: attachement; filename=source.tar
&lt; Server: Aleph/0.4.4
&lt; Connection: Keep-Alive
&lt; Date: Tue, 05 Mar 2019 13:40:48 GMT
&lt; transfer-encoding: chunked
&lt;
• Connection #0 to host localhost left intact

Dunno if it may help, im using compojure-api for the routing

https://github.com/bob-cd/bob/blob/master/src/bob/api/routes.clj these are the routes where im creating a new one

Any chance to upgrade at least to 0.4.6?

I can

Let’s try, this version is ancient 😉

it is at 0.4.6

< Server: Aleph/0.4.4

https://github.com/bob-cd/bob/blob/master/build.boot#L27

transitive deps?

you’re cURL response indicates another version

not sure where to put it.

yeah ive noticed it too. saw it with plain standalone aleph projects too

but afaik im using 0.4.6

ls ~/.m2/repository/aleph/aleph/
0.4.6

Ah, I see

https://github.com/ztellman/aleph/blame/626bec56dea84ddc5930f8bd9852fc1a236e23f7/src/aleph/http/server.clj#L113 this was changed only in 0.4.7-alpha2

:thisisfine:

aha 😄

im not sure if its an aleph problem or something else, have a lot of moving parts in my code. can try it without the compojure-api routing

I doubt it’s about routing

Nevertheless, I will try to reproduce locally with just a single tar stream to see how it works

Thanks a lot for the help! 😄 you have all the repos there too. lemme know if you need more info

@rahul080327 another question, you said you’re using TarArchiveInputStream but the filename is *.tar… what do you want to archive? read file and return as a tar archive? or read an archive and return as unzipped file?

I wanna return the file itself. a single download, as the path may contain multiple files

the docker lib returns a tar stream of the path. i want to return an archive of that path over http

if they’re reading directory to create a tar… shouldn’t that return TarArchiveOutputStream?

https://github.com/spotify/docker-client/blob/master/docs/user_manual.md#get-an-archive-of-a-filesystem-resource-in-a-container

should i convert to an output stream? i was under the impression aleph needs an input stream to send stuff over http

no, but you don’t need to wrap it in TarArchiveInputStream either

https://github.com/spotify/docker-client/blob/14b4464fb7acec25073c8c9c564136188f94562c/src/main/java/com/spotify/docker/client/DockerClient.java#L1348 this returns you an InputStream already (they just do a copy and then open you a stream to a copied archive)

You need TarArchiveInputStream *only* in case you want to get raw files (decompressed)

ah okay. But will this cause an issue for Aleph? TarArchiveInputStream is an InputStream too, so i thought it would be okay

this works

Tried with FileInputStream, - works fine

Try this, let me know if it works

okay

👍

@kachayev But these are the constructors of FileInputStream FileInputStream(File file) FileInputStream(FileDescriptor fdObj) FileInputStream(String name) and i have a InputStream

Right

I’ve just showed that as an example

You can set :body to your InputStream

okay without the wrapping

Yep, exactly

Ah yesss, it works!!

Thanks a ton! 😄 😄

Sure! Let me know if you have more questions. Glad to help

will do. great learnings and kudos for having aleph around!