Clojurians Log v2

Hi! Quick question: how do you add an EOF to a string you provide to :in for planck.shell/sh ?

@bloemelau Looks like a bug. Is this similar to what you are encountering: https://github.com/mfikes/planck/issues/516

Yes, exactly

Guess I found a bug :P

@bloemelau Indeed. It looks like it is not even implemented in the C port (on the native side of Planck)

Ah, I was looking for the relevant code, but could find it. That's why.

@bloemelau Me too. I’ll see if I can add it. It is easy enough to see the string on the native side. The trick would be to set it up as the input pipe contents for the process. Let me see if that is not too difficult…

How well is piping supported in planck? Do functions get input as soon as there's output in the :in provider? Because if that's a goal the implementation can be a lot harder...

@bloemelau Well, it is entirely under our control, given it is written in C. I’ve succeeded in piping in some text into cat. Checking to see if I can get it to properly handle lots of text, where there might be blocking. What I don’t understand is how the API is supposed to distinguish between a regular string, vs. one that represents a file source.

(I didn’t write the initial Planck shell, implementation, and glossed over this aspect.)

Generally, the goal is to make it act just like clojure.java.shell if possible.

Ahh… clojure.java.shell/sh handles types. Planck has types for all of the things mentioned in the Clojure docstring

Does that have a lazy implementation? Is that even possible?

I mean in bash { echo hi; sleep 5; } | grep hi outputs hi immediately

Would something like (sh "grep" "hi" :in (str/join (map :out [(sh "echo" "hi") (sh "sleep" "5")]))) do the same?

Or isn't that a design goal?

@bloemelau In the implementation I just scratched together, it blocks for 5 seconds before printing the result

cljs.user=> (sh "grep" "hi" :in (str/join (map :out [(sh "echo" "hi") (sh "sleep" "5")])))
{:exit 0, :out "hi\n", :err ""}

Since JavaScript doesn’t have threads, Planck has a sh-async that can do nice things with calls like your sleep example

As expected. What I mean is, is there any goal in making :out/:err a stream instead of a string, so you can read as soon as there is something there?

Because with the current implementation you could'nt for example run a server and meanwhile parse the log messages or something. The process needs to end before you have output and you'll have the whole batch at once

So, I’m thinking that, for some things, like if you pass an instance of <http://planck.io/File|planck.io/File>, then on the native side it could read the file, and handle all of the streaming on the native side.

Ahh, right… you are alluding to https://github.com/mfikes/planck/issues/54

There might be some clever way to make it all work, with the JavaScript engine simply blocking if you, say, try to read from a stream that has no data yet.

This branch, for example has experimental code that streams with sockets, but it is really easy to get it to lock up if it blocks https://github.com/mfikes/planck/commits/socket-io-support

I’ve hooked up :in in a naïve way for strings in Planck master with https://github.com/mfikes/planck/commit/5f75b3eda1bb0a9cf07c87bb6ade79053c0d08d8 I think the docstring needs to change so that it can know when it is dealing with other values for :in

Works wonderfully. Thanks for the quick patch :)

Yeah, the streaming idea needs some serious thinking for it to work out, I think. But it's good to see there's people involved in it.

Yeah, happy to take patches for this stuff. I'll probably get around to it sooner or later (I often fix Planck if I'm using it for something it can't do yet.)

in the commit message you say smallish string, why's that? Limitation of value_to_c_string?

@bloemelau Nah. Just a presumption on my part that with something like cat it might block reading stdin after a while if you don’t read from its stdout. The way the code is structured, it tries to write everything to the process first, expecting it to hold everything in the pipes or RAM or somewhere. In fact, it doesn’t even properly check that all of the characters have been written.

Something akin to this loop https://github.com/mfikes/planck/blob/master/planck-c/sockets.c#L14-L43 should probably be worked into the select loop below the current write call

Ah. Well It seems I've already hit the wall :P

In other words, I suspect that the code could select on the subprocess stdin to know if it can write data

I bet the code on Planck master can’t write more than 4K

42944 bytes it stopped here

And emacs died in the process