Clojurians Log v2

Hello, i'm trying to use babashka.process for a little thing. I would like to be able to launch a process that do not terminate and do something whenever some new output occurs from it. Is there a builtin way to do this? I've try this example from the documentation:

(future
  (loop []
    (spit "log.txt" (str (rand-int 10) "\n") :append true)
    (Thread/sleep 10)
    (recur)))

(pipeline (pb '[tail -f "log.txt"])
          (pb '[cat])
          (pb '[grep "5"] {:out :inherit}))

@pbaille The readme explains what the problem is: buffering.

oh now you are using pipeline ok

sorry I've edited the message

Yes, so this works:

(require '[babashka.process :refer [pipeline pb]])

(future
  (loop []
    (spit "log.txt" (str (rand-int 10) "\n") :append true)
    (Thread/sleep 10)
    (recur)))


(-> (pipeline (pb '[tail -f "log.txt"])
              (pb '[cat])
              (pb '[grep "5"] {:out :inherit}))
    last
    deref)

thank you @borkdude, I've just tried this snippet in my editor (intellij cursive) on Java 15 and clojure 1.10.0. Am I supposed to see something printed in the repl ?

@pbaille :inherit prints directly to stdout (`System/out`)

Just try (process ["ls"] {:inherit true}) to try it out what happens in your editor

yes it prints the output

in the repl

ok, then that works correctly.

ok, so I should be able to see some output from your first snippet ?

yes, I think so

maybe simplify the example for debugging purposes

it hangs on the last expression

you can remove the deref to make it not hang. I just added that to make the bb script not terminate

let's continue in thread

ok

I got this but nothing prints

Can you please try this in a standalone bb script and then execute it from the terminal?

Or clojure script if you are using this lib on the JVM

yes I will do this

it works nicely as a script

Perhaps try :out *out* in your editor for the last process in the pipeline

else, I don't know :)

I've tried this

but didn't change anything

ok, thank you for your time @borkdude! I will investigate on my own and let you know if I find something

Can I connect to the bb repl in VSC Calva without manually entering it every time? it does not pick it up from .nrepl-port file

@dennisa Babashka does not write an .nrepl-port file since that can conflict with a Clojure process.

I can write it myself, just need Calva to pick it up from somewhere, otherwise getting this promt with no port. I have to restart the bb repl rather often as it slows down and it's a chore to type every time 1667 : )

@dennisa There is a snippet about writing this port file here: https://book.babashka.org/#_nrepl

(I think this script can be simplified/refactored now since there is babashka.process)

Have someone managed to Calva to pick it up? Or am I in the wrong channe?l :)

@pez ^

what's the babashka.process : ) ?

nevemind, found it in the BBook )

I haven’t tried this myself, but you can probably configure a connect sequence, nReplPortFile with the name of whatever nrepl-port file you write. See: https://calva.io/connect-sequences/

^ @dennisa

Now tried it. This setting works:

"calva.replConnectSequences": [
        {
            "name": "bb",
            "projectType": "generic",
            "nReplPortFile": [".bb-nrepl-port"],
            "cljsType": "none"
        }
    ]

echo 1667 > .bb-nrepl-port && bb --nrepl-server

Continuing from #clojure in a thread: I'd like to add babashka support to https://github.com/IGJoshua/farolero The key thing that would prevent it from working as-is is the requirement to be able to throw an object that isn't caught by ex-info catch clauses.

If there's a way to throw an arbitrary object or extend a throwable marker protocol or something that'd be ideal, but since babashka is likely to be used in smaller projects with fewer dependencies, I may if need be just use ex-info, but I'd prefer to not as there's a requirement to re-throw exceptions from catch blocks based on a condition, and a normal catch block for ex-infos won't re-throw based on that condition.

@borkdude

How do you solve this in normal Clojure?

I'll brb

I solve this by making the farolero-signal artifact on maven that contains a single class that extends java.lang.Error (which extends from Throwable but not Exception), https://github.com/IGJoshua/farolero/blob/master/signal/src/farolero/signal/Signal.java Then in the actual farolero lib I just extend-protocol to this class to add the functionality for storing the objects that need to get passed around.

And the reason that it's in a different artifact altogether is to allow git dependencies on farolero.

@suskeyhose So your library, does it catch only this specific exception and handle that one? Or does it install some global exception handler?

It only catches this one specific exception, and it never throws the exception except if it knows for a fact that it's currently in a dynamic context where it will be caught.

For the CLJS support it catches all objects thrown and re-throws them if they don't implement the protocol.

In particular there is exactly one construct that throws or catches this exception, and it is block, and everything else is implemented on top of that.

(block the-block
  (some code)
  (return-from the-block :returned)
  :unreachable)

In CLJS, can you throw any arbitrary object?

Yeah, it's a feature of JS.

A workaround I can come up with for bb is: Make one singleton Exception instance, e.g.:

(defonce FaroleroEx (Exception. "my-special"))

or just use ex-info with a special field in it to mark it as a farolero exception

I think that may just work best for bb

That's a good thought, I might do that. The key reason that I would want to throw arbitrary objects or something would just be to prevent catch blocks from interfering with this mechanism, but if I can't do that in bb, that's fine.

The features already won't be 100% parity anyway because I doubt bb implements locking and so the debugger won't be the same.

bb does implement locking

btw another thought. I see you mix specs with your code. If they are non-essential to your lib, e.g. only for development, I recommend putting them aside in a farolero.specs namespace so they can be optionally loaded. This saves startup time

bb currently doesn't come with clojure.spec.alpha since ... it's alpha

but there is https://github.com/borkdude/spartan.spec as a drop-in replacement

Oh, cool, maybe it can be on part with the debugger then. The main part I was concerned about is this highlighted portion: https://github.com/IGJoshua/farolero/blob/master/src/cljc/farolero/core.cljc#L1192-L1269 I guess it's worthwhile to see if it'll run unmodified on bb though. Thanks! As for the exception mechanism, I think I can just do what you suggested with an extra field on an ex-info

Oh, yeah, the specs. Good to know. The reason the specs are there is just to help with macroexpansion, they aren't used at runtime at all. Moving them to a separate namespace is far from out of the question.

Hmm, dosync and ref etc are not in babashka. Nobody ever asked for this. Those features were cool in the beginning of Clojure but nowadays pretty much everyone uses atoms

Yeah, this was the first time I'd ever found a use for refs.

What is the debugger object?

I might be able to refactor it to use atoms, but it could be a challenge, this is all pretty sensitive code and prone to race conditions on changes.

you call e.g. .wait and .notify on it, what is the type?

It's a unique cons of the condition and arguments.

line 1283

I don't think .wait and .notify currently work, but I think they could be made to work, with some work ;)

not sure how hard it is to port dosync etc

probably it's possible

Fair enough. I'd like to take a peek in the source of babashka/sci to see how hard adding notify and wait are. dosync etc I might be able to avoid by refactoring to use an atom, although we'll see if that ends up worth it.

oh gosh it does work already:

(locking 1 (.wait 1))

Oh nice!

Are numbers interned objects in bb?

hm, this didn't work:

user=> (def c (cons 1 2))
user=> (locking c (.wait c))
java.lang.IllegalArgumentException: No matching method wait found taking 0 args for class clojure.lang.PersistentList [at <repl>:1:8]

Are numbers interned: I don't do anything special with numbers, so I don't know

Ah, I see.

Yeah, I was able to get just a bare object to work as expected with wait and notify

user=> (identical? 1 1)
true

user=> (identical? 1 (+ 1 0))
true

Fair enough. Anyway, it looks like wait and notify don't work on clojure's collections, yeah, so that'd probably be where the work on wait and notify would have to be put.

I'm going afk now, as it's getting late. Let's revisit some other time and make issues for the things that could be examined / improved

Also I'm looking at this and while the ref code could be re-expressed in terms of atoms, it'd be clunky at best.

Sounds good!

btw, I think I know what the issue is. bb supports reflection on a selection of classes. e.g. this works:

user=> (def c (java.io.File. "foo"))
#'user/c
user=> (locking c (.wait c))

so if I add explicit support for a type then it should work

it just doesn't include the wait/notify methods for PersistentList

Fair enough. I'll make an issue for this now.

As well as a discussion issue about ref/dosync

:thumbsup:

As an aside, farolero kind of completed my personal trek through the multithreading facilities that clojure provides, since before this I'd used atoms of course, core.async channels, reducers, and agents for per-endpoint ratelimits on a discord client. Finally with farolero I've also used refs. Now I guess it's time for me to learn manifold, lol.

I think manifold could use some help, so that would be super cool

Honestly the one thing that I felt was really missing from core.async though was a promise channel that you could deref. So I've had to write one. 😛 https://github.com/IGJoshua/discljord/blob/master/src/discljord/util.clj#L93-L151