Clojurians Log v2

We are using clojure.java.shell/sh for a long running command. Is there a way to stream the output of the command to stdout as it is running? We are hoping to get visual feedback of the command's progress instead of having to wait for it to complete to check the return map's :out key.

I've not used it myself, but maybe @borkdude's https://github.com/babashka/process can help

@mitchell_clojure Yes. Use (babashka.process/process ["your-program" "arg"] {:inherit true})

you are the best! tyvm everyone

So if I wanted to capture the output to be processed by my script while also outputting it to stdout, I would have to open a reader which prints to stdout and also appends to a string

correct

you could just use a loop to read from the out stream

and read lines from that and append to some string writer + stdout

Something like:

(binding [*in*  (io/reader (:out tail))
          *out* (io/writer (:in cat-and-grep))]
  (loop []
    (when-let [x (read-line)]
      (println x)
      (recur))))

(this is from the README)

What's the idiomatic way to say (if (str/blank? dir) nil dir)

Seems fine to me. There’s also:

(when-not (str/blank? dir) dir)

this is also easy in normal clojure via interop:

hammurabi$ clj
Clojure 1.10.1
(cmd)user=> (-> ["ls"] (ProcessBuilder.) (.inheritIO) (.start) (.waitFor))
CHANGELOG.md  examples	project.clj  resources	target	todo.otl
doc	      LICENSE	README.md    src	test
0
user=> 

the .waitFor makes the repl wait until the process exits before printing the next prompt

there's also (not-empty dir)

(ins)user=> (not-empty "")
nil
(ins)user=> (not-empty "foo")
"foo"

I'd argue not-empty is most idiomatic as it is specifically designed for this use case (nil on empty, without changing input type to seq)

not-empty won't defend against a blank string, which dir could be

did you look at my example?

it explicitly returns nil for empty string

oh - you mean whitespace?

right, if whitespace is a concern not-empty? doesn't help

yes, I assumed they wanted to defend against a "blank" string as opposed to an empty one

yes, but if you want to read from the process's stdout you should not use inheritIO

he wants to get the string, but also write to out

ahh, that's different, yes

Hi .. i want to change the log level on the command line when i run tests (sometimes) I tried JVM_OPTS='-Dlog.console.threshold=INFO" lein test and also using `LEIN_JAVA_OPTS instead of JVM_OPTS`

definitely more complex

(let [pb (ProcessBuilder. ["ls"])
      p (.start pb)
      in (.getInputStream p)
      result (StringBuilder.)]
  (loop []
    (let [b (.read in)]
      (if-not (pos? b)
        (str result)
        (let [c (char b)]
          (.append result c)
          (print c)
          (recur))))))

@clojuregeek I would usually handle this by making a separate lein profile with :quiet {:jvm-opts ["-Dlog.console.threshold=INFO"]} and use with-profile to add that profile lein with-profile +quiet test

oh thats a good idea

that way it combines correctly with other additions you might want to make to the java command line

(require '[babashka.process :refer [pipeline pb]]
         '[<http://clojure.java.io|clojure.java.io> :as io])

(let [[catp teep] (pipeline (pb ["cat"]) (pb ["tee" "log.txt"] {:out :inherit}))]
  ;; write to cat every 100 milliseconds 10 times
  (future
    (with-open [writer (io/writer (:in catp))]
      (loop [i 0]
        (when (&lt; i 10)
          (binding [*out* writer]
            (println i)
            (flush))
          (Thread/sleep 100)
          (recur (inc i))))))

  @teep ;; wait for the tee process to finish
  ;; finally we print the file's content
  (println (slurp "log.txt")))

(shutdown-agents)

This works if you have tee on your system

the first process is cat, to which we write 10 times (just to simulate some output)

and this output is captured by tee, which writes it to stdout AND to a file

I think it could be useful to have this built into the process lib

I'm trying to setup kaocha, and I tried running the following line in the very beginning of the kaocha install docs and got this error:

❯ clojure -Sdeps '{:deps {lambdaisland/kaocha {:mvn/version "1.0.700"}}}' -m kaocha.runner --test-help
Execution error (FileNotFoundException) at kaocha.specs/fn (specs.clj:82).
Could not locate clojure/test/check/generators__init.class, clojure/test/check/generators.clj or clojure/test/check/generators.cljc on classpath.

you need to pull in the generative deps sepearately to use that feature of spec

org.clojure/test.check {:mvn/version "1.1.0"}

That worked! Thanks @noisesmith! I didn't see that library dependency anywhere in the kaocha docs. It looks like clojure/test/check is a reference to the test.check library? Can I assume that future error messages that has that kind of path will help me identify missing dependencies?

right - the deal is that clojure.spec contains code that dynamically looks for that code

it's not a "hard dependency" because if you don't want to use those features you don't need to include the extra dep

it's a gotcha because most libs don't work this way

Got it! I just started playing with kaocha, and I didn't know that kaocha.runner needed spec 😕

IMHO spec should intercept the error and include a message like "to use this optional spec feature you need the following package ..."

spec comes with clojure

certain spec features require this other optional dep that doesn't come with clojure

oh! my bad! I guess kaocha.runner calls spec, then spec in turns calls test.check?

right

cool, understood! Thanks for the tip! 😄

I posted some questions on reddit. Thought I'd post them here too so I get more answers https://www.reddit.com/r/Clojure/comments/jed5z9/new_clojurians_ask_anything/g9gpmly?utm_source=share&utm_medium=web2x&context=3

How big of a problem is dependency conflict in Clojure? I've seen a lot of libraries that have not been updated for 3 - 4 years. I'd imagine in a scenario where use one such library which depends on an older version of another library (library-x for example), and I also use a more updated library which depends on a recent version of library-x, would that have high probability of causing troubles?

I've found that in Clojure, I always want to make the perfect API upfront because being a dynamic typed language, it's harder to refactor after. I come from a background of typescript where I just write function that work first then come back to change the interface, maybe even much later along the road, because it's very easy. What have been your experience?

I corrupt the REPL state a lot, especially on newer projects when everything is not so figured out (dependencies, etc...). Can you share tips on working with REPL so that it doesn't hinder productivity.

What need to happen so that a typical clojure web project (jdbc, ring api) can be run under graalvm? I know in Java land, there are Quarkus and Micronaut which support graal pretty well. Fast startup and low memory usage are sought after features these days because of microservices. Do you think we Clojurians will have these in the future?

Thank you very much~

most clojure libraries change API very little, even as versions are updated. The main version problems come from poorly designed deps coming from java (eg. jackson, a recurring problem for many projects). If fast startup of your server is a goal I'd switch to another language, graal native image comes with a huge set of problems for clojure usage

My experience with REPL state is that I change dependencies very rarely compared to the amount of development I do within one REPL with one dependency set.

NB about server startup: my baseline assumption is that you are using an uberjar, and running your server via the java command or a java process runner, if you use eg. deps.edn or lein on your server itself you are doing it wrong

one more about the microservices: reduced memory usage is not a feature under development (both the vm itself and clojure have multiple design decisions implemented that trade more memory usage for speed)

thank you for taking the time to answer, these are valuable insights

To answer the first, clojure library authors tend to prefer stability. Meaning that a library gets very little breaking changes, and versions that are years old will still work perfectly fine now. Some libraries go as far as just starting a new namespace when they introduce breaking changes, meaning you can use either the ‘old’ api, or the new

Compared to the JS/TS communities, Clojure libraries tend to include less breaking changes (if any), and, I’d personally say thats also part of the … how do you say … style? culture? of Clojure

Its like the Python community has their zen of Python right, part of the zen of Clojure is to write slow moving software 🙂 If that makes sense

That makes total sense. And I think that also answers my second question: it's normal to want to put more thoughts into the initial API, in the spirit of writing stable software. I am gonna have to be much more disciplined when writing Clojure code and ditch that "just gonna put this thing here for now" attitude that I'm so used to. Thank you!

you can get very far with using immutable data structures with descriptive keys for nearly everything, the spec system ignores unknown keys so it's automatically "future proof" in that respect

overuse of deftype / defrecord is something to look out for coming from ts