Clojurians Log v2

Thank you so much! Will give it a spin.

@karol.wojcik Perhaps I will remove the logging stuff which I pretty much butchered together and will just expose *task-name*. So you can do:

(println ">>>" *task-name*)

Can we use some kind of middleware/interceptor pattern for tasks (similar to :init)? That way we could globally define and customize things like printing task names, logging responses, etc.

hmm, interesting!

:pre-task (fn [...] ...) :post-task (fn [...] ...) ?

I guess if you want to save some state, you can accumulate it in some atom you define in :init

nice idea

:interceptors [{:name "optional"
                :pre-task (fn [..] ...)
                :post-task (fn [..] ...)}]

^ perhaps something like that, and you can probably ship with a couple default interceptors built-in

so the previous discussions would boil down to:

:interceptors [log-task-names, log-task-outputs]

https://github.com/babashka/babashka/issues/805

Please allow binding interceptors from task script as well.

What do you mean exactly?

I want to add interceptors from script not via bb.edn:

(ns holy-lambda.tasks
  "This namespace contains tasks!"
  (:require
   [clojure.string :as s]
   [clojure.pprint :as pprint]
   [clojure.java.shell :as csh]
   [clojure.edn :as edn]
   [babashka.tasks :as tasks]
   [babashka.deps :as deps]
   [babashka.fs :as fs]
   [babashka.curl :as curl]
   [babashka.process :as p]
   [<http://clojure.java.io|clojure.java.io> :as io]))

(tasks/add-interceptors [{:name "Wrap task"
                          :enter (fn [task-name] (println task-name))
                          :leave (fn [task-name] (println task-name))}])

What is the reason you want this in addition to declaring them in bb.edn?

Hmm, we could add the value of each task execution into the middleware map as it executes

and then each middleware fn gets to see the middleware map

and can add/remove to it

> Hmm, we could add the value of each task execution into the middleware map as it executes hmm, but this is already supported by :depends as well, but perhaps it doesn't hurt to have some more flexibility

baz {:enter (fn [m1] (assoc m :foobar 2)) :task 66 :leave (fn [m2] ...)}

Interceptors is an implementation detail for me. I don't want the user to change them.

What is "the user"

The one who interacts with holy-lambda via executing provided bb tasks.

Is holy lambda generating a template project for the user with a bb.edn in it?

or how does it work?

When I'll be done with tasks I will just reference to holy-lambda-babashka-tasks/bb.edn from template. Running

lein new holy-lambda <some-project>

bb version

ok but note that a user of the template is as much a "power" user of the bb.edn as you are, a template is only a starting point of a project after which you make your own modifications

so whatever changes they want to make to the bb.edn, that should be allowed

Actually that's a good point

and btw, version is also a subcommand of babashka which you are overriding

but that shouldn't be a problem

I will use tasks:version then

Thanks!

Instead of calling it interceptors maybe a more basic "hook" system works better (also in combination with parallel tasks). You can define a global :enter and :leave hook (which gets to mutate the global context map) and you can override those per task. If you want to call the global hook, just call it manually in the override.

This way you don't get any weird API edge cases with interceptor ordering/insertion/appending.

Pretty much the same as (sync) Ring but two hooks, one before and one after

or can we get away with just one?

I think it's more flexible to have one before and one after hook probably

possibly you can also decide to abort/skip the task in the :enter hook based on some condition

Is there a built in way to stream out a processes output whilst the process is running? Right now I am just doing it myself which is fine. Just checking if I was missing out on some included batteries?

(defn attach-out-printer
  [proc]
  (thread
    (with-open [rdr (<http://clojure.java.io/reader|clojure.java.io/reader> (:out proc))]
      (let [lines (line-seq rdr)]
        (doseq [line lines]
          (println line)))))
  proc)

(-&gt; (process ["./gradlew" "jvmFatJar"] {:dir kgpu-dir})
        (attach-out-printer))

yeah, even logic like should a "failing" task continue quietly or Sys/exit could be customized by a :leave hook

@slack1003 {:out :inherit :err :inherit :in :inherit} or {:inherit true} for short

nice! i’ll try that

ah missed that. just started using your lib

yup that works great

in the process of converting the few little helper build scripts in my project over to your tasks system. Very smooth sailing so far!

@slack1003 cool. The tasks system also has a shell function which you can use like:

(shell {:dir kgpu-dir} "./gradlew jvmFatJar")

alright. good to know. right now tasks has just been calling my prexisting bb scripts but I guess I may just do some shell stuff directly

both approaches are totally fine. in bb 0.3.6 I just added a new :continue option so shell won't automatically exit with a non-zero exit code

that will be a nice addition. though all my stuff so far is happy to bail on non-zeroes

yeah, that's the default in shell

is cheshire.factory included in babashka.json? I have a need of customizing the underlying object-mapper object. This is kind of nice in e.g. jsonista since you can pass it your own object-mapper.

https://github.com/babashka/babashka/blob/bbf144fbce66c6986253119eb81392a440cb17c6/src/babashka/impl/cheshire.clj

the answer is no

😅

indeed. I'm not planning to since I'm not certain if cheshire is the right json tool in 2021 now that jsonista and clojure.data.json are also gaining popularity

so I'd like to keep the surface area somewhat minimal so it might be easier to port to some other solution in the future

could you convince me of the usefulness of providing your own object-mapper? examples?

Options when parsing json. Right now I'm getting: ; clojure.lang.ExceptionInfo: [line 1, col 51] Unsupported escape character: \U. This would not be a problem if I configure the underlying object-mapper. e.g with jsonista:

(def object-mapper
  (-> jsonista/keyword-keys-object-mapper
      (.configure (.mappedFeature JsonReadFeature/ALLOW_UNESCAPED_CONTROL_CHARS) true)))

so now, when using jsonista I can do: (jsonista/read-value json-string my-object-mapper)

so basically this blocks me from using babashka from our ci-pipeline, but im going to figure out if I can use jsonista some way.

you can't currently use jsonista in babashka. Is there a way to fix the json first with str/replace?

Gonna see, the json we receive is malformed from a system which we do not control. This option is kind of hacky indeed 🙂, I'll check it out. We still use BB for other parts of our CI and it's a pleasure to work with!

its basically a nested json structure containing a json string as a value of a key which we do not want to parse directly.

Perhaps you can use a tool like jq to fix the JSON (if that tool supports it...)

Can you give a small example of such JSON?

How can I change in docker where .deps.clj/ClojureTools are downloaded? Here is what I tried:

ENV _JAVA_OPTIONS=-Duser.home=/project/.holy-lambda/
ENV XDG_CACHE_HOME=/project/.holy-lambda/
ENV XDG_CONFIG_HOME=/project/.holy-lambda/
ENV CLJ_CACHE=/project/.holy-lambda/
ENV CLOJURE_TOOLS_CP=/project/.holy-lambda/
ENV HOME=/project/.holy-lambda/

RUN curl <https://raw.githubusercontent.com/borkdude/deps.clj/master/install> | bash
RUN bash -c "cd project/.holy-lambda &amp;&amp; deps -P"

CLOJURE_TOOLS_DIR is the thing! 🙂

Should the :enter hook be executed before or after dependencies... :thinking_face:

What is CLOJURE_TOOLS_CP ?

Before I started reading deps.clj source code I tried my luck with this env var 😄

Have you ever seen babashka load/pod to write to '?' folder?

This environment variable is also documented in the README

no

Ok. It's really weird. Here is repro: https://github.com/FieryCod/holy-lambda/commit/22e18f5f434cfff484db5dd019a541ac04705062 git clone git@github.com:FieryCod/holy-lambda.git && make build-docker && cd modules/holy-lambda-babashka-tasks && bb stack:sync After command finishes you will see '?' folder 😮

could it be related about the order of these commands?

RUN mkdir -p /.babashka
RUN chmod -R 777 /clojure
RUN chmod -R 777 /.m2
RUN chmod -R 777 /.babashka

WORKDIR /project

shouldn't WORKDIR go before mkdir?

I'm just guessing, I don't know your project well enough

Nope. It's something with resolver. I got folder .holy-lambda/.babashka/pods/repository/org.babashka/aws/0.0.5/, but bin is downloaded to '?' ?/.babashka/pods/repository/org.babashka/aws/0.0.5/

$ bb stack:sync
----- Error --------------------------------------------------------------------
Type:     java.lang.Exception
Message:  File does not exist: stack:sync

In which folder are you?

oh right

> After command finishes you will see '?' folder how do I see this folder?

oh yes, I see it

Uff. It's good that I'm not crazy 😄

I don't have time to debug your program, so can you make a small repro?

Sure

@borkdude Here it is: https://github.com/FieryCod/babashka-docker-path-repro

Just run make

Do you have a more minimal one? There is a lot happening in this Dockerfile

Oh I see you have stripped it already

It's stripped.

I can use alpine if you want me to to make it even more minimal

But I guess you have graalvm-ce in cache so build should not take long

I don't use docker locally for graalvm, but I'm going to try. I do see the question mark dir, but how do I know this is not an issue with your docker config vs pods?

😄 I'm already using the same strategy for mountig .aws directory no question mark there

FWIW when I make a .babashka directory inside your repo, then I don't get the question mark dir

Hmm isn't it proof that it's something wrong with babashka resolver then?

I don't say there isn't but right now I'm debugging your docker issues while I have another job to do, so I'd say this is not a clear repro yet

Sure. I fixed the repro. It should be very minimal right now. You can spot '?' in error:

----- Context ------------------------------------------------------------------
 7: 
 8: (def PODS {'org.babashka/aws "0.0.5"})
 9: 
10: (doseq [[s v] PODS]
11:   (pods/load-pod s v))
      ^--- ?/.babashka/pods/repository/org.babashka/aws/0.0.5/manifest.edn (No such file or directory)

----- Locals -------------------------------------------------------------------
seq_2:   ([org.babashka/aws "0.0.5"])
chunk_3: nil
count_4: 0
i_5:     0
vec__9:  [org.babashka/aws "0.0.5"]
s:       org.babashka/aws
v:       "0.0.5"

----- Stack trace --------------------------------------------------------------
user - /usr/bin/download_pods:11:3

@karol.wojcik When I print in the script:

(prn (or
            (System/getenv "XDG_CACHE_HOME")
            (System/getProperty "user.home")))

When you run make you will see "?"

yes. it seems your XDG_CACHE_HOME env variable isn't properly set in the script. so it picks the value of user.home which is a question mark.

so it turns out I have been debugging your Docker issues instead of pods. :/

Hmm.. With this it still outputs question mark:

FROM <http://ghcr.io/graalvm/graalvm-ce:ol8-java8-21.1.0|ghcr.io/graalvm/graalvm-ce:ol8-java8-21.1.0>
MAINTAINER Karol Wójcik &lt;Karol Wójcik&gt;

WORKDIR /

RUN microdnf install wget

RUN mkdir -p /project
ENV _JAVA_OPTIONS=-Duser.home=/project
ENV XDG_CACHE_HOME=/project
ENV XDG_CONFIG_HOME=/project
ENV CLJ_CACHE=/project
ENV CLOJURE_TOOLS_DIR=/project
ENV HOME=/project
ARG BABASHKA_VERSION=0.3.6

RUN wget -c <https://github.com/babashka/babashka/releases/download/v$BABASHKA_VERSION/babashka-$BABASHKA_VERSION-linux-amd64.tar.gz> -O - | tar -xz
RUN chmod +x bb
RUN mv bb /bin/bb

COPY download_pods .
RUN chmod +x download_pods
RUN mv download_pods /bin/download_pods

I think I have provided you with enough debugging now. I won't look at this anymore unless you can provide a repro without Docker, etc.

😕

I don't want to sound like an asshole, but I what I mean is: this is a contextual problem between your Docker setup and bb. You can do more diagnosis before you poke me about it. E.g. bash into the container and inspect:

bash-4.4$ bb
Babashka v0.3.6 REPL.
Use :repl/quit or :repl/exit to quit the REPL.
Clojure rocks, Bash reaches.

user=> (System/getProperty "user.home")
"?"
user=> (System/getenv "XDG_CACHE_HOME")
nil

My first repro lacked one env variable which is: XDG_DATA_HOME

It sounded like you being asshole, but no worries I deserved it 😄

I should have studied resolver code more. I did not spot XDG_DATA_HOME I'm really sorry!

no worries ;)

I'm deeply sorry. I will spend more time on reading babashka code! 🙂 No more pokes today 🙂 Have a great day!

I don't mind the pokes, I think you're making a very interesting project. It's just that in case the bugs aren't directly related to bb a more clear cut repro could be made. Feel free to poke me anytime after you've done your part of the research, so it will save me time.

and if I think you should make it more clear cut, I will just say so (hopefully without sounding like an asshole :-D)

I also enjoy your feedback on the tasks, so thanks for that

at least if you run as a user, which doesn't have a /etc/passwdentry, then you get ? for (System/getProperty "user.home") (just came to my mind from some previous docker adventure)

Huh. I didn't know that. Thanks @viesti

You can force this property in bb by doing bb -Duser.home=/foobar

Hi I am trying to use babashka.process to run a subprocess, but I don't want to capture its stdin/stdout/stderr (like subprocess.run from Python by default). Is there anyway to do this :thinking_face: ? I did not found anything in the documentation.

@thiagokokada https://clojurians.slack.com/archives/CLX41ASCS/p1619438205104400 :)

if you didn't find that in the documentation, then the docs need improving

Huh... Did try this but it isn't printing anything on REPL

Ah, of course

It will not print anything on REPL 😛 (since it is not redirecting the output)

I did find the :inherit option, but it wasn't really clear what it does by the explanation But looking at the examples it seems ok

cool

It is working fine, thanks @borkdude

I think I'll keep it more basic, in the same fashion :tasks are just expressions:

Re-considering, I think this is all that's needed to support logging of task names:

add *task-name* dynamic var
global :enter (println :> *task-name*)
global :leave (println :< *task-name*)
able to replace :enter and :leave on the task level
This is all that's needed, even to support some kind of middleware pattern, since you can define an atom in :init where :enter and :leave can mutate something to.

Have to say, porting some internal tools from my company from Bash/Python to Babashka

It is awesome!

Is there any logging library available in babashka :thinking_face: ?

@thiagokokada At the moment, no. But here is a tiny useful snippet perhaps: https://gist.github.com/borkdude/c97da85da67c7bcc5671765aef5a89ad

It is printing the correct line but it always print the namespace user

Weird, the gist still seems to work for me where it prints the ns bar

I am using the -m flag

Since it is a full Clojure project (with src, etc)

ah!

Maybe because of this :thinking_face:?

(ns user (:require [foo.core])) (apply foo.core/-main *command-line-args*)

I was trying to create a new project to demonstrate, but this got to my attention

(It is an exception btw)

Can you try this instead?

(defmacro log [& msgs]
  (let [m (meta &form)
        file *file*]
    `(binding [*out* *err*] ;; or bind to (io/writer log-file)
       (println (str ~file ":"
                     ~(:line m) ":"
                     ~(:column m))
                ~@msgs))))

pulling *ns* outside of the macro body might also work

Yep, the one with *file* works

BTW, if you want to reproduce it

Run bb -m foo.core

But this alternative with *file* seems even more useful to me

:thanks:

ok, you like file better than ns?

(ns logger)

(defmacro log [& msgs]
  (let [m (meta &form)
        ns *ns*]
    `(binding [*out* *err*] ;; or bind to (io/writer log-file)
       (println (str ~ns ":"
                     ~(:line m) ":"
                     ~(:column m))
                ~@msgs))))

Yeah, now that I think about it maybe ns is better (the printed line is smaller)

Don't know yet, I will do a experiment here

But thanks anyway

ok, adapted the gist

From my search I don't think there is a way, but asking anyway (since I didn't found anything for neither Clojure nor Java) Is there anyway to set an environment variable from Babashka?