Clojurians Log v2

Hello!

If I were to write a reporter which would print some extra information when a test fails, but which otherwise works as normal, is there a guide I could follow?

that's pretty specific, there's no documented recipe for that, but it should be straightforward. A reporter is just a function. You would write one which deals with the cases you want to deal with, and otherwise delegates to an existing reporter.

This is what I thought I’d do. Maybe I’m not reading the docs closely enough, but I’m in doubt about how to delegate. (I was thinking that it was just a matter of calling (existing-reporter m), but I’m in doubt about which existing report to refer to, and whether that would fit into the flow)

(defn my-reporter [event]
  (if (kaocha.hierarchy/fail-type? event)
    (println "oh no that's not good")
    (kaocha.report/dots* event)))

something to be aware of here is that in kaocha you can specifiy multiple reporters, they will get merged into a single function, this is what happens in case of dots and documentation

(def dots
  "Reporter that prints progress as a sequence of dots and letters."
  [dots* result])

(def documentation
  "Reporter that prints an overview of all tests bein run using indentation."
  [doc result])

they share a result reporter which prints out the test summary in the end. So either you delegate to that too, or you add kaocha.report/result as an extra reporter in tests.edn

#kaocha/v1
{:reporter [my.project/my-reporter kaocha.report/result]}

does that make sense?

That makes great sense, thanks @plexus!Now to iterate on it and see if I can do what I want 🙂

good look! the kaocha.report/debug reporter can be helpful to get a sense of what data you're getting

and as demonstrated in my example it's recommended to use kaocha.hierarchy instead of checking for concrete :type values, so as to be maximally compatible with different test types and assertions

A "writing reporters" section in the docs would certainly be welcome. If you have the appetite for it it would be great if you could gather up these tips and stick them in a markdown file 🙂

That’s good to know. I was expecting to match directly with =.

If this goes well, I might do a writeup 🙂

awesome!

reporters are simple in theory but complex in practice, they're a concept we took directly from clojure.test, but then kind of ran with it. We stick a lot more information in the event map than clojure.test does, and well behaved kaocha reporters use that when available. This is also because we need to support many test types (cucumber, cljs), with sometimes quite specific requirements

some examples

(defmethod doc :kaocha/begin-test [m]
  (t/with-test-out
    (let [desc (or (some-> m :kaocha/testable :kaocha.testable/desc)
                   (some-> m :var meta :name))]
      (print (str "\n  " desc))
      (flush))))

:var is a clojure.test specific thing, the more general thing is to use a testable's description

(defmethod fail-summary :kaocha/fail-type [{:keys [testing-contexts testing-vars] :as m}]
  (println (str "\n" (output/colored :red "FAIL") " in") (testing-vars-str m))
  (when (seq testing-contexts)
    (println (str/join " " (reverse testing-contexts))))
  (when-let [message (:message m)]
    (println message))
  (if-let [expr (::printed-expression m)]
    (print expr)
    (print-expr m))
  (print-output m))

More info is great, and it’s probably what will enable me to do what I want. I just learned that I can’t go ahead and do (pprint/pprint m), that blows the heap space 😄 Too much info in one portion!

for clojurescript tests we do the formatting on the clojurescript side, because we can't always properly serialize/deserialize clojurescript assertions into clojurescript. So if :kaocha.report/printed-expression is present then the reporter simply prints that instead of trying to parse and format/print the assertion

yeah you have the full testable in there which can be huge. That's why the debug reporter limits the keys that it prints

(defn debug [m]
  (t/with-test-out
    (prn (cond-> (select-keys m [:type
                                 :file
                                 :line
                                 :var
                                 :ns
                                 :expected
                                 :actual
                                 :message
                                 :kaocha/testable
                                 :debug
                                 ::printed-expression])
           (:kaocha/testable m)
           (update :kaocha/testable select-keys [:kaocha.testable/id :kaocha.testable/type])))))

also notice the with-test-out, that's another clojure.test thing that allows you to rebind where test output is printed. It defaults to just *out*. We don't do much with it but users might. We really only use it to capture output in kaocha's own tests.

I was just messing around with printing parts of m in a custom reporter, but I’ll try the debug reporter now, as iterating on it is a bit slow. (I wonder how I could get lein test to start up faster…)

use kaocha.repl! no need to start a new process every time

Even better 😄

Seems like you’ve thought of everything 🙂

Do you happen to know how to get the path for a source file based on the ns? I was hoping that the path was available in the Kaocha map, but :file only seems to refer to the filename itself.

Maybe I can also get it via the filename.

(let [ns 'foo.bar-test]
  (io/resource (str (.. (name ns)
                        (replace \- \_)
                        (replace \. \/))
                    ".clj")))

not sure if that's really the best way but that's roughly the idea. Might be some other option through tools.namespace

🙏 thank you very much!

@plexus I don’t understand this example. I tried it verbatim, but it seems like even though I have a test failure (as reported by the summary: #:kaocha.result{:count 1, :pass 0, :error 0, :fail 1, :pending 0}), (kaocha.hierarchy/fail-type? event)` does not return true during my run.

Ah wait, I am mistaken.

The printing confused me.

(run 'query.opgavefabrikken-test.config-test
     {:reporter 'util.reporters/show-owner})
[()]
Randomized with --seed 1576729021(:pending :file :type :fail :line :error :kaocha/testable :pass :kaocha/test-plan :test)

=> #:kaocha.result{:count 1, :pass 0, :error 0, :fail 1, :pending 0}

No wait, I confused myself! The code should be like this:

(ns util.reporters
  (:require [kaocha.hierarchy :as hierarchy]
            [kaocha.report :as report]))

(defn show-owner [{:keys [] :as m}]
  (if (hierarchy/fail-type? m)
    (println "oh oh!")
    (report/dots* m)))

And the output is then:

(run 'query.opgavefabrikken-test.config-test
     {:reporter 'util.reporters/show-owner})
[()]
Randomized with --seed 1706915743
=> #:kaocha.result{:count 1, :pass 0, :error 0, :fail 1, :pending 0}

might be output capturing? try adding the t/with-test-out

also add :kaocha.plugin.capture-output/capture-output? false to your config when calling run, that's generally a good idea when working on stuff like this to make sure things don't get hidden that you really would want to see

Both things work separately. I mean, for some definition of “work” 😄 The output is a bit funny looking, but at least it includes my prn now.

(run 'query.opgavefabrikken-test.config-test
     {:reporter 'util.reporters/show-owner
      :kaocha.plugin.capture-output/capture-output? false})
[(oh oh!
)]
Randomized with --seed 278474731
=> #:kaocha.result{:count 1, :pass 0, :error 0, :fail 1, :pending 0}

Will it always output whatever I prn as a part of that dots-like vector?

if you are using the dots reporter, yes. keep in mind that these reporters aren't very smart, they just get handed event after event and write some output to stdout as a result. The dot reporter prints "[" when a suite starts, "(" when a group starts, etc.

what are you trying to achieve?

Specifically, I think it’d be neat to use something (perhaps a reporter) to react to failing tests by printing the names of the people who last touched the file that the ns belongs to, possibly via some git command that uses the file name.

It seems like all the pieces of the puzzle are available, provided that git is available in the environment (and it can be)

I think that will be easier with a plugin/hook

since that way you can just print a "report" at the end of the test run

I’ll give that a go 🙂

Iterating with the repl is, as always, great

I'll write up something to get you started

I’ve hooked into the :kaocha.hooks/post-run, but the m doesnt’ seem to have the info I want

Nor for :kaocha.hooks/pre-report

it's all there, just give me a moment

🙂

managed to freeze up my emacs by printing a too big a datastructure 🙂

😉

(ns my.project.kaocha-hooks
  (:require [kaocha.testable :as testable]
            [kaocha.hierarchy :as hierarchy]
            [kaocha.result :as result]))

;; use as a post-summary hook
(defn blame-report [result]
  (let [clojure-test-suites (filter (comp #{:kaocha.type/clojure.test} :kaocha.testable/type)
                                    (:kaocha.result/tests result))]
    (doseq [suite clojure-test-suites
            ns-testable (:kaocha.result/tests suite)
            :when (result/failed? ns-testable)]
      (prn (:kaocha.ns/name ns-testable)))
    )
  )


(comment
  (require '[kaocha.repl :as repl]
           '[kaocha.api :as api])

  (def test-result (api/run (repl/config {})))

  (blame-report test-result)
  )

this specifically assumes clojure.test, since you can't really generalize this, unless we make test types add a :kaocha.testable/file to their testables, which might be a good idea

I’ll let you be the judge of that. You’re deep into datastructures that I only have surface knowledge of 🙂

Just trying this now

(defn blame-report [result]
  (let [clojure-test-suites (filter (comp #{:kaocha.type/clojure.test} :kaocha.testable/type)
                                    (:kaocha.result/tests result))]
    (doseq [suite clojure-test-suites
            ns-testable (:kaocha.result/tests suite)
            :when (result/failed? ns-testable)
            :let [ns-name (:kaocha.ns/name ns-testable)]]
      (println
       ns-name "last touched by"
       (re-find #"Author:.*"
                (:out
                 (sh/sh "git" "log" "-1" (str/replace (str (or (io/resource (str (.. (name ns-name)
                                                                                     (replace \- \_)
                                                                                     (replace \. \/))
                                                                                 ".clj"))
                                                               (io/resource (str (.. (name ns-name)
                                                                                     (replace \- \_)
                                                                                     (replace \. \/))
                                                                                 ".cljc"))))
                                                      "file:" ""))))))))

I’m trying to hook it in as a post-summary hook. I think I’m either doing it wrong, or on the wrong version of kaocha.

(run 'query.opgavefabrikken-test.config-test
     {:plugins [:kaocha.plugin/hooks]
      :kaocha.hooks/post-summary [util.hooks/blame-report]
      :kaocha.plugin.capture-output/capture-output? false})

no sign of the output of blame-report in the output

ah right, post-summary doesn't get run when running via kaocha.repl, a post-run should also work

I’m on "0.0-541" 😮

owwww better do something about that 🙂

yeah that's almost a year old

and exactly 100 commits behind 1.0.641 🙂

😄

Time flies in this project. I feel like Kaocha was introduced just yesterday.

Brilliant! Now I get proper output

commit 723ff0e2ed7c63d8daf36ee39b6da57186d8686a
Author: Arne Brasseur <arne@arnebrasseur.net>
Date:   Thu Apr 12 09:16:35 2018 +0200

    Initial skeleton

2.3333333 years old 🙂

that is darn cool

I’m iterating towards your code, seeing that it works along the way

Yes! This is fantastic. Thanks Arne!

I’ll refine this and add it to our repo, it should help out whenever our CICD fails.

Is the missing run from post-summary when running via kaocha.repl intentional, or just something that hasn’t been fixed yet?

I’d like to write in our docs about how to test this stuff, and if post-summary is more appropriate, it’d be cool to be able to test that in the REPL as well.

it's kind of intentional, post-summary is handled in kaocha.runner, not in kaocha.api, to handle cases like --print-test-result which does a test run via kaocha.api but then does completely different output handling, and we don't want plugins that print stuff at the end of the test run to clobber that

now we could call it from kaocha.repl as well, which I think would be appropriate

so yes, post-summary is the right place to print this kind of extra info at the end of a test run

that's also for instance what the profiling plugin uses

Call what and when? kaocha.runner when testing in the REPL, or?

no, call the post-summary hook from kaocha.repl

basically kaocha.repl and kaocha.runner both are ways to invoke kaocha.api. We don't want post-summary to be called from api, because there are use cases where api is used directly where it's not appropriate to call post-summary (this is the main difference between post-summary and post-run, the former is called outside kaocha.api, the latter inside)

@plexus how do you specify the equivalent of

{:plugins [:kaocha.plugin/hooks]
        :kaocha.hooks/post-run [util.hooks/blame-report]}

Alrighty 🙂 So that’s a feature request for kaocha, I guess.

I guess by using a tests.edn. Trying that now.

Seems to work great! Though kaocha.hooks/post-summary does not seem to work, only kaocha.hooks/post-run, like we tried earlier. Kaocha is not documented to have post-summary either. Are you ahead of the published code when talking about post-summary? 😅

Hm, no, I see it in the source.

Expected:
  true
Actual:
  -true +false
171 tests, 675 assertions, 1 failures.
Error encountered performing task 'run' with profile(s): 'base,system,user,provided,dev,kaocha'
Suppressed exit