Clojurians Log v2

I have some test data (input and expected output) in a text file. I would like to parse the examples from this file and somehow get clojure.test to tell me which examples are failing. Should I try to do something like (doseq [...] (deftest ...)) or (deftest ... (are [x y] (= x y) ...)? I am also hoping to get the test runner to print the actual data from the files, rather than printing the code verbatim that I used to pull data from the files. E.g. I want (deftest file-parsing (is (= (parse "input-file.txt") (slurp "output-file.txt")))) to print the output file contents as the expected value rather than to print literally "(slurp \"output-file.txt\")".

I'm not sure I follow @ben.grabow…

user=> (deftest parsing (is (= (parse "foo") (slurp "user.clj"))))
#'user/parsing
user=> (parsing)

FAIL in (parsing) (NO_SOURCE_FILE:1)
expected: (= (parse "foo") (slurp "user.clj"))
  actual: (not (= "test" "(println \"top-level user.clj was loaded\")\n"))
nil

Ah, you don't like the expected value formatting -- clojure.test shows you the specific form from your code that you expected to the truthy and then the actual values that the assertion failed with (for = tests).

Ah, makes sense. I had some really noisy stuff on the "expected:" line that was distracting me from the fact that the "actual:" line already has what I want.

Any advice for the best way to generate multiple assertions or tests from a file that has multiple examples in it?

I wanted to use are but it seems it works at macroexpansion time so I don't know how to do that without literal values.

clojure.test isn't really designed for the sort of test you're trying to build -- it assumes a statically known set of assertions in code. I think your best bet is to write some code to read the two files and then write out a deftest with are and the input/output forms in the body -- write that to a file and then load-file it to read it back in and actually define the test.

In other words, preprocess your input/output files into a Clojure file that you can then run like any normal test...

Hmm... well... I guess you could write a deftest that sucked in the two files and then did a doseq over is but I'm not sure what your failures would look like...

No, I think that wouldn't work here -- your inputs are code fragments/expressions, yes?

And the outputs are (just) values. So you'd really need the (is …) to include the source from the input so you could see what was going on.

I think you have to go the file generation route for that.

I've parsed my files into a structure like [[input1 ouptut1][input2 output2] ...].

But is the input a piece of code you need to evaluate, or just a plain value?

This seems to give me what I'm looking for:

(deftest all-examples
  (doseq [[input output] examples]
    (is (= output (fn-under-test input)))))

Just a plain value.

(where examples is the data structure with the above format)

Oh, so you're not really "parsing" the files -- you're just reading them in as values. They don't contain code?

Yes

And it's all to test a specific function?

Yep!

OK, then the approach you have there is fine 🙂

Great, thanks for the feedback!

If you did a doseq over deftest, it would (attempt to) define a named test for each iteration (so you'd need different names) -- and then you'd still have to both execute that code and run tests afterward.

But doseq over is gives you multiple assertions inside a single named test.

I think the one thing this is lacking is the ability to see in the test runner's output what the value of input was. In the "expected:" line it shows the literal code "(fn-under-test input)" and in the "actual:" line it shows the resulting value from calling fn-under-test.

Try (is (= [input output] [input (fn-under-test input)]))

That may give you too much output to be useful but it should at least show both input and output values.

That's a good idea. Another route seems to be to add a testing context label around the assertion: (testing input (is ...))

Ah, I always think of that as taking a literal string but I guess it can taking computed values too?

Sure looks like it! Here's my final test code that I'm satisfied with.

(deftest all-examples
  (doseq [[input output] examples]
    (testing (with-out-str (clojure.pprint/pprint input))
      (is (= output (fn-under-test input))))))

Thanks for the help, Sean. You rock.