kaocha

Official support channel: https://clojureverse.org/c/projects/kaocha
2020-04-01T16:54:39.059400Z

Has anyone seen this error from kaocha/expound before?

Exception: clojure.lang.ExceptionInfo: Call to expound.alpha/printer did not conform to spec:
alpha.clj:258

-- Spec failed --------------------

Function arguments

  (nil)
   ^^^

should satisfy

  map?

-------------------------
Detected 1 error

{:clojure.spec.alpha/problems [{:path [:args :explain-data], :pred clojure.core/map?, :val nil, :via [], :in [0]}], :clojure.spec.alpha/spec #object[clojure.spec.alpha$regex_spec_impl$reify__2509 0x1b78c761 "clojure.spec.alpha$regex_spec_impl$reify__2509@1b78c761"], :clojure.spec.alpha/value (nil), :clojure.spec.alpha/args (nil), :clojure.spec.alpha/failure :instrument, :orchestra.spec.test/caller {:file "alpha.clj", :line 258, :var-scope clojure.spec.alpha/explain-out}, :orchestra.spec/var #'expound.alpha/printer}
 at orchestra.spec.test$spec_checking_fn$conform_BANG___3298.invoke (test.cljc:115)
    ...

2020-04-01T17:08:45.060400Z

I think this is being caused by a test of mine calling (s/explain-str :a.spec.of/ours data)

2020-04-02T10:47:15.073100Z

@bbrinck Thanks for looking into this for me. What is it that is currently defining that fdef?

2020-04-02T10:47:32.073300Z

I can’t seem to find it in expound; or kaocha

2020-04-02T11:05:30.073900Z

So I guess the reason this happens and hasn’t been discovered before is because expounds printer assumes it never needs to print success messages? i.e. the binding approach doesn’t currently assume that users are going to call s/explain-str themselves within that binding form?

bbrinck 2020-04-02T13:23:33.074800Z

@rickmoynihan I suspect the reason this hasn’t been reported earlier is that you need 3 things to be true. 1. You must use explain-str (not expound-str) 2. You must have a case where you are calling explain-str for data that matches the spec (a common use case is to only call this if s/valid? returns false) 3. You must have instrumented expound/printer I suspect that’s quite rare

👍 1
bbrinck 2020-04-02T13:25:32.075500Z

;; Using expound directly works
  (expound/expound-str :foobar/name "")
  ;; => "Success!\n"  

  ;; A common pattern of using valid? + explain-str works
  (binding [s/*explain-out* expound/printer]
    (if-not (s/valid? :foobar/name "")
      (s/explain-str :foobar/name ""))
    )
  ;; => nil

  ;; If instrumentation is off, it works
  (binding [s/*explain-out* expound/printer]
    (s/explain-str :foobar/name "")
    )
  ;; => "Success!\n"  )

  ;; But if instrumentation is turned on, it fails!
  (st/instrument)
  (binding [s/*explain-out* expound/printer]
    (s/explain-str :foobar/name "")
    )
  ;; error

  ;; Even w/ instrumentation, expound-str works
  (expound/expound-str :foobar/name "")
  ;; => "Success!\n"

  ;; ... as does pattern of using valid? + explain-str

  (binding [s/*explain-out* expound/printer]
    (if-not (s/valid? :foobar/name "")
      (s/explain-str :foobar/name ""))
    )
  ;; => nil

2020-04-02T15:16:45.076700Z

Yeah I suspect it is quite rare… The code in question here was actually a clojure.test assertion that some test data conformed to the clojure spec…

(= "Success!\n" (s/explain-str :mut.download/job-instance-schema test-job))
It’s pretty hacky, however it was written that way because it gives a better clojure.test error when the spec fails.

2020-04-02T15:18:07.077100Z

an s/valid? call just gives a true/false error etc.

2020-04-02T15:20:30.077300Z

Instrumenting is a better approach generally to this kind of thing too… but there are some problems with instrumentation in this code base. Mainly that we use integrant and load-namespaces quite extensively; so not all specs in use are loaded upfront; because the code loading is dynamic.

bbrinck 2020-04-02T15:26:24.077500Z

I think the following would work

(is (s/valid? :foobar/name 1)
        (s/explain-str :foobar/name 1)
        ))
The above is repetitive, but I suspect some macro magic could clean it up 🙂

bbrinck 2020-04-02T15:27:13.077700Z

(basically test for valid but then use explain-str for the is failure message)

2020-04-02T15:27:27.077900Z

hmm nice idea

2020-04-02T15:27:49.078100Z

I guess that’ll still cause the expound problem though 🙂

bbrinck 2020-04-02T15:28:03.078300Z

It doesn’t seem to

bbrinck 2020-04-02T15:28:49.078500Z

I haven’t dug into the clojure.test code, but my suspicion is that the message is not evaluated unless is check fails

2020-04-02T15:29:29.078700Z

yeah I guess the messages are evaluated later outside the binding

bbrinck 2020-04-02T15:31:46.078900Z

Oh, nevermind. I still had the “workaround” spec declared. The above won’t work 😞

bbrinck 2020-04-02T15:32:06.079100Z

I suppose you could use a macro to delay eval of the message but at that point it’s getting pretty complicated

2020-04-02T15:33:15.079300Z

can we not just correct the printer spec in expound; to allow nil? Or does that imply having the printer print success messages?

2020-04-02T15:34:10.079500Z

i.e. your workaround; I’m curious why that’s not a fix.

bbrinck 2020-04-02T15:34:17.079700Z

It is the fix

bbrinck 2020-04-02T15:34:34.079900Z

I just haven’t had time to apply it and cut a release 🙂

2020-04-02T15:34:38.080100Z

ahh cool

2020-04-02T15:34:48.080300Z

ok, well I can wait. It’s not a big deal.

2020-04-02T15:35:08.080500Z

I’d already commented out the assertion for now anyway.

👍 1
bbrinck 2020-04-02T15:35:13.080700Z

FWIW, I haven’t used it, but there are also fancier tools to assert that some data matches a spec. YMMV https://github.com/nedap/utils.spec/blob/3c9c0f32d3bb436aa8cc6b975b3d843ca95968ab/src/nedap/utils/spec/api.cljc#L9-L15

2020-04-02T15:38:08.081100Z

ahh thanks good to know — this code is pretty old. I had written similar things in the past, but never extracted it into something to reuse as it was always a bit hacky.

2020-04-02T15:38:36.081300Z

and made do with the string check in this instance

bbrinck 2020-04-02T15:41:05.081500Z

Yeah, it makes sense. The string check is a good solution - if there wasn’t this bug 😆 . In any case, thanks for reporting it, I’ll try to get it fixed soon.

2020-04-02T15:41:38.081700Z

Well thanks a million anyway, it was insightful 😁

bbrinck 2020-04-01T17:09:39.060600Z

Hm, for some reason it looks like the printer is being called with nil for explain-data.

2020-04-01T17:09:49.060800Z

yeah

bbrinck 2020-04-01T17:10:06.061Z

Is it possible that somehow explain-str is being called even though the data actually matches the spec?

bbrinck 2020-04-01T17:10:27.061200Z

(s/explain-data int? 1) ;; nil

2020-04-01T17:11:38.061400Z

I am explicitly calling s/explain-str in my test

bbrinck 2020-04-01T17:11:56.061700Z

what does (s/valid? :a.spec.of/ours data) return

2020-04-01T17:13:23.062Z

true

2020-04-01T17:13:54.062200Z

s/explain-data returns nil

2020-04-01T17:14:03.062400Z

as you’d expect

2020-04-01T17:15:35.062600Z

It looks like expound in kaocha is interfering with spec usage in my test. At a repl it works; but via ./bin/kaocha it fails with the expound error

bbrinck 2020-04-01T17:16:31.062900Z

Hm, I’d be curious to see how expound is set up in your codebase and/or kaocha. e.g. presumably there is a (set! s/*explain-out* expound/printer) somewhere

2020-04-01T17:17:13.063100Z

I can’t find any reference to expound in my app anywhere

2020-04-01T17:17:18.063300Z

that was my first thought too

2020-04-01T17:18:06.063500Z

but kaocha includes expound & orchestra now

bbrinck 2020-04-01T17:20:10.064100Z

Ok I’ll see if I can build a local repro this evening

2020-04-01T17:21:56.064300Z

I think this issue might be relevant: https://github.com/lambdaisland/kaocha/issues/104

bbrinck 2020-04-01T19:54:43.065100Z

If you can run same test with another test runner, that’d be useful as well.

bbrinck 2020-04-01T19:55:58.067100Z

I realize you tried the REPL but a REPL potentially has other state, so it’d be useful to try “lein test” or an equivalent