clojure-spec

About: http://clojure.org/about/spec Guide: http://clojure.org/guides/spec API: https://clojure.github.io/spec.alpha/clojure.spec.alpha-api.html
johanatan 2020-08-20T19:01:52.000900Z

Hi, I'm running into a crash in some substrate of my testing environment. When calling stest/check on a particular function, if I increase the number of tests beyond a point or increase the size of the collections involved, i get the following (unfortunately not very illuminating output):

> clojure -A:test
2020-08-20 11:45:17.836:INFO::main: Logging initialized @5575ms to org.eclipse.jetty.util.log.StdErrLog
[Figwheel] Validating figwheel-main.edn
[Figwheel] figwheel-main.edn is valid \(ツ)/
[Figwheel] Compiling build test to "target/public/cljs-out/test-main.js"
[Figwheel] Successfully compiled build test to "target/public/cljs-out/test-main.js" in 5.052 seconds.
Launching Javascript environment with script:  "./scripts/launch_headless.sh"
Environment output being logged to: target/public/cljs-out/test/js-environment.log
#error {:message "ClojureScript test run failed", :data {:type :end-run-tests, :fail 1, :error 0, :pass 18, :test 4}}
Error: ClojureScript test run failed
    at new cljs$core$ExceptionInfo (<http://localhost:9500/cljs-out/test/cljs/core.js:36627:10>)
    at Function.cljs$core$IFn$_invoke$arity$3 (<http://localhost:9500/cljs-out/test/cljs/core.js:36688:9>)
    at Function.cljs$core$IFn$_invoke$arity$2 (<http://localhost:9500/cljs-out/test/cljs/core.js:36684:26>)
    at cljs$core$ex_info (<http://localhost:9500/cljs-out/test/cljs/core.js:36670:26>)
    at Function.eval [as cljs$core$IFn$_invoke$arity$variadic] (eval at figwheel$repl$eval_javascript_STAR__STAR_ (<http://localhost:9500/cljs-out/test/figwheel/repl.js:763:24>), &lt;anonymous&gt;:64:25)
    at redacted$test_runner$_main (eval at figwheel$repl$eval_javascript_STAR__STAR_ (<http://localhost:9500/cljs-out/test/figwheel/repl.js:763:24>), &lt;anonymous&gt;:18:33)
    at eval (eval at figwheel$repl$eval_javascript_STAR__STAR_ (<http://localhost:9500/cljs-out/test/figwheel/repl.js:763:24>), &lt;anonymous&gt;:1:26)
    at figwheel$repl$eval_javascript_STAR__STAR_ (<http://localhost:9500/cljs-out/test/figwheel/repl.js:763:24>)
    at <http://localhost:9500/cljs-out/test/figwheel/repl.js:811:56>
    at Object.G__12816__2 (<http://localhost:9500/cljs-out/test/cljs/core.js:35676:106>)
Execution error (ExceptionInfo) at cljs.repl/evaluate-form (repl.cljc:578).
#error {:message "ClojureScript test run failed", :data {:type :end-run-tests, :fail 1, :error 0, :pass 18, :test 4}}
Full report at:
/var/folders/jz/920rhb8h8xj864001s7_grh80000gn/T/clojure-8321569605138574991.edn
Neither the full report nor the js environment log contains anything of interest. Just stack traces in figwheel main.
(s/def ::any-coll
  (s/with-gen
    (s/coll-of any?)
    #(s/gen (s/coll-of any? :max-count 5))))

(s/fdef concatv
        :args (s/cat :x ::any-coll :rest (s/* ::any-coll))
        :fn #(let [r (:ret %1)
                   x (-&gt; %1 :args :x)
                   rest (-&gt; %1 :args :rest)]
               (println (count r) (count x) (count rest))
               (= r (apply (partial concat x) rest)))
        :ret (s/coll-of any? :kind vector?))

(defn concatv
  "Strict version of concat."
  [x &amp; rest]
  (case (count rest)
    0 x
    1 (into x (first rest))
    (into x (apply concatv rest))))
I can currently execute fine with num tests of 20 but when increasing to 100 or more, the crash occurs.

johanatan 2020-08-20T19:02:25.001500Z

The crash only seems to occur under headless Chrome; when using figwheel-extra-main/auto-testing in the actual browser, it doesn't occur.

johanatan 2020-08-20T19:04:17.002300Z

Unfortunately I don't get access to the "seed" when the problem occurs and my printlns do not get executed either so actually diagnosing is difficult.

seancorfield 2020-08-20T19:22:55.004100Z

@johanatan At a guess, since it seems both environment-specific and size-specific, I wonder if your (apply (partial concat x) rest) in the :fn spec is causing a stack overflow due to a buildup of lazy concat calls?

johanatan 2020-08-20T19:23:39.004700Z

oooh! that's possible. in fact that was the issue that spurred the creation of concatv to begin with

johanatan 2020-08-20T19:23:54.005Z

would (mapcat identity colls) be a better impl ?

seancorfield 2020-08-20T19:24:49.005700Z

I think I would approach testing this via test.check and properties/generators, rather than trying to use fdef with :fn.

seancorfield 2020-08-20T19:25:23.006700Z

After all, your :fn is pretty much re-implementing the function being tested, but using lazy concat instead.

johanatan 2020-08-20T19:25:46.007100Z

this does use test.check and generators though. yes, but that's pretty much how all :fn s end up

seancorfield 2020-08-20T19:26:03.007600Z

You're misunderstanding my point I think.

seancorfield 2020-08-20T19:26:39.008400Z

You're basically testing a vector-based concat implementation against concat itself -- so your test is going to run into the same problems as concat.

johanatan 2020-08-20T19:27:15.009600Z

ah, true. that's why i mentioned (mapcat identity ...) though. so, basically what we need are "two strict implementations of concat". one to test the other

seancorfield 2020-08-20T19:27:18.009800Z

What you should be testing are properties of the result, e.g., count of the result is sum of count of each argument.

seancorfield 2020-08-20T19:27:43.010500Z

Set of values of result is union of set of values in each argument.

seancorfield 2020-08-20T19:28:44.012Z

(I'm currently working through Eric Normand's three courses on Property-Based testing so this is top of my mind right now)

seancorfield 2020-08-20T19:29:02.012700Z

In the beginner course, he has a specific example of testing some properties of concat.

johanatan 2020-08-20T19:29:07.012900Z

well i agree that testing the actual sets is better. that supercedes the individual properties

johanatan 2020-08-20T19:30:06.013700Z

i think there is a difference of philosophy here. if you test for actual equality then you don't need to test for the properties

johanatan 2020-08-20T19:30:24.014Z

this is what i typically do with my :fn tests

seancorfield 2020-08-20T19:30:40.014700Z

The key is to not re-implement the function under test though.

johanatan 2020-08-20T19:30:45.014800Z

i see the other sort of property-based tests as strictly less powerful / more limited / less robust

johanatan 2020-08-20T19:30:54.015Z

no, re implementation is fine

seancorfield 2020-08-20T19:31:10.015600Z

Clearly it isn't 🙂 That's what is causing this problem.

johanatan 2020-08-20T19:31:16.015900Z

as long as the implementation is in a completely different fashion

johanatan 2020-08-20T19:31:28.016400Z

well laziness is (likely) what's causing this problem

seancorfield 2020-08-20T19:31:47.017100Z

And also, you may end up with bugs in your function under test because you accidentally replicated them in your :fn re-implementation of it.

johanatan 2020-08-20T19:31:47.017200Z

so if we have two completely separate strict implementations that both produce the same result, then we're good

johanatan 2020-08-20T19:32:08.017700Z

sure, but i don't do that 🙂 or i typically keep iterating until i eliminate those

seancorfield 2020-08-20T19:32:15.018Z

You would probably change your position on this if you took Eric's course.

johanatan 2020-08-20T19:33:35.019100Z

i find a lot of his material to be a bit more "entry level" than fits my needs. (just in general, having seen a few of his other videos. there are definitely philosophical differences here and his audience is, like you said, beginner-esque)

johanatan 2020-08-20T19:34:52.019800Z

i've done property-based testing for a while now. it's not like my thoughts on this just formed recently.

johanatan 2020-08-20T19:35:29.020400Z

i do agree that there are times where checking properties is fine: the canonical example being checking the correctness of an NP-complete problem's solution

johanatan 2020-08-20T19:35:49.020800Z

but a lot of problems in practice do not have "easy checks for correctness"

johanatan 2020-08-20T19:36:04.021Z

but yea i'll give it a listen

johanatan 2020-08-20T19:37:29.022Z

still yet, and we're assuming that laziness is the problem here (which it very well could be), there would seem to be a "bug" in the way the failure is presented if you will. i.e., there is literally no trace of helpful information in this case in the actual output 🙂

johanatan 2020-08-20T19:48:29.023Z

and actually just realized mapcat won't help here since it is a thin wrapper around apply concat

johanatan 2020-08-20T19:50:18.024400Z

also, btw, nothing about :fn says that you must do a full reimplementation. you can still do "property" checks in there. of course with the downside being perhaps less readable output upon a subexpression failure. i'm willing to make that tradeoff though for the streamlined / inline specification

johanatan 2020-08-20T19:54:02.025100Z

i think what i'll do in this case since a second "strict" version of concat seems elusive is a simple pairwise traversal comparing elements along the way

seancorfield 2020-08-20T20:20:19.026400Z

> nothing about :fn says that you must do a full reimplementation. you can still do "property" checks in there I would say property checks are better than a re-implementation.

seancorfield 2020-08-20T20:22:15.028500Z

> i find a lot of his material to be a bit more "entry level" than fits my needs I've been doing Clojure in production for a decade and I'm still picking up new ideas from even his "entry level" courses -- but I agree that his style is aimed at beginners in many courses. He has three PBT courses: beginner, intermediate, and advanced -- so I figured since I have a subscription, I might as well watch all of them (I run them at 1.5x speed).

johanatan 2020-08-20T20:22:38.028900Z

cool, sounds good. i'll give it a look. thanks for your help on this!

seancorfield 2020-08-20T20:27:38.029600Z

It sounds like fdef is going to get substantially reworked in Spec 2, based on what Alex has been saying about that. But it's still all in "design mode" right now.

johanatan 2020-08-20T21:01:13.030100Z

hm, in what way? btw, which properties would you test here if you were going the "property route" ?

johanatan 2020-08-20T21:01:24.030400Z

everything i'm trying to do is hitting the same problem / crash

johanatan 2020-08-20T21:01:35.030700Z

like sum of counts, sum of hashes etc

johanatan 2020-08-20T21:02:03.030900Z

e.g., this crashes:

(= (count r) (apply (partial + (count x)) (mapv count rest)))

johanatan 2020-08-20T21:05:38.031800Z

perhaps the s/* is causing this to blow up really large. i've read somewhere that it can happen and that it's hard to tweak as "recursion depth" is not all that precise of a control on it

seancorfield 2020-08-20T21:05:53.032200Z

> (`fdef`) hm, in what way? Alex hasn't said... just that it's going to be substantially reworked.

alexmiller 2020-08-20T21:15:07.032700Z

Rich is working on it, there have been many ideas explored, not sure where it's going to end up