Clojurians Log v2

Hi, I'm running into a crash in some substrate of my testing environment. When calling stest/check on a particular function, if I increase the number of tests beyond a point or increase the size of the collections involved, i get the following (unfortunately not very illuminating output):

> clojure -A:test
2020-08-20 11:45:17.836:INFO::main: Logging initialized @5575ms to org.eclipse.jetty.util.log.StdErrLog
[Figwheel] Validating figwheel-main.edn
[Figwheel] figwheel-main.edn is valid \(ツ)/
[Figwheel] Compiling build test to "target/public/cljs-out/test-main.js"
[Figwheel] Successfully compiled build test to "target/public/cljs-out/test-main.js" in 5.052 seconds.
Launching Javascript environment with script:  "./scripts/launch_headless.sh"
Environment output being logged to: target/public/cljs-out/test/js-environment.log
#error {:message "ClojureScript test run failed", :data {:type :end-run-tests, :fail 1, :error 0, :pass 18, :test 4}}
Error: ClojureScript test run failed
    at new cljs$core$ExceptionInfo (<http://localhost:9500/cljs-out/test/cljs/core.js:36627:10>)
    at Function.cljs$core$IFn$_invoke$arity$3 (<http://localhost:9500/cljs-out/test/cljs/core.js:36688:9>)
    at Function.cljs$core$IFn$_invoke$arity$2 (<http://localhost:9500/cljs-out/test/cljs/core.js:36684:26>)
    at cljs$core$ex_info (<http://localhost:9500/cljs-out/test/cljs/core.js:36670:26>)
    at Function.eval [as cljs$core$IFn$_invoke$arity$variadic] (eval at figwheel$repl$eval_javascript_STAR__STAR_ (<http://localhost:9500/cljs-out/test/figwheel/repl.js:763:24>), &lt;anonymous&gt;:64:25)
    at redacted$test_runner$_main (eval at figwheel$repl$eval_javascript_STAR__STAR_ (<http://localhost:9500/cljs-out/test/figwheel/repl.js:763:24>), &lt;anonymous&gt;:18:33)
    at eval (eval at figwheel$repl$eval_javascript_STAR__STAR_ (<http://localhost:9500/cljs-out/test/figwheel/repl.js:763:24>), &lt;anonymous&gt;:1:26)
    at figwheel$repl$eval_javascript_STAR__STAR_ (<http://localhost:9500/cljs-out/test/figwheel/repl.js:763:24>)
    at <http://localhost:9500/cljs-out/test/figwheel/repl.js:811:56>
    at Object.G__12816__2 (<http://localhost:9500/cljs-out/test/cljs/core.js:35676:106>)
Execution error (ExceptionInfo) at cljs.repl/evaluate-form (repl.cljc:578).
#error {:message "ClojureScript test run failed", :data {:type :end-run-tests, :fail 1, :error 0, :pass 18, :test 4}}
Full report at:
/var/folders/jz/920rhb8h8xj864001s7_grh80000gn/T/clojure-8321569605138574991.edn

(s/def ::any-coll
  (s/with-gen
    (s/coll-of any?)
    #(s/gen (s/coll-of any? :max-count 5))))

(s/fdef concatv
        :args (s/cat :x ::any-coll :rest (s/* ::any-coll))
        :fn #(let [r (:ret %1)
                   x (-> %1 :args :x)
                   rest (-> %1 :args :rest)]
               (println (count r) (count x) (count rest))
               (= r (apply (partial concat x) rest)))
        :ret (s/coll-of any? :kind vector?))

(defn concatv
  "Strict version of concat."
  [x & rest]
  (case (count rest)
    0 x
    1 (into x (first rest))
    (into x (apply concatv rest))))

The crash only seems to occur under headless Chrome; when using figwheel-extra-main/auto-testing in the actual browser, it doesn't occur.

Unfortunately I don't get access to the "seed" when the problem occurs and my printlns do not get executed either so actually diagnosing is difficult.

@johanatan At a guess, since it seems both environment-specific and size-specific, I wonder if your (apply (partial concat x) rest) in the :fn spec is causing a stack overflow due to a buildup of lazy concat calls?

oooh! that's possible. in fact that was the issue that spurred the creation of concatv to begin with

would (mapcat identity colls) be a better impl ?

I think I would approach testing this via test.check and properties/generators, rather than trying to use fdef with :fn.

After all, your :fn is pretty much re-implementing the function being tested, but using lazy concat instead.

this does use test.check and generators though. yes, but that's pretty much how all :fn s end up

You're misunderstanding my point I think.

You're basically testing a vector-based concat implementation against concat itself -- so your test is going to run into the same problems as concat.

ah, true. that's why i mentioned (mapcat identity ...) though. so, basically what we need are "two strict implementations of concat". one to test the other

What you should be testing are properties of the result, e.g., count of the result is sum of count of each argument.

Set of values of result is union of set of values in each argument.

(I'm currently working through Eric Normand's three courses on Property-Based testing so this is top of my mind right now)

In the beginner course, he has a specific example of testing some properties of concat.

well i agree that testing the actual sets is better. that supercedes the individual properties

i think there is a difference of philosophy here. if you test for actual equality then you don't need to test for the properties

this is what i typically do with my :fn tests

The key is to not re-implement the function under test though.

i see the other sort of property-based tests as strictly less powerful / more limited / less robust

no, re implementation is fine

Clearly it isn't 🙂 That's what is causing this problem.

as long as the implementation is in a completely different fashion

well laziness is (likely) what's causing this problem

And also, you may end up with bugs in your function under test because you accidentally replicated them in your :fn re-implementation of it.

so if we have two completely separate strict implementations that both produce the same result, then we're good

sure, but i don't do that 🙂 or i typically keep iterating until i eliminate those

You would probably change your position on this if you took Eric's course.

i find a lot of his material to be a bit more "entry level" than fits my needs. (just in general, having seen a few of his other videos. there are definitely philosophical differences here and his audience is, like you said, beginner-esque)

i've done property-based testing for a while now. it's not like my thoughts on this just formed recently.

i do agree that there are times where checking properties is fine: the canonical example being checking the correctness of an NP-complete problem's solution

but a lot of problems in practice do not have "easy checks for correctness"

but yea i'll give it a listen

still yet, and we're assuming that laziness is the problem here (which it very well could be), there would seem to be a "bug" in the way the failure is presented if you will. i.e., there is literally no trace of helpful information in this case in the actual output 🙂

and actually just realized mapcat won't help here since it is a thin wrapper around apply concat

also, btw, nothing about :fn says that you must do a full reimplementation. you can still do "property" checks in there. of course with the downside being perhaps less readable output upon a subexpression failure. i'm willing to make that tradeoff though for the streamlined / inline specification

i think what i'll do in this case since a second "strict" version of concat seems elusive is a simple pairwise traversal comparing elements along the way

> nothing about :fn says that you must do a full reimplementation. you can still do "property" checks in there I would say property checks are better than a re-implementation.

> i find a lot of his material to be a bit more "entry level" than fits my needs I've been doing Clojure in production for a decade and I'm still picking up new ideas from even his "entry level" courses -- but I agree that his style is aimed at beginners in many courses. He has three PBT courses: beginner, intermediate, and advanced -- so I figured since I have a subscription, I might as well watch all of them (I run them at 1.5x speed).

cool, sounds good. i'll give it a look. thanks for your help on this!

It sounds like fdef is going to get substantially reworked in Spec 2, based on what Alex has been saying about that. But it's still all in "design mode" right now.

hm, in what way? btw, which properties would you test here if you were going the "property route" ?

everything i'm trying to do is hitting the same problem / crash

like sum of counts, sum of hashes etc

e.g., this crashes:

(= (count r) (apply (partial + (count x)) (mapv count rest)))

perhaps the s/* is causing this to blow up really large. i've read somewhere that it can happen and that it's hard to tweak as "recursion depth" is not all that precise of a control on it

> (`fdef`) hm, in what way? Alex hasn't said... just that it's going to be substantially reworked.

Rich is working on it, there have been many ideas explored, not sure where it's going to end up