Clojurians Log v2

I am using the collections-check library, which currently depends upon not-the-most-recent version of test.chuck and test.check libraries

In trying to verify that it can catch certain kinds of errors, I am intentionally introducing some small errors into a collection implementation. In one case, I make it throw an exception.

collections-check makes a call to test.chuck's checking macro, like this:

(chuck/checking "vector-like" num-tests-or-options
    [actions (gen-vector-actions element-generator (transient? empty-coll) ordered?)]
    (let [[a b actions] (build-collections empty-coll base true actions)]
      (assert-equivalent-vectors a b)))

The function assert-equivalent-vectors is a series of clojure.test is macro invocations.

When I change the implementation so that it throws an exception, the exception comes not from calling assert-equivalent-vectors, but calling build-collections.

I put a print statement with a counter inside of assert-equivalent-vectors to see what parameters it was being called with, and how many times, and when there is no exception thrown, I see the checking macro limiting itself to num-tests-or-options iterations (100 or 200 in my short tests).

When build-collections throws an exception, something happens where I see assert-equivalent-vectors continue to be called many, many times, perhaps an infinite loop, but a println I added within test.check's quick-check macro (or function? I forget) no longer prints each time through the loop. Either the println gets messed up in some weird way (very odd if so), or somehow that quick-check loop is somehow causing assert-equivalent-vectors to be called in an infinite loop, without calling build-collections again.

This may all be rubber ducking if I figure this out, but wanted to ask if:

Shrinking is part of this, right?

(a) Is test.chuck's checking macro intended to be used with a checking function that calls clojure.test/is one or more times, and its return value isn't important?

Is there a way I can tell whether shrinking is involved? I'm happy adding extra println's anywhere here.

Oh, perhaps shrinking is causing many calls to assert-equivalent-vectors, and they are all failing, and shrinking is "stuck" somehow?

is is the whole point of checking

I don't have any theories for why you'd see it stop printing

ok. Will keep looking a bit longer.

You can money patch that to confirm it's shrinking

I was hoping that checking would quickly show some output of a failure if an exception was thrown, rather than going into an infinite loop (if it is infinite -- not sure whether it is infinite or just long)

Phone handier than links today? 🙂

Yep 🙂

That's in the base t.c namespace

I can try updating to latest versions of test.chuck and test.check, but initially when I tried that there seemed to be some incorrect args being passed around somewhere, and hadn't figured that out yet to make it work well enough to debug.

Thanks for the leads

n.p.; happy to help out if you run into something

Should latest test.check and test.chuck be compatible with each other, as far as you know?

I think collections-check may need some changes to work with latest test.chuck, but would prefer not to guess too many times at which pairs of test.chuck/test.check version combinations should work with each other.

Yes

Both libraries have generally avoided breaking changes

Wow. I think I found it. collections-check is passing a double for the num-tests-or-options, which test.chuck's times function in the latest version returns nil for.

I mean, I found the thing that I thought would take 30 seconds in about 20 mins 🙂. Not the final answer.

This thing returns nil???

This thing:

(defn times [num-tests-or-options]
  (cond (map? num-tests-or-options)     (:num-tests num-tests-or-options tc.clojure-test/*default-test-count*)
        (integer? num-tests-or-options) num-tests-or-options))

Oh geez there's two of them

It was weird that collection-check was using doubles anyway -- probably Zach T liked 1e2, 1e3 for brevity on test counts.

This is why we can't have nice static type systems

The shrink-loop is definitely going for far more iterations than I would have expected.

What sort of data is it?

collections-check generates sequences of operations on data structures, in this case on vectors, like assoc, conj, transient, persistent!, seq, etc.

So I think it must be attempting to shrink that sequence of operations down to a case where it passes, and then increase it a bit?

I am printing out the length of the vectors produced in the function that does multiple is calls to compare the results of two different vector implementations, each produced with two different vector implementations, and they do fluctuate a bit, but they are not getting consistently smaller.

I guess I could figure out how to look at the sequence of operations being tried in the shrink-loop to see if it is getting smaller/simpler

The pattern isn't necessarily simple

Especially if the generator uses bind

This is a wild guess, but it also generates the values to be added to the vector via gen/int, and there are about 500 of them in the failing vector, so if it is spending any work trying to simplify those numeric values, most of those are going to give the same result, since the error I put into the bad vector implementation fails over about 520 vector elements, unless pretty much most of the vector elements are equal to each other.

What problem are you actually trying to solve?

Good question. In this case, I wanted to verify that collections-check could detect a simple bug in an implementation, because I wanted to better trust the results when it returns no bugs found.

Ah okay, and the fact that it's spinning instead of returning failure is concerning

So I'm really done verifying that it can find this example bug I've introduced, but only because I've added extra debug prints. If I don't have those, I just get what appears to be an infinite loop, with no failing test case given.

Wrap the gen in gen/no-shrink 😛 😉

I suppose I could eliminate the possibility of gen/int shrinking slowing things down, by forcing vector elements in the sequence of operations to be increasing numbers.

OK, didn't know about that, so thanks.

will try

If there's not a feature for limiting shrink time then there oughta be

Just (gen/no-shrink gen/int) where I currently have gen/int I guess?

You could. Or you could wrap the whole collection generator

You get different effects for each

Is there anything in shrink-loop that represents the value passed to the function being tested?

i.e. that I could print and look at?

I know it isn't head and it is not result because those have the wrong class

Actually no

I thought so

But the structure here is more function oriented than data oriented

You know what I think there's a callback feature you'd like

The value of (:args (rose/root head)) might be what I'm seeking

Use reporter-fn

Pass one in

I'm comfortable hacking on my own modified copy of test.check and test.chuck if it helps, which is how I got to that result.

Thanks for the help. Will be away from keyboard for a bit here.