Clojurians Log v2

Is there a way to get test-check to continue running a check even if one fails? I'm looking to create a large list of failure values for further analysis. Going to try using with-redefs on results/pass? . Curious if anyone else has done something like this.

Are you using defspec in particular? Do you want shrunk values or just original failures?

@gfredericks Just quick-check on a prop/for-all.

In this case, I don't really care all that much on shrunk vs original.

Wouldn't it be pretty easy to run it in a loop then?

Loop over quick-check N times, collecting failures? Would that cover a good amount of surface area if the iterations to failure is small?

You're worried about there being some small likely failure that keeps getting found and obscuring other larger failures?

Right

I think it's likely to fail in < 5 tests.

I'm looking to find holes in my specs, verifying that the hole is correct or the spec is wrong.

So the tc failure doesn't represent an actual bug?

Yes - it may or may not.

Getting a PBT library to generate distinct failures without iteratively fixing the earlier ones is pretty hard/advanced

Any small failure is likely to be a component of larger failures

For sure. Distinct can be impossible to define too 🙂

Yep that's part of it

You can get it to start at larger sizes if you think that will help

gen/scale

With min

To be more specific about my problem... We have a wrapper around some data pulled from an external API. We query data out using flat maps. Sometimes your query may have no results. This is the "problem". The solution is either: 1) Our generator for the query generated a map that should never be possible. We need to fix the generator (this can be hard) or acknowledge that this case will return no results 2) The external API is missing data for a particular query that should be there. We need to open a support case with the 3rd party, asking them why this is the case and what should be done.

Hmm, that's interesting. I'll try your loop approach first and see what results it brings. If it seems to be missing some cases I can try scaling the gen. Perhaps even using a random scale with each loop.

Sure

Thanks 🙂