test-check

Joe Lane 2018-04-10T16:04:09.000063Z

Is there an api in test.check to get the collection of inputs and outputs used when exercising a test? For example, when running

(defspec first-element-is-min-after-sorting ;; the name of the test
         100 ;; the number of iterations for test.check to test
         (prop/for-all [v (gen/not-empty (gen/vector gen/int))]
           (= (apply min v)
              (first (sort v)))))
getting a collection of
({:input [1] :output [1]},
 {:input [2 1] :output [1 2]}
 ...
I suspect it the data I’m looking for is :args and :result-data or :result from the quickcheck function in check.cljc. Is there a blessed way to get access to this information?

ghadi 2018-04-10T16:11:03.000820Z

you can recreate it using the random seed that quickcheck gives you

Joe Lane 2018-04-10T16:20:44.000479Z

@ghadi I can get the data literals using the seed? I know I can re-run the test again, i’m curious about the data itself though

ghadi 2018-04-10T16:21:37.000006Z

right, I'm not sure the exact process

ghadi 2018-04-10T16:21:53.000198Z

the seed is there so that you can get reproducibility

nberger 2018-04-10T16:24:59.000327Z

@lanejo01 may I ask what do you plan to do with that data? I'm truly curious, mostly because a while ago I wanted to add a way to get a glimpse on what's the distribution of the generated values

2018-04-10T16:32:53.000097Z

args is the generated data

2018-04-10T16:33:36.000530Z

The return value isn't given, you can just call your function yourself once you have the args

ghadi 2018-04-10T16:34:19.000385Z

@gfredericks can you do something like sample-seq while providing the seed?

Joe Lane 2018-04-10T16:34:45.000233Z

I had another instance of a recurring dream last night. The word cloud looks like this: clojure.spec generative testing + consumer driven contracts (like http://pact.io) except for libraries, not web apis + spec-ish type-system-ish code inspection/analysis (e.g. (-> (makes-random-int) expects-only-odds) would be analyzed and the system would show there is a problem) + service for publishing specs to a public repository. The big thing here is consumer driven contracts for function apis. A library author could know the expectations of its consumers based on properties they provide, and then know if they make a breaking change or not.

Joe Lane 2018-04-10T16:35:19.000754Z

There are a LOT of conflated ideas here, please dont take this as a proper description of all goals.

Joe Lane 2018-04-10T16:39:13.000016Z

I suppose the ideas are 1. consumer driven contracts for public library functions 2. analysis of clojure programs using spec to determine areas for potential bugs 3. maven for specs.

Joe Lane 2018-04-10T16:40:35.000586Z

The reason I’m asking is about inputs and outputs is to create a system like haskell’s hoogle (https://www.haskell.org/hoogle/) with spec.

Joe Lane 2018-04-10T16:41:27.000801Z

Thanks @gfredericks!

grzm 2018-04-10T16:55:12.000535Z

@lanejo01 The way something like that could work is if consumers published specs that the upstream author could consume, and those specs would have to be more constrained versions of the specs the author already publishes. The upstream author could swap those more specialized specs in their own tests. That reminds me of "specialization by constraint" in Date and Darwen's "Databases, Types, and the Relational Model". I'm not well-versed in type-theory in general, so it might have other names, but that's where I came across it.

grzm 2018-04-10T16:56:28.000625Z

The idea there is that any proper subtype is only a subtype if it satisfies the definition of its parent type plus some additional specialization constraint.

Joe Lane 2018-04-10T16:57:11.000052Z

That’s exactly how http://pact.io does it, except with json and regex, not something more expressive like spec + edn.

grzm 2018-04-10T17:01:44.000333Z

How would "maven for specs" be different from maven (or similar tool) in general? Specs are code and just like when you use a library, you still need to know the names of the functions (or specs) once you have it.

2018-04-10T17:06:53.000736Z

It could exist, but might not be immediately useful since you couldn't easily pick out the failing args. Do you have another use case?

Joe Lane 2018-04-10T17:13:46.000491Z

It was a bit of a mischaracterization of what I’m going after ( which, is admittedly, a moving target). It’s more about a place for consumer “contracts” to be placed. In addition to that, the idea of a central place that I can look up what functions take this input and produce this output could be really exciting for a variety of reasons. Not so much maven for specs. Sorry for the confusion.