Clojurians Log v2

As long as you're not building anything with it that is more than a personal toy project @rafael...

hi everyone, is it possible to add meta to spec?

(s/def ::my-custom-int ^{:error-fmt #(str % " is not valid int")} int?)

@fmnoise I don’t know the answer, but I’m pretty sure you’re checking the wrong thing. Try (meta ::my-custom-int)

(s/def ::my-int (with-meta (s/spec int?) {:err-format #(str % " is not a valid int")}))

@fmnoise then (meta (sp/get-spec ::my-int)) will work

thanks @vlaaad, this works!

While this may work, please that it won’t work on all specs and some spec transformations like with-gen will not preserve meta. This is more accidental than intentional

Adding meta and doc string support to specs is the most requested thing in the tracker and I expect we will add it in spec 2

I have a design question - I have a Kafka Streams topology where I use conformance to automatically sanitise input, then split the stream into two - those where conformance resulted in ::s/invalid and a happy path. In the happy path, I call a pure function that transforms the conformed message. This transformer has a function spec attached to it, but because I have already conformed the input according to the arg spec before calling the function, it will fail the conformance check.

I seem to be facing two options - either remove the function spec, or create yet another spec for the already-conformed input, a "derived spec" of sorts. Neither really feels satisfactory.

Yet another alternative would be to simply validate with s/valid? earlier in the stream, then conform inside the pure function - which feels like the right thing to do, but it also feels like I am validating the data twice effectively. What would be the cleanest solution here?

@sgerguri The approach we've taken at work is to treat that initial Spec layer as a boundary and assume it maps from "external data format" to "internal data format" and is applied consistently. Thus everything "inside" is Spec'd in terms of the (already conformed) data produced at the boundary.

So if I understand it correctly you have a second spec layer for the internal representation, that represents things that have been conformed, correct?

Yeah, an API Spec, a Domain Spec, and actually a Persistence Spec. The API Spec is conforming, from external data (often strings) to internal data (numeric, Boolean, date, etc). The Domain Spec describes the data formats used inside the API. The Persistence Spec describes the data that goes into the database (flat hash maps with JDBC-compatible types). We don't use all three in all cases, but it seems to have become a useful way to split things up -- and it clearly identifies two boundaries (API input and database output).

That's excellent, thanks. I have used a similar approach in another service, the only thing I'm struggling with in this one is the fact that the internal layer is very thin and quite close to the input layer, thus it feels like duplicating the specs, but might be worth it for the greater clarity.

spec-coerce is quite good for this too

I would caution against leaning too heavily on conforming with Spec -- I don't really like what spec-coerce enables.

By default it will do the right thing to coerce x to your spec expeceted values, and the other way around (serializing to db type, ex keyword -> string) just requires to override those cases

No spec duplication that way

No polluted specs either (no conform abuse)

I have used it for some fairly tricky transformations in the past, but I've found it really works well for emulating pattern matching along with multimethods by tagging individual cases through an s/or. In this particular example I only wanted to trim whitespace but for some reason am having trouble fitting it neatly in without overhauling one part of the service or another.

This particular situation also left me wondering whether conformance should either be the final station in your transformation/validation stack or whether it should yield something that also has a spec for it (without further conformers), like @seancorfield described.

In the grand scheme of things I could just trim the fields I need to but with spec around that just feels like the wrong approach.

There are folks on both sides of that decision 🙂

Some feel that input should be cleaned and parsed first, then checked for validity with Spec (and therefore next to no conforming). Others feel that input conformance to valid data is a reasonable use of Spec.

I used to have dual specs like you mention, but it requires more work and is more brittle imho

Then as you say ymmv

I am slightly on the conforming side of center. spec-coerce is quite a lot further out on the conforming side. I think Alex (and maybe others at Cognitect) are on the non-conforming side of center.

I actually like spec-coerce :)

Really? I was sure you had complained about it in principle in the past? 🙂

I don't like spec-tools

Food for thought. Thank you everyone, always a pleasure to come here for another opinion on how folk do things. 🙇

spec-coerce builds a separate registry of coercions that leverages specs, which I think is a good approach

I haven't used it in anger but that seems in line with how I would approach it

It's gotten better lately, it was missing multi-specs, merge & tuple until recently but now it's quite feature complete

It's also 300ish lines of code, easy to modify, fit to your taste if needed

@alexmiller Ah, thanks for the clarification. Then I suspect I'm getting the two libraries confused and maybe I should look at spec-coerce in more detail? 🙂

You contributed to it a few years ago https://github.com/wilkerlucio/spec-coerce/pull/11 :)

Hello, I hope it's ok to post some code. I'm trying to spec a function that has ehmm, this optional arguments thing? I'm not sure how it's called in Clojure. I'm having some trouble with the spec for it. Right now I specifically use (st/instrument) in a test as suggested and the following code breaks with the following exceptions: (ship (:id p)) ; (:id p) is surely a uuid || :cause "class java.util.UUID cannot be cast to class java.lang.Number (java.util.UUID and java.lang.Number are in module java.base of loader 'bootstrap')"

It seems like it's related to the ::speed thing, being an integer. I must have made something funny in the spec, if someone can notice. Thanks!

Ah yes, thet exception I gave is invoked on the s/def of ::speed 😕

what is (random-id) ?

It's my own function, which calls (UUID/randomUUID) wth java.util.UUID

Wow, I don't even remember that. Looks like I'd started to look at spec-coerce as a possible alternative to our existing "web specs" at World Singles Networks -- and I guess by the time that PR got accepted I'd decided not to go down that particular path for some reason...

The code itself works without the spec, I must be defining something badly

I've made a note to revisit it, but here's what we ended up doing at work https://github.com/worldsingles/web-specs

what about default-ship-speed?

(defonce default-ship-speed 20)

What is DateTime?

(:import [org.joda.time DateTime]
           [java.util UUID]))

Though I use clj-time, so I might be wrong here when I think of it

'k... Joda Time is deprecated. If you're on Java 8 or later, you should use Java Time really.

clj-time is also deprecated.

(for the same reason)

Oh, thanks for letting me know. I'll google for Java Time

(s/def ::speed (fn [v] (and (int? v) (> v 0)))) this definition works

but dunno why it assumes uuid there by default when doing coercion

The problem is that all three optional arguments are optional independently

So dest-planet-id could be omitted and the speed and departure-time could still be provided.

In other words, it tries to check (ship src-planet-id src-planet-id) against a signature that is essentially [uuid? #(> % 0)] and that's causing the exception.

And that's why adding (and (int? %) ...) into ::speed "works" -- it guards the > operation from being called on non-numeric values.

Because then the ::speed spec successfully fails to match (instead of blowing up) and Spec goes on to try the other options.

So...

I'd suggest spec'ing ship a bit differently, perhaps using s/alt over the four arities, each spec'd with no optional parameters.

Or at least across the first three and leave just :departure-time as s/?

Thanks a lot! However I don't fully understand how this happens: > dest-planet-id could be omitted and the `speed` and `departure-time` could still be provided.

I'm sorry I have to go now as it's getting late. I hope it's ok if I ping you tomorrow in this thread if I still have some trouble. Thanks!

You have src dest? speed? time? -- each of those three are optional, so each could be omitted while the others are passed.

so src speed is "valid", as is src time or src dest time or src speed time.

That's what your fdef says.

So when Spec sees (ship src-planet-id src-planet-id) it's going to try src speed, src time, and src dest in some random order.

Since your speed spec just tries to compare the value > it will throw an exception if passed a non-number: which a UUID is.

Because Spec encounters an exception, it won't try the other options -- it just propagates the exception.

(! 629)-> clj -A:test -Sdeps '{:deps {clj-time {:mvn/version "RELEASE"}}}'
Clojure 1.10.1
user=> (require '[clojure.spec.alpha :as s] '[clojure.spec.test.alpha :as st])
nil
user=> (import '(org.joda.time DateTime) '(java.util UUID))
java.util.UUID
user=> (defonce default-ship-speed 20)
#'user/default-ship-speed
user=> (defn random-id [] (UUID/randomUUID))
#'user/random-id
user=> (s/def ::id uuid?)
:user/id
user=> (s/def ::src-planet-id uuid?)
:user/src-planet-id
user=> (s/def ::dest-planet-id uuid?)
:user/dest-planet-id
user=> (s/def ::speed #(> % 0))
:user/speed
user=> (s/def ::departure-time #(or (nil? %) (instance? DateTime %)))
:user/departure-time
user=> (s/def ::resources #(>= % 0))
:user/resources
user=> (s/def ::ship (s/keys :req-un [::id ::src-planet-id ::dest-planet-id ::speed ::departure-time ::resources]))
:user/ship
user=> (defrecord Ship [id src-planet-id dest-planet-id speed departure-time resources])
user.Ship
user=> (s/fdef ship
        :args (s/alt :arity1 (s/cat :src-planet-id ::src-planet-id)
                     :arity2 (s/cat :src-planet-id ::src-planet-id :dest-planet-id ::dest-planet-id)
                     :arityN (s/cat :src-planet-id ::src-planet-id :dest-planet-id ::dest-planet-id :ship-speed ::speed :departure-time (s/? ::departure-time)))
        :ret ::ship)
user/ship
user=> (defn ship
  ([src-planet-id]
   (ship src-planet-id src-planet-id))

  ([src-planet-id dest-planet-id]
   (ship src-planet-id dest-planet-id default-ship-speed))

  ([src-planet-id dest-planet-id ship-speed]
   (ship src-planet-id dest-planet-id ship-speed nil))

  ([src-planet-id dest-planet-id ship-speed departure-time]
   (->Ship (random-id) src-planet-id dest-planet-id (if (nil? ship-speed) default-ship-speed ship-speed) departure-time 0)))
#'user/ship
user=> (st/instrument)
[user/ship]
user=> (ship (random-id))
#user.Ship{:id #uuid "3ad92050-2b9b-499e-b7e3-f933053d7b1c", :src-planet-id #uuid "79cd0367-d78d-4774-9f5f-3a29a8c77851", :dest-planet-id #uuid "79cd0367-d78d-4774-9f5f-3a29a8c77851", :speed 20, :departure-time nil, :resources 0}
user=> 

^ That shows the s/alt structure that would work @amir