Clojurians Log v2

Patterns of what?

Spec has regex ops for recognizing patterns of data structures, the implementation of those most closely matches taking the derivative of the regular language, which is a nice simple method for implementing regular expressions bit doesn't get used much

I believe https://github.com/ztellman/automat uses the traditional NFA transformation to recognize sequences of values

There is another library whose name escapes me that is more deliberately regexes on top of clojure sequences, I don't know what the internals of that are like

If you just need to implement regexes, the derivative route is way easier

Strings

if you have a java lib you can just use it

Hi! I am working to understand clojure.spec on a project and hoping to find a bit of illumination on the exercise-fn and the test/check functions. Just as a toy example, I wrote a function that takes an email and returns a string.

(defn toy-fn [email] (str "the email is " email))

I then wrote a spec for email as

(s/def ::email (s/and string? #(re-matches email-regex %))

and I wrote a function spec as

(s/fdef toy-fn
:args (s/cat :email ::email)
:ret string?)

When I do either a (test/check toy-fn) or (s/exercise-fn toy-fn) I get an error of: Couldn’t satisfy such-that predicate after 100 tries

You'll need a custom generator on your ::email Spec.

I read in this spec faq (https://blog.taylorwood.io/2018/10/15/clojure-spec-faq.html) that this is because it’s trying to match just for string

The problem is that string? generates a very broad range of random strings and the vast majority will not satisfy your regex predicate.

right right! I was wondering what would be the idiomatic way of doing this, or a suggested way of doing this. Is it more common to write the generator as part o the spec, or as something I feed into a test function?

Gary Fredericks has a library called test.chuck that includes a regex generator which you could use to generator strings that would conform to your Spec. Or you could just generate some "canned" emails for testing (e.g., using a set of email addresses).

I have a few specs like this, and some that are maps of these things (e.g. a map of a username and an email, with both fitting a regex)…and I was excited about writing less code with clojure.spec , but it seemed like i’d be writing custom generators for much of my specs…and wanted to check there wasn’t a better way.

If you write the generator as part of the Spec, then aggregates of those Specs should generate automatically.

Here's an example from our codebase at work:

(s/def ::email (s/with-gen (s/and (bounded-string 5 128)
                                  wstr/valid-email?)
                 (wgen/fn-string-from-regex wstr/email-regex)))

(defn fn-string-from-regex
  "Return a function that produces a generator for the given
  regular expression string."
  [regex]
  (fn []
    (let [string-from-regex (requiring-resolve 'com.gfredericks.test.chuck.generators/string-from-regex)]
      (string-from-regex regex))))

and here's our email regex:

(def email-regex
  "Sophisticated regex for validating an email address."
  (re-pattern
   (str "(([^<>()\\[\\]\\\\.,;:\\s@\"]+(\\.[^<>()\\[\\]\\\\.,;:\\s@\"]+)*)|"
        "(\".+\"))@((\\[[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\])|"
        "(([a-zA-Z\\-0-9]+\\.)+[a-zA-Z]{2,}))")))

oh my gosh this is handy.

And because yr generator is defined as part of the spec, if you had (`test/check email)` and (`test/check coll-of-emails)`, you don’t have to feed sample emails into both tests.

It generates gnarly stuff:

user=> (s/exercise ::email 10)
(["\"𭏩\"@[991.8.76.94]" "\"𭏩\"@[991.8.76.94]"]
      ["\"񜑣𶤍\"@-a.JX" "\"񜑣𶤍\"@-a.JX"]
      ["𛴹.񓬝󾆜򆘔.񎔞@[0.2.306.04]" "𛴹.񓬝󾆜򆘔.񎔞@[0.2.306.04]"]
      ["󝖣󄂪@[5.293.2.240]" "󝖣󄂪@[5.293.2.240]"]
      ["򪤷.𵲥藮@[96.9.946.5]" "򪤷.𵲥藮@[96.9.946.5]"]
      ["𰔀򟡙񛼏�.󶰿񛧂򥙭󅟜.򲰴􌭨𭄁􏔛󗤠񡄃@E.NN6.erjjWW" "𰔀򟡙񛼏�.󶰿񛧂򥙭󅟜.򲰴􌭨𭄁􏔛󗤠񡄃@E.NN6.erjjWW"]
      ["\"􆓑𬸸򯐜򿳪񜺈讴񃪕\"@J.w0P.21x.hRxb" "\"􆓑𬸸򯐜򿳪񜺈讴񃪕\"@J.w0P.21x.hRxb"]
      ["񯸯.𓱨񷤖񕂉𥺼򇔵򡜕򞝹𸏙.󂶹.󿎛򨐪愸񱞶.󺖆򬃞󼾽󍎲.򴹧볿𲄆.󋲽𵛍򩠐󌡇򣛳󥊰򢍕󰅚.񊍁𲒣񊽫򘱌𨇪@[2.393.765.1]" "񯸯.𓱨񷤖񕂉𥺼򇔵򡜕򞝹𸏙.󂶹.󿎛򨐪愸񱞶.󺖆򬃞󼾽󍎲.򴹧볿𲄆.󋲽𵛍򩠐󌡇򣛳󥊰򢍕󰅚.񊍁𲒣񊽫򘱌𨇪@[2.393.765.1]"]
      ["󲶍򂇇򤍩󐞨󂁚񵚌.ⵜ񫱽𬀄򘖲󺫟򆳫鶆񗈡玉.񣰮󆪕�􋆉򦻼@l0u.C6d4qPVy.R3s.F3P2kkwd.br62Nkb4.aE-JqLl.bO.IKffxYXLW.gqyIcL" "󲶍򂇇򤍩󐞨󂁚񵚌.ⵜ񫱽𬀄򘖲󺫟򆳫鶆񗈡玉.񣰮󆪕�􋆉򦻼@l0u.C6d4qPVy.R3s.F3P2kkwd.br62Nkb4.aE-JqLl.bO.IKffxYXLW.gqyIcL"]
      ["\"񅸦򪅢댴򨱹𒊱򰍉\"@lzFjoe.aWJ9k.-9CxZ1b.TdUWCK.Nn.urDeD.FDpBg" "\"񅸦򪅢댴򨱹𒊱򰍉\"@lzFjoe.aWJ9k.-9CxZ1b.TdUWCK.Nn.urDeD.FDpBg"

Right, so

user=>  (s/def ::emails (s/coll-of ::email :kind vector?))
user=>  (s/exercise ::emails 3)
([["򸢧@[6.84.4.329]" "󐿎@[8.4.444.65]" "\"񳾈\"@[59.775.8.4]" "\"򒙊\"@5.Ub" "񂚝@L.Sq" "򒏙@[3.87.8.441]" "\"􆄅\"@u.fn" "󾟅@[618.644.46.084]" "\"񓥹\"@S.HM" "򎹉@[8.56.028.56]" "\"󖧩\"@[4.30.361.922]" "\"󕪤\"@w.WG" "\"𒨴\"@[307.508.705.640]" "\"󰡻\"@s.tq" "󙴏@M.pw" "�@I.tb" "\"񿴛\"@[04.9.3.2]" "\"񁽰\"@n.Eg" "\"􄚯\"@A.vm" "\"󆆻\"@[2.4.95.3]"]
       ["򸢧@[6.84.4.329]" "󐿎@[8.4.444.65]" "\"񳾈\"@[59.775.8.4]" "\"򒙊\"@5.Ub" "񂚝@L.Sq" "򒏙@[3.87.8.441]" "\"􆄅\"@u.fn" "󾟅@[618.644.46.084]" "\"񓥹\"@S.HM" "򎹉@[8.56.028.56]" "\"󖧩\"@[4.30.361.922]" "\"󕪤\"@w.WG" "\"𒨴\"@[307.508.705.640]" "\"󰡻\"@s.tq" "󙴏@M.pw" "�@I.tb" "\"񿴛\"@[04.9.3.2]" "\"񁽰\"@n.Eg" "\"􄚯\"@A.vm" "\"󆆻\"@[2.4.95.3]"]]
      [["󘖺.�񟑀@DC.fsX" "\"񐦲\"@[27.378.381.633]" "\"񪵐񺛕\"@Y.Sv.Hsf" "\"󱽠򚌳\"@[02.73.791.23]" "\"󤗭󱥀\"@[24.9.7.46]" "񅏪󆯊@[093.6.1.4]" "\"𹁮\"@[8.31.839.0]" "\"󺭵𵫎\"@E.dIk" "\"򚎾򆳈\"@[3.751.87.647]" "\"򝁾\"@5.wdM"]
       ["󘖺.�񟑀@DC.fsX" "\"񐦲\"@[27.378.381.633]" "\"񪵐񺛕\"@Y.Sv.Hsf" "\"󱽠򚌳\"@[02.73.791.23]" "\"󤗭󱥀\"@[24.9.7.46]" "񅏪󆯊@[093.6.1.4]" "\"𹁮\"@[8.31.839.0]" "\"󺭵𵫎\"@E.dIk" "\"򚎾򆳈\"@[3.751.87.647]" "\"򝁾\"@5.wdM"]]
      [["\"󜟞򅿅󓄹\"@o-.QQfc" "\"񟮻쇪򾆯\"@aUe.I.Db.FmjY" "񐭿.򻺩򰩠@[37.4.6.976]" "񸯛񷊾󌹃@uT.6aZ.fsX" "\"󣅼\"@YJ0.uh" "\"񼈙򬎈\"@z.j.DNle" "\"򓢅򷳮\"@J.RsE" "\"񴌜\"@ZUW.owBf" "\"񜞯񿴨򁯵\"@u.U.USjW" "򊂮.𐊹𠱣𡷗.񏂶@lrw.zqn.gN" "𫸽񽅆򱕞@[31.9.15.13]" "𱌼󋥌.򏙘�񌺊@[00.441.127.31]" "󪫮.񭞪򪟝@m.E5U.Ab.bfy"]
       ["\"󜟞򅿅󓄹\"@o-.QQfc" "\"񟮻쇪򾆯\"@aUe.I.Db.FmjY" "񐭿.򻺩򰩠@[37.4.6.976]" "񸯛񷊾󌹃@uT.6aZ.fsX"...

Sometimes you may want to override a generator for a specific test case -- you might want readable test data, for example.

That makes sense. I read a mention of overriding in the blog I linked above, but I couldn’t find an example of this in the spec guide.

it said to feed a map to test/check…would it be something like (test/check email-fn {:gen [my list of emails]})

:gen takes a hash map from symbols to generators so it's a bit more involved than that

(assuming you mean clojure.spec.test.alpha/check there)

:gen map from spec names to generator overrides

from https://clojure.github.io/spec.alpha/clojure.spec.test.alpha-api.html#clojure.spec.test.alpha/check

gotcha. Thank you!

@tskardal, that gives the result. Thanks!

@simongray, @denis.baudinot Ordering is not a concern. Can you help with an example on how I’d go about using a set to achieve the result? Thanks!

@sova, I’m not sure how I’d go about the structure you mention to achieve the resulting structure. The base-collection in the example is from an API response. Are you suggesting I transform the response to the structure you mention, and that should make the extraction of emails /phones simpler?

@alexmiller, Didn’t think of it that way. I can see the parallels now.

I'm trying to figure out how one would go about executing cljs builds using clojure tools/deps.edn without shadow, lein etc. Something like this (I'm aware of the fact that you can run cljs.main with :main-opts set)

{:paths ["src"]
 
 :deps 
 {org.clojure/clojurescript {:mvn/version "1.10.758"}}


 :aliases
 {:build/release 
  {:exec-fn cljs.build.api/build
   :exec-args {:src "src" :opts {:optimizations :advanced} } }}}

I'm trying to write a generative test, but get an exception "Wrong number of args (10) passed to: jobtech-taxonomy-api.test.generative-test/send-concept-request" when running (spec/exercise-fn `send-concept-request) in the repl. I want to debug why this has happend, but I can't find a way to inspect the exception stack. I can see the stack with *e, but not the values passed between each frame. In C I would have written 'bt full' in gdb to get the full stack. But how to you do that in Clojure? I want to see the 10 arguments, and how they were generated and passed down.

Hm fun one .. I start a message queue worker in a (def) and I think that is why lein uberjar is hanging

I'm trying to make tests in my API for study, using nubank matcher combinators. But, in my "post" route test the output show this error:

java.lang.Exception: JSON error (unexpected character): N

first thought is that N is the beginning of Not found or something that's not json

Check your payload, that's an error you'll get from json/read-str

(require 'clojure.data.json)
nil
user=> (clojure.data.json/read-str "N")
Execution error at clojure.data.json/-read (json.clj:230).
JSON error (unexpected character): N

I'd start with figuring out the java command that invokes the cljs compiler in the way you want, then translate to deps.edn config. Also there are features that shadow provides beyond compiling your clojurescript, and it can be used with deps.edn.

The jvm stack trace object doesn't carry that data, sadly. The jdb command has many of the features of gdb, but will also require being familiar with the constructs that clojure bytecode creates. Alternatively you can easily wrap send-concept-request so that it captures its args for debugging.

right, any side effects in namespaces are invoked when you aot compile

this is a very common trap for beginners to fall into

Hello Clojurists 😄 What's the easiest way to achieve simple parallelism with clojure? By that i mean, running two functions together at the same time. Someone recommended me to use promise but i can't see how this is achieved through promises, because when i try to deref the first, it will block until it's computation is done, and only then proceed to the next one.

promise doesn't do parallelism

future does

oh, right, reading the docs, makes sense

perhaps they were thinking of js, which does have promises but no parallelism

Thanks, I thought it was just that

Thanks, you helped me a lot

i have a very newbie question, because i don't have much knowledge about parallelism and so on. But for example:

(ns user)

(require '[throttler.core :refer [throttle-chan throttle-fn]])

(def +# (throttle-fn + 100 :second))

(let [p1 (promise)
      p2 (promise)
      p3 (promise)]
  (deliver p1 (time (dotimes [_ 300] (+# 1 1))))
  (deliver p2 (time (dotimes [_ 300] (+# 1 1))))
  (deliver p3 (time (dotimes [_ 300] (+# 1 1)))))

;; "Elapsed time: 3044.38462 msecs"
;; "Elapsed time: 3079.034443 msecs"
;; "Elapsed time: 3075.497436 msecs"

(let [p1 (future (time (dotimes [_ 300] (+# 1 1))))
      p2 (future (time (dotimes [_ 300] (+# 1 1))))
      p3 (future (time (dotimes [_ 300] (+# 1 1))))])

;; "Elapsed time: 9145.961132 msecs"
;; "Elapsed time: 9186.730517 msecs"
;; "Elapsed time: 9196.688568 msecs"

the timing is happening in parallel though

sorry, I misread for a moment

this is a classic problem with parallelism

so that throttling - is it being applied per invocation, or globally?

i think it's per invocation, if per invocation means that it's applied when you call the function

I mean does it limit "maximum throughput for this function, across all call sites" or "maximum throughput at this call"

because those numbers make it look like the former

@matthewlisp yeah, looking at the project, it looks like what this lib does is slow down your function globally, making parallelism pointless https://github.com/brunoV/throttler

it creates a single "spigot" controlling throughput of your function across all calls

try just using (defn slow-plus [& args] (Thread/sleep 1000) (apply + args))

that sleeps 1000 ms on each invocation, but the slowdown is per call, not globally, so parallelism will actually do something

@matthewlisp

(cmd)user=> (time (let [a (promise) b (promise) c (promise)] (doseq [p [a b c]] (deliver p (slow-plus 1 1)))))
"Elapsed time: 3000.869278 msecs"
nil
(cmd)user=> (time (let [a (future (slow-plus 1 1)) b (future (slow-plus 1 1)) c (future (slow-plus 1 1))] (doseq [f [a b c]] (deref f))))
"Elapsed time: 1000.468 msecs"
nil

that throttler lib does expose an important point though: if your bottle neck is a global resource, rather than CPU, parallelism won't help

also, this usage of promises is weird - they are quite useful if you need to coordinate between threads and don't need the full complexity of core.async, but here they are doing nothing

ohhh right, i got it

thanks for the help

I'm using deps.edn, and clojure reports 1.10.1 when i start the repl. Yet on my mac it accepts -m to run my tasks... and linux (also reporting same clj version) it has to use -a ... why does it report the same version but yet have these minor differences?

1.10.1 is the Clojure (language) version. if you do clj -Sdescribe you can see the version of your clojure tools

any version of the Clojure tools can use any version of Clojure (forward and backward) but they share a number because that's the default you'll get if you don't specify (and the version the tools themselves are using)

I assume your linux version is older

ok thanks, i'll check the tool version

@clojuregeek The change in behavior (around -M / -A -- uppercase) happened in 1.10.1.697 in the stable releases of the tools per: https://clojure.org/releases/tools (there were a whole bunch of prerelease versions between 1.10.1.561 and that).

I ended up using linuxbrew for installing the Clojure CLI on Linux so that i can brew upgrade clojure/tools/clojure on both macOS and Linux to keep up with the latest versions.

(well, if I was on stable, that's what I'd do -- I actually use brew install <mailto:clojure/tools/clojure@1.10.1.xxx|clojure/tools/clojure@1.10.1.xxx> to install specific versions since I'm always running the prerelease builds)

Thanks, i'll look at Linxu brew. I was installing clojure via instructions on Eric Normands site https://purelyfunctional.tv/guide/how-to-install-clojure/#linux ... but Linux Brew might be better solution here 🙂

I switch between macOS and Ubuntu (WSL2 on Windows) all the time so I just find it so much easier to be able to use brew everywhere and not have to think about which platform I'm one! 🙂

@jr0cket 😲 I saw this before when I'm searching for repl. Do you have a twitter account?

HI, I'm trying to solve this problem on codewars in Clojure https://www.codewars.com/kata/52a78825cdfc2cfc87000005/train/clojure I have the tests almost passing but I'm failing on things like

Test Failed
expected: (= (sol/calc t2) (calc t2))
  actual: (not (= 1.1818181 1.1818181818181819))

There are several specs and spec arguments (primitives ?) that take concrete values, such as s/int-in and :min-count etc. Is it weird/wrong to provide these values via a binding and create specs dynamically for validation or parsing (with conform). For example it seems useful to read them from a database and then register a spec based on that. A slightly more involved example would be to create a spec for a map or tuple where one value is dependant on the other.

1.0 is a double literal in clojure, not a float literal, I'd use Double/parseDouble

@denis.baudinot I wouldn't say it is weird or wrong, but it is hard with Spec 1. It's much easier with Spec 2 (but that's not ready for production use yet).