Clojurians Log v2

What are the design tradeoffs to building a custom keyword-based global registry like Spec or Re-frame handlers/subs, versus simply using Clojure's vars? (has anyone written articles or posts on this subject?)

Well, for spec, the whole process is defining a spec for a keyword

Can’t speak to reframe. I’m not sure what they do with them or why.

Really it’s massively use-case dependent. I don’t think there are any general rules. Do you have a use case you’re curious about?

It's more a general question I was wondering about - I saw this "keyword registry" pattern in a few other libraries like Malli

Spec's different for sure, just referring to the idea of using a global registry to map keywords to some domain-specific data

I wonder if it's simply a matter of ergonomics (being more DSL-ey and not having to explicitly require and refer namespaces ), but you end up having to implicitly make sure that the ns registering those keywords has been loaded

At a certain point once your DSL gets too complicated and bloated, you end up essentially reimplementing Clojure itself badly

People are uncomfortable with serializing code as a rule sometimes, but DSL's with enough expressivity and the intent-to-run are code

My tendency is to default to clojure vars until there is a concrete reason to move to keyword references.

I think most people drift towards keywords first because they are more familiar.

But when you end up with a bunch of keywords that end up invoking functions elsewhere, you’ve kinda clearly gone too far.

That makes sense - so if I have an extensible domain-specific list of 'actions' that are meant to be interpreted in several places and invoke different functions, like

[{:type ::action/START
  :attr ::color/red}
 {:type ::action/PAUSE}
 ...]

(require '[foo.actions :refer [START PAUSE]])

[{:type START
  :attr ::color/red}
 {:type PAUSE}
 ...]

or actually just

[(START {:attr ::color/red}})
 (PAUSE)
 ...]

Yeah in cases like these I always start with the data, then maybe move to a function for concision.

So say you had this:

[(START {:attr ::color/red
         :state ::state/wait
         :name "hello!"})
 (PAUSE)]

I would make that more concise by doing this:

[(START (init-start ::color/red
                    ::state/wait
                    "hello!"))
 (PAUSE)]

where init-start returns:

{:attr ::color/red
 :state ::state/wait
 :name "hello!"}

hmm, so each action takes the same arglist format (for composability?) and you define preprocessor functions to pipe into them

ie. why not

(START ::color/red
       ::state/wait
       "hello!")

I guess there's also the tension between "expanding" the DSL early vs keeping it user-readable through intermediate stages

;; Hiccup DSL
[:div {} args]

;; vs

(div {} args)

=> {:type ::dom-nodes/div
    :render-fn #function[0xabc3d]
    :html-name "div"
    :lots-of :noise
    ...}

Yeah I don’t know what the START fn does/returns.

I was just riffing on your initial suggestion of:

(START {:attr ::color/red})

But yeah, I also lean toward “expanding early.” Removes a lot of rigamarole.

You can always remove things for readability. Can’t add things back in when you need them.

ymmv — that’s just my tendency

Thanks a lot for your perspective - I'm not experienced in this sort of library design and wasn't able to find any best practice guidelines

I’m reminded of the difference between Compojure and Reitit

random(not serious) question: what is the Haskell's equivalent of Lisp? (the answer should not include clojure) edit for "not sure if i get the question": the question comes from the context that, Haskell is not really a mainstream language, often considered academic, but somewhat foundational to learn and do some "real" programming - this can be said of Lisp as well. yet, Haskell has got a good community, and the language is comparably more active, we can see it being used in production to address specific use cases - this cannot be said about Lisp** You can hardly find someone or a company that is using Common Lisp ( I found Grammarly after some Google search) Scheme used at.. Naughty Dog? Clojure is good except it has a good audience complaining about being a hosted language. We can ignore them or try to convince them, but can't deny it.

not sure if i get the question... will try to attempt anyway... so if clojure is a nice descendant of lisp... probably elm is a nice descendant of haskell... 🙂

and if we're talking about origins... for haskell... it's (drum roll)

This might be the series of blog posts for you https://crypto.stanford.edu/~blynn/compiler/

Took me a while to get that… Miranda… but that’s a simplification: Haskell has its roots in a lot of languages that UK university researchers had been creating for many years before the committee got together to create Haskell.

NPL, Hope, KRC, SASL, ML, SURE, … many, many FP languages were created in the ’70s and ’80s before Haskell appeared at the beginning of the ’90s.

One interesting design point is that afaik, spec only provides functions that work with the global registry. if you call (s/valid? ::my-spec m) and m includes an attribute, that attribute will be validated even if that attribute is never mentioned by ::my-spec. This is in contrast to the hierarchies for multi methods where both using a global registry or a local registry is supported. re-frame also has a global registry, but I think it's mostly a side-effect of the event system provided by the DOM (see https://github.com/day8/re-frame/issues/137). It's also probably partially influenced by clojurescript's reduced var support. If you're designing a system where the goal is for names and constraints to be enforced globally (like spec), then keywords support that more directly. By global, I don't mean just mean across a whole program, but across multiple programs or systems. For example, you could store a spec keyword in a file or send it across the network which is less direct if you were trying to use vars as names across programs and systems. This video touches on some of the reasons spec uses a global keyword registry rather than vars, https://vimeo.com/195711510

is there a nifty way to check if a given http://java.io.file is a child of another? (may be N levels deep, so a naive check doesn't suffice)

@vemv (.getParent ...) returns nil if it doesn't have a parent I think?

oh you want to say (parent-of? x y) ?

(= x (.getParent y)) ?

> oh you want to say (parent-of? x y) ? yes > (= x (.getParent y)) skips the N levels requirement ;p but thanks to :duckie: I think a solution is apparent now

I would start with .toPath

absolutize paths to strings, check with str/starts-with?

I think java.nio.Path is the way to do just that properly

@nilern what specifics of Path makes this easier?

Paths as strings are not so robust and portable

@nilern that doesn't answer my question though :)

I am still looking

$ bb -e '(str/starts-with? (str (fs/real-path "README.md")) (str (fs/real-path ".")))'
true

@vemv fyi, this function is available in https://babashka.org/fs/babashka.fs.html which is based on java.nio

fs/real-path returns a path without a trailing slash. Meaning, the check is not robust at all if "." is "/tmp/a" and "README.md" is actually "/tmp/aa/README.md".

true that. so you could use (seq (fs/real-path ...)) instead (this returns all the components) but is there similar starts-with? for seqs in clojure?

Apparently, Path has startsWith.

(-> (io/file "OpenSource")
    .toPath 
    (.toRealPath (make-array java.nio.file.LinkOption 0))
    (.startsWith (-> (io/file "/home")
                     .toPath
                     (.toRealPath (make-array java.nio.file.LinkOption 0)))))

ah, path has startsWith? nice

$ bb -e '(.startsWith (fs/real-path "README.md") (fs/real-path "."))'
true

oh!

$ bb -e '(fs/starts-with? (fs/real-path "README.md") (fs/real-path "."))'
true

Having suffered through that interop, fs looks really sweet

@borkdude Wait, you had fs/starts-with? all along? :D

yeah :)

It's this one: https://github.com/babashka/fs

Here are the docs: https://babashka.org/fs/babashka.fs.html

Recently added this fun function:

$ bb -e '(map str (fs/modified-since "test" "src"))'
("src/babashka/impl/tasks.clj")

is there a way to add a watcher to an atom that is fired not when the state change (like via add-watch), but instead when an atom is deref fed?

If you create a custom atom-like type, yes. Reagent does exactly that with its ratoms and reactions.

ah exactly, thanks 🙂

not natively but it’s possible to extend IAtom

the behaviour i am looking for is basically same as a TTL cache so i use that

(def my-deref (let [a (atom 1)
                    deref-fn (fn [] (prn :foo))]
                (reify
                  clojure.lang.IDeref
                  (deref [this] (deref-fn) @a)
                  clojure.lang.IAtom
                  (swap [this f]
                    (swap! a f)))))

@my-deref ;; prints :foo
(swap! my-deref inc)

nice!

genius

might just use that instead of ttl cache

any copyright? 🙂

I hereby permit the right to copy

haha thanks

Good afternoon everyone! I have recently purchased a new Mac M1 with Apple silicon and it looks like kaocha doent work in there. Can anyone help me or point to the right channel?

I have a Mac Mini M1. I can run kaocha without any issues.

You will have to elaborate further on the symptoms you are experiencing. There is the #kaocha channel too.

I think Ill go there! Thanks a lot!

Have others found the ::thing/id qualified keyword sugar to be harmful? Everything just works so much smoother for me when I type out qualified keywords in full…specifically surrounding: • Circular dependencies • Confusing lint tools into thinking namespaces are unused (mainly with cljc) • Fragility when moving functions/data to new namespaces (ie ::xyz can’t “travel” at will) • grep for aliased keywords may be unreliable • …others?

i've settled on a pattern that i mostly never use auto-resolve keywords from other namespaces. i treat auto-resolving keywords to the current namespace as marking that keyword as "private" and then use more generic namespaced keywords (not tied in any way to a ns) when they need to be referenced from multiple namespaces

I agree on the practical flaws, but for me that indicates that the tools should be improved, instead of making my clojure idioms more rudimentary more often than not they can be improved with a very moderate effort being necessary

the rebirth of rewrite-clj in particular will be a game-changer

grep will never be aware of namespaces but file bugs for the linters?

I foresee https://github.com/borkdude/grasp or similar tools being the future of clojure grepping :) far more semantic, reducing false negatives/positives

Nice. This makes sense. Then it’s a convenience for a keyword being internal to a namespace, with zero risk of colliding with any user keys that happen to flow through. Just have to be really careful that they never “escape”.

it's not escape per se but no one else cares. and in that sense it's quite easy to do. Think of a middleware for a handler. if it wants to annotate the request with a start time and then an end time afterwards, it can add ::start-ms or something, and this is a sign that only this namespace is aware of, and cares about it. If it's introducing something that other things should care about, say user details it should use :cached/user or :context/user or whatever you think makes the most sense. Then it's clear this is introduced for others. In both cases the keyword "escapes" but the use is clear if its intended for others to access or not

I disagree a bit with wanting tooling to step up. More rudimentary clojure idioms actually seems kind of nice…it affords simplicity for the cost of a little convenience. I’m getting pretty used to the extra keystrokes and visual noise. The auto-completion helps quite a bit tho, which is admittedly a tool ha.

Interesting. How would this work with something like datomic schema, where my keywords are scoped to my domain? Like :org.company.model.user/id? It’s not really private to any one namespace, but still deserves the fully qualified name.

Isn't the approach there simply to always refer to those with the full name :org.company.model.user/id ?

You can do that, regardless of whether a particular project has a namespace org.company.model.user or not, I thought.

in that case you've already gone down the road of tying keywords to namespaces. I would ignore the fact that there happens to be a ns that currently shares the same name and always use the :org.company.model.user/id

> Isn’t the approach there simply to always refer to those with the full name That’s what I do yea. I was more curious whether it was an exception to the above guideline. Another example, pathom places a lot of fully domain qualified keywords into the parsing environment, with the intent that you can modify them and analyze them. So they’re not really “private” per se (or maybe they are and I shouldn’t be poking them lol).

yeah more than one way to skin a cat :) the circular dependencies point is interesting as it is tooling-independent. A pattern that has worked well for me over the years is a namespace dedicated solely to specs/keywords. I call it kws.clj (shorthand for "keywords"). Could be called specs.clj as well (although I don't like that choice for unrelated reasons) ...It's not merely a workaround, but a design choice that can yield a clearer API/impl separation

There is no need to create a namespace that matches the qualifier part of a qualified keyword, right? i.e. you can use the keyword :foo.bar/baz regardless of whether there is a namespace foo.bar or not.

as long as you are willing to copy/paste (or type, or use IDE auto-completion) its full name

yes, but 90% of my ns-qualified kw usage is spec-related. The spec semantics imply an underlying implementation, which increases the chances for having an actual circular dep

I’ve experimented with that approach as well. I still can’t quite get my head around how to use spec. What’s always irked me is that a keyword isn’t really always the same “shape” everywhere, as much as spec would like it to be (with its global registry). Here’s an example: say I spec a keyword called :order/line-items. This keyword can have any number of values depending on the context where it’s used. As a vector of entities: {:order/line-items [{:li/id "1"}, {:li/id "2"}]} but also as a vector of strings if we’re using datomic temp-ids: {:order/line-items ["li-1", "li-2"]}, or sometimes not even as a vector, and just a single temp-id {:order/line-items "li-1"}… So I never quite perfected the approach of putting them all in one place as a source of truth, because they’re just so contextual by nature. Could be why qualifying them is difficult.

should be :order/line-items and :order/line-item-ids

@cjsauer re: https://clojurians.slack.com/archives/C03S1KBA2/p1618329309463400 use #lsp - find references will find all your related keywords, no matter their surface form. This is powered by clj-kondo analysis, which you can also use as a library if needed.

circular deps are not an issue with keywords at all

i have a strong aversion to map entries having ambiguous cardinality. {:order/line-items "id"} vs {:order/line-items ["id"]}. that is doubly so when the string of the single item is seqable

That won’t work tho. Because then I can’t validate my transaction map that uses temp-ids. I have to use the keyword as it exists in my database schema.

specs are not contextual, that was talked about recently https://clojurians.slack.com/archives/C1B1BB2Q3/p1617647754017300

they are only an issue if you insist on closely coupling your code namespaces to keyword namespaces, and then tie your code namespaces into knots

no you don't

Nice, that’s pretty nifty

I tend to want this. :)

don't it is silly

I do agree that ::foo/bar and #:foo{:bar ...} etc have made it harder on clojure tooling. I don't even like the #:foo{:bar ...} notation myself, I prefer not to use it

(d/transact conn {:tx-data [{:order/line-items ["tempid-1" "tempid-2"]}]})

I can’t use :order/line-item-ids in that tx-map

it can be seen as an enforcement of decoupling, which is the opposite of having knots

a transaction may be different than an entity

Exactly, yet they use the same keyword. That’s my point. The same keyword can take different values depending on the context.

at the point you're transacting you're past the point of using spec, though

you can see it that way, but the fact that it ties your code base in knots is evidence that seeing it that way is not correct

I think that’s specific to that one example. Take the case of using idents. {:order/line-items [:li/id "123"]}. This could very well be pre-transact.

You could use s/or in some way, but I don’t think that really covers it, because now it’s too admissive. There might be contexts in which idents really aren’t valid.

I believe the main point of spec is to have a stable definition within your program of what a key might be. e.g.

(defn order-total
  [{:order/keys [line-items]}]
  ,,,)

the things you're talking about are necessary because at the edges of your program, it might not be in a useful shape yet to use throughout your program

I prefer to have (and solve) an explicit knot to an implicit one else the circular dep is there, it may just not manifest itself until much later

between the database and the meat of your program there's typically code that will transform the shape of data into what it should look like.

Yea. I suppose the lines of “your program” can get a bit blurry in web applications too, because you’re managing so many things at once. Imo, a global registry is just wrong. There’s simply no such thing as a “stable definition” at that level. Reductio ad absurdum: scope “your program” to be just a single function. That’s really the only level of granularity that allows a keyword to be stable.

e.g. you might transform {:order/line-items [:li/id "123"]} to {:order/line-items [{:li/id "123"}]}

it isn't

I think if the novelty of your program is shuffling data to and from a database then the majority of your code will be handling your data in its un-conforming state

a keyword is a value, like the number 5, so when you using :foo/bar you don't implicitly have a dependency on the namespace foo, and you don't have a dependency on everywhere that keyword is used

which does make spec less attractice

just like using the value 5 doesn't mean you have some kind of dependency on wherever else 5 is used

Yea it’s a weird one. The schema is what allows for the disambiguation.

I've worked in webapps basically my whole career and that hasn't usually been the case; eventually there's a bunch of interesting business logic or some other process that needs to occur, e.g. taking data from one place, transforming it and annotating it and doing some logic, and then putting it somewhere else, which having a stable definition of what a key should look like or what an entity looks like is quite useful in that in-between step

and separating the I/O and the data shape required to effect changes with those two different systems is useful

With spec in play though, consuming :foo/bar, where :foo/bar is backed by a spec, implies an implicit dependency on whatever ns is performing s/def :foo/bar

and you're right, spec isn't going to ensure that my datomic transaction is correct, but it could check to see that my data is correct before I create the transaction

so it's not useful in all contexts

only if you are using spec features (s/valid? what have you)

and spec dependencies are much more lenient than clojure namespace/var dependencies

That’s really the core of my struggle with it. It’s such a useful thing to have that specification to point at in those in-between steps like you mention. But I really only want to make that specification at that location. As implemented, spec forces you to make one grand statement about a keyword, and in my experience I always end up getting stuck. Basically, what if you didn’t use s/def at all, and only whipped up specs “on demand”? Like what https://github.com/metosin/malli does (specs as data).

> only if you are using spec features using a spec-backed defn is common enough yeah well I don't particularly appreciate that leniency for one thing it can result in spec checks not being performed because some ns didn't happen to be required by the consumer also can result in circular deps iirc

I dunno, I basically never spec fns, defn or otherwise

I do think that having a global definition is a feature, not a bug. but I agree that it would be useful to opt in to - or override locally - that global definition

at work we have 34 s/fdefs and 1011 s/defs

Yea I’ve never actually written out #:foo{:bar …}, but I see it printed quite a bit.

yeah varies widly per team. I tend to use s/fdef (or a variation) for most 'public api' defns. Accordingly I try to keep APIs small

It's useful when you have a large static map where many keywords have the same ns.

But then you have the one odd-ball keyword that’s in a completely different namespace, and now you have to rewrite the whole map, or resort to merge haha.

user=> #:aaa {:b 1, :c/d 2}
{:aaa/b 1, :c/d 2}

Oh wow. I didn’t know that was possible.

:D Did I just convert you?

opens repl to test…

whelp, I had no idea you could opt out of the sugar like that!

I might have just been converted

you can also write

#:foo{:_/a 1}
{:a 1}

Now it's my turn to use 🤯

This actually makes typing out long namespaces (and avoiding ::sugar) that much easier.

It’s something that takes practice I think. I’ve attempted it several times and usually end up scratching my head. With a global registry, you have to be very intentional ahead of time what exact “level” you’re going to be running specs at. From your examples that would be at the validation layer of the stack.

With a web stack tho, there’s just so many layers…and several of them might benefit from specs “a la carte”

#:_{:a 1 :_/b 2 #_:c}
;; => {:_/a 1, :b 2}

I wonder what a Obfuscated Clojure contest would look like

@borkdude playing with #:foo{:bar …} syntax in vs code. It seems to confuse #lsp when using “Find references” on the fully qualified keyword. The line numbers are way off. Where would I file this issue?

@cjsauer filed: https://github.com/clj-kondo/clj-kondo/issues/1251

interesting, thanks for mentioning this if I’m correctly understanding what this does, the binary could be used in multiple repositories for searching all occurrences of some symbol, which is quite helpful for finding deprecated functions usages

sounds correct! a classpath https://github.com/borkdude/grasp/tree/464f86af5866a134374af9ef7ed152882846b531#grasp-a-classpath can easily span multiple repos

This may be a reach, but does anyone have a regex handy that matches the version numbers of clojure core? Trying to make a spec for matching Clojure versions

@donyorm Clojure has something better: *clojure-version* which is data

(source clojure-version) should give a structure of what to expect in the version string output from (clojure-version)