Clojurians Log v2

I'll try this test again with the JIT off

what is the best way to understand the different kinds of function definition argument lists and the corresponding call-site argument lists? Simple function definitions with a fixed number of named arguments are easy. But in Clojure, function application also allows exotic argument lists involving optional arguments and key arguments.

basically if I want to implement a macro which extends the semantics of fn what are ALL the cases I need to cover as far as argument binding and syntax of call-sites ?

@jimka.issy There is really one "special" thing in the arg list, which is & which separates the required (fixed number) args from the variadic args.

These variadic args are always received as a sequence. But as a convenience you can use destructuring in the arglist, same as in other places like let.

I’m referring to the recent @didibus comment concerning an argument list like [{:keys [^Int bar]}]

So when you have x y & {:keys [a b]} and you pass 1 2 :a 1 :b 2: x = 1, y = 2, rest = (:a 1 :b 2) which is destructured

great! so are those rules stated anywhere, or does the programmer (me) have to understand lots and lots of interrelated concepts to understand it?

Clojure has functions like destructure and maybe-destructured (https://github.com/clojure/clojure/blob/b1b88dd25373a86e41310a525a21b497799dbbf2/src/clj/clojure/core.clj#L4504) which are very helpful for re-implementing fn

There are only two interrelated concepts here: variadic arguments and destructuring

what is wrong with this function invocation?

((fn [& {:keys bar}]
   (list bar)) :bar 3)

{:keys [bar]}

OIC, the missing [] yes indeed

so apparently this syntax [& {:keys [bar]}] allows any number of :bar value pairs at the call site, as well as any number of other unmentioned keys. The right-most value associated with an :bar is bound to the bar variable within the function.

what if I want to disallow repeated keys and disallow unmentioned keys?

in Common Lisp I can specify &allow-other-keys but I cannot preclude repeated keys

Clojure has an open world philosophy where additional other keys are usually not warned against

This is where libraries like spec come in. spec2 will have an option to specify this

Although at this point it's unclear if and when spec2 will see the light of day

Libraries like malli or prismatic/schema also support this

Not sure whether you’ve seen the post by @didibus, but which of the following would be applicable ?

(dsfn foo
 ([& {:keys [^Int bar]}]
  (list 'int 'bar bar))
 ([& {:keys [^Int baz]}]
  (list 'int 'baz baz))
 ([& {:keys [^String bar]}]
  (list 'string 'bar bar)))

It depends on the rules of your dsfn macro, I guess you are the master of your macro

Yes, I think that the caller/programmer WOULD NOT EXPECT normal clojure evaluation rules to apply.

In practice, I think the signature of that function would be unusual: one branch is interested in one key, but another branch is interested in only another key with no overlap. Usually you have a common set of required keys + some optional ones.

on the other hand, I think there are other specifiers other than :keys which can be used in this syntax. right?

(dsfn foo ([& {:required [^String bar]}]))

you can make up whatever you want in macros, although this requires custom destructuring logic

I guess you could merge the commonalities into one arity, destructure and then check

Somewhere I say some extra syntax for inside these braces. syntax that provides default values for example. Is there a section about destructuring in https://clojure.org/reference

found it https://clojure.org/guides/destructuring

In standard Clojure JVM you're not allowed more than one variadic overload, so normally that scenario would throw:

CompilerException java.lang.RuntimeException: Can't have more than 1 variadic overload

But that's similar to how you're also not allowed two overloads of the same arity. With your macro though, since it can dispatch on type, for this example I do think the expectations would be it calls the second one.

And if someone called it like so (foo :other 10) then I'd expect the first one to apply. Since your macro does already prioritize the first match left to right.

And I'd expect the third one if someone typed (foo :bar "hello")

And finally if someone typed: (foo :other "hello") I'd expect the first one again.

I'm nowhere of an expert on language/library design but I think this is what Rich Hickey meant in Simple Made Easy when talking about pattern matching and switch statements being sources of complexity - you can have powerful rules for dispatching different behavior, but having separate semantics of arity, type, key inclusion all mixed together in a single spec leads to these sort of puzzles and edge cases

Where the person reading the code has to basically draw an inheritance chart or run a type inferencer in their heads to figure out what their code does, and the notion of "what the caller expects" may not always be clear

Destructuring reference is at https://clojure.org/reference/special_forms

@didibus I have a question for you about a user’s intuition w.r.t. destructuring keyword argument lists

As I understand Clojure is in general permissive about ignoring information it doesn’t care about. Why does the following form trigger an error? Shouldn’t it simply ignore the unrecognized key/value pair?

((fn [& {:keys [bar] :ignore-other-keys true}]
                        (list bar)) :bar 3)

why am I disallowed from putting extra keys in this map?

When something triggers an error, as a rule you must post the error :)

I wouldn't say Clojure is always permissive about ignoring extra things in syntax. If you are getting a bunch of error messages when you try to do that beginning with "Syntax error macroexpanding clojure.core/fn at ..." then that is from the spec syntax checking of some macro invocations, in this case fn, that was added in Clojure 1.9.0

Another example in a different area:

user=> (if true 5 7 8)
Syntax error compiling if at (REPL:1:1).
Too many arguments to if

why should fn care if there’s an entry in the map which it doesn’t care about?

Of course I can filter it out before I expand to the call to (fn …)

Probably because Clojure's destructuring facilities would never use that for anything, and it is likely a sign of a bug in the user's program that they are passing syntax that the Clojure compiler would ignore completely.

Post. The. Error

Clojure 1.10.1
user=>       ((fn [& {:keys [bar] :ignore-other-keys true}]
                        (list bar)) :bar 3)
Syntax error macroexpanding clojure.core/fn at (REPL:1:2).
:ignore-other-keys - failed: simple-symbol? at: [:fn-tail :arity-1 :params :var-params :var-form :map-destructure :map-binding 0 :local-symbol] spec: :clojure.core.specs.alpha/local-name
:ignore-other-keys - failed: vector? at: [:fn-tail :arity-1 :params :var-params :var-form :map-destructure :map-binding 0 :seq-destructure] spec: :clojure.core.specs.alpha/seq-binding-form
:ignore-other-keys - failed: map? at: [:fn-tail :arity-1 :params :var-params :var-form :map-destructure :map-binding 0 :map-destructure] spec: :clojure.core.specs.alpha/map-bindings
:ignore-other-keys - failed: map? at: [:fn-tail :arity-1 :params :var-params :var-form :map-destructure :map-binding 0 :map-destructure] spec: :clojure.core.specs.alpha/map-special-binding
:ignore-other-keys - failed: qualified-keyword? at: [:fn-tail :arity-1 :params :var-params :var-form :map-destructure :qualified-keys-or-syms 0] spec: :clojure.core.specs.alpha/ns-keys
true - failed: vector? at: [:fn-tail :arity-1 :params :var-params :var-form :map-destructure :qualified-keys-or-syms 1] spec: :clojure.core.specs.alpha/ns-keys
:ignore-other-keys - failed: #{:as :or :syms :keys :strs} at: [:fn-tail :arity-1 :params :var-params :var-form :map-destructure :special-binding 0] spec: :clojure.core.specs.alpha/map-bindings
{:keys [bar], :ignore-other-keys true} - failed: simple-symbol? at: [:fn-tail :arity-1 :params :var-params :var-form :local-symbol] spec: :clojure.core.specs.alpha/local-name
{:keys [bar], :ignore-other-keys true} - failed: vector? at: [:fn-tail :arity-1 :params :var-params :var-form :seq-destructure] spec: :clojure.core.specs.alpha/seq-binding-form
& - failed: vector? at: [:fn-tail :arity-n :params] spec: :clojure.core.specs.alpha/param-list

Sorry, that was long enough I probably should have put it in a thread comment

1. this error represents clojure.spec checking the valid grammar of params and destructuring forms

2. these specs get checked at macroexpansion time for certain macros (defn/fn/let)

3. it returns a mouthful because of ambiguous parsing

and it shows the potential ways something could be short/long/incorrect

yes, @ghadi, the question is why doesnt the syntax check allow me to put a key/value in the hash that it doesn’t care about, and that hurts nothing?

because it does hurt something

it's an invalid map destructuring pattern

{binding key} <- is valid map destructuring

:allow-other-keys` means nothing to fn, so it would be nice if the syntax checker simply ignored it

binding = symbol, and you've provided a keyword

ad 2. for every macro I think, it's all or nothing, can be disabled using a system property (or dynamic var?)

let's say the invalid syntax was accepted, who/what would act on that flag?

I believe it would be possible to write your own macro, similar to but not the same as fn or defn that are built in, that do different syntax checking than they do.

@ghade, if nothing would act on it then it would be harmless. The idea is that I have a macro which expands to fn, my macro understands :allow-other-keys and I’d like to simply pass along my map to fn without having to rebuild it.

of course I can code walk the expression and remove :allow-other-keys, it just seems unnecessary

In this case, the Clojure compiler / macro-expander is trying to be helpful to developers who misunderstand the correct kinds of destructuring forms that are allowed, and letting them know that they did something useless, which is likely a bug in their code.

I think you should code walk as it will trip up people's programs otherwise. In these very fundamental building blocks it's actually nice that clojure has some sanity checks (although the output can be overwhelming to some people)

Jim, you started this discussion with the sentence "As I understand Clojure is in general permissive about ignoring information it doesn’t care about." I tried to reply that this is not a general rule that applies to everything in Clojure. Yes, there are some things it will ignore, but there are others where it checks and gives errors/warnings.

It applies to data in/out but not so much to fundamental syntax like defn or ns

I'm sure there are several open JIRAs with requests to tighten up checking even more than it is checked now. The kinds of checks done aren't quite static, but they do not change rapidly across Clojure versions, either. Clojure 1.9 introduced quite a bit of additional syntax checks like this.

I'm glad that Clojure throws a compile-time error for that particular case - 99% of the time I destructure using :keys, and the other 1% I'll often accidentally reverse the order of the bindings just because having the keyword on the left is more familiar:

(let [{:a a} {:a 1}]
  (inc a))

You commented that you would like dsfn to be able to handle something like the following.

(dsfn 
 ([& {:keys [^Int bar]}]
  (list 'int 'bar bar))
 ([& {:keys [^Int baz]}]
  (list 'int 'baz baz))
 ([& {:keys [^String bar]}]
  (list 'string 'bar bar)))

(dsfn 
 ([& {:keys [^Int bar] :allow-other-keys true}]
  (list 'int 'bar bar))
 ([& {:keys [^Int baz] :allow-other-keys false}]
  (list 'int 'baz baz))
 ([& {:keys [^String bar]}]
  (list 'string 'bar bar)))

yes, I’ll just filter it out before passing the map on to the clojure primitive.

why shouldn’t the second apply if you typed (foo :bar "hello") This seems to be implied by your suggesting that the first should apply on (foo :other 10) , what’s the difference between the two cases?

Imagine that the two ladder cases were not there:

(dsfn foo
 ([& {:keys [^Int bar]}]
  (list 'int 'bar bar)))

The way I’ve implemented this is I’ve added an additional keyword :allow-other-keys which can be used as follows

(dsfn foo
 ([& {:keys [^Int bar] :allow-other-keys true}]
  (list 'int 'bar bar)))

:allow-other-keys defaults to false, so the clause won’t match if there are keys at the callsite not mentioned in :keys [ … ].

I want to send an http request to my localhost:3000 server from a custom ip address. And apparently this can be done with a custom dns resolver.

(client/get "<http://localhost:3000/api/v1/subscribe/ETH-USD>" {:dns-resolver (doto (InMemoryDnsResolver.)
                                                                              (.add "localhost" (into-array[(InetAddress/getByAddress (byte-array [127 0 0 1]))])))})  

A custom DNS resolver is supposed to resolve localhost as something else. You're trying to resolve it as 127.0.0.1, which should already be done by your system. I don't think you can properly simulate multiple incoming IP addresses this way, although I'm not an expert here.