Clojurians Log v2

Somebody did it already https://clojuredocs.org/clojure.core/update-in#example-54298289e4b09282a148f202

A question related to using Clojure’s multimethod at scale. Multimethod inlcudes a caching mechanism. In my code I called a multimethod with lots of different values (around 20 millions) and it caused a memory leak. Is there a way to disable multimethod caching or to limit the cache size? Is it advised not to use multi methods when there are too many different calls?

I'm not sure if it's possible with multimethods but I think https://github.com/camsaul/methodical supports passing a custom cache

You can call (.reset multi-fn) on each call. But it's quite a bit strange that, seemingly, MultiFn caches all values that end up in calling the default method.

Happens here if you're curious https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/MultiFn.java#L161

Just asked: https://ask.clojure.org/index.php/10532/memory-leak-using-the-default-method-of-a-multimethod

And of course I'm wrong about using reset because it also reset the hierarchy. So a workaround would be to call defmethod again - it resets the cache as well.

My use case is different. The dispatch value is always the same. And the memory leak is not as easy to reproduce as in the case that your reported

The dispatch value is the same in my case as well - it's :default.

in your case the dispatch value is the by array

I guess there are two perspectives. :) > The dispatch value is always the same "Same" as in identical? or equal??

Although, judging by the Java code, you would still have to be hitting the :default branch for the issue to surface.

Another guess that I can hazard without having a minimal reproducible example is that your objects can be equal? without having their hashes be equal. It would mean wrong hash implementation, and can cause all sorts of problems in general, not just memory leaks when using defmulti.

Actually, my case was exactly the same as yours but not on purpose. It took me lots of time to understand it. Here is my buggy code

(defmulti foo :a)
(defmethod foo :default [_ _] nil)

(foo {} (Math/random))

My intent is to dispatch on the valu associated to :a in the first arg.

(defmulti foo (fn [x y]
                (:a x)))

But what I wrote was equivalent to this:

(defmulti foo (fn [x y]
                (:a x y)))

As a consequence, each time I called foo with a different second arg, I increased the cache of the multi method

It drove me crazy

Because my code behaved as I expected in term of functionallity

Anyway, I think that the cache size of the multi method should have a limit

I want to import some dynamic var from another namespace but I also want to allow users to define a var with the same name, without breaking (it's on them to rename their own vars in order to be able to access the one from the prelude). E.g. my prelude is

(require '[babashka.tasks :refer [*task*]])

(def *task* {})
IllegalStateException

(intern *ns* '*task* ...)

I feel it could be useful if clojure had something like a proxy-var, which just proxied everything to another var. This would also solve the potemkin/import-vars stuff.

Or you could just not ever do that

Somehow I’ve managed to avoid it in 10 years of writing Clojure

potemkin/import-vars was just an additional benefit that could be solved at the same time, but not the main issue (FWIW: I have never needed potemkin/import-vars myself, not a big fan, but the problem it addresses could be solved better if clojure supported "forwarding" to other vars). Let me try to explain my use case. This is a config-like program, where I want to be able to expose *task* without breaking existing configs. It is not a typical every day Clojure program, but more a framework/DSL-like setup. Not common.

There is a logical error here 🙂 And I say this with the utmost respect, hoping you don't mind it. https://en.wikipedia.org/wiki/Denying_the_antecedent

I guess I could hack around it by making the dynvar part of clojure.core, but I'm not a fan of that either. https://github.com/clojure/clojure/blob/b1b88dd25373a86e41310a525a21b497799dbbf2/src/jvm/clojure/lang/Namespace.java#L87

why do you need to refer *task* instead of b.t/*task* ?

Maybe a better way is to make everything opt-in (or namespaced) but that will create some more boilerplate.

maybe it’s fair to require user programs to not define *task* for this system? what’s the problem you are trying to solve?

> maybe it’s fair to require user programs to not define `*task*` for this system? that is a fair restriction if you can control the future

there might already configs around that are using *task* for something else today

if they do define it it's on them to use an alias instead of referring

I realised I don’t know enough about the context where this is defined to give advice 😄

you can also :rename {*task* *bs-task*}

What I've done so far is this:

(when-not (resolve 'clojure)
  ;; we don't use refer so users can override this
  (intern *ns* 'clojure babashka.tasks/clojure))

(when-not (resolve 'shell)
  (intern *ns* 'shell babashka.tasks/shell))

not a fan ¯\(ツ)/¯

is a prefix that bad?

yes, in the sense that that prefix might also be already taken by the user

what problem are you trying to solve?

e.g. I had tasks as the default prefix, but I reverted this because a user already had taken this :)

@vlaaad I am making a task runner / Makefile like system where people can define tasks using a DSL

oh, like gulp and grunt in js? 😄

i think requiring users to change existing code is perfectly reasonable, given the relatively age of babashka's task runner feature

@nbtheduke The problem is more general: even when stable, it might introduce more of these things in the future.

I guess the problem is similar to [clojure.test :refer :all] and clojure.test introducing new things in the future which breaks certain programs when upgrading clojure

and if clojure doesn't see this breakage as problematic, maybe I shouldn't either?

@vlaaad to summarize: the problem I'm trying to solve is: avoid breakage like with the above example

but maybe my programs shouldn't try to be "don't break"-holier than clojure

i realize one of the niceties of babashka is that a lot of stuff is imported by default, but could you make this one explicit?

nothing is imported by default, except in the user namespace it pre-defines a couple of aliases to be used on the command line. e.g. (json/generate-string ...)

shouldn’t this also be done with default aliases?

@vlaaad what do you mean by this?

instead of referring *task* you can add default alias to babashka.tasks

@vlaaad even that was breaking some user's program where the user had already chosen the alias tasks

we could just make everything explicit via a required [babashka.tasks :refer [shell clojure *task*]] instead.

or :refer :all at the risk of the user

I guess we could keep supporting normal fns like previous but additional stuff must be explicitly imported. Perhaps task or current-task could also be a normal fn which derefs *task* which is not uncommon.

Perhaps of interest, https://github.com/apenwarr/redo

Did you also comment this in the Mach repo?

Then I've probably seen it

In clojure, fn allows me to define a multi-funciton, i.e. a function with several possible lambda lists. at call-time, the most appropriate function is selected by matching the arguments to the appropriate lambda list. Is the mechanism of selection documented somewhere? For example, certain sequences of lambda lists are not allowed. Case in point [a b] and [a [b c]] are not allowed simultaneously. What else is allowed and not allowed?

(defn foo 
  ([a [b c]] a)
  ([a b] (list a b)))

Syntax error compiling fn* at (clojure-rte:localhost:54179(clj)*:133:23).
Can't have 2 overloads with same arity

The logic is pretty simple basic: it's only based on arg count, regardless of destructuring.

And there is a rule around varargs.

@jimka.issy [a b] and [a [b c]] are both 2 args fns.

really? only argument count?

yes. arg count only.

@potetm, yes I understand why this case fails. But is this the only such failing case? that’s my question. according to @borkdude that’s the only selection criteria, the arity

you should macroexpand the function to see what is happening:

(macroexpand '(fn ([a [b c]] a)
                  ([a b] (list a b))))

->

(fn* ([a p__140] (clojure.core/let [[b c] p__140] a)) ([a b] (list a b)))

@jimka.issy

user=> (macroexpand '(fn [[a b]]))
(fn* ([p__138] (clojure.core/let [[a b] p__138])))

term of art here is "function with multiple arities"

lambda lists

fn* doesn’t seem to be documented in https://clojuredocs.org/

fn* is a special form in the compiler

it's the familiar fn minus destructuring

http://clojuredocs.org is very useful, but non-authoritative

@ghadi, there seems to be some case about varargs, is the “term of art” really multiple arities even in the case of varargs?

varargs is a type of arity

you can have fixed arities + a vararg arity (which must be > than the length of the longest fixed arity)

[a b] [a b c] [a b c & more]

for the varargs arity, under the hood there is a mechanism that rolls up extra args and presents it to the arity as a seq bound to "more"

ahhh, now we are getting some rules. If http://clojuredocs.org is not authoritative, where is the authority?

https://clojure.org/reference/special_forms#fn

The `IFn` interface defines an `invoke()` function that is overloaded with arity ranging from 0-20. A single fn object can implement one or more invoke methods, and thus be overloaded on arity. One and only one overload can itself be variadic, by specifying the ampersand followed by a single rest-param. Such a variadic entry point, when called with arguments that exceed the positional params, collects them in a seq which is bound to, or destructured by, the rest param. If the supplied args do not exceed the positional params, the rest param will be `nil`. [taken from http://clojure.org]

destructuring is orthogonal to any of the above

purely macro sugar to tear apart the arguments

it is a copy-paste from http://clojure.org, isn’t it?

that destructuring is orthogonal makes since, yes, just two feature that work well together

BTW what is the difference between the following two?

(defn foo 
  ([a b c] a)
  ([^Boolean a b] (list a b)))

(defn foo 
  ([a b c] a)
  ([a b] (list a b)))

In that example: in practice nothing, since the type hint is only relevant to Java interop here

Doesn't this represent a potential DOS vulnerability that a lot of people might not be aware of? Presumably the cache entries are not weak-interned, so you could use this to consume all available memory by providing many unique inputs that may happen to be used as dispatch vals? I'm sure many large unique inputs will cause pollution of caches all over the place not just multi-methods in many apps but I would still sleep easier if I knew :default matches were not cached.

e.g. ^String a (.length a)

does the compile compile a special branch for the case that a is a Boolean? and another for when it is not a Boolean?

no, the type hint is just leveraged when doing interop and in other cases, just not used

there are other hints which can prevent boxed math

someone mentioned above that lambda-list is not a term used in the clojure community. what is the correct word for the vector which specifies the parameters of a function and their semantics wrt position, destructuring, and optionality?

@jimka.issy Although not entirely accurate, a Clojure decompiler could be educating to see how the Clojure compiler turns s-expressions into bytecode in these cases https://github.com/clojure-goes-fast/clj-java-decompiler

arglists is a term commonly used and this is also the name in the metadata on the var

and what is the list of arguments called, if arglist has a different meaning?

((fn [a b c] …) 1 2 3) for me [a b c] is the lambda list and (1 2 3) is the arglist

arguments?

arglist != list of arguments ???? curious

user=> (defn foo [a b])
#'user/foo
user=> (meta #'foo)
{:arglists ([a b]), :line 1, ...

A few years back I used one lisp which called (a b c) the formal parameter list and (1 2 3) the arg list

don't get caught up on names, just accept them, they won't ever change :)

Not sure how easy it'd be to exploit with most apps but I bet a bunch have multi-methods that take user input as a dispatch val (e.g string "type" json field) assuming that hitting :default comes with no risk.

Arglist

Argvec

It’s a vector, but often called the arglist

what are the type annotations called in a definition like this: (defn [^Boolean a b] …) are they called type hints, or type annotations or what?

they are not annotations, annotations are a bytecode feature

(at least I'd like to keep that term specific)

type hint is the term I've seen

they are type hints

the only type hints that affect the actual compilation of the function signature are ^long and ^double

otherwise they are always Object

Oh interesting. Is there a technical reason for this?

For more details and official documentation of this behavior: https://clojure.org/reference/java_interop#primitives

see also the definition of IFn - the specializations are only on long and double args, otherwise the sig is always Object https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/IFn.java

Sorry, I don't know what the Mach repo is

Aha that clarifies

Why is there an assymetry between assoc! and conj!? (conj!) returns a transient vector (assoc!) does not return a transient map (conj! .. 1 2 3) is not varargs (assoc! (transient {}) :a 1 :b 2) is varargs

I am surprised if assoc! does not return a transient map

how could it work if not?

assoc! matches assoc

I think there is a ticket to make conj! match conj

he means the 0-arities of conj/conj! are defined

while of assoc/assoc! aren't

https://ask.clojure.org/index.php/2370/make-conj-assoc-dissoc-transient-versions-handle-similarly?show=2370#q2370 is one but I'm pretty sure there may be other tickets

I know there are some with other proposals that would interact with this one

ah, sorry - I see you meant specifically those, I think that's covered in this ticket

I think the answer to the original question is: there is no good reason

Calling conj! with no arguments is meaningful for its usage as a reducing function

transduce calls that arity when no init provided

there is no such kv reducing context that calls the no-arity, so i suspect that there was no pressure in giving assoc! the same treatment

that ask.clojure issue has 0 votes, feel free to vote if you'd like to see it move up the priority list!

voted thanks.

Jira has now force migrated people to their “new issue view” which inexplicably does not contain vote information. All the jira jiras about it are closed and seem to suggest it is available but perhaps only on “next gen” projects, not “classic” projects. To migrate, you have to create a new project, presumably with a different project id, which would break all existing links. Cool.

"All the jira jiras ..." 🙂

I’m just teeing you up to ask how I voted for that jira jira

"How did you vote for that jira jira, Alex?"

Because they didn’t use the new ui

Because it sucks

Sounds like a case of "normally we would eat our own dog food here, but in this case our own dog food wasn't good enough."

Eating your own dog food and holding your nose or something

I mean like, I get why software is hard. I really get it. But this new ui has been like 2 years in the making and it just moves some fields to the right and got rid of actually useful things (afaict). I dunno.

Clubhouse (the Clojure project thing, not the audio social media thing) is pretty cool

I guess it’s a good job we now have http://ask.clojure.org as a proxy for Jira so we can still vote on things by proxy :rolling_on_the_floor_laughing:

FWIW, if I’m logged into Jira, I can still see the voting button for Clojure issues:

(That’s CLJ-2556)

oh, I might have just missed that completely

oh, I was on one with no votes there's no number so it's easy to miss