Clojurians Log v2

I always forget where the documentation for the parameter list of functions, (fn, defn, defn-, etc) is. Its not in the https://clojuredocs.org/clojure.core/fn of fn 😞

it is in documentation

(fn name? [params*] exprs*) (fn name? ([params*] exprs*) +)

This doesn't explain keyword arguments.

neither https://clojure.org/guides/learn/functions.

keyword arguments?

yes some functions can be called like (f a :x 100 :y 200)

there's a way to define these giving default values of x and y

it is a sequence of args

https://clojure.org/guides/destructuring

the descructuring rules for let are different from those of fn . Is this really not explained anywhere?

they are the same

no, I cannot define a function like this, for example. (fn [a & others :as all] ...)

nor can I define a function like this (fn [[[x]]] x)

right?

they are very similar but not the same.

you can

wow. I didn't know that one. cool.

but :as doesn't work as far as I know

(fn [& [a & other :as all]])

but not (fn [a & other :as all] (list a other all))

clojure-rte.rte-core> ((fn [a & other :as all] (list a other all)) 3)
Syntax error macroexpanding clojure.core/fn at (clojure-rte:localhost:59787(clj)*:17647:24).
(:as all) - failed: Extra input at: [:fn-tail :arity-1 :params] spec: :clojure.core.specs.alpha/param-list
a - failed: vector? at: [:fn-tail :arity-n :params] spec: :clojure.core.specs.alpha/param-list
clojure-rte.rte-core>  

because this form is violating a spec for arglist

exactly, the format for arglist is different than the format for destructuring. But I don't find an explanation of this difference in the documentation anywhere

note that [ and ] are not a part of arglist, it is a form that is inside of brackets

BTW, thanks for the link. I found this useful example>

exactly, and what is inside the brackets is different than destructuring rules.

(fn name? [params*] exprs*)

no

Let me try again. I can destructure with [a], [a & others] , and [a & others :as all]

but I cannot define functions those three ways.

I see what you're saying though.

you're claiming that the arglist is not destructuring, but each variable in the argument list can be replaced by a destructuring form.

and you cannot use :as to get the entire value of the argument list, only the entire value of any element of the argument list.

right

I suspect this is really a bug in clojure core which has simply become a feature. Imagine writing a function which allows :as in destructuring forms, but explicitly disallows it in argument lists. My vague suspicion is that this is really just an overzealous syntax checker on fn.

question about destructuring:

(defmethod rte-match :sequential
  [pattern items & {:keys [promise-disjoint]}]
  (rte-match (rte-compile pattern) items))

I think this works.

(defmethod rte-match :sequential
  [pattern items & {:keys [promise-disjoint] :as keys}]
  (apply rte-match (rte-compile pattern) items keys))

(defmethod rte-match :sequential
  [pattern items & {:keys [promise-disjoint] :as opts}]
  (apply (partial rte-match (rte-compile pattern) items) opts))

I didn't use partial , is it necessary?

no

bold assumption

I added these examples to https://clojuredocs.org/clojure.core/fn

i want to test if the length of a lazy-seq is larger than some value, without exhausting the seq by using count

what’s the proper approach ?

(take n) and (count) ?

(seq (drop n l-coll))

maybe (bounded-count n coll)

I'm looking to name a parameter to one of my functions, and the semantics are potentially difficult for the caller to understand, but could nevertheless have a big impact on the performance of the function. The function has two modes, 1) it can internally lazily-compile the user's data structure into a form which is very fast. Unfortunately the compilation is slow. 2) it can use a semantically equivalent interpreter which is slow. #2 is better if the user intents to only do it once. But if the user intends to call the function multiple times with different arguments, it may be better to use option #1. What could I call the parameter to make this more apparent to the caller?

I was thinking of calling it :hot-spot , implying that the more you run it the faster it will become.

otherwise what is the motivation for intentionally making a function's argument list processor, simply a destructuror?

I agree, it is a very bold assumption on my part

> otherwise what is the motivation for intentionally making a function’s argument list processor, simply a destructuror? I guess to simplify reading and compiling processes. As an example: (fn ([x]) ([x :as y]) …) not valid because those forms of the same arity but to throw there compiler need to know about destructuring

compare with [param*] where each param can be a simple symbol or destructuring form

split into two functions with proper documentation for each “mode”

Another question about Destructuring and keyword arguments. My experimentation shows that Destructuring works a particular way, which I like, but I don't know whether this is accidental or by design. If a function is defined as follows:

(defn foo [& {:keys [x y]
              :or {x 12 y 13}}]
  (list x y ))

(apply foo some-map}

(apply foo :x 300 some-map)

some-map can not be a map in that case

(apply foo :x 300 {:x 200})
Execution error (IllegalArgumentException) at user/foo (REPL:8).
No value supplied for key: [:x 200]

Here's the function. I don't see how splitting it into two functions solves the problem. Since the function is pretty large, the common code would be factored anyway into a common function, and there'd still need to be a hot-spot argument. Right?

(defmethod rte-match :Dfa
  [dfa items & {:keys [
                       ;; if the caller promises that never are two transitions in the Dfa
                       ;;   labeled with intersecting types, the use
                       ;;   :promise-disjoint true, in this case rte-match
                       ;;   can be more efficient and can assume that the
                       ;;   clauses can be tested in any order.  If the transitions
                       ;;   are not guaranteed disjoint, then rte-match must
                       ;;   build new type designators each one containing an and-not
                       ;;   of the previously seen types. 
                       promise-disjoint
                       ;; hot-spot = true -- lazily compile the type checks into Bdds
                       ;;    which is slow-going but becomes faster the more often you
                       ;;    re-use the same pattern, either because of loops in the
                       ;;    Dfa, or when the same Dfa is used to match different
                       ;;    input sequences.
                       ;;    
                       ;; hot-spot = false -- always interpret the type checks rather
                       ;;    than converting them to Bdd.  This option is probably faster
                       ;;    if there are few loops in the Dfa, or if you only use the
                       ;;    pattern once to check a single input sequence.
                       hot-spot
                       ]}]
  (let [state-vec (:states dfa)
        sink-states (set (dfa/find-sink-states dfa))]
    (if (empty? sink-states)
      (rte-match (dfa/extend-with-sink-state dfa) items :promise-disjoint promise-disjoint)
      (let [sink-state-id (:index (first sink-states))]
        ;; There are two possible transition functions
        ;;   slow-transition-function -- this is faster if the caller intends to match
        ;;       the pattern only once.   The pattern is matched by an interpreter,
        ;;       and it is possible that the same type predicate will be tested multiple
        ;;       times on the same candidate objects.  If one of the type predicates
        ;;       is (satisfies slow-predicate) then that slow-predicate may be called
        ;;       multiple times, resulting in poor performance, especially if the
        ;;       pattern is used to test multiple sequences.
        ;;   fast-transition-function -- this is faster if the caller intends to match
        ;;       the pattern multiple times with different input sequences.  The
        ;;       pattern is *compiled* into a form where type-designators are converted
        ;;       to Bdds thus each type check guarantees to never check the same
        ;;       type predicate multiple times, and sometimes not at all.
        (letfn [(slow-transition-function [transitions]
                  (fn [candidate sink-state-id]
                    (some (fn [[type next-state-index]]
                            (if (gns/typep candidate type)
                              next-state-index
                              false))
                          transitions)))
                (fast-transition-function [transitions]
                  (dfa/optimized-transition-function transitions promise-disjoint sink-state-id))
                (transition-function [transitions]
                  (if hot-spot
                    (fast-transition-function transitions)
                    (slow-transition-function transitions)))
                (consume [state-index item]
                  (let [state-obj (state-vec state-index)]
                    (cl/cl-cond
                     ((member state-obj sink-states)
                      (reduced false))
                     (((transition-function (:transitions state-obj)) item sink-state-id))
                     (:else (reduced false)))))]
          (let [final-state (reduce consume 0 items)]
            ;; final-state may be integer desgnating the state which was
            ;;  reached on iterating successfully through the input
            ;;  sequence, items.  Or final-state may false, if the
            ;;  iteration finished without iterating through the entire
            ;;  sequence, either because we found ourselves in a
            ;;  sink-state, or we encountered a item for which no transition
            ;;  was possible.
            (cond
              (= false final-state) false
              (:accepting (state-vec final-state)) ((:exit-map dfa) final-state)
              :else false)))))))

i’m not suggesting duplication of code but making two “public/interface” function for each corresponding mode (rte-match-simple, rte-match-full as an example) each should prepare and set as a default value for that argument

it doesn't error if there are an even number of keys in the some-map, but it's not doing what you want

(do (defn foo [& {:keys [x y]
                  :or {x 12 y 13} :as m}]
      (println x y m))
    (apply foo :x 1 {:x 2 :y 3}))
=> 1 13 {:x 1, [:x 2] [:y 3]}

> map in clojure is a sequence of key-value pairs

the fact that map with even number of keys is not throwing exception doesn’t mean the logic of function is correct)

then we have the same question, right? that argument needs a name.

no, you can change the names of functions to describe what they are doing instead

but the functions call a common function, and pass an argument. that argument needs a name. Or did I misunderstand?

details of implementation doesn’t need to have “absolutely meaningful” names as long as they hidden from the caller and properly documented for maintainance reasons

ahh, I'm glad I asked. This means the note that I added to fn documentation some time ago is actually wrong.

(defmethod foo :a [a b & {:keys [x y z] :as all-keys}]
  (apply foo (f a) (f b) all-keys))

So what's the correct way to call a function, re-passing the keys which were received, or overriding some of them?

bounded-count fits here. but the function itself looks twisted to reason

do not use “keyword arguments” maybe?

(defmethod foo :a [a b {:keys [x y z] :as all-keys}]
  (foo (f a) (f b) all-keys))

I'm tempted to write my own version of apply which has more reasonable semantics with optional arguments. couldn't be so hard actually. It's just lisp.

hmmm it seems clojure doesn't allow me to list the same key multiple times in a call to foo 😞

I'd expect a call like (foo :x 100 :y 200 :x 300) cause x to be bound to 100 within foo, instead it throws an exception.

Execution error (IllegalArgumentException) at clojure-rte.rte-core/eval8637 (form-init3719502167719786565.clj:93).
Wrong number of args passed to keyword: :x

1. why 100 and not 300? 2. it should working normally

(defn foo [& {:keys [x y]
              :or {x 12 y 13}}]
  (list x y ))

I'd expect leading values to override later values.

that way when dealing with argument lists programmatically, you can simply prepend overriding values without having to copy the entire list.

That means if you have a list of the optional args like my-optional-args, I can call the function as (apply foo my-optional-args) but you would also be able to override some values like this (apply foo :x 200 my-optional-args)

I would rather prepare my arguments before the call

with my approach no preparation is necessary. The approach is used very often in Common Lisp, just another surprise that something I thought of as fundamental is different in clojure. Not wrong, I suppose, just different.

What do you mean by that? the implementation of bounded-count is twisted?

I think wrong is thinking that Common List is “more fundamental” than clojure ) you could also replace with any other pair of programming languages in the sentence above )

(defn count< [n coll]
  (< n (bounded-count n coll)))

I think the mismatch here has to do with the fundamental reliance in Common Lisp on the concrete cons cell - that is why prepending arguments seems more "natural"

probably right. if the cons cell is primary, than adding to the beginning is efficient, and becomes commonplace.

whereas in Clojure you might use vectors instead to store arguments, and conj to the end

(also note that structures in Clojure are persistent and immutable, so the distinction of "copying the entire list" doesn't really apply)

I mean the meaning of the return value of bounded-count.

ah, I think of it as a sort of (min n actual-count)

The idiomatic way to override a value would be (apply foo (assoc some-map :x 300))

can someone explain what destructure does? Its https://clojuredocs.org/clojure.core/destructure is not so illuminating. neither does there seem to be a useful comment in the code where it's defined.

Destructuring is the process of extracting values from collections. Say you have a vector of two elements, [1 2], you can use a let to destructure that in the first and second value:

(let [[a b] my-vector] ...)

(let [{:keys [a b]} my-map] ...)

I’m pretty sure there is a better page about it

https://clojure.org/guides/destructuring https://gist.github.com/john2x/e1dca953548bfdfb9844

Ah nevermind, I guess you are specifically asking about the function

indeed

I didn't see a reference to restructure in that link. Did I miss it?

I misunderstood what you asked

I was hoping it might be a programmatic interface to destructuring. In the coming weeks, I think I'm going to have to write a lambda-list parser which mimics the semantics of function lambda-lists. I was hoping the destructure function might be of help.

can’t say much why there is no documentation for destructure in clojure.core this is more like internal helper that accidentally become public (this is my opinion, don’t have any proof that this is true)

yes I think my mac spell corrected destructure -> destructuring in my OP

it is programmatic interface but in the form of helper for macro writers

https://clojure.org/guides/destructuring#_macros at the very end it's talking about the destructure fn: > However, in rare cases you might want to instead resolve the destructuring yourself in a macro. In this case, use the (undocumented) clojure.core/destructure function[...]

clojure.core/destructure is a function, not macro

yes, right

looks like destructure produces the first operand of let. But it also gives me a sequence whose even elements are the names of the variables which the destructing will bind. That could be very useful.

you can think about it as a function to return bindings (destructure '[[x y] z]) => [gen_0 z, x (nth gen_0 0 nil), y (nth gen_0 1 nil)] ^ this is simplified form

so in one sentence it transform destructure syntax into clojure syntax

Is there a idiomatic/preferred binary tree in Clojure?

@watchtheblur sorted-map is a persistent red-black tree

Thanks @ghadi. I was looking for something that I could traverse/search children with. I realize I can implement a binary tree using a list and its indices, but I wondered if there was a better way (maybe objects)?

(use a vector and indices, lists are not for looking up by position)

@watchtheblur on a sorted collection (eg sorted-map) the subseqfunction function can be used for efficient splitting, sadly it doesn't return a pair of new sorted collections though

@noisesmith seems like the censuses is to use a map instead of a vector for a binary tree? So if I wanted to grab the 2 children of a node, I would need to give indicies to the map to get the child, or is the map already structured as a tree (i.e. nested maps)?

there's also sorted-set, but neither directly exposes its tree structure https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/PersistentTreeSet.java

actually this might be what you want https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/PersistentTreeMap.java#L223

it requires interop, but at least it's public

I can't imagine that's what OP actually wants, but that raises the question - what do you want? :) are you looking for a "by the book here's how to implement a binary tree" or are you looking to store some data in a tree for a reason, and if so there are probably better default answers

We have an app running in wildfly at work and it started to get unresponsive with users reporting that they are getting "BadGateway" issues. Looking at the logs I noticed quite a few errors. Main ones that stick out were "OutOfMemoryError: Metaspace", "NoClassDefFound", "I/O Exception Broken Pipe (Write failed)" and "Postgres connection attempt failed". Looking at NewRelic, I noticed that the UnloadedClass count started to increase along with a huge spike in GC (ConcurrentMarkSweep), where at the highest point, is where the application went down and users started reporting the errors. Operations restarted the servers and the app was working again. But this had happened last week as well. Currently I am researching but this is honestly a little over my head/experience level. I am thinking there is perhaps a memory leak going on but I am not so sure. Does anyone have any idea what could be causing this or could point me in the right direction?

Oh, another thing, which could be coincidence, but the servers both went down on Monday/afternoonish

sounds like you are calling eval a lot

Hmm, what makes you say that?

I don't have experience doing this myself, but there are several attach-them-to-the-running-JVM-process memory profiling tools that can aid in looking for what classes of objects are using the most memory.

That could be it actually, since some new code that went out about two weeks, uses the eval function

eval compiles clojure code to jvm classes, classes are allocated in metaspace

the correct amount to be calling eval is usually 0

If you really need to have some form of eval in your code, there’s always sci

Actually, the code does not use eval. I thought it did but it does not after second look

https://github.com/borkdude/sci

Yea I was trying to hook up VisualVM but was getting issues with it freezing

hprof is much lighter weight, and visualvm can load the dump file it creates

@alexmiller I'm looking to practice leetcode-type questions with clojure. Since leetcode doesn't support clojure, I wanted a simple binary tree data structure to run algorithms on. I wanted to check with the community on a idiomatic approach

@mario.cordova.862 are you trying to limit the metaspace? That seems to be a common mistake for folks coming from earlier JDKs where limiting the non-heap part of memory sometimes made sense (it's been a while so I've forgotten exactly what setting I'm thinking of on earlier JDKs)

(PermGen!)

(it used to be a good idea to limit PermGen -- it is not a good idea to limit Metaspace, as I recall...)

yeah, in my experience setting limits on metaspace is cargo-cult and/or mindless copying of old config from ancient jvms

it's not a useful setting

I am just trying to figure out why this issue occurred. If its a problem with new code changes or something else.

something is allocating memory off heap, that's done by some libraries, it's done when you create new classes

in general, the best way to work with Clojure is to lean on the built-in data structures, so I'd probably make each node a map with slots for :left and :right (or whatever). Or a vector of 2 items. not sure which way works out better.

the biggest culprits I know of are using eval to make new functions at runtime, and badly written interop with platform level stuff (eg. a memory leak in initialization of some adaptor to C code)

I guess "at runtime" is when all functions are created, but I mean in a deployed app after startup

I think it might be the eval but it's odd that it just started to happen. Going to check if there is a limit on metaspace

IIRC that's the only way to get that specific error

The metaspace one?

right, without the metaspace limit you get a different flavor of "vm out of memory"

and you'd need to create a lot of functions before that came up (I guess that's one benefit of a metaspace limit, catching metaspace leaks early...)

and now I wonder if there's any such thing as garbage collection of Classes (probably not...)

there is, but it is relatively new (I forget how new, or maybe if it is just a jep that hasn't landed yet)

I guess it isn't that new, java 8 or so

so something must be not only creating new functions via eval but also holding onto them I guess

I am seeing this in the standalone.conf -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=2G

This is setting a maxsize on metaspace, which should not have a limit?

I've never seen a drawback to just leaving it unlimited, but you probably also want to find out what's filling your metaspace and not getting cleared up

visualvm / yourkit / hprof can show you what's filling metaspace

Going to investigate it then, thanks for the help guys! :thumbsup:

luckily anonymous functions / partial / reify / proxy etc. create class names that will lead you to the code creating them

Partial's is just something like clojure.core$partial$fn

So eval is okay to use when "defining" something but during run-time it is an issue?

It is important to understand, and an important design feature, that clojure only creates classes during compilation, running code doesn't create new classes unless you call the compiler (calling eval)

Got it, thanks @alexmiller!

So using any of the features @noisesmith mentioned will only generate a single class (and partial doesn't even do that, because partial doesn't really belong in that set) at compile time, and at runtime you will just get multiple instances of that class

The only way to generate an unbounded number of classes is by calling eval over and over

right, I shouldn't have included partial, but was thinking that instead of creating fns via eval, they might be creating reify or proxy

Reify and proxy will only generate new classes when compiled

right, inside eval

it would be weird but it's possible

So if I do (eval '(+ 1 1)) this creates a class?

I would not try to define cases when eval is ok to use and when it isn't. I would say because clojure provides it neatly wrapped in a single function you don't realize the complexity of it, and whatever you are doing almost certainly doesn't require it

To execute byte code on the jvm it has to live in a class file

Live in a class, the class may just exist in memory

Eval has some special cases for very trivial expressions where it just acts as an interpreter and doesn't generate byte code, but in general for anything but the most trivial expressions, you will generate at least one class when calling it

Im not so much trying to define cases when is it okay to use but I am trying to reason about some code. I am trying to verify that a certain portion of the code is the culprit. Since this part of the code will be calling eval during runtime where as the other portion is using eval but not during runtime

Remove the calls to eval

When there is a foot gun in your code you remove it or debate where it is pointing, and for #beginners the response has to be removing it. Debating about where it is pointed belongs somewhere else

sounds like https://clojure.atlassian.net/browse/CLJ-1152

I dont feel comfortable just yet removing it, since it is legacy code and its essentially the heart of the application so a little risky

Hah, I know that guy

If the application is written such that it can be shut down in an orderly fashion, and/or has multiple instances running in parallel that can take over for each other dynamically, e.g. like many highly available web services, you could potentially consider the workaround of monitoring the PermGen space usage, and when it reaches a threshold, force that instance to restart. That would be a workaround, mind you, with its own introduced operational difficulties.

hello, I'm writing a little tool to automate some of my daily task on Clojure and Conch (https://github.com/Raynes/conch), however I have a function that I can't make pass a test. Here the code for both:

(defn exec-cmd [expression]
  (let [bin (:bin expression)
        args (:args expression)
        config (:config expression)]
    (sh/with-programs [echo mvn docker-compose]
                      (case bin
                        "echo" (echo args config)
                        "mvn" (mvn args config)
                        "docker-compose" (docker-compose args config)))))

(deftest test-echo-cmd-exec
  (testing "Execution of echo command OK"
    (is (= (exec-cmd {:bin "echo", :args '("hello" "world!"), :config {:seq true}})
           '("hello world!")))))

by the way, the test fails with:

FAIL in (test-echo-cmd-exec) (core_test.clj:7)
Execution of echo command OK
expected: (= (exec-cmd {:bin "echo", :args (quote ("hello" "world!")), :config {:seq true}}) (quote ("hello world!")))
  actual: (not (= ("") ("hello world!")))

Ran 1 tests containing 1 assertions.
1 failures, 0 errors.
Tests failed.

@sdmoralesma ymmv but I find conch to be over engineered and I prefer to use ProcessBuilder / Process directly:

user=> (let [b (ProcessBuilder. ["echo" "hello"])
             p (.start b)
             res (.getInputStream p)]
         (.waitFor p)
         (slurp res))
"hello\n"

thank you for the answer, I'll give it a try, also I noticed that is possible to pass a folder for execution which helps in my case. 👍

yeah, the API is pretty nice actually, especially if you already know the posix exec api

except for the part where it doesn't let you know the PID :/

using the inputStream and outputStream together you can even run an interactive program, which is much harder to do in most languages

with the PID at least is not an issue for me... yet. About the streams, that might improve the workflow... after all the idea is to simplify my day, thanks for the hint

In Java 9 they added a .pid() method