Clojurians Log v2

https://gist.github.com/hiredman/a68a2add9751eb8de3d2776363219e13 is an attempt to golf a swimm style cluster membership / failure detection core.async thing that uses wrapped timeouts

cool, thanks! you always have the best core.async gists

cooool

the CML join function could be a protocol (that allowed you extend it to Process or CompletableFuture), and it would return you a channel.

returns you a readport, technically

CML is the Simpsons of concurrency, they did it all

I dunno if you've seen https://clojure.atlassian.net/browse/ASYNC-234 (which relates to ASYNC-225) but it turns out in some circumstances alts! can leak memory when used with something like a timeout channel, because there is a global reference to the timeout channel, and a callback that closes over data may be waiting on that channel even after an alts! selected a different channel

so you actually need nacks to not leak memory

no, wow

restart REPL 😛

I do use refresh, but I also don't always trust it, so when I really want to be sure I eventually restart my REPL

Or I run an AOT compile

I mean, you just described (refresh)

nice writeup

All it does is unmap namespaces and their vars, then reload them from source

it was a fun little puzzle. I was chatting with someone about ASYNC-225 which led me to the conclusion the leak must exist, and then a lot of digging trying to figure out if I didn't see a leak in a simple test case because I was wrong or because there was something else confounding things

It's a sledgehammer. And it tries to rebuild "everything". Which is why it can break stuff.

I occasionally (and surgically) remove and then load a single ns in the very rare situations where I need just that.

I don't know, its pretty clear in what it does. Remove all namespaces that were modified since the last time you run refresh and then load them again in dependency order. Or, remove all namespaces and reload them in dependency order.

I am very specifically advocating tight/narrow manual control over your REPL and eschewing "magic" that tries to do clever stuff / too much stuff.

And I usually don't have tools.namespace as a dependency 🙂

I agree, but for this particular use case, I think refresh-all is the best tool. You've moved things around, you want to know, wait... could any of my code be using something that only exists in the REPL but not in my source files? Well, to know that, you need to remove all loaded namespaces, and reload your source files.

If I can agree on one thing, is I wish refresh was more stupid actually. The whole "trying to keep your state and clean up things" is where maybe it starts to mess up.

I disagree. Automated refresh is lazy and "magic" and breaks unexpectedly for a lot of people -- witness all the beginners here on Slack who trip over problems with it. We should teach better hygiene instead of just calling in a mob of cleanup crew folks.

I'd need to review that, I've never had it break unexpectedly on me, I suspect the only unexpected thing is that their new code is broken, but their old code worked

Like as soon as you want to check that your REPL state doesn't contain things that's missing from your source files, you've entered a difficult place. Its pretty easy to not remember what you changed where, and lose track of what you should remove-ns and load again to be sure. That's when I'll use refresh, as a faster way than restarting the REPL.

Like I said: good hygiene, evaluate every change, think carefully about your refactoring. If you're moving functions between namespaces, you're changing the "public API" of those namespaces, so that should already give you pause. I'm not saying "don't refactor" -- I'm saying be methodical, be careful, take the smallest steps possible, and keep your REPL in sync at every step (if possible). Needing to "refresh" a namespace should be rare and it is easy to do manually -- no library needed. Needing to "refresh everything" means you screwed up somewhere in your process and need a better process. Having bad work practices and just papering over them with a "nuclear" library is the wrong approach (IMO).

(and also like I have said several times: my REPL runs for days, or weeks without restarts so... 🙂 )

I think I agree with you mostly, except that, once you've decided to rename something, or move a function from one namespace to another. Keeping that hygiene is pretty hard. Should you try to avoid renaming and moving things around for fun, sure. But when you decide to do it, you do have an issue. What if you forgot to update a reference? Ok, clj-kondo does help a lot, but that's new stuff 😛 We didn't always have it before. Even then, clj-kondo won't work across namespaces. So it can't guarantee you didn't forget to update a reference.

clj-kondo does work across namespaces because it reparses and caches information about each ns as you make changes.

I kinda of wasn't sure, but I just tried it, and it did not

When I've updated a function arity at the definition and I switch to another ns, it highlights the now-incorrect arity in calls.

Arity maybe, let me try that. It didn't for rename

Well, this has been fascinating in parts but I have cats to attend to so I'm off (and I am going to unsub from this thread as it seems to have run its course at this point).

Also in general, it seems if I do:

(ns foo
  (:require [bar :as b]))

(b/this-does-not-exist-in-bar)

Fair enough, have a nice evening

Heroku is ideal for that. Even has some clojure examples

I'm confused about these results - shouldn't I get a primitive array? It looks like an array of objects

(into-array Integer/TYPE [1 2 3])
;; => #object["[I" 0x39b658f6 "[I@39b658f6"]
(int-array [1 2 3])
;; => #object["[I" 0x3d5bb38f "[I@3d5bb38f"]

I think the "[I" indicates that it is an array of ints

@jumar `Integer` is an object. Irrelevant: Integer/TYPE is int which actually surprised me (I've been doing Clojure too long!)

Ah, so this is interesting:

(type (to-array [1 2 3]))
;; => [Ljava.lang.Object;
(type (into-array [1 2 3]))
;; => [Ljava.lang.Long;
(type (into-array Integer/TYPE [1 2 3]))
;; => [I
(type (int-array [1 2 3]))
;; => [I
(type (aget (int-array [1 2 3])
            0))
;; => java.lang.Integer

Yeah, I think it makes sense - see more examples in the thread

the last one with aget - is that always an object?

Just saw that, yes. And the printed representation of objects has changed since JoC.

I don't think so:

> (type (int 1))
java.lang.Integer

> (identical? (int 1) (Integer. 1))
false
> (identical? (int 1) (int 1))
true

I'm not sure how test for boxed Integer vs int

user=> (let [is (into-array Integer [1 2 3])] (class is))
Execution error (IllegalArgumentException) at java.lang.reflect.Array/set (Array.java:-2).
array element type mismatch
user=> (let [is (into-array Integer [(int 1) (int 2) (int 3)])] (class is))
[Ljava.lang.Integer;
user=>

As for identical? I believe Java caches small values of numbers so (int 1) is always the same object, but (Integer. 1) is going to construct a new instance (with the same value) but a unique identity?

the docs say aget /`aset` are overloaded for arrays of primitives, https://clojure.org/reference/java_interop#primitives

regarding identical?, it looks like it just uses the java == operator, https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Util.java#L134

which I think is value equality for primitives like int

user=> (take 1 (drop-while #(identical? (int %) (int %)) (range)))
(128)

So the first 128 (0..127) are cached.

This is true for byte and long as well.

I think working with primitives is very tricky, every time arguments are passed to a function, primitives are boxed again. So I wonder if the call to type will box the return of aget

See https://insideclojure.org/2014/12/15/warn-on-boxed/

ohhh, so https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Util.java#L134 is using the == operator, but only after converting the ints to Integers?

because the method signature is identical(Object k1, Object k2)

rather than also having identical(int k1, int k2)

Ya, identical also boxes, all function by default will box their arguments and return value, unless they are type hinted as long or double (no other primitive type hint will work to remove the boxing)

Unless they also have an overload in Java, like aget does

I don't think type can actually return a primitive type, right?

I guess it couldn't, since it probably can't actually be passed one as argument

(a great discussion, thanks!)

And like that article sean posted says: > You probably noticed we didn’t see the boxing occurring on the way in/out of the function. That’s not actually boxed math, just function invocation, so it’s a little harder to detect and warn on automatically Which is why its extra tricky, cause :warn-on-boxed won't show you if you're boxing by accident through function application

But also, in a primitive context, you can't use any of Clojure core, except for arithmetics and the array functions, everything else will box again.

also, referencing @seancorfield’s comment about small Integers being cached. you can use that to make 1+1=3, https://pedrorijo.com/blog/java-integer-cache/

Hahaha... 🙂

Is it possible to walk a lazy-seq without creating intermediate data?

Say I know I'll consume it all, and just want to sum the numbers in it.

It depends on what you been by "lazy-seq" and "intermediate data"

Walking a lazy seq will realize any unrealized bits, and if a lazy seq is lazily built on top of another lazy seq (like with map) then it needs to realize that lazy seq as well

Ya, I'm a little fuzzy on that myself. But basically, I know some of the overhead of walking a lazy-seq is that each next that is beyond the chunk size, we will create a new lazy-seq no?

Transducers and CollReduce are the way to pipeline operations without creating intermediate stuff between the pipeline steps

Those would assume I am starting with a collection though right? But say I have the result of (map inc [1 2 3])

Imagine I had like 100 elements in there. Now if I understand, traversing that will create intermediate representation every chunk right? And there's no way around it?

They do not assume that

For example range has a very complicated implementation of ReduceInit (the java interface version of CollReduce) which turns reducing into a loop with a counter

https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/LongRange.java#L229

It has a complicated implementation, the ReduceInit impl is pretty simple

Hum, so its possible that reducing over a lazy-seq will take a faster route? Like in the case of range?

Even when its wrapped like above in a call to map?

Range can return a complicated object

One that is a seq, but also has a separate code path for optimized reducing

So no, if you operate on it like a seq, you get it as a seq, if you use transducers you get the optimized reduce path

Clojure's reducing operations are defined over a number of different interfaces and types, it actually does a lot to try and avoid bottoming out at reducing over a sequence

You can implement ReduceInit or CollReduce and never bother with ISeq and be reducible

I think next.jdbc does that for things like result sets

https://github.com/seancorfield/next-jdbc/blob/develop/src/next/jdbc/result_set.clj#L747

Yes, next.jdbc avoids all of the overhead of creating hash maps etc if you can reduce rows to simple values -- and the reduction automatically manages the opening and closing of the connection.

If a "collection" (conceptually) knows how to reduce itself efficiently, via ReduceInit then reduce will go down that path, else it'll use CollReduce which will ultimately fall back to walking the sequence if no faster approach is defined.

https://github.com/clojure/clojure/blob/master/src/clj/clojure/core/protocols.clj#L75 is where CollReduce is extended to different types/interfaces to provide faster reducing then walking seqs, Iterable for instance

Ok, thanks, I'll have a look at this. If I follow though... (map inc coll) would not be returning something that has a faster path for reducing correct?

It would need to be (eduction (map inc) coll) or something that uses transducers instead, or a direct call to reduce with a collection type that can be fastly reduced?

Correct, reduce is the fast path, the way to build transformation pipelines with reduce is transducers

Sounds good, thanks

@didibus If you would like to see the JVM objects returned by various function calls, to see what lazy seqs look like in memory, there are a couple of examples in this article using my cljol library: https://github.com/jafingerhut/cljol/blob/master/doc/README-gallery.md

Question about PersistentHashMap. If I the map contain kvpair1 and I assoc in the same pair does the amount of memory the map consumes change or is assoc smart enough to know that no change is actually needed?

(def foo {:a 1 :k2})
(assoc foo :a 1)

the current impl returns the original map if both k and v are already present

but it's an impl detail

not a guarantee

@bronsa Thanks!!

What is everyone using nowadays for validating ECDSA sigs

buddy? https://funcool.github.io/buddy-core/latest/api/04-dsa.html

Do operations over lazy sequences in clojure work the same as in kotlin where each element goes through the whole "pipeline" at once, instead of each function being applied to the whole data in sequence?

Yes, but, I don't have enough Kotlin experience to know for sure what they do, I suspect that Kotlin is not lazy, and does something more like Java streams. So lazy-seq would not be the same. Transducers are more likely to be similar to what Kotlin does I'd suspect

That said, lazy seq will not go through the whole data in sequence, each element is pulled and then goes through the whole pipeline of transformation, and then the next element is pulled, etc.

Awesome, that's exactly what I wanted to know, thanks!

Hum, a quick read of the Kotlin doc on sequences, so maybe they are much more like Clojure sequences.

And not like Java streams

one difference might be with chunking - clojure has chunked collections which are evaluatred N elements (often 32) at a time, rather than one by one

this won't matter if you don't use laziness for side effects (don't use laziness for side effects)

Anyways, I'm not sure on the Kotlin side, but I'm sure on the Clojure side 😛 From Clojure, think of it as "pulling". When you define a pipeline:

(->> [1 2 3]
  (map inc)
  (filter odd?)
  (map #(* 2 %)))

And like noisesmith said, there's a small caveat in that because Clojure support ChunkedSeq as well (not all lazy-seq are chunked but most are). Well when you have a ChunkedSeq you can't ask for the next element, you can only ask for the next chunk. So the default chunk size is 32 it means even if you say please give me (first ...) you're actually saying please give me the first result of the first chunk. Which means 32 elements will go through the pipeline, but not one at a time, it'll be 32 go through the first step fully, and then 32 go through the second step, etc.

This is a performance optimization, but it also means if you really want to be one at a time lazy, you have to be careful you're not using ChunkedSeq.

\end-wall-of-text

Ok, the Kotlin docs might put the Clojure docs to shame: https://kotlinlang.org/docs/reference/sequences.html

Damn 😞

> you can't ask for the next element, you can only ask for the next chunk you can ask for one item, but a chunk at a time is realized and cached on the implementation side

> you're actually saying please give me the first result of the first chunk

I'm talking specifically of their infographics

Also, I might wager that Kotlin was inspired directly by Clojure for this feature, I mean all the names are the same, Sequence, filter, map, take, etc.

@ggfpc12495 Ya so the infographic at the end of the Kotlin doc https://kotlinlang.org/docs/reference/sequences.html this is exactly how the Clojure lazy sequence work when not chunked.

Would be nice if we made a similar infographic to explain lazy-seq, chunkedseq, transducers, etc... Hum, add it to my backlog of todos that I never get too 😛

well the model is way more complicated once you factor in chunking and transients

but filing an issue on https://github.com/clojure/clojure-site/issues with concrete requests is a great step. we have resources to do this kind of work when time allows (more skillful people than me :)

I have a question about the path set by deps.edn. Our deps.edn file includes:

:test
  {:extra-paths ["test"]
   :extra-deps {com.cognitect/test-runner {:git/url "<https://github.com/cognitect-labs/test-runner.git>"
                                           :sha "209b64504cb3bd3b99ecfec7937b358a879f55c1"}}
   :main-opts ["-m" "cognitect.test-runner"]}

Execution error (FileNotFoundException) at clojure.main/main (main.java:40).
Could not locate cognitect/test_runner__init.class, cognitect/test_runner.clj or cognitect/test_runner.cljc on classpath. Please check that namespaces with dashes use underscores in the Clojure file name.

.gitlibs/libs/com.cognitect/test-runner/209b64504cb3bd3b99ecfec7937b358a879f55c1/src

Does clojure -Sforce -M:dev:test solve the problem?

Just tried… and no

clojure -Sdescribe -- what version of the CLI are you running?

I suspect you're using an older version where -M doesn't bring in dependencies (it used to just run :main-opts).

{:version "1.10.1.492"
 :config-files ["/usr/local/Cellar/clojure/1.10.1.492/deps.edn" "/Users/pgearon/.clojure/deps.edn" "deps.edn" ]
 :config-user "/Users/pgearon/.clojure/deps.edn"
 :config-project "deps.edn"
 :install-dir "/usr/local/Cellar/clojure/1.10.1.492"
 :config-dir "/Users/pgearon/.clojure"
 :cache-dir ".cpcache"
 :force false
 :repro false
 :resolve-aliases ""
 :classpath-aliases ""
 :jvm-aliases ""
 :main-aliases ""
 :all-aliases ""}

Yup, that's too old for the new -M behavior.

that's really old

ah, thank you

1.10.1.763 is current. A lot of changes happened after 1.10.1.536

I’ve only been paying attention to the 1.10.1

Yup, pulling it in now. Thank you!

https://clojure.org/releases/tools lists all the changes.

While it’s trying to pull it in now, it’s failing to do so:

Cloning: <https://github.com/cognitect-labs/test-runner.git>
Error building classpath. Destination path "test-runner" already exists and is not an empty directory

I suspect that's in ~/.gitlibs

thank you!

either you had a failed thing there that's messing with it or you are encountering a known race condition with downloading gitlibs

adding -Sthreads 1 avoids the latter for the moment

OK, I’ve hurt myself here. I blew away the ~/.gitlibs directory, but now it says:

Cloning: <https://github.com/cognitect-labs/test-runner.git>
Error building classpath. git@github.com:cognitect-labs/test-runner.git: USERAUTH fail

maybe it’s my 2FA on github

it's not finding your user auth. 2fa isn't specifically an issue

https://clojure.org/reference/deps_and_cli#_git has some further tips

Thank you

first, check

ssh-add -l

then most thing wrong is in ~/.ssh/config

using IdentityFile ~/.ssh/id_rsa there is usually problematic - try commenting that out (with #)

hang on, I think it's probably something else

you're using https (public) but it's trying git (private)

are you on CI or local?

sorry, I don’t follow that question

oh, continuous integration

local

this is all my local command line

do you have a ~/.gitconfig with a insteadOf directive?

enabled with something like git config --global url."git@github.com:".insteadOf <https://github.com/>

(don't run that, but that's how it would be enabled)

grep for insteadOf in ~/.gitconfig

could also be set on a per-project basis

ah, yes, I do. Last set up some years ago, so I forgot it

yeah, that's forcing it to try to use your ssh key

Finally got it. Thank you!

(actually running the tests now)

cool

appreciate it all. That was a lot 🙂

Anyone know how to create a middleware-like pattern using Jetty Servlets? I gave a go to HandlerWrapper and HandlerCollection, but they are a bit different from what I expected: when the original (i.e. wrapped) handler completes, the HTTP connection is succesfully terminated with its result. Whereas I'm seeking something more similar to Ring middleware, where a given member can alter the previous member's body/status/headers prior to HTTP termination.

that isn't actually how ring middle works

it looks like it because of the commonly used idioms for wrapping handlers in middleware

but a ring middleware is (fn [f] (fn [req] (f req)))

given a handler, return a new handler that applies the given handler to a request

it has been a long time since I've used servlets, I think the equivalent for them is something like a servlet that takes another "inner" servlet when constructed, and then whenever its doGet or whatever method is called, fiddles with the parameters it gets, maybe replacing some streams with bytearrayinputstreams or whatever, and then passes those parameters to the "inner" servlet, possibly waiting around for it to complete, then examining the baos and doing whatever

I see. Sounds very fiddly indeed :) if this kind of transparent composition isn't encapsulated under a handy technique or class like HandlerWrapper/HandlerCollection I'm not sure I'd want to give it a shot. Closest thing I found (just now) is HttpOutput.Interceptor. In principle it only alters bodies, although maybe status/headers will remain mutable using usual methods

hello, one question on performance, I'm trying to make my custom version of reduce-kv, before adding my things to the process I'm trying to figure what is the fastest way to iterate on a map, other than reduce-kv, I ran a few experiments, the fastest option I found out was using the map iterator directly, but that still 1.2x slower than reduce-kv, is there a simple way to loop on a map that's closer in performance to reduce-kv?

this my iterator based implementation:

(defn kv-loop-iterator [f init coll]
  (let [it (clojure.lang.RT/iter coll)]
    (loop [out init]
      (if (.hasNext it)
        (let [entry (.next it)
              k     (key entry)
              v     (val entry)]
          (recur (f out k v)))
        out))))

bench results (also add some other things I tried):

| title                   | mean          | bar                  | variance-str |
|-------------------------+---------------+----------------------+--------------|
| reduce-kv-native        | 136.966296 ns | █████▏               | 0.000x       |
| reduce-kv-loop-iterator | 301.646678 ns | ███████████▍         | 1.202x       |
| reduce-kv-simple-reduce | 412.819192 ns | ███████████████▋     | 2.014x       |
| reduce-kv-loop-seq      | 530.523741 ns | ████████████████████ | 2.873x       |

On the clojure survey, what does "namespaces" relate to in Q22. Rate the priority of making improvements to Clojure in these areas.

I took it as meaning "all things related to namespaces" -- so if there's anything around how we write/use namespaces, mark how important that is for you, and then add a specific comment in the free text box on the last page.

I took the same as @seancorfield, one example that comes to my mind is this issue: https://clojure.atlassian.net/browse/CLJ-2123

Thanks