Clojurians Log v2

Just watched @tcrayford video https://www.youtube.com/watch?v=0tUrbf6Uzu8 What is the performance of small collections in Clojure now? @alexmiller @ztellman @ptaoussanis do we still need https://clojure.atlassian.net/browse/CLJ-1517 and https://clojure.atlassian.net/browse/CLJ-1610 ?

Perhaps an update of this talk would be in order 😉 Some performance tips on ClojureScript would also be great.

The unrolled vecs/maps were a case of bad benchmarking

Something that sounded good but didn’t prove out in the wild

Ok, seems to be still half open/ there is no closing words for that proposal if I understand it correctly.

IIRC, the benchmarks were oriented around monomorphic calls, which HotSpot excels at optimizing, but the increased # of classes seen at critical call sites (e.g. RT/assoc) turned those call sites megamorphic. Those kinds of call sites have the worst performance

microbenchmarks stress monomorphism... real world programs have more dynamism

It would be nice to close those tickets with a nice summary

some really good analysis of a similar failed optimization effort here https://github.com/google/guava/issues/1268 @adam.kalisz

> Guava ImmutableList (and others) offer awful performance in some cases due to size-optmized specializations

Thanks! I have added a comment below the YouTube talk. I would appreciate a summary, these tickets leave a sour taste when the case wasn't so clear after all. 🙂

I don’t think the idea is dead. Rich would still love to have tuples. They just need to be performant. There are lots of new tools in Java since this work that may offer new choices

@adam.kalisz I think Rich’s comments in CLJ-1517 are pretty clear on the performance problems (but CLJ-1610 could benefit from a comment at least pointing to why CLJ-1517's original approach wasn’t accepted).

(and it’s nice to see the link to the Guava issue — in Colin’s comment on CLJ-1517 — I had stopped following the issue by that point I think so it’s very interesting to read about that failed optimization too!)

It was weirdly contemporaneous iirc

I don't think anything about Hash Array Mapped Trie is needed for the O(1) count though, Clojure just literally counts elements as calls to assoc are made.

I don't think anyone has tried v8-style or SELF-style hidden classes in Clojure

(such a map optimization would still have to defeat the megamorphic issue mentioned above)

In learning clojure, what is the ONE thing that if you practice regularly, you will become a a great clojure programmer?

You might get better responses in #beginners since folks there have opted in to helping/teaching new folks…

But, to answer your question: REPL-Driven Development.

Always work in your editor, never type into the REPL. Always evaluate every change you make as you make it (you don’t even need to save the file!). Use Rich Comment Forms (i.e., (comment ..)) for all of your “scratch” ideas and exploration — and usually keep it in place in the final code so you can see how you got there. Be able to run tests via the REPL from your editor, so you’re not switching contexts. Develop good REPL hygiene so you can keep a REPL running for days (or weeks, or even months) without restarts or “refresh” tooling.

Thank you, @seancorfield! I will ask in #beginners but thank you a lot for the response!

> Always work in your editor, never type into the REPL. As an autodidact I dabbled in Clojure for at least a year before I really understood this, or REPL-driven development generally. Wish I'd heard it stated plainly like this! 😅

I hadn’t really thought about it being an issue until Stu Halloway said in one of his talks that he was “always baffled when he saw people typing into the REPL”…

https://github.com/matthiasn/talk-transcripts/blob/master/Halloway_Stuart/REPLDrivenDevelopment.md — “Save everything. Your interactions with the REPL -- I am baffled when people type things into the REPL. We will talk about that a bit more in a second.”

(the whole talk is excellent)

Thanks, I'll check it out.

> Develop good REPL hygiene so you can keep a REPL running for days (or weeks, or even months) without restarts or “refresh” tooling. While we're on the subject, do you have any recommended reading about this? I've been using mount and generally getting better at this but my progress has been pretty piecemeal. Usually have to restart my REPL at least once a day to hack around various things I don't fully grok yet.

hah, he's drinking a Lagunitas...I think I'll go grab a beer 😎

> In learning clojure, what is the ONE thing that if you practice regularly, you will become a a great clojure programmer? Jumping to the source to understand what you're invoking. And favoring the use of said source over querying documentation, issue trackers, etc

@ctamayo “While we’re on the subject, do you have any recommended reading about this?” Well, there’s the whole https://clojure.org/guides/repl/introduction series — especially the last two sections (Enhancing…, Guidelines…). Stu’s talk is excellent, @holyjak just posted a video of RDD (in the #news-and-articles channel: https://clojurians.slack.com/archives/C8NUSGWG6/p1623857070131800 ) and that thread contains more links — including Stu’s talk and my demo to London Clojurians from December showing RDD. And there’s also a link in that thread to this set of resources: https://clojure.org/guides/repl/annex_community_resources

@didibus can you point me to the loc’s where this mutation on count’s value happens?

With regards to unrolling, I did find significant performance improvements when unrolling rest-args or keys sequences when they're known at call site. The difference between (get-in m [k1 k2]) and (-> m (get k1) (get k2)) is significant and surprising. Same with assoc

get-in being faster i'm assuming?

about 2x slower

you can see the results here: https://github.com/bsless/clj-fast/blob/master/doc/results.md#get-in It's basic loop unrolling

not to mention that iterating over the keys with reduce1 is slower than reduce

Ya, if you look here: https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/PersistentHashMap.java#L139 For maps, each assoc implementation returns a new map with the count parameter set to count + 1 and here for vector on cons is the same: https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/PersistentVector.java#L228

Thanks so much, really appreciate your efforts. 🙂

Did you check if it has that same megamorpgic issue the ticket refers too for real-life use cases? (No idea what megamorpgic refers too so I don't really know of relevant to get-in)

https://en.m.wikipedia.org/wiki/Inline_caching jitted jvm method calls will emit certain instruction sequences depending on if a call is monophonic (only invoked with a single target type), polymorphic (some small, maybe 2, different target types), or megamorphic (many target types, full real dispatch through a method table)

I got error Clojure can't invoke "java.util.concurrent.Future.get()" because "fut" is null.

Here is the details, the source code caused error:

(defn long-func []
  (let [p (promise)]
    (.start (Thread. (fn []
                       (Thread/sleep 5000)
                       (deliver p "hello!"))))))

;;; this expr will wait for 5 seconds.
(deref (long-func))

Show: Project-Only All 
  Hide: Clojure Java REPL Tooling Duplicates  (13 frames hidden)

1. Unhandled java.lang.NullPointerException
   Cannot invoke "java.util.concurrent.Future.get()" because "fut" is null

                  core.clj: 2304  clojure.core/deref-future
                  core.clj: 2324  clojure.core/deref
                  core.clj: 2310  clojure.core/deref
                      REPL:    8  user/eval7232
                      REPL:    8  user/eval7232
             Compiler.java: 7181  clojure.lang.Compiler/eval
             Compiler.java: 7136  clojure.lang.Compiler/eval
                  core.clj: 3202  clojure.core/eval
                  core.clj: 3198  clojure.core/eval
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn/fn
                  AFn.java:  152  clojure.lang.AFn/applyToHelper
                  AFn.java:  144  clojure.lang.AFn/applyTo
                  core.clj:  667  clojure.core/apply
                  core.clj: 1977  clojure.core/with-bindings*
                  core.clj: 1977  clojure.core/with-bindings*
               RestFn.java:  425  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn
                  main.clj:  437  clojure.main/repl/read-eval-print/fn
                  main.clj:  437  clojure.main/repl/read-eval-print
                  main.clj:  458  clojure.main/repl/fn
                  main.clj:  458  clojure.main/repl
                  main.clj:  368  clojure.main/repl
               RestFn.java: 1523  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   84  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:   56  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:  152  nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
                  AFn.java:   22  clojure.lang.AFn/run
               session.clj:  202  nrepl.middleware.session/session-exec/main-loop/fn
               session.clj:  201  nrepl.middleware.session/session-exec/main-loop
                  AFn.java:   22  clojure.lang.AFn/run
               Thread.java:  831  java.lang.Thread/run

You have to return the promise

Which you're not doing

(defn long-func []
  (let [p (promise)]
    (.start (Thread. (fn []
                       (Thread/sleep 5000)
                       (deliver p "hello!")))
p
)))

I see. Let me take a try. Thanks @karol.wojcik

Problem solved. the returned p need to put out side of (.start …) sexp. Thanks a lot! I thought this might be a bug…. how stupid.

The memory model stuff (memory segments, writing to native memory) I don't use much aside to get an integer pointer as I already use sun.misc.unsafe for writing/reading to native memory. The ffi stuff, CLinker and friends I use when I generate jdk-16 FFI bindings and I do this using a library named insn https://github.com/cnuernber/dtype-next/blob/master/src/tech/v3/datatype/ffi/mmodel.clj. There could conceivably be a JDK-16 specific version of dtype-next that doesn't use unsafe at all and thus doesn't required restricted access but some parts of the memory model specification - regions - I have absolutely no use for as dtype-next uses the https://github.com/techascent/tech.resource to bind objects that need a dispose-or-free-type operation to the GC or stack frame as needed.

One thing to note as that graalvm has two completely different and unrelated FFI pathways of which I support one of.

unrelated to your question, the core future function returns a dereffable thing to get the result of a thread (like you have here) but with proper dynamic binding behavior and using a thread pool for efficiency

You might enjoy looking at https://github.com/redplanetlabs/specter which has similar performance to unrolled, but has an interface which is even nicer than the higher-level clojure functions, especially when dealing with highly-nested data.

I'm familiar with spectre but my design goal was to stick as close as possible to Clojure's semantics, with the ability of providing a drop in replacement, so users don't have to learn a new language. My implementation passes the core test suit.

I see. Thanks for your suggestion. I will learn.

Hey guys, Clojure programming in Emacs, any tips? Of course I know about Cider, but apart from that? I understand the Clojure people are keen on ParInfer (for Emacs that'd be https://github.com/justinbarclay/parinfer-rust-mode then I imagine) more so than on the default Emacs paredit? (I'm yet to try ParInfer, I used to use Lispy and the Lispy EVIL thing before when working in Common/Emacs Lisp.) I'm an EVIL user (the Vim emulation for Emacs that is). So working in multiple modes is really preferred, being all the time in insert mode is really not my preference. Still – I'm happy to try anything that has potential.

I am not a fan of parinfer and stay with paredit. Simple, always works, and I direct it rather than it trying to figure out what indentation i need. it seems basically backwards. things should indent on the nesting i provide, not assume the nesting based on indentation. as for modal editing, I don't know of any reason that you should have to switch to non-modal editing

You can become pretty productive if you just learn 2-3 paredit type commands. Some that I use all the time that I recommend learning the keybinds for: • Slurp forwards (makes it easy to surround code with an if/when/let, etc) • Splice Sexp killing backwards (makes it easy to undo the above) • Kill sexp

I think the main thing with clojure people being keen on parinfer is that it's a useful tool for beginners to get into the language if they're unused to lispy syntax. Almost every clojure developer I've spoken to who started with parinfer has "graduated" to other structural editing packages.

Right, I wasn't aware of that. So basically on front of structural editing, I might as well stick to Lispy/Lispyville then.

So structural editing + Cider would then be the typical way of working with Clojure in Emacs then I reckon?

@suskeyhose I've been using parinfer for years now. Can't work with anything else productively

its the main reason I haven't really tried calva

I’ve gone back and forth between Paredit and Parinfer multiple times over the 11 years I’ve been doing Clojure. I like some aspects of both and I dislike some aspects of both. For the last several years, I was using them together, first in Atom, then in VS Code, but now I’m using Calva (+ Chlorine) so I’m only using Paredit. I think Paredit takes a lot more getting used to and it’s a lot more powerful — but you need to learn/remember a lot more commands to be productive. I found Parinfer mostly did what I expected/needed but missed the more powerful structural stuff that Paredit adds. When I used them together, I disabled some of Paredit’s indent/paste stuff in order to let Parinfer do its thing.

Calva Peredit plays pretty well with the Parinfer extension. You'll need to disable Calva's indent-as-you-type, though.

In cursive with paredit, you can be up to normal code-editing productivity, if not higher, just by knowing 1 keybind (Ctrl-W for "Expand selection")

That is how I work. Lispy(ville) + Cider. I think if you can get really good lispy muscle-memory, that is about as good as any structural editing support out there.

i’ll second lispy(ville), it’s nice.

@ben.sless https://www.youtube.com/watch?v=Yt4zQqndLdQ Every annual Clojure survey trashes the stack traces. Java? Excellent choice of bar to clear! 👏 Sadly, the real bar is set by Common Lisp. @p-himik Yes, old Google hits should be vacuumed, agreed. The weird thing? No one asked how I saved the sale. Or congratulated me on doing so. You yobbos never change....

I just live off indentation and re-indentation, and parens-balancing highlighting. I know how to edit, thanks. But I have been Lisping for 25 years... I had a fling with Paredit, not sure what ended the relationship. I might have forgotten its birthday...

Hmm, The Google is failing me. I saw some ADR templates somewhere, cannot find it now. Any recommendations? 🙏

https://github.com/joelparkerhenderson/architecture-decision-record ?

I found aggressive-indent-mode + adjust-parens-mode in Emacs is my favourite combo. Works better than paredit and pareinfer for me. Though I use a few smartparens commands on top.