Clojurians Log v2

I think native can go smaller because the program isn't allocated to memory by java anymore, so there's a little extra space

Would be interesting to see if it could be measured as a whole

Then again, java binary is probably larger anyway.

If you want to know how GraalVM wins in terms of memory, there’s a podcast with the main author

Ooh, that sounds interesting. Where is that?

Graal or native image, just to clarify?

https://www.infoq.com/podcasts/graalvm-optimizing-java/

actually I listened to this one: https://airhacks.fm/#episode_78 but the above one might also be interesting

it's about both native and JVM

is there any way for me to setup a development environment using clj, on a computer that doesn't have direct access to the internet?

I want to be able to: 1. install clojure (it's a windows machine) 2. pre-download all dependencies and transfer them from my local machine to that computer

so that I'm able to run clj on that machine without it trying to download dependencies

setting up a local maven repo might be a solution, but I have no idea how to do that / how hard it is to do

hmm, how do add a writer for native Java arrays for Transit?

user> (require '[cognitect.transit :as transit])
(import [<http://java.io|java.io> ByteArrayInputStream ByteArrayOutputStream])
niljava.io.ByteArrayOutputStream
user&gt; (def out (ByteArrayOutputStream. 4096))
#'user/out
user&gt; (def writer (transit/writer out :json))
#'user/writer
user&gt; (transit/write writer [1 2])
nil
user&gt; (.toString out)
"[1,2]"
user&gt; (transit/write writer (into-array Long [1 2]))
Execution error (NullPointerException) at com.cognitect.transit.impl.AbstractEmitter/marshalTop (AbstractEmitter.java:203).
null

So, I have been reading a few things about Clojure and GraalVM. There is not much about using both for AWS lambdas, but it seems promising. Has anyone been experimenting with this? I am especially more interested in knowing the tradeoffs Vs using Cljs (performance-wise, not regarding the library ecosystem)

I don't know anything about lambda functions specifically (I'm guessing you mean AWS lambda?), but overall graalvm vs cljs the tradeoff is: • building graalvm takes a long time • startup is ~same (I think <0.01s) • graalvm faster runtime overall

these are my experiences, don't have any hard numbers to back it up 🙂

I think @sogaiu has more to say on the subject

I heard that GraalVM became quite a bit more friendly towards Clojure (eg. plays nicer with common libs), how true is that in your experience?

i think a good person to ask that question to would be borkdude. on the subject of clojure working better with graalvm, i think there was some work on the clojure side toward this end. see here for details: https://github.com/lread/clj-graal-docs

personally, I've mostly used the "talk to c" capabilities of graalvm: https://github.com/Saikyun/clobits so don't know much about clojure libs

this page lists some of the libs that are compatible with graalvm 🙂 https://github.com/BrunoBonacci/graalvm-clojure

probably it's good to inquire at #graalvm for this type of thing

in the prerelease version of clj, i think there's a -P -- you might try #tools-deps for related inquiries

@sogaiu You're right, my bad

thanks : )

Is anyone successfully running Clojure on Big Sur, please?

I haven’t had the chance to dabble much with Clojure lately, but I’ve upgraded to Big Sur and the Clojure REPL works as expected. What are the problems you’re facing?

@hkjels I was unable to install clojure via homebrew. solved it by installing manually.

Ahh, OK. Nice that it panned out!

hello all, Is there any way to limit memory usage of clojure program?

@fabrao -Xmx and similar Java options are one way to limit memory

like java -Xmx512m ?

limit using 512Mb?

Yes. That's only for the heap size though, there are also other resources that take memory

so, my clojure program can get more than 512 Mb?

what is the other resources?

Unable how?

RoamResearch is written in Clojure/Script?

I have a DAG of Model records that I need to serialize. Each model has an :id and some attributes that may contain another model, a collection of models, a map of keyword -> model, or any other Clojure value. Right now I have this code:

(defrecord Ref [id])

(let [acc (atom (transient {}))]
  (postwalk (fn [item]
              (if (satisfies? Model item)
                (let [id (:id item)]
                  (swap! acc assoc! id item)
                  (Ref. id))
                item))
            roots)
  (persistent! @acc))

From https://roamresearch.com/#/app/help/page/Vu1MmjinS: > The Roam stack: > React > Clojure and Clojurescript > Datalog

Thanks @p-himik . I just registered today on Roam after reading a few times headlines about it. After getting excited I checked for the team and found a page about Conor...

No wonder why its rocking 😆

Cool shit 💪:skin-tone-2:

you often see things like this with walking/updating state while you do it and I think it's perfectly fine if you're using walk (you might find volatiles to have slightly lower overhead though and sufficient for single-threaded use case)

another option is to use zippers but I'm not sure it's any better for this

this is a pattern I've also used in a couple of places (with volatile!) (e.g. discovering used params in a function body)

actually, the entire clj-kondo analysis is basically a walk + swap! to a couple of atoms

Thanks! That's reassuring. :) Right, I completely forgot about volatiles.

recursive for or mapcat are nice functional options, too @p-himik

Thanks!

The above use of swap! for atoms that contain mutable things (transients in the example above) is safe for single-threaded use, but can lead to incorrect results if you try to do it from multiple threads, I believe? I am not saying "Don't do it in a single-threaded case", just trying to make sure for myself when it is unsafe.

putting mutable things in atoms is safe re visibility (atoms guarantee visibility)

atoms are an issue only if you're trying to coordinate changes in multiple mutable things (as they don't do coordination)

what do you mean by visibility here?

changes in non-volatile fields are not necessarily visible to other threads for arbitrary periods of time. Clojure largely protects you from issues around this by a) using immutable values (so changes are not a thing in the first place) and b) providing tools for safe change (atoms, refs, etc)

transients are kind of a special case - they are "local" mutability

@andy.fingerhut swap! is safe, only e.g. deref + reset! is not, I guess?

that's still "safe" in terms of well defined semantics

but it is a race condition

Perhaps that race condition might be what @andy.fingerhut meant

Doing assoc! on a transient collection from multiple threads would be unsafe in many cases, true?

yes

you need some way to guarantee that a) the result of the transient op is used for the next op and b) other threads see those changes

basically, serializability is the answer to both, which atoms give you

Sure, atoms give you visibility of changes, but assoc! by multiple threads on the same transient is going to give you visibility of garbage. (edit: not garbage in the gc sense, but in the sense of "a mutable object that has been banged on by multiple threads in parallel, and they have likely messed up its internal state)

it's not the same transient if you're using assoc! in a swap!

swap! is replacing the value with the result of the assoc!

I guess I should it may or may not be the same transient, but the important thing is that you follow the same usage pattern as assoc

I find atom+transient harder to reason about

atom + persistent -> no brainer

even though the atom usage above is scoped to a single function and doesn't escape threads... Clojure makes no promise that your swap! will be called only once

Is the transient guaranteed to be different when you call assoc!? I thought it was just that it may return a different value

@isak it will not always be a different value - sometimes the current one is modified and returned

@ghadi that's a good point and could indeed cause problems

atom+transient is inappropriate, IMHO

I recant! :)

Ok thought so

How about volatile + transient though?

I think it is ok, because then the expectactions are different

volatile is only about usage from 1 thread, right?

not necessarily

I got that impression from this part of the release notes: > volatiles - there are a new set of functions (volatile!, vswap!, vreset!, volatile?) to create and use volatile "boxes" to hold state in stateful transducers. Volatiles are faster than atoms but give up atomicity guarantees so should only be used with thread isolation.

https://github.com/clojure/clojure/blob/97c1210713cbbe0bacbb59004a91e25515d74ab6/changes.md#21-transducers

@ghadi I was thinking that atom+transient (or any other mutable object in general) + single-thread only might be safe, which is the context where it was originally brought up in this discussion. It is atom+transient (or any other mutable object in general) + multiple threads concurrently that I was double-checking my understanding about. But maybe I'm wrong that single-thread is safe there.

hypothetically if we used weakCompareAndSet instead of compareAndSet under the hood, it could possibly retry even in one thread

besides, your function should have a budget of one mutative thing :)

I wonder if vswap! is guaranteed to only run once, unlike swap!

Looks like it, at least with the current implementation: https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Volatile.java#L25-L27

Called from here: https://github.com/clojure/clojure/blob/master/src/clj/clojure/core.clj#L2541-L2548

sorry, missed your message. basically brew tells you this package is not compatible with big slur, so good bye

thread isolation here means 1 thread at a time not only one thread so you should be careful about the difference

Huh, hadn't seen weakCompareAndSet before. Sounds like something only an expert implementer of concurrency mechanisms might ever use?

Got it, thanks

That is some rule-of-thumb for mutable data? Seems like a reasonable one.

Good example of how introducing some local state can sometimes make functions much easier to read 🙂

I'm being a bit unfair, the big difference in readability might be mostly due to the use of postwalk compared to the custom traversal in the functional style.

Yeah, it's a useful function in this case. But on the other hand, with a custom traversal code I could do that traversal in a more efficient way.

Yes, it is always the same trade-off in programming :)

Is this a superfluous indent in select-keys, or am I missing something? https://github.com/clojure/clojure/blob/master/src/clj/clojure/core.clj#L1545

Clojure source is the result of a large number of patches so it's not particularly regular

looks like that particular one was a mass re-format from Rich, must have been something off with whatever tool was used

https://github.com/clojure/clojure/commit/703c847b1a470100d2113ae76352647400d32476

https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/util/concurrent/atomic/AtomicReference.java#L206-L215

AtomicReference itself uses a weak CAS internally

yes, expert stuff

that was 11 yrs ago, so who knows

I was also wondering why there are large chunks of commented-out code still around? Ex: https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/PersistentVector.java#L819 https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Numbers.java#L2014 I assumed it was just a “move fast, break things” kind of result, but perhaps there’s another reason? In any event, not being critical, just curious.

@jeffrey.wayne.evans that’s a rich comment

Not in a Java file

e.g. (comment …bunch-of-helpful-forms) at the bottom of a file

The linked bit is just commented out Java code

It is the closest one can get in a Java file to a Rich comment containing test code for the code in the same file, I think.

@dpsutton does it make a diff?

“here is some code you might find helpful when messing w/ this class”

Ha a little bit but I thought it was outdated but didn’t realize it was a separate main for testing

yep

more interesting is this: https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/PersistentVector.java#L437-L466

which I assume was the result of some exploration

It looks like the last one is in inner class ChunkedSeq, so potentially a copy-and-paste from similar inner classes, then realized that he never wanted to do assoc or conj to increase the size of a ChunkedSeq object, so commented it out. Hard to tell sometimes the history/reasons, unless it is somehow clear from the git commit history, but even then not always.

yep

i've seen these blocks before and always just assumed it was abandoned features or commented out and didn't think they were kinda debug/introspection bits

Yeah, that was sort of my assumption as well, but I figured there would be a comment just above that with a short explanation as to why. You can glean some of that by checking the history, of course.

In general, there is very little answer to 'why' in Clojure's implementation code that I have found. There are some answers to those questions in some of Rich's talks, which you can find transcripts for most of them and links to videos here: https://github.com/matthiasn/talk-transcripts. None of those answers are 'why' at the detail line-by-line code level, but at a higher level of explanation. For line-by-line kinds of questions, the best source I know of is asking here or in #clojure-dev to see if anyone knows. There might be some answers in the book Clojure: The Essential Reference, but I do not know how closely any reasons given there correspond to the reasons of the Clojure developers.

lots of the commented out static main methods are benchmarks

sometimes Rich keeps explorations around in comments. it's Rich's world, we just live in it.

Hello all. is there anyway to get to the underlying map type for a lazy seq? (type (doall (lazy-seq {:1 2 :3 4})))

@gary001 what are you actually trying to achieve?

I have a return value that might be a vector or map and I need to dispatch off that type

you can check if something is a vector? or map?

but that is separate from being included in a lazy-seq

yes but its a lazy-seq

I need to realize the lazy-seq to find its type.

sorry I'm not quite following

an element knows not what it is contained in

(type (doall (lazy-seq {:1 2 :3 4})))
=> clojure.lang.LazySeq
(type (doall (lazy-seq [1 2 3 4])))
=> clojure.lang.LazySeq

Once you've turned the map into a lazy sequence, it's a sequence (of MapEntry items) -- it is no longer a map. Same with turning a vector into a lazy sequence -- it is no longer a vector. (and this feels more suitable for #beginners -- not sure where you are on your journey @gary001?)

i think this is a bit like the underlying type of the result of (+ (float 1) (float 1)). its a double regardless of how you got there. lazy-seq produces a new value

@seancorfield Thanks thats what I thought, I am consuming from an external library so have to find another way.. thanks

@gary001 So you're getting the lazy sequence from the library? Which library is it? I'm curious as to how/where the map vs vector dilemma comes into such code...

@seancorfield http response from my server can return data in different shapes and I am trying to dispatch my output based off of the type so I can format the results.

@gary001 Still not following how the lazy seq aspect plays into this... can you elaborate? (and I figured we'd take it to a thread rather than clog up the main channel at this point)

Which library is returning a lazy seq where you need to find our what the lazy seq was originally built from?

seems strange to get a lazy-seq back as a response from a webserver. that doesn't really make sense to me