Clojurians Log v2

(deftest init-game-test
  (testing "Init the game state"
    (let [state {}
          stack [:foo :bar :baz]
          state' (sw/init-game! (atom state) stack)]
      (is (= true (contains? (:stack @state') :foo))))))

I want to convert a clojure service to Go. What is the most optimal way to do this?

Why? Performance?

"Optimal" in what way?

if the clojure service has integration tests, you might hack them so that they work over http so that you can exercise the Go version is in fact equivalent to the old one

If this is a big (e.g. multi-week migration) project, you could do it gradually by creating some clojure<->go interop/bridge code then slowly replacing the service piece by piece.

It would be interesting to hear more about the why. If you want low latency (startup) and lower memory footprint, GraalVM native-image might offer you that with the code you have right now (or minor changes)

forthcoming GCs promise sub-ms latency as well, so if that's the reason you might do with GC tuning for the time being and no-setup GC with newer JDKs

Golang also uses GC, and probably less advanced or less options compared to Java

although typically for Clojure programs you would generate more garbage due to lazy seqs / FP style programming

GOGC=off that's Go's only option :)

From my experience, Clojure applications with performance issues have a metric ton of performance lying on the floor. There are lots of easy improvements without sacrificing readability or idiomatic code

ZGC and Shenandoah gc offers SUB ms pauses. Right now we have all of our services running with ZGC with JDK16, and the 99% of pauses (the percentile 0.99) is 0.03ms (analyzing the gc log of today).

@niwinz how big is your heap, if you don't mind sharing?

the concrete log is for 2gb heap, but I have similar/identical situation for services that runs with 16gb, ZGC does not depends on heap size, in fact, it has better pauses and better behavior with bigger heaps.

That's what my question was aiming at. Since Clojure is more GC intensive than other langs I was wondering what kind of heaps you need to keep with ZGC for it to not keel over under the GC pressure

Or is it Shenandoah which keels over when the allocation rate gets too high?

Probably relevant: > Pause times do not increase with the heap, live-set or root-set size https://wiki.openjdk.java.net/display/zgc/Main

It depends on the application, In my opinion, shenandoah behavior under high memory pressure is better than ZGC, but has a bit higher pauses (avg 1-2ms) (only tested under JDK15, so don't know if it is improved with jdk16 changes). With JDK16, ZGC improves behavior on small heaps and removes the scalability issue related to the root size (number of threads, etc...) so the GC pause was reduced from AVG 1ms to <1ms.

I'm pretty sure that right now shenandoah and zgc behavior will be very similar

In our case, we process large json/transit objects and serialize/deserialize them constantly, generating pretty big ammount of garbage on each request

Large maps or large sequences of maps?

@aditi.mishra clojure and go are different enough that I don't think the answer will be a code oriented one. If I were undertaking that translation I'd start with diagram (on paper or whiteboard) of the big pieces, and the relationships between them (what they share, how they transfer data to one another, which code crosses the boundaries between those pieces). often times with real world apps the answer is that the boundaries are crossed everywhere and you don't actually have separate pieces, but in that case the process of making the diagram helps you realize how you should have written it and can guide the new version. of course this is a clojure forum so most answers are going to be attempts to argue you don't need to switch

also, logging can help a lot - put the same log calls in the same places, feed the same input, and look at differences in the log output

similarly, it can be useful to use the existing codebase to generate data for example based tests (look at what the existing code returns, make a test in the other impl that expects the equivalent result)

and once again, if your existing codebase doesn't play well with that kind of data-in / data-out testing, that's another indicator of something to fix in the reimplementation

another trick I've used is to do a "bad rewrite" - code that is not idiomatic / readable in the new language but translates 1->1 with the old one. verify that the result works, then do the "readability / not being a pile of crap" edits as a second iteration

because in particular here, a lot of clojure idioms are going to be terrible / unreadable / very poor in performance when directly copied to go

lazy-seqs will probably become a queue consumer(?) - I think this is enough brain dump on my part now, I just find this kind of question interesting in general

> another trick I've used is to do a "bad rewrite" - code that is not idiomatic / readable in the new language but translates 1->1 with the old one. one would end up with "aphyr interview code" :)

right, but that's not code I'd ever publish (or at least I'd delete the branch as soon as possible) :D

it's an intermediate step, but breaking into steps with limited roles can be helpful with big complex tasks

"do one thing at a time" and all that

general clojure question, what's the proper way to emulate a for loop with a conditional break statement in clojure?

reduce to replicate the loop, returning reduced values when you want to early exit.

Without knowing a bit more context and specifics, it's hard to be precise, but generally a reduce that can return reduced for the conditional "break" should work.

And that sort of question is going to get you better answers in #beginners (because the folks there have opted in to helping folks learning Clojure in depth -- which is not the case in this channel).

depending on if you want a for loop for iteration or for side effects, the for macro also accepts a :while to "break" early:

> (for [i (range)
        :while (< i 5)]
   i)
(0 1 2 3 4)

for side effects:

> (doseq [i (range)
          :while (< i 5)]
    (println i))
0
1
2
3
4
nil

(which is why we encourage this sort of question in #beginners because there's a lot of context needed to get the best answer for a given situation)