Clojurians Log v2

Seems like a good use-case for (the recently released) https://github.com/nextjournal/clojure-mode

https://chriskiehl.com/article/thoughts-after-6-years

Hah it seems that someone put things from my head on a paper! Anyway didn’t we have a call two weeks ago, did we?

Somewhat agree, but I have to say that these lists have a certain circle-jerky flavour to them. What did we really learn reading any of those bullet points? The statements are never qualified in any way. This exact blog post or something close to it floats to the top of Hacker News nearly every single day. It’s the kind of stuff I will read when I sit on the toilet to pass the time.

"Cargo-cults are bad. Anyways, here's my strong opinions about a bunch of stuff without any explanation." 😄

Precisely.

I don’t think that these bullets try to qualify anything, they are based probably by some experience that’s it 🙂

IME when you’re 6 years into development you go through the phase of denouncing some “accepted wisdom” that you consumed uncritically at the beginning. It helps 🙂

• “90% – maybe 93% – of project managers, could probably disappear tomorrow to either no effect or a net gain in efficiency.” Even if true, IMO it’s a challenge determining which are the 7-10% worth keeping around.

Argh, just getting this off my chest: C#’s record and ImmutableDictionary are so terribly implemented, I am almost tempted to include ClojureCLR into the C# project just for collection classes with proper equality semantics and composabilty.

Hah, as someone with PM duties, I think that people really underestimate the role PMs play — or haven’t been in any reasonable large or long project with no PM, and haven’t seen what a shitshow it can be 🙂

Using Clojure for a while, then looking in detail at how other languages implement these things, can certainly be a way to see why Clojure's implementation of equality semantics can be measurably slower than those others, I think.

Clojure's equality semantics is definitely doing some extra work for you, e.g. in Java land at least, to compare different storage sizes of integers as equal in integer numeric value to each other, not equal in (type, integer numeric value) combination as Java equals() method does.

@andy.fingerhut that is true. On the other hand (in my not so big experience with Clojure), it is so much more robust and predictable in the wayit works than what happens in other languages, it frees up a lot of mental resources and lines of code. Also, it seems to me, if it comes to a point where Clojure's equality semantics is a performance bottleneck, probably rethinking the whole idea of simply equating nested data structures for this specific use case would yield much better results than any possible optimizations.

I agree with you on the freeing up mental resources, no question. I have often looked into some of the implementation details and performance, which is the only reason I am aware that there is some performance cost for Clojure's = behavior. My current guess is that most of that performance cost is not due to the nested data structures reason (if you wanted that deep-equals behavior, you couldn't optimize it away in any way I know of, anyway), but the "making different JVM types be equal, despite their different types". Take that with a grain of salt, as I haven't done actual experiments/measurements myself to authoritatively say that X% of the cost of Clojure's = is due to reason X

I'd like to set up a reverse proxy (nginx or some such) to go to a different backend depending on the HTTP method - POSTs go to an instance optimized for writes, GETs to one optimized for reads. Is there a name for this specific technique?

I've heard the term "sharding" used in the context of databases, but I think that term is even more specific and has to do with how indexes are divided up. Or maybe I'm splitting hairs...

the most common version of this I've found is with CDN used for get of static assets - I don't know the technical term you want, but adding CDN to your search query might help

this is more specific than just segregating GET, it also looks at paths to determine which are static

Can there be such a thing as too off-topic? 😉 Our Clojure servers are communicating through Apache Pulsar. This was setup by somebody who has left the company. Is there anyone here who has a very good understanding how to setup a Kafka cluster and is willing to help us out?

sounds like a good case for hiring a contractor. kafka is a distributed immutable log, it can do some of the things that pulsar is used for, but isn't well suited for most pulsar uses IMHO having used both.

if what you want is persistent communication between services that spin up / change servers, pulsar is simpler than kakfa

if you want immutable data logs that can be used flexibly, kafka does the job (but is much more finicky to keep running)

pulsar is about messaging (which means it has logic about messages, senders, and recipients baked in), kafka is a distributed log, using it for messaging requires a lot of hacks and conventions and leads to weird corner cases in setup

Yeah hiring a contractor is definitely an option. Pulsar is serving us well for the most part, we’re not really looking to change to something else. But when there is a network outage or something in the data center, or on one of the nodes things go down due to out-of-memory, they take all of the other nodes down as well. That was not the point of having a four-node cluster I thought 🙂

in distsys even numbers tend to be a bad idea generally - you need majority so 4 nodes perform worse than 3 (or at least no better)

more relevant to your original question: confluent offers a docker compose config to spin up kafka locally, and you can purchase a kafka cluster as a managed service from aws

Confluent also has an open source operator for Kubernetes that seems to work fine. But like @noisesmith said, the typical use case for Pulsar is quite different from Kafka. The major drawback of Kafka is you can't commit a single message, and also can't easily read a single message. It's really made for streaming.

also IIRC pulsar has straightforward request - > response, which is something service clients tend to take for granted, making request->response on top of kafka gets super weird and messy