Clojurians Log v2

Morning

¡morening!

👋

Good Morning!

Morn'

Morning... 🤞 my copy of Software Design for Flexibility arrives this morning. I might need to go buy more coffee grounds to set myself up for reading it

Oooh, would like your review of it when you are ready

morning

oof - so one of our kafka-streams app ran out of capacity this morning - the 60 partitions i'd assigned to the topic a while back (thinking that it would be enough for a long time in the future) are not nearly enough

do any of y'all have any experience with this - is it safe to add partitions to the existing topic, or is it better to create a new topic and migrate everything to there ?

paging @carr0t

New topic, 100%, IMO

not a good idea to add partitions to an existing topic if you want to maintain partition assignment for keys

If you add additional partitions to an existing topic, you lose guarantees about ordering if reprocessing in future or similar, because the message key -> partition algorithm is based on the number of partitions, so changing that means keys will move to different partitions

If you're not going to reprocess old data, and you're all caught up, or you have some other means of ordering/deduping which means you don't care if a given key suddenly changes partition, or you just don't care about processing order in general, then OK

But you want to be really sure about that

we need the ordering guarantees of partitions - otherwise users will see out of order messages, badly ordered lists of conversations and general inconsistency

so it sounds like i should [1] create a new topic, [2] switch routing over to send to new topic, but don't start processing from the new topic [3] drain old topic [4] start processing on new topic

'drain the old topic'? I have to admit I've not used Kafka streams specifically. But in the past when i've wanted to do this my pattern has been [1] Create new topic, [2] Create 'duplicator', which runs through the old topic consuming all messages and writing them to the new topic (per-key order is maintained), [3] When the duplicator has caught up and is basically copying messages as soon as they appear on the old topic, switch off the app, wait for the duplicator to process potentially the last few messages (literally milliseconds, mostly likely), and switch the app back on now pointing at the new topic Streams having state in rocksDB and such may change how viable that is, it might mean it takes a lot longer to start back up on the new topic or whatever

Yes, I've done what Dan has done in the past too

that process would work for us... what do you use for duplicating topics ? confluent replicator, or something else ?

I wrote a bit of code in Clojure 🙂

Likewise. It wouldn't surprise me if there was a Confluent tool to do it for you, that comes as part of the paid enterprise package, but at the time we were using the free OSS version

thanks @carr0t @dharrigan - all moved over to a new topic with 1440 partitions (vs 60 on the old) and everything is running smoothly again

:partywombat:

Is this your own install, or MSK/similar?

it's strimzi on k8s

👍:skin-tone-2:

our own

strimzi has been great

especially the topic management nouns

well that was a "good" bug - an unexpectedly unbounded recursive call in some promise-based code ate the world