I have a question about how to deploy several topologies in clojure projects. Usually you have a system component
which fires up all the topologies defined in the code?
While its possible to start multiple topologies for a test, in production, we typically build these as separate uberjars.
uhmmm.. interesting... but that option will not be hard to manage after some time? I have never used Kafka in production systems, however the examples I see on the internet, the topology is a single "stream-processing-units", what I mean is each topology accomplishes a single "business objective" which could be proecssing a stream and publishing to another topic. But, the idea is to have many of these, right?
Maybe the definition of topology is what I might be missing at
A processor topology is just a collection of processor nodes you attach to a StreamsBuilder when defining your app. With the Kafka Streams DSL, these can be quite complex.
Our core accounting ledger has 40 processor nodes defined as one “topology”. It’s technically more than one topology because we use through
but that happens under the hood.
We have separate topologies to handle different aspects of accounting. These could all be one “thing” but we wanted to able to deploy them to production separately.
You can develop all your “apps” in the same code repository if you want. We have about five “apps” in one ours.
Apologies if this has been posted before, but I found this article well worth reading... https://clojure-conundrums.co.uk/posts/kafka-streams-the-clojure-way/