jackdaw

https://github.com/FundingCircle/jackdaw
bartuka 2019-09-09T03:22:13.008800Z

I have a question about how to deploy several topologies in clojure projects. Usually you have a system component which fires up all the topologies defined in the code?

2019-09-19T00:00:54.019900Z

While its possible to start multiple topologies for a test, in production, we typically build these as separate uberjars.

bartuka 2019-09-22T16:26:32.022900Z

uhmmm.. interesting... but that option will not be hard to manage after some time? I have never used Kafka in production systems, however the examples I see on the internet, the topology is a single "stream-processing-units", what I mean is each topology accomplishes a single "business objective" which could be proecssing a stream and publishing to another topic. But, the idea is to have many of these, right?

bartuka 2019-09-22T16:26:48.023100Z

Maybe the definition of topology is what I might be missing at

2019-09-23T15:55:27.023300Z

A processor topology is just a collection of processor nodes you attach to a StreamsBuilder when defining your app. With the Kafka Streams DSL, these can be quite complex.

2019-09-23T15:57:44.023500Z

Our core accounting ledger has 40 processor nodes defined as one “topology”. It’s technically more than one topology because we use through but that happens under the hood.

2019-09-23T15:59:36.023700Z

We have separate topologies to handle different aspects of accounting. These could all be one “thing” but we wanted to able to deploy them to production separately.

2019-09-23T16:00:26.023900Z

You can develop all your “apps” in the same code repository if you want. We have about five “apps” in one ours.

2019-09-09T04:11:13.009700Z

Apologies if this has been posted before, but I found this article well worth reading... https://clojure-conundrums.co.uk/posts/kafka-streams-the-clojure-way/