jackdaw

https://github.com/FundingCircle/jackdaw
Quest 2019-11-30T01:32:59.046800Z

Could anyone speak to the latency of Kafka Streams? I'd like to know the amount of milliseconds between publishing a message to a topic until it's received by a consuming Streams app (assuming negligible network latency.) Not finding much on Google

gklijs 2019-11-30T08:49:58.046900Z

Depends on a lot of things. But using default settings, and having a low througput you have a latency between calling the send method in the producer and it actually be send to the broker going up to 100ms. This is because http://linger.ms is by default 100 in Kafka Streams to help throughput. Apart from that, and assuming connections already made, it should be on the order of a couple of milliseconds.

gklijs 2019-11-30T08:53:15.047200Z

If latency is more important then throughput you can set it lower. It will also cause more stress for the brokers, because less messages are combined. From what I saw doing benchmarks I wouldn't set it lower then 10.

gklijs 2019-11-30T08:58:35.047400Z

Some of them can be found here, when 0 the is a lot more load on the brokers, https://graphql.gklijs.tech/results/linger-ms-config/average-kb-cpu

👍 2
Quest 2019-11-30T13:25:45.047600Z

Hmm, sub 40ms is probably find and achievable for my use case. Thanks for the graph and insight! 👍