Clojurians Log v2

Where eactly are you polling here?

It's hidden in the 'own' library client.clj inside the topology. I created this project 4 years ago, at the time there was not really a good client for Clojure. So it's running

(defn consumer-loop [keep-running ^KafkaConsumer consumer function]
  (if @keep-running
    (do
      (poll-execute consumer function)
      (recur keep-running consumer function))
    (.close consumer)))

@kishorekaranam99 yes, that sounds about right. In my case a client should do a query to get the current items, and new items are retrieved by the subscription.

Well, you can. If you create a Consumer for each streamer. But that would not scale.

What happens when you poll, https://github.com/apache/kafka/blob/03690d7a1f5e90fb85690352d7f2b7651e554dc3/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L1169

This might be helpful https://chrzaszcz.dev/2019/06/16/kafka-consumer-poll/ This is the quote that matters for your issue > Due to the fact that consumer internally is not thread-safe, so it ensures that only one thread at the time can access it, hence acquiring lock here. In case you call methods from different threads, you’ll get an exception in one of them.

Thank you @dstephens.