jackdaw

https://github.com/FundingCircle/jackdaw
finchharold 2021-05-06T07:33:31.147700Z

Where eactly are you polling here?

gklijs 2021-05-06T10:56:55.148700Z

It's hidden in the 'own' library client.clj inside the topology. I created this project 4 years ago, at the time there was not really a good client for Clojure. So it's running

(defn consumer-loop [keep-running ^KafkaConsumer consumer function]
  (if @keep-running
    (do
      (poll-execute consumer function)
      (recur keep-running consumer function))
    (.close consumer)))

gklijs 2021-05-06T10:59:41.149Z

@kishorekaranam99 yes, that sounds about right. In my case a client should do a query to get the current items, and new items are retrieved by the subscription.

gklijs 2021-05-06T11:14:16.149200Z

Well, you can. If you create a Consumer for each streamer. But that would not scale.

Daniel Stephens 2021-05-06T13:53:50.149700Z

This might be helpful https://chrzaszcz.dev/2019/06/16/kafka-consumer-poll/ This is the quote that matters for your issue > Due to the fact that consumer internally is not thread-safe, so it ensures that only one thread at the time can access it, hence acquiring lock here. In case you call methods from different threads, you’ll get an exception in one of them.

finchharold 2021-05-06T16:37:07.152100Z

Thank you @dstephens.

👍 1