Clojurians Log v2

Hey @dominicm, sorry I missed this last week. Do you mean this get-records? https://github.com/FundingCircle/jackdaw/blob/9b5bbd0d4582213ee43eb6af3ea37dfd6129d82a/dev/jackdaw/repl.clj#L70. Did you figure it out?

I didn't figure it out, no 🤕. I did borrow that get-records, yeah.

Do you remember what your steps were?

I'll upload the code I was messing around with to a bin

https://bin.disroot.org/?6b122f22678a1bac#HJgQujM4wuvNuttPQx6TLjKJAB2F7byEkwWwEr9uBUvS

Hm, that actually worked for me. To provide the kafka env, I used the docker-compose setup described here https://grumpyhacker.com/test-machine-test-env/, and I had to adjust the bootstrap server setting to be "localhost:19092" instead of "localhost:9092", but once I did that, and after evaluating the line that publishes the line of text, when I evaluated get-records, I got the records that had been published (4 records, one for each unique word).

I connected to an in-memory one that I ran inside the JVM. Docker doesn't work for me. Do you think the problem is likely to be in the kafka server?

I think running it embedded should work ok. Is there one of those bundled with crux? Is that what you're using?

@cddr yeah, I borrowed the code from there

It just uses the KafkaServer class.

Giving this a spin now

io.dominic.jackdaw=> (publish (:input topic-metadata) "hello there kafka streams 2")
nil
io.dominic.jackdaw=> (get-records (:input topic-metadata))
()
io.dominic.jackdaw=> 

(note input twice)

Hm, when I used the embedded server from crux (https://github.com/juxt/crux/blob/master/crux-kafka-embedded/src/crux/kafka/embedded.clj#L86), I can't even create the topics. Seems to just hang.

I mean the server seems to start up fine, but the call to create-topics! hangs

Interesting. I managed to create the topics, no problem.

Crux definitely uses this and works, and this is exactly what the cli calls into. Hmm.

Yeah I'm sure it does. I think it must be something weird. Anyway, I'll play around for a bit and see if I can get it working with the embedded server. There's probably some logging somewhere indicating why it's not working. This is why I think I ended up preferring docker-compose. It seemed easier to find the logs and distinguish them from my own app logs.

Ah, so with logging enabled, I think I see what the problem is (at least for me not being able to create topics)....

2020-06-29 14:23:27 ERROR KafkaApis - [KafkaApi-0] Error when handling request: clientId=adminclient-1, correlationId=18, api=METADATA, body={topics=[],allow_auto_topic_creation=true,include_cluster_authorized_operations=false,include_topic_authorized_operations=false}
java.lang.NoSuchMethodError: 'org.apache.kafka.common.requests.MetadataResponse org.apache.kafka.common.requests.MetadataResponse.prepareResponse(int, java.util.List, java.lang.String, int, java.util.List, int)'

Seems like there's a version conflict between the kafka stack used by the embedded version of kafka, and that being used by jackdaw.

So I was able to reproduce the problem of get-records returning empty by adding "kafka-clients" to the exclusions list when depending on jackdaw. However, for me, the data really did exist on the output topic which could be confirmed by using the standard kafka-console-consumer command line client to read the topic from the beginning.

ah, hmm. I thought I had them all lined up.

I'm using kafka 2.3.0 of kafka-clients, streams, streams-test-utils and kafka_2.12. So those should all be OK, no?

But it seems like if I sift through the logs I should be able to find my answer

{:deps
 {fundingcircle/jackdaw {:mvn/version "0.7.4"}
  org.clojure/clojure {:mvn/version "1.10.1"}
  org.apache.kafka/kafka-streams {:mvn/version "2.3.0"}
  org.apache.kafka/kafka-streams-test-utils {:mvn/version "2.3.0"}
  org.apache.kafka/kafka-clients {:mvn/version "2.3.0"}
  org.apache.kafka/kafka_2.12 {:mvn/version "2.3.0"}
  org.slf4j/slf4j-simple {:mvn/version "1.7.30"}
  org.apache.zookeeper/zookeeper {:mvn/version "3.5.7"
                                  :exclusions [io.netty/netty jline org.apache.yetus/audience-annotations org.slf4j/slf4j-log4j12 log4j]}
  org.clojure/data.csv {:mvn/version "1.0.0"}}

 :mvn/repos
 {"confluent" {:url "<https://packages.confluent.io/maven/>"}}}