Clojurians Log v2
jackdaw
Hello everyone, I want to use a key.serializer and value.serializer for kafka producer other than Kafka own serializers with jackdaw. this is my code for producer
(def serialize nippy/freeze)
(def Kafka-Edn-Serializer
(reify Serializer
(serialize [this topic headers data]
(serialize data))
(serialize [this topic data]
(nippy/freeze data))
(configure [this _ _])
(close [this])))
(def producer-config
{"bootstrap.servers" "localhost:9092"
"key.serializer" (-> Kafka-Edn-Serializer
.getClass
.getName)
"value.serializer" (-> Kafka-Edn-Serializer
.getClass
.getName)
"acks" "all"
"client.id" "foo"})
(defn -main []
(jc/producer producer-config)
)Invalid value producer_example$reify__12915 for configuration key.serializer: Class producer_example$reify__12915 could not be found.
pretty sure you can give it as an instance instead of as the class name which might fix your issue @raziyehmohajer.rms did you try:
{"bootstrap.servers" "localhost:9092"
"key.serializer" Kafka-Edn-Serializer
"value.serializer" Kafka-Edn-Serializer
"acks" "all"
"client.id" "foo"}Ahh seems my use is slightly different:
(client/producer
{:bootstrap.servers brokers
:client.id client-id
:acks "all"}
{:key-serde mj/key-serde
:value-serde mj/value-serde})These are expected to be serdes but hopefully that's okay, they only have the .serializer side called in honesty
is jackdaw still alive 😬 I've been hoping https://github.com/FundingCircle/jackdaw/pull/270 would get merged for quite a while
last i heard (months ago) covid hit the admin's schedule hard or something. iirc it was @noisesmith who said that
That sounds like a great idea, keen for not too many forks to be made for this.
Agree, I guess it might be unfair to expect all the original maintainers to come along, but considering the number of PRs it seems like there would be plenty of other people willing to help, myself included!
The thing is that the last commit was only 15. July. So if they're just busy a bit that's perfectly acceptable imho. But if it's really the case that quite a few people would like to contribute but can't because it's still under the fc company account where nobody has access to, then it's different. Personally I just see a bunch of PRs that look good and which I would like to see merged.
I think you still misunderstand - dev moved to london, there's still an active eng team, that team might not be as invested in jackdaw as the us team was but the company still has clojure devs, and last I heard was hiring, just consolidating in london
2👍I can't really speak on the rest
ahh thanks @noisesmith I had indeed misunderstood. That's more promising, might just need to give it some time after the upheaval to get going again 👍
ok. thanks for the clarification @noisesmith. Hope they pick up development of jackdaw soon. Quite a few PRs seem ready and just waiting for the merge button to be pressed 😉
i too am interested in a couple of the pr's currently open against the lib
it was work related to covid that displaced jackdaw work, I (along with the rest of the former US eng team for fc) am no longer working with fc so can't tell you the current situation
thanks for the update, it was what i remembered. sorry about the loss of a job. hopefully you found something quick enough
Thanks, both, for the information 👍
@dstephens yes I tried that too I got this error
Invalid value producer_example$reify__38516@76bd8d56 for configuration
key.serializer: Expected a Class instance or class name.So does that mean that most (all?) people that developed jackdaw don't have write access to the repo anymore to maintain it? If so, then maybe it would even make sense to fork it and put the original authors back as maintainers? I like jackdaw and currently use it for my project. I, too, would like an update to a newer Kafka version and also very interested in the transducer PR.
2👍