Clojurians Log v2

@cddr seems like I'm a bit confused about serdes. I checked src/jackdaw/specs.clj as you suggested and that says that :key-serde must be a map EG {:serde-keyword ...}. OTOH looking at examples/simple-ledger/src/simple_ledger.clj, topic-config defaults :key-serde to (jse/serde). The problem I'm trying to resolve is that although I've passed a (Serdes/serdeFrom (key-serializer) (key-deserializer))) as :key-serde, it seems to be falling back to the default key serializer. Any ideas please?

Ah yeah. The serdes are definitely the most confusing part of this thing. The {:serde-keyword ...} syntax is to be used in conjunction with the resolver which will use the data to find a schema required for the serde to actually serialize a message. When the resolver runs, it assigns an actual Serde object to the :key-serde/:value-serde slots.

Is the resolver so you can set serdes from config files or something? I'm not using it so it should be OK to put real serdes into :key-serde/:value-serde and pass that topic map to EG j/kstream?

This is my code; I should maybe mention I'm using this in a TestTopologyDriver

(-> (j/kstream b (sut/topic sut/account-topic))
         (j/group-by (fn [[_k v]] (log/spy (get v ::pool/id))))
         (j/aggregate (constantly {})
                      (fn [a [_k v]] (merge a v))
                      (sut/topic "account-numbers.account-rekey")))

The account-topic works OK but it fails to aggregate

(aggregate
    [_ initializer-fn aggregator-fn {:keys [topic-name value-serde]}]

Any idea why the key-serde might have been ignored here please?

and would you like a PR?

I think it's just because we haven't needed it ourselves yet. When we upgraded versions of kafka, we did "just enough" to get it working without really adding support for all the stuff that has been added. Looks like someone did add (doto (Materialized/as ^String topic-name) (.withValueSerde value-serde))))) but didn't allow for the key serde to be customized at the same time. But yes, if you have the time a PR would be great. One of my colleagues recently made a spreadsheet to document all the missing functionality so it will help us get that cleaned up. Here's the contributing guide if you hadn't noticed it already. https://github.com/FundingCircle/jackdaw/blob/master/CONTRIBUTING.md

Do the Jackdaw tests rely on a local Kafka?

Some of them do. And we use circleci docker containers to provide that at CI time. https://github.com/FundingCircle/jackdaw/blob/master/.circleci/config.yml#L43 For dev time, I tend to use docker-compose.

Do you have the commands available please?

I've not used circle ci or docker-compose very much

Here's an old PR which tried to add docker-compose to the project. https://github.com/FundingCircle/jackdaw/pull/55/ If you take the docker-compose.yml file from there, copy it to your repo and gitignore it globally, then you should be able to do docker-compose up -d to get a working kafka cluster with all the services required by the tests. Beware that one of the rest-proxy tests is a little bit flakey at the moment so if that fails don't worry.

OK thanks

https://github.com/FundingCircle/jackdaw/pull/172

When you say in the PR > This doesn't seem to work... What do you mean? You might want to try adding a test. There's a few already for the various types of aggregations so you might be able to copy one of those and adapt it to the problem of using a custom key-serde. https://github.com/FundingCircle/jackdaw/blob/master/test/jackdaw/streams_test.clj#L713