Clojurians Log v2

Does jackdaw have support for json-schema serdes or could anyone point me to any examples of how to build a json-schema serdes for jackdaw?

What would you want it to do? Simply validate messages against some schema or are you thinking about something which keeps track of versions and checks compatibility like the confluent schema registry?

Mainly the former. My issue is that I’ve got a topic with plain old json messages that I want to write to an aws lambda connector that requires json-schema formatted messages (or avro). If you pass in plain json it munges the data so I need some sort of schema solution.

Today I learned that json and json schema aren’t binary compatible. I guess my other solution would be to do avro or protobuf.

https://json-schema.org/ is this what you mean by json-schema?

Referring to this: https://rmoff.net/2020/07/03/why-json-isnt-the-same-as-json-schema-in-kafka-connect-converters-and-ksqldb-viewing-kafka-messages-bytes-as-hex/

Not json schema as in “a json schema” but there is an actual format for the data that isn’t just json strings.

Or, do you have a simple example of using avro in clojure? I may just do that if it’s pretty straighforward.

Is it as simple as registering a serdes using https://cljdoc.org/d/fundingcircle/jackdaw/0.7.6/api/jackdaw.serdes.avro.confluent? That seems way to easy.

Yeah that's pretty much it. But you need to have a schema-registry to make that work

I am using the confluent registry.

Does it handle everything over the wire?

Seems magic.

It does handle the wire format.

An alternative option for avro is https://github.com/cddr/edn-avro. In the README I tried to explain the rationale for the different approach.

cool

I’ll check it out. Thanks!

What is the correct way to invoke this:

(def serde-resolver
  (partial
    resolver/serde-resolver
    :schema-registry-url
    "<https://schemaregistry.aws.confluent.cloud>"))

Actually, I think jackdaw.serdes.avro.confluent is more what I want. Is the schema param the actual schema itself or the name of the schema in the registry?

(avro-confluent/serde
  "<https://schemaregistry.aws.confluent.cloud>"
  "orders-value"
  false)

I guess it’s the raw string for the schema. Shouldn’t it look that up from the registry?

It needs the schema only for the producer side. It's a bit of a wart that the interface makes you specify for both

Your producer should probably have access to the schema either via JSON stored as a resource, or via the class generated by the avro compiler

With edn-avro do you not specify a serdes or do you use it as a serdes?

So just:

;This
{:topic-name         "my-topic-name"
 :partition-count    1
 :replication-factor 3}

;vs something like this
{:topic-name         "my-topic-name"
 :partition-count    1
 :replication-factor 3
 :key-serde   (avro-serdes/serde)
 :value-serde (avro-serdes/serde)}