Clojurians Log v2

Hi, wondering if anyone could give me some advise on using avro serdes? I'm trying to pull a schema from schema reg with basic auth. My code looks like this:

(def +topic-metadata+
  {"input"
   {:topic-name         "my_topic"
    :partition-count    1
    :replication-factor 1
    :key-serde          {:serde-keyword :jackdaw.serdes/string-serde}
    :value-serde        {:serde-keyword   :jackdaw.serdes.avro.confluent/serde
                         :schema "name_of_my_value_schema_on_remote_schema_registry"
                         :key?            false}}})

(def serde-resolver
  (partial resolver/serde-resolver :schema-registry-url "https://<user>:<pass>https://<blankedout>@my_remove_schema_reg.com:13583"))

(def topic-metadata
  (memoize (fn []
             (fmap #(assoc % :key-serde ((serde-resolver) (:key-serde %))
                             :value-serde ((serde-resolver) (:value-serde %)))
                   +topic-metadata+))))

Yeah this part of the API is a bit tricky IMO. The :schema keyword is for when you want to specify a schema that is stored locally to the program being run. At funding circle, our apps usually have a copy of the schema and the registry is used to check compatibility rather than to obtain the schema at message write time.

What is your use-case for acquiring the schema at this time? Do you have some other trusted mechanism that would ensure that the schema you intend to use has been registered before you use it?

Before trying to use jackdaw I've been using vanilla kafka apis but with a custom avro serde which upon reading a avro message from kafka will pull out the schema id from the header and go look up the schema in the registry then deserialise with abracad, and memoizes the id/schema. A lot of our services pull from many other teams so it just seems easy not to take a copy of all the schemas, and have to manually update them and things but maybe I've been thinking about it wrong? I started to move to jackdaw and thought it would be good to get rid of this custom avro serde as abracad etc isn't supported anymore.

Ah ok. From a consumer's point of view you should still be able to do this. The only time a schema is actually needed is for producers. I think you could probably put any old garbage in there but I'm not sure about that.

Oh wait. I think I need to re-read your message to understand properly :thinking_face:

> a custom avro serde which upon reading a avro message from kafka will pull out the schema id from the header and go look up the schema in the registry then deserialise with abracad, and memoizes the id/schema. This sounds like something we implemented ourselves a while back. > A lot of our services pull from many other teams so it just seems easy not to take a copy of all the schemas, and have to manually update them and things but maybe I've been thinking about it wrong? I think this should work in principal but it's not something we've done ourselves. I'd try putting "{:type \"string\"}" in the :schema attribute and see if it just works. But I agree it would be better if you could just leave it blank.

Thanks, it seems to get further, seems like I'm banging into an issue outside of jackdaw now io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null') suggests I'm getting html from schema reg, but the url it should be calling works fine from a browser. Seems like the confluent schema registry might not like basic auth in the url, as I know a 401 will return html. Anyway thanks I think the jackdaw query is solved 🙂