onyx

FYI: alternative Onyx :onyx: chat is at <https://gitter.im/onyx-platform/onyx> ; log can be found at <https://clojurians-log.clojureverse.org/onyx/index.html>
lucasbradstreet 2018-12-10T04:35:52.004500Z

I gave it another run. Chrome starts up with this:

lucasbradstreet 2018-12-10T04:36:04.004600Z

lucasbradstreet 2018-12-10T04:36:44.005Z

https://imgur.com/a/1jh0M2e

lucasbradstreet 2018-12-10T04:37:26.005300Z

Test output: https://gist.github.com/lbradstreet/8a481771fde676ac8d166dc89fd0312f

lucasbradstreet 2018-12-10T04:38:46.005900Z

If I load the page manually, I see: https://imgur.com/a/PvvDiQk

lucasbradstreet 2018-12-10T04:39:02.006400Z

cc/@manuel

manuel 2018-12-10T07:05:09.007200Z

@lucasbradstreet this is what I get on lein test: https://gist.github.com/manuel-uberti/a91db5a83e4bcaffc12c0eb14964b82e

youvere 2018-12-10T21:07:22.014300Z

https://github.com/onyx-platform/onyx-kafka#attributes :kafka/deserializer-fn what is the recommended way to add some state (not serializable) in onyx ? The way I’ve found so far would be to set a global state and get the global state inside the deserializer function. I’ve also tried to create my own deserializer in my own task, but I have the same issue. You must use a namespace keyword. When you use a namespace keyword you can’t inject a state. I can use lifecycle, create a new task for deserialization and inject the state using the lifecycle I could also create a new def at runtime with the state, but all those solutions seems bad

(defn build-catalog [kafka-zookeeper kafka-topic bootstrap-servers]
  [{:onyx/name               :kafka-in
    :onyx/type               :input
    :onyx/medium             :kafka
    :onyx/plugin             :onyx.plugin.kafka/read-messages
    :onyx/max-peers          1
    :onyx/batch-size         50
    :kafka/zookeeper         kafka-zookeeper
    :kafka/topic             kafka-topic
    :kafka/deserializer-fn   ::deserializer-avro
    :kafka/offset-reset      :earliest}
(defn set-deserializer [schema-registry]
  (defn deserializer-avro [data]
    (avro/decode schema-registry data)))