Clojurians Log v2

I think the docs on onyx-kafka should mention which keys are required. I just ran into a NPE due to not having set :onyx/n-peers or :onyx/max-peers caused by https://github.com/onyx-platform/onyx-kafka/blob/86f422389d32d9feb526355954ec4ffa08638df5/src/onyx/plugin/kafka.clj#L80. It would've been helpful to have known I need to specify one of those.

@kenny thanks. Could you please open an issue? That should have been detected much earlier.

Sure

Thanks. I think we assumed a max is enforced by the onyx schema, but that’s not true.

Either way, it should never NPE

Agreed 🙂

When using the Kafka output task, do you guys usually attach an :onyx/fn to transform your segment into the format :onyx.plugin.kafka/write-messages expects?

It depends on whether data comes from multiple tasks. If so you may need to do it on the previous tasks, as the format may differ.

Not sure what you mean. I assume you mean the format of my segment?

Yeah.

Main point is you can do it on either the writer task or the previous task. Which is best depends on your job.

I’d prefer to put it closer to the output task in general though.

Right, makes sense. Thanks!