Clojurians Log v2

@lucasbradstreet a small question about plugin design rationale, i'm trying to figure out what's behind it. in kafka, you start the consumer inside the start plugin protocol function: https://github.com/onyx-platform/onyx-kafka/blob/0.12.x/src/onyx/plugin/kafka.clj#L134 in sqs, you start it outside the plugin protocol: https://github.com/onyx-platform/onyx-amazon-sqs/blob/0.12.x/src/onyx/plugin/sqs_input.clj#L94 what's the rationale behind it ? legacy artifact ? sqs being not restartable ?

i get the feeling that starting inside the start protocol function is the cleaner approach, and in sqs it only works because Amazon's async client is kind of like a big singleton that always persists. is this true ?

I find myself repeating a pattern when making stateful flows, where I want to only have the emitted state flow downstream, with the input segments to the stateful task not flowing to downstream tasks. Is there a setting or best practice to achieve this?

@dave.dixon :onyx/type :reduce will not emit transformed segments downstream

@dave.dixon with constraint that :trigger/emit must be set if it’s an intermediate node.

@lmergen to be honest it’s mostly a legacy artifact. I think it’d be better off in the start, but when I was converting over all the plugins I took some shortcuts since there was a lot of work to do.

@dave.dixon you can also use an :onyx/type :reduce on a terminal node and it won’t force you to use a plugin (though that’s not what you want here).

@dave.dixon We really need an onyx-example for :onyx/type :reduce. It was a recent addition and we have been busy with the business side.

@lucasbradstreet Many thanks. I must have read that section in the docs a thousand times and managed to miss that.

Do you think it needs fleshing out?

Probably. The explanation you gave above seems more clear.

OK, thanks.

There was a mistake in the doc anyway :onyx/task -> :onyx/type

Improved https://github.com/onyx-platform/onyx/commit/64785f6fc7b3552dd569a70365323fd31cdfc7ab. I might add an onyx-example and a link later tonight.