onyx

FYI: alternative Onyx :onyx: chat is at <https://gitter.im/onyx-platform/onyx> ; log can be found at <https://clojurians-log.clojureverse.org/onyx/index.html>
2018-03-29T13:30:24.000032Z

I feel like I should be able to answer this question from the documentation but I am not seeing it: In the case of output tasks how do you recover when a segment can’t be written to storage?

2018-03-29T13:31:11.000672Z

for example an output task that writes to a db has a conflict on a particular segment

2018-03-29T13:34:30.000057Z

we’re on onyx 0.9.15 currently (upgrading isn’t a viable options right now)

michaeldrogalis 2018-03-29T14:32:39.000243Z

@jgerman If the task on that job is configured to reboot when an exception is thrown, it'll retry the exact same operation. I'm not sure how you would recover from that specific scenario, Onyx aside.

2018-03-29T14:36:10.000689Z

my concern is if I have a stream of incoming documents to persist, and one has a conflict, I’d like to catch that message and handle it

2018-03-29T14:36:20.000334Z

I’ll keep playing around and see if I can find a solution

2018-03-29T14:36:53.000451Z

I shouldn’t use the word stream there, if I have a batch of segments coming into a plugin and only one has a conflict

michaeldrogalis 2018-03-29T16:30:57.000335Z

@jgerman You can use lifecycles to retry on failure. The other thing you can do is supply an :onyx/fn to any task - input or output included, and modify the outgoing segments

michaeldrogalis 2018-03-29T16:31:11.000260Z

Perhaps some pre-emptive handling in there could help

michaeldrogalis 2018-03-29T16:31:31.000192Z

Hard to help more since this seems very application-specific, but happy to keep bouncing ideas around.

2018-03-29T16:37:39.000293Z

you can supply :onyx/fn to any task? I was under the impression that once the segments went into an output task that’s where we basically lost our view into them

2018-03-29T16:38:12.000130Z

it’s definitely application specific, thanks for the ideas!

lucasbradstreet 2018-03-29T16:38:41.000146Z

Yes, any task can perform transforms with onyx/fn

2018-03-29T16:41:48.000060Z

I’ll look at the docs again, I’m slightly confused by when that fn would be applied in an output task, the output task doesn’t pass any segments on because it has to be a leaf node in the job right?

lucasbradstreet 2018-03-29T17:01:39.000284Z

The onyx/fn is applied to the segments that are sent to it from upstream, so it’s a way to apply some transformations immediately prior to being placed in storage.

lucasbradstreet 2018-03-29T17:02:40.000177Z

You can see the phases in the process-batch section of the state machine, it’ll read a batch from the upstream tasks, then apply-fn