Clojurians Log v2

Would a downstream task emit twice if it has two upstream tasks given that it has a trigger that would only fire if it sees a job-completed event-type?

@dbernal Nope, it wouldn't. A job-completed event doesn't "flow" through the tasks

ah ok. So when would a trigger observe a job-completed event-type?

@dbernal a job completed event will be fired when all of the input sources signal “completed”, and all messages are fully flushed through the pipeline.

@dbernal the only time a job-completed even could/should be seen more than once is if there is a fault before it fully completes, and the job restarts from a checkpoint.

ok that makes sense. So if the input tasks complete then a trigger listening for a job-complete would fire off right? Let's say I have input tasks leading to function tasks that lead to an aggregation task. In this instance how would I be able to determine that all the segments have flowed through the function tasks into the aggregation task?

That’s right but as I said a day or two ago, you can’t currently depend on job-completed for trigger/emit, only trigger/sync

To do so would require onyx adding an extra stage to the job completion workflow, to ensure the segments get flushed downstream.

Support for that is currently on our backlog.

gotcha. Thanks for the explanation, it's taking a few times of hearing it to understand what's happening 😄. So is there any way that I can tell the downstream tasks when to fire? Maybe using a punctuation trigger listening on a particular "end" segment?

That’d do it. That’s what we’ve done in other projects thus far to work around the problem, but it does have some limitations in that you need to ensure that all the channels (inputs, tasks, etc) are flushed in order to do it right.

ok got it. I'll look into that then. Thanks for all the help!