Thanks Lucas. So for a given task that has both upstream and downstream tasks, then it will basically use whichever publisher
or subscriber
timeout is lower, so curious what is the value in setting those separately? Is that just to target input or output tasks? We'll look into writing an output plugin. It would work for our use-case but since it involves triggering 3rd-party REST api we didn't assume it was appropriate as it wasn't saving or synchronizing data. Will start rewriting.
@eriktjacobsen it depends on which peer is the one monitoring. So if you have [[:A :B] [:B :C]]
:C
tasks will evict :B
tasks via the publisher timeout, and :A
will timeout :B
tasks using the subscriber timeout. Honestly, it doesn’t make much sense to have two different timeouts - I just set it up that way for flexibility since it’d be a breaking change to go to two timeouts later.
@eriktjacobsen an added benefit of building an output plugin is that you can implement the checkpoint protocol function, so you can store state in cases where you want to retry the request at a later time.