Clojurians Log v2

We are trying to diagnose a difficult to reproduce error and this is a shot in the dark: Are there any guarantees around a trigger with a threshold of a single segment and create/apply aggregations for that window? Our apply fn clears out a specific key in the state and based on logic might re-populate that key. Our emit fn for trigger runs side-effects based on the contents of that key. We might be seeing multiple emits, which could be explained by the emit fn being called for a segment before that segment's associated apply was finished executing.

This is all in a single peer / virtual peer env.

@eriktjacobsen the task lifecycle is batched, so you can see the create/apply be performed for a given window more than once in a single batch, during which time it’ll collect the segments to emit downstream

Example: the apply does this: (dissoc state :txts-to-send)(if logic (assoc state :txts-to-send phone-numbers) state) and the trigger is set to {:trigger/on :onyx.triggers/segment :trigger/threshold [1 :elements]}. Could we see trigger emit fn executed multiple times with duplicate state(without the apply having been run to clear out the key)?

The apply should always have cleared out the key after each segment.

Sorry, what I just said is not true.

It will create / apply, then call emit-fn and collect the segment to be emitted, repeating that process for each segment in the batch

then it will collect all of the emitted segments and send them to the next task / plugin output as a batch

The emit will have access to the exact state after the apply on each emit, so I believe it should all work how you want.

are you using refinements, or eviction?

The typical approach you would use is a create to build the state transition, apply to apply that entry to the state, then a refinement to get rid of any state that you don’t want after the trigger fires.

no, we are on 0.12.0 and trigger is

:windows [{:window/id :notifications-handler
                     :window/task task-name
                     :window/type :global
                     :window/aggregation ::notification-handler-update-aggregation}]
          :triggers [{:trigger/window-id :notifications-handler
                      :trigger/id :notifications-handler0
                      :trigger/emit ::emit-notifications
                      :trigger/on :onyx.triggers/segment
                      :trigger/threshold [1 :elements]}]}

Could you define the specific create and apply that you’re using?

It’s a bit fuzzy with the dissoc and if logic there

Alright. We can switch to that approach. This code started on 0.10 Also, this is still a shot in the dark as to whether the trigger firing multiple times without the apply clearing could be the cause. Just was curious if that was a known possibility

(defn create-notifications-state-update
  [window segment]
  (if (:component-configs segment)
    segment
    (when (:component-changes segment)
      segment)))

(defn apply-notifications-state-update
  [window state update-message]
  (let [state (dissoc state :notifications-to-send)]
    (if-let [configs (:component-configs update-message)]
      (initialize-state-config-map state configs)
      (if (:component-changes update-message)
        (handle-notifications-for-changes state update-message)
        state))))

ok, right, so the emit-fn will be passed the state after apply updates it

then if you need to clear out any state after the emit, a refinement is the way to go. I think you’re essentially doing that up front in the apply though.

handle-notifications-for-changes would populate the :notifications-to-send key.... yeah 99% of the time it seems 1apply-1emit seems to hold and we don't get duplicates. It's still possible this is in our business logic, just trying to check onyx trigger-behavior off the list.

You shouldn’t see a trigger firing multiple times without an apply being called in between

K, let me know how you end up

Thanks for your quick help

No worries