Clojurians Log v2

When firing up i) onyx ii) with onyx-kafka , iii) on a single machine, onyx.log spits out this message.

17-12-22 23:39:14 6c4ec751622d INFO [onyx.peer.virtual-peer:17] - Starting Virtual Peer a8fad2b9-f0d9-e015-1c76-62ab461c076d
17-12-22 23:39:14 6c4ec751622d INFO [onyx.log.zookeeper:151] - Stopping ZooKeeper client connection
17-12-22 23:39:16 6c4ec751622d INFO [onyx.log.commands.submit-job:91] - Job ID 76d33ae1-79df-8d07-daac-1d556c3346a0 has been submitted with tenancy ID dev, but received no virtual peers to start its execution.
                     Tasks each require at least one peer to be started, and may require more if :onyx/n-peers or :onyx/min-peers is set.
                     If you were expecting your job to start now, either start more virtual peers or choose a different job scheduler, or wait for existing jobs to finish.

I have logs for the app, kafka and zk if needed.

… this is all in docker, btw.

Seems like the job was either submitted on a different tenancy id, or the peers haven’t started up fully (possibly because they can’t connect to Aeron - it will warn you about this)

@lucasbradstreet This is my :peer-config, which is what I use when calling onyx.api/start-peer-group then onyx.api/start-peers, then onyx.api/submit-job.

{:onyx/tenancy-id "dev"
  :zookeeper/address "zookeeper:2181"
  :onyx.peer/job-scheduler :onyx.job-scheduler/balanced
  :onyx.peer/zookeeper-timeout 5000
  :onyx.messaging/impl :aeron
  :onyx.messaging/bind-addr "0.0.0.0"
  :onyx.messaging/external-addr "0.0.0.0"
  :onyx.messaging/peer-port 40200
  :onyx.messaging.aeron/embedded-driver? true}

Do I need to wait before calling onyx.api/submit-job?

Hmm, that looks fine. No, you don’t need to wait, but it may give that warning if the peers haven’t started up or you don’t have enough yet. Did you start enough peers for the job?

There’s one box, and I set the peer count to 1.

Lemme double check…

Well, that’ll be it. It needs enough for all the tasks as described in the warning

So I need more than I machine, in other words?

Nope, but the onyx.api/start-peers call needs to start enough to run the job.

I mean virtual-peers not peers, sorry

np

So a peer-count of 1 is not enough then. B/c for the moment, I’m just firing 1 onyx.api/submit-job once.

(let [{:keys [zookeeper-url] :as config} (read-config (io/resource config-file-string))
        peer-config (:peer-config config)
        peer-group (onyx.api/start-peer-group peer-config)
        peer-count 1
        v-peers (onyx.api/start-peers peer-count peer-group)]

    (for [[the-workflow the-lifecycles the-catalog] [[psc/workflow
                                                      (psc/lifecycles :kafka)
                                                      (psc/catalog zookeeper-url "scanner-command"
                                                                   peer-count :kafka)]]]

      (do (println "the-catalog: " the-catalog)
          (let [job {:workflow the-workflow
                     :catalog the-catalog
                     :lifecycles the-lifecycles
                     :task-scheduler :onyx.task-scheduler/balanced}
                {:keys [job-id task-ids] :as submitted-job} (onyx.api/submit-job peer-config job)]

            submitted-job))))

You only need one submit job, but yes, you need a higher peer count there.

You just need to add up max-peers/n-peers for all the tasks

Ah ha, so if my catalog has 4 “things” (with :onyx/max-peers) in it, then the peer count is 4.

If one is min-peers 2, one is n-peers 1, and another max-peers 2, it will need 2 + 1 + 1

Right…

Hmm, using onyx / onyx-kafka, I’m not seeing any output from my workflow (onyx.logs here: https://pastebin.com/ktQrFDdJ).

I have the workflow and catalog if needed.

Besides setting :onyx.log/config what other ways are there to troubleshoot / inspect job execution?

If you’re using 0.12 you can try out “onyx.api/job-state. Plays back the log for a given tenancy-id and job-id and returns the current state of the job.”

Looks like it did start though. If you use onyx-http-peer-query you can also query /metrics and find out what it’s up to

Ok yeah, I don’t think onyx.api/job-state or onyx-http-peer-query have made it into the cheatsheet yet… Let me take a look at the source.

Http query requires https://github.com/onyx-platform/onyx-peer-http-query

Ah ha

Gotta run - Xmas travel. Good luck

The thing I’d most be interested in is the epoch_Value metrics - those should be increasing over time.

Oooh, Merry Christmas :)

Hmm, yeah bothapproaches look interesting.

I’ll start with onyx.api/job-state and see how far I get. Thanks !

@lucasbradstreet Have a basic workflow happening with onyx and kafka.

Also working for multiple jobs and workflows. So 2 birds with 1 late night in. Many thanks :)

Any idea what the problem was?

@lucasbradstreet After I got i) a correct peer-count, ii) I realized that my :kafka/key-deserializer-fn had the wrong arity.

I also upgraded to org.onyxplatform/onyx "0.12.0" and org.onyxplatform/onyx-kafka "0.12.0.0". But that probably tangential.

I figured out both errors, while watching onyx.log. So that was the breakthrough, after getting stuck watching console logs for zookeeper, kafka, my app. I think what threw me off were those NoNode exceptions in zk. I spent a lot of time digging into that, which was a non-issue.

Cool, thanks

I’m using the onyx-seq plugin to read a file from disk. how can I fire a trigger when I’ve reached the end of the seq? I want to do a batch computation over all the data and only emit once, when all the data is processed

figured out a solution: use a punctuation trigger with a pred checking if (= event-type :job-completed)

Am I correct that these segments cannot be used by downstream task due to https://github.com/onyx-platform/onyx/issues/779?