Clojurians Log v2

I went through all the learn-onyx challenges & have been trying to get the onyx-template working from https://github.com/onyx-platform/onyx-template Just using the local in-memory zookeeper, I figured out how to compile & run Onyx via:

java -cp target/peer.jar myapp.core start-peers 2 -c resources/config.edn
java -cp target/peer.jar myapp.core submit-job "basic-job" -c resources/config.edn

Unfortunately, when I try to actually submit the job it is accepted by the Onyx cluster but never executed. The Onyx log gives off "Job ID 6a1f7400-f5d3-3d49-ebde-487f6964ab8e has been submitted with tenancy ID 1, but received no virtual peers to start its execution."

I got the Onyx Dashboard working and I can see the accepted job in the cluster, but it's indefinitely stalled. The Clojurians Slack logs for the previous time this came up have sadly been lost --

https://clojurians-log.clojureverse.org/onyx/2017-12-23 is a GOTO NPE 🙂

Do you have enough peers defined to run your job?

I'm rusty so I can't remember all the details but I believe there is essentially an npeers setting

I'm still a bit green, but I believe so. The job is the default from the onyx-template, which has the following catalog (pulled from the dashboard):

[{:onyx/plugin :onyx.plugin.core-async/input,
  :onyx/medium :core.async,
  :onyx/batch-timeout 1000,
  :onyx/type :input,
  :onyx/name :in,
  :onyx/max-peers 1,
  :onyx/doc "Reads segments from a core.async channel",
  :onyx/batch-size 1}
  ... ;; rest of tasks also have :onyx/max-peers 1

Need enough vpeers for each step of your work flow

Hmm, I'll take a look at what options I can pass to the first peer.jar instantiation that starts the in-mem Zookeeper, thanks for the tip

Np

I think that was basically it -- using too small N for number of peers. I started up with more & it took the job. From looking at the dashboard, I suspect it wants at least N >= 3. I think I had the misconception that a sequential 3-task workflow could be executed sequentially with just one peer, but that doesn't seem to be true

Correct, need one peer per task or more if it requires multiple

It using Kafka you need at least one peer per partition in the topic as well

Huh. Thanks for the explanation. Glad I have 32gb RAM with the amount of JVMs in play here 😅

I'll make a small PR to the README of the template, might help other newbies to document the findings & invocations here

Virtual peers i don't believe are extra jvms

You can n number of physical peers with lots of vpeers