onyx

FYI: alternative Onyx :onyx: chat is at <https://gitter.im/onyx-platform/onyx> ; log can be found at <https://clojurians-log.clojureverse.org/onyx/index.html>
Quest 2019-07-26T19:08:27.039400Z

I submitted https://github.com/onyx-platform/onyx-template/pull/77 with the java invocation required to get the template running, but I'm still a little confused on one point. If my :in for the job is a clojure.async channel, how do I go actually submitting data to this clojure.async channel? The test (https://github.com/onyx-platform/onyx-template/blob/0.14.x/src/leiningen/new/onyx_app/test/onyx_app/jobs/basic_test.clj) uses (&gt;!! in segment), but I don't have an nREPL into the peergroup in order to try & access this channel

Travis 2019-07-26T19:10:48.040700Z

Core async is more for unit/integration testing

Travis 2019-07-26T19:11:37.042300Z

Probably want to use kafka for a real job

Quest 2019-07-26T19:11:40.042500Z

Ahh. Yeah, my hunch is that the job isn't inter-actable and never completes (though the (exit 0 "Job Completed")) in core.clj led me to think otherwise

Quest 2019-07-26T19:12:29.043200Z

I did get a real Zookeeper instance running this morning -- already have the compiled Kafka. Despite the age, I've gotten great instruction out of https://dataissexy.wordpress.com/2016/07/31/using-onyx-template-to-craft-a-kafka-streaming-application-part-1-clojure-onyx-kafka-streaming-data/ so far

Quest 2019-07-26T19:13:13.044300Z

I'll continue onto Part 2 & go ahead with getting Kafka setup. This seems more sane than having the peer group initialize an HTTP server & pipe data into the async channel, which is a lot of throw-away work

Quest 2019-07-26T20:49:45.047100Z

onyx-kafka plugin has changed significantly -- the kafka-opts for the task from the 2016 tutorial needed a lot of tweaking. There was one key that's supposed to be allowed -- :kafka/commit-interval -- and it's not marked as deprecated at https://github.com/onyx-platform/onyx-kafka/blob/b092ad0e96bd52bb38e9b447834d78c072a0fb47/src/onyx/kafka/information_model.cljc However, keeping it in the task options was failing the Schema validation check every time. I'm guessing this key was intended to be deprecated? If nobody in channel knows within a day or so, I'll file an issue.

Quest 2019-07-30T16:54:17.047500Z

created https://github.com/onyx-platform/onyx-kafka/issues/62