Clojurians Log v2

i’m trying upgrade to onyx-0.12.x, getting this exception when trying to read from kafka using kafka plugin. scala.collection.immutable.Map$Map1 cannot be cast to scala.collection.Seq Caught exception inside task lifecycle :lifecycle/initializing. Rebooting the task. -> Exception type: java.lang.ClassCastException. Exception message: scala.collection.immutable.Map$Map1 cannot be cast to scala.collection.Seq Any help?

Coming from someone who has experience with kafka, but none with onyx yet, is it possible the kafka client has a higher version then the broker/server?

kafka breaking changes is notorious inded

https://clojurians-log.clojureverse.org/onyx/2017-09-21.html

@sreekanth What Kafka version are you using and what’s the Kafka plugin version?

I’ve not done any updating since 0.10 so I’d need to spend some time having a look. Need to get up to speed on the newer versions.

@jasonbell 0.11.0.1

[org.apache.kafka/kafka_2.11 “0.10.1.1”]

And the Onyx Kafka plugin version?

[org.onyxplatform/onyx-kafka "0.12.2.0"]

before was using 0.10.beta-x

I’ll try and find some time and have a play later on. Nothing springs to mind apart from agreeing with @gklijs and @lmergen

thanks @jasonbell

I’ll be starting to prepare some hands-on tonight (CET), using latest server, and the kafka endpoint. Hoping it ‘just works’ but if not I’ll let it know here.

@sreekanth Onyx-kafka is using [org.apache.kafka/kafka_2.11 "0.11.0.0"]

https://github.com/onyx-platform/onyx-kafka/blob/0.12.x/project.clj#L17-L18

Is there a way with metamorphic to set a match to a prior event? I'm parsing an event stream or re-frame actions, and I know that a match has finished when an event happens, but I actually want the end of the match to be the event one-prior

Kind of a look-back and say "Oh, know I know what I know here, the match actually finished back there"

I'm guessing it's not possible to do this efficiently

@danielcompton Can you simply axe off the last event from the returned match?

That was my plan, but I need to then grab the n-1 and add that to the match

I think I know what the event preceding the finish would be, so I could optionally match on that, but I don't know if I've covered all of the edge cases

That's probably the best option, and having a fixup path if I don't have id=n-1 and id=n as the last two events in a match

It's a tricky problem domain. I don't think the algorithm can express what you're asking it to directly because it needs to keep all of the events in its tracking path linked together.

Yeah, I think the real answer is to figure out what the preceding events could be. My use for metamorphic is more for slicing an event stream, rather than parsing it

does anyone know if there is an easy way to see consumer offsets via 0.9.x onyx?

^ sorry, kafka consumer offsets. we're having a bit of trouble gauging whether we are keeping pace or not

Also on metamorphic, what is the cost of getting the history in an event match influenced by? Is it number of events that were matched, or in-flight matches, or number of total events?

I need to look back at history to detect the start of a match, and am just trying to figure out how big a deal derefing history is

Not that it hugely matters cause it's a dev tool operating on 100's to 1,000's of traces, but good to know what influences that cost

@djjolicoeur It's stored in ZooKeeper for <= 0.9

@danielcompton Number of actual matched events.

do you happen to know where in ZK. we've been poking around trying to find it with no luck

It discards non-matches immediately

Great, thanks!

it did work after changing to 0.11.0.0. thanks @michaeldrogalis

@djjolicoeur https://github.com/onyx-platform/onyx-kafka/blob/0.9.x/src/onyx/plugin/kafka.clj#L43

damnnit. I looked right past it. thanks appreciate it.

Np