Clojurians Log v2

Can org.clojure/test.check be set to :scope "test" and org.clojure/clojurescript & org.clojure/clojure set to :scope "provided" in onyx-spec?

Yes, that seems fine

Just updated our cluster after a crash, getting this exception. Any idea what that means?

@dave.dixon looks like that’s a case where we fail fast and your container / monitored java should start back up

It seems to be due to initial failures connecting to Zk, and then having it fail again when it tries to start the connection. We may be able to improve this case.

It's stuck in CrashLoopBackOff in k8s, error keeps repeating. I'll check ZK.

Actually this part of the message suggests something else: “org.apache.zookeeper.KeeperException.create KeeperException.java: 103 org.apache.zookeeper.KeeperException$UnimplementedException: KeeperErrorCode = Unimplemented for /onyx code: -6 path: "/onyx"

UnimplementedException - May need to know what version of onyx and ZK you’re on.

Onyx 0.13.0.0. We're running ZK as a hosted service via CloudKarafka, I'll try to find out what's happening there. It was all working fine just an hour ago.

Ah weird then. Either way it definitely seems like something to do with ZK but I’m not much help past that.

Can you think of anything I might have been able to do to cause this?

One possibility is that we did bump the ZK client library in 0.13.0 so that we could support SSL. It’s supposed to be backwards compatible, but it’s possible that there is a bad interaction with the hosted ZK version

That might be preventing you from seeing the true error

You could try excluding org.apache.curator/curator-client and org.apache.curator/curator-framework from the onyx dependency, and provide [org.apache.curator/curator-framework "2.9.1"] in your deps to get back the old library, and try again

That yields this. Do I need to add a curator-client dep as well?

Oof. Looks like there are other deps that will need to be overridden. I’ll check with the diff again

FWIW - I'm pretty sure the previous working version was onyx 0.13.0.0 as well.

Yeah, I’m mostly hoping to resolve the unimplemented exception issue so we can get at the real error

You may need to exclude org.apache.zookeeper/zookeeper and use “3.4.10” instead

All my testing thus far has found the curator/zk deps we bumped to be fine, but I don’t really have any other ideas for debugging it

I guess you could try connecting to zk with the cli and listing /onyx and see if you’re able to poke around there

I rolled back to the container image that was working before, and it still works, so it must be something that we changed recently.

Ah ok. Dependencies seem the most likely cause.

BTW, the big change is that we moved to using tools-deps, which seems to change how deps are resolved.

hello all - I'm using a docker-compose setup based closely on the one generated by the +docker flag to the leiningen onyx template - and the config.edn is currently the one which was generated. I'm seeing a lot of DEBUG level logs from the peer container though, which this

:onyx.log/config #profile {:default nil
                             :docker {:level :info}}

it's this kind of thing:

DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x1644d383a3d0001 after 1ms

any ideas how to turn down logging to INFO?