Clojurians Log v2

ZooKeeper is running on localhost:2181?

Yes

If you telnet to it and type “ruok” does it respond? Looks like a pretty clear connection issue

It responds.

K. That’s pretty odd then. Does it ever work?

It always happens after a certain amount of work is sent through onyx. Since I'm working locally, ZK is used for checkpoints. perhaps related to that?

Yeah, that would make sense, especially if you’re using windows.

There’s a 1MB limit on the ZK znode size, and also if you’re not using onyx.api/gc-checkpoints then you will keep adding data to ZK

There’s a reason that insanely is in the config option to override it. I don’t really have a great way to prevent people from blowing their feet off there

Any recommendations for other checkpoint storage to use for local dev?

Others have had luck with a local S3 server written in go. I forget what its name was.

This one I think? https://github.com/minio/minio

Maybe we should prevent the ZK override and just push people towards minio

You’ll still want to use gc-checkpoints though, because the checkpoints will keep growing otherwise

You can override the s3 endpoint setting to point to the local minio endpoint http://www.onyxplatform.org/docs/cheat-sheet/latest/#peer-config/:onyx.peer/storage.s3.endpoint

I'll give it a try. I am running checkpoint GC. but do have a lot of state being held, so it seems likely I'm drowning ZK.

Mmm

Please let me know if it works out. Thanks!

Will do. Thanks for your help.

@lucasbradstreet minio worked great. Many thanks, that solved a huge headache.

Awesome. I might just disable the ZK windows feature and force people to use it instead.

i have ran into various problems in dev with storing ABS state in zookeeper as well.. took me a while to realize the cause for my crashing ZK nodes

btw i'm reading the onyx-plugin template, and it's still a bit unclear to me what the difference between synced? and completed? means. sementally, i would say that the difference is that completed? means our internal buffer is empty, and synced? means that the remote database/whatever has actually written everything to disk ? still, it's a bit unclear to me what i should do where

in other news: i've just pushed the first version of a Google Cloud Pub/Sub input and output plugin live, available here -- https://github.com/solatis/onyx-google-pubsub would love it if someone gave it a test ride as well, and provide some feedback / review the code. it's modeled after the amazon SQS plugin.

Can someone please explain to me, how does redeployment work in onyx? Should i just build new version uberjar and restart all my nodes one by one? Should i do some work manually, like stoping tasks etc?

@aleh_atsman you most likely want to use ABS snapshots and recover state of your jobs

http://www.onyxplatform.org/docs/user-guide/0.12.x/#resume-point

^ you can do that using resume points

thx i see, so i need to change my job and add resume-point to it before submit

@aleh_atsman yes, you can setup a basic structure when you build job to only add the resume point if one exists

Though for certain jobs you should probably just fail there because it’s probably an error if it can’t find one on a redeploy

I might bang up an onyx-examples sample soon.

I have a few "paths" in my workflow which don't depend on each other, but I need them to arrive at a certain order in the final step do I need to have just one "path" in order to have this order?

customers need to be transacted before addresses, for example

That is certainly one way (probably the simplest way) to go. Another way would be to collect messages in a window and flush them as the dependencies are sorted out. I wouldn’t recommend this unless you have other reasons to do so though.

e.g. it’s really hard to handle the dependencies in other ways.

You can scale things out in a linear workflow like the above one by using :onyx/group-by(fn,key) which will make sure all of your messages with a certain key ends up on the same peer

yes, I was worrying about scalability. I'll follow your advice 🙂

Don’t be afraid to perform multiple operations in one task though. It’s better not to have too many tasks doing small things. Better to have slightly fatter tasks that you scale out in a parallel way

Maybe the docstrings in the protocol definitions help with understanding both functions a little better? https://github.com/onyx-platform/onyx/blob/0.12.x/src/onyx/plugin/protocols.clj#L28-L33

@lucasbradstreet let me know if you have other recommendations for me. I hope I am not being too annoying with these requests 😛

s/annoying/awesome/

I’ll have a look for a next task. One thing I’ve been wanting to do for a while is include onyx-examples for onyx/type reduce, as well as a resume point example.

that sounds good! I'll gladly look into that if you want.

Great! Let’s start with onyx/type :reduce

One sec.

The first type I’m thinking is where you have a reduce task as a terminal node (rather than an output task) https://github.com/onyx-platform/onyx/blob/0.12.x/test/onyx/windowing/reduce_test.clj

The idea there is that you don’t have a plugin set, and you collect into a window and then use something like trigger/sync to output

If we could have an example like that in the style of https://github.com/onyx-platform/onyx-examples, that’d be great

Then I think we can link to it from the reduce docs too

I have another type of onyx/type :reduce example for you, but I’ll wait until this one is done before describing it.

I provide that reduce-test as an example of what’s going on, but you may want to change the example in ways that explain what’s going on better.

Thanks for the explanation! 👍

No worries. Will be around if you have any more questions.

onyx is fantastic, yesterday I knew basically nothing and today I'm importing data from Prestashop into my Datomic-backed app at the speed of light