Clojurians Log v2

hey guys … basic question here about REPL usage

I’m working with one of the examples (session windows to be specific)

and when I load the REPL I get a ZooKeeper error

where is says the address is in use

@raymcdermott uploaded a file: https://clojurians.slack.com/files/U04V5V0V4/F8TG0HDPY/image.png

any ideas / tips on how I can run the examples in the REPL?

@raymcdermott it seems like everything has already been eval’d by the time you start evaluating things? It’s not that great for repl play since you need to eval things in turn.

@lucasbradstreet er, yeah I just took the example and loaded it into the REPL … my point is only that it seems like ZK is started twice and I don’t get that

Ah, this is before you even eval anything?

(manually I mean)

ah … it might be that cursive evals it when starting the REPL and than I load it … so it’s done twice

Right, that’s what I was thinking.

They’d be better off in tests I think.

But this way you can go through line by line and see what happens.

yes, that’s what it was

agreed yes although I guess your examples are made for lein run rather than the REPL

I now get this message

@raymcdermott uploaded a file: https://clojurians.slack.com/files/U04V5V0V4/F8S54A4HF/-.clj

but I don’t get that from lein run

it might just be a timing thing, because it can’t tell whether the media driver was dead or had just been shut down recently.

when I submit the job (plucked from the main fn) it all works

And not if not?

this is the session window example, right?

yes

k, lemme try

I mean it all still works regardless of the aeron error / warning

Ah

it’s probably just not shutting things down afterwards, since the shutdowns are in the main entry point

but there are none in the repl eval sections

i executed them manually … I can try again from the top

yeah when I reload after doing the shutdown manually, no warnings

Cool, it’s just stale state then. If you can think of a better way to keep the repl bits while also cleaning up the state, I’m all ears

This is certainly non-ideal

Maybe I could add a (comment) wrapping things and some helpers to start up, and do the final shutdown. It’s really easy to get into a state where you have to reload the repl

That’s why I prefer tests for general dev

sure I’m just trying to get my hands dirty for the first time with Onyx in anger tbh

🙂

thanks @lucasbradstreet also for Onyx 🙂

Of course 🙂

one other question while I’m here

sure

one of our goals is to dump data to mongodb at the end of each session window

I guess that is done via the trigger fn?

you can do it with trigger/sync, or you can use trigger/emit to emit the window contents to a downstream task (which you could write a simple mongo output plugin for)

output plugins are pretty easy to write

ah, that sounds cool …. are there some examples?

I’ve been meaning to fix up the onyx-plugin template.

please let me give you work 😉

Ha, too much to do 😛

I guess it’s a protocol?

It is. Everything below this line is an SQL output plugin https://github.com/onyx-platform/onyx-sql/blob/0.12.x/src/onyx/plugin/sql.clj#L187

sweet, I think I can do one of those for mongodb using monger …

yea, shouldn’t be too hard.

I might need to come back for the wiring

sure, any questions let me know.

thanks man

sorry to come back so soon 🙂

looking in more detail at the sql plugin, it’s all about outputting segments

but I want to transform the data into a pre-defined JSON doc format

so it’s not a 1:1 case

if I have 10 segments I only want to output 1 doc, summarising the data

Well, you can:

1. transform the state in your trigger/emit function before you send it downstream

2. alternatively or additionally batch your outputs via the output plugin (this is mostly for perf optimisation, as you have no guarantees about which segments will batch together)

I’m guessing you want 1

seem like 1 is the most appropriate in this case

of course mongo is easier than sql

as there is no schema

you just need to use the right aggregation for your window

You could either collect via something like conj and transform in trigger/emit, or you can build your document incrementally with your own aggregation

just beware that with sessions you will need to merge sessions together too

I think conj + transform is simpler

how do I merge the sessions?

I’m reading https://github.com/onyx-platform/onyx/blob/0.12.x/test/onyx/windowing/window_session_merge_test.clj

The conj aggregation will do it for you. It’s just a function that the aggregation has to define

http://www.onyxplatform.org/docs/cheat-sheet/latest/#/state-aggregation

oh sorry, I thought you meant I had extra work to do

[ I could not see anything in the merge tests ]

Not unless you want to merge it as you go

With some different logic

ah ok, I see … in that case you need to merge with the existing session state?

Yeah, when two sessions combine, it’ll merge the two states

ok, so you can either conj or have some merge function that is executed when one or more segments are added

in my case I think just using conj and then doing the transform at the session boundary is more straightforward

Agree, since it’s less work

ok, I think I’ve got it … so I’ll leave you to enjoy Sunday 🙂

also I have to sleep !

and I can sleep soundly now