Clojurians Log v2

Is there a limit on the number of downstream segments that can be generated by a task? Say there's one segment coming in and from that segment the task generates 30K segments that need to be passed downstream, would Onyx be able to handle this use case?

@dbernal There's not a conceptual limit, but you're going to want to be cognizant of the memory limits on the peer doing that kind of expansion - especially since segments are batched.

@michaeldrogalis Do you know what kind of error might be thrown when that's the case? I'm seeing and ArrayOutOfBounds error on a task that does this and haven't been able to figure out what's causing it. It happens during a prepare-batch lifecycle and the index in question is 20,000 so I thought that it might be some sort of limit

@dbernal Can I get the full stacktrace?

You might be running into the default configured limit of the maximum size of an Aeron message.

Ah I know what this is. This is a hardcoded limit in onyx.

I didn't know that, hah.

+1 Mr. @lucasbradstreet https://github.com/onyx-platform/onyx/blob/0.12.x/src/onyx/plugin/messaging_output.clj#L138

Indeed. We can either add a parameter or we can expand it. You should be a little worried about generating that many messages from very few messages, as all of the messages generated will be inbetween two barriers.

If it’s an occasional thing then it will probably drain correctly through the barriers. I’d be worried if it was, say, 20000 messages creating 30000 messages for example

@lucasbradstreet @michaeldrogalis ah ok good to know! In my case it's 1 segment generating something close to 100,000 in some cases. It might make more sense to read in the 100,000 segments from SQL instead of generating them so it's possible for me to change the workflow to do so

Yeah, would highly recommend having a more proportional set of messages flowing throughout. It'll be much easier to manage.

sounds good. I'll go ahead and do that instead. Thanks for all the help once again!

Have you guys seen this exception?

Exception in thread "Thread-30" java.lang.NoClassDefFoundError: org/agrona/LangUtil
        at org.agrona.IoUtil.delete(IoUtil.java:141)
        at io.aeron.CommonContext.deleteAeronDirectory(CommonContext.java:359)
        at onyx.messaging.aeron.embedded_media_driver$delete_aeron_directory_safe.invokeStatic(embedded_media_driver.clj:8)
        at onyx.messaging.aeron.embedded_media_driver$delete_aeron_directory_safe.invoke(embedded_media_driver.clj:7)
        at onyx.messaging.aeron.embedded_media_driver.EmbeddedMediaDriver$fn__33945.invoke(embedded_media_driver.clj:36)
        at clojure.lang.AFn.run(AFn.java:22)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.agrona.LangUtil
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

@kenny that’s pretty odd. That class should exist.

Which onyx version?

@lucasbradstreet 0.12.7.

It occurs after all the tests have ran.

Ran 2 tests containing 9 assertions.
0 failures, 0 errors.
Exception in thread "Thread-32" java.lang.NoClassDefFoundError: org/agrona/LangUtil
        at org.agrona.IoUtil.delete(IoUtil.java:141)
        at io.aeron.CommonContext.deleteAeronDirectory(CommonContext.java:359)
        at onyx.messaging.aeron.embedded_media_driver$delete_aeron_directory_safe.invokeStatic(embedded_media_driver.clj:8)
        at onyx.messaging.aeron.embedded_media_driver$delete_aeron_directory_safe.invoke(embedded_media_driver.clj:7)
        at onyx.messaging.aeron.embedded_media_driver.EmbeddedMediaDriver$fn__38268.invoke(embedded_media_driver.clj:36)
        at clojure.lang.AFn.run(AFn.java:22)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.agrona.LangUtil
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 7 more

Thanks, I’ll give it a look

It's from this call https://github.com/onyx-platform/onyx/blob/0.12.x/src/onyx/messaging/aeron/embedded_media_driver.clj#L36. I am running the tests with boot test. If I run them with lein test I do not get the error. Boot must cause some issues with shutdown hooks?

Interesting. Yeah, might be a problem where boot pod isolation is preventing access to the agrona jar?

The weird thing is that it’s a missing class error

Right, it is strange. Boot run tests in a pod and pods have a special API for shutdown hooks: https://github.com/boot-clj/boot/blob/master/doc/boot.pod.md#shutdown-hooks.

Ah. Sorry, we won’t be able to spend any time on that as we’re not boot users, but if you manage to track it down let us know.

Hey all. Anyone want to remind me where I saw the web page where you could build an Onyx Workflow in the browser?

@campeterson I think this was the one https://colinhicks.github.io/onyx-blueprint/resources/public/workflow.html. It hasn’t been updated in a while though cc/@colinhicks

yep, that’s it. Thanks

is it impossible to run onyx with 1.9?

It should work fine. We use it with 1.9 internally.

it's working now, idk why but I was getting something about com.google.common.util.concurrent.MoreExecutors.sameThreadExecutor

K. That wouldn’t be our code anyway