Clojurians Log v2

Has anyone ran into this exception?

Exception in thread "Thread-36" java.lang.NoClassDefFoundError: org/agrona/LangUtil
        at org.agrona.IoUtil.delete(IoUtil.java:141)
        at io.aeron.CommonContext.deleteAeronDirectory(CommonContext.java:354)
        at onyx.messaging.aeron.embedded_media_driver$delete_aeron_directory_safe.invokeStatic(embedded_media_driver.clj:8)
        at onyx.messaging.aeron.embedded_media_driver$delete_aeron_directory_safe.invoke(embedded_media_driver.clj:7)
        at onyx.messaging.aeron.embedded_media_driver.EmbeddedMediaDriver$fn__30987.invoke(embedded_media_driver.clj:35)
        at clojure.lang.AFn.run(AFn.java:22)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.agrona.LangUtil
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 7 more

How close are we to a full 0.12.0 release?

Today.

@kenny that one is odd

@kenny java.lang.NoClassDefFoundError generally indicates two versions of that lib are on the classpath

Are you by some chance bringing in Agrona separately?

No. I'll check the classpath for conflicts though. Aside: http://www.onyxplatform.org/docs/user-guide/0.12.x/#_peer_fails_to_start_and_throws says to increase the shm-size if you run into that exception. How drastically? I believe it defaults to 64M.

Strangely, Agrona is not on the classpath at all.

Is that a dep Onyx should be bringing in?

Both should be being brought in by aeron-all

A worst case estimate is (number of peers in a job / 2) ^ 2 3 http://www.onyxplatform.org/docs/cheat-sheet/latest/#peer-config/:onyx.messaging/term-buffer-size.segment

Basically each task on a peer needs a connection to each peer on a downstream task, and each of those needs 3 segments * that buffer size

No aeron either?

oh wait, I’m looking at lib-onyx

agrona is brought in by aeron-all

There's this from Onyx [io.aeron/aeron-all "1.5.1"]. Perhaps it's not printing everything?

I believe they’ve packaged them both into one library.

Is it possible you’re building the media driver with a reduced set of dependencies? I guess not.

I don't think we are doing anything abnormally.

A bit side tracked on that issue because it's not a blocker right now. I set my shm-size to 1g and I am still getting io.aeron.exceptions.RegistrationException: IllegalStateException : Insufficient usable storage for new log of length=50335744 in /dev/shm (shm). Is there something else I am missing?

I can docker inspect the containers and they all have a ShmSize of 1073741824.

Ahh, it’s because you’re using 0.11

We really reduced the default storage sizes for the buffers.

I can give you a java property to reduce it until you upgrade.

Sure. I'll probably upgrade today if it's released today 🙂

You can give -Daeron.term.buffer.length=2097152 a go until then

I’m releasing rc2 right now. If all looks good I’ll release 0.12 final right after.

That's just set on the Java process you start Onyx from?

Yeah.

Yeah that fixed the shm exception.