Clojurians Log v2

Congratulations onyx team.

I have the Java prop -Daeron.term.buffer.length=536870912 set but Onyx still sends Warning: space is running low in /dev/shm (shm). Is there some other configuration I need to set to get it to pick up the property?

are you running in a docker container by any chance ?

^ if so the shm-size option in docker (or memory containers in your orchestration equivalent) is the way to go

Running with docker-compose. I have shm_size: 1024M in my service. Running df -h shows 1.0G available.

hmm could or be that the warning is triggered when it gets somewhere near the 512MB free range? have you tried increasing shm size to something larger such as 4GB?

you’re using the correct docker settings, and this works fine for me in comparable setups

I'll try that. The part that throws me is this message: Warning: space is running low in /dev/shm (shm) threshold=167,772,160 usable=1,372,160. I assume threshold is the max size and it's definitely not set to 167,772,160.

I think the threshold probably defines when it is supposed to emit the warning, but this is just a guess.

Can't set aeron.term.buffer.length to 4g because that's larger than Integer.MAX_VALUE. Will try something smaller.

It probably fills up the 512MB quickly. Are you sending large messages by any chance ?

Those term buffers are huge. Each connection will create three of them to create a connection

Yes. Setting to 1gb seems to do it.

I think the default is 16mb

Which is arguably too big as it is, as we create a lot of connections to build a job DAG

point being, 1gb is still too large. @kenny

512mb did not work.

I think 1gb is unlikely to work either. If you get these warnings, you’ll most likely want to decrease the aeron term buffer length rather than increase it