Clojurians Log v2

I must have done something wrong with docker.

I'm getting a bunch of http async request exceptions on a Spandex output plugin that I built for writing to Elasticsearch. Is there a certain strategy I could employ to avoid these timeouts? Maybe something similar to the http-client? And more generally is this an issue with how thread heavy the peers are? I'm getting the exception pretty consistently after I input about 100 segments into the job.

I think it might be an issue with the client and not Onyx. nvm

Ah k. If it comes up again and you’re making a lot of async requests you may want to add some back pressure into prepare-batch/write-batch

is there a good example of how to properly do back pressure somewhere? Running into some issues trying to set it up

You can see an example of onyx-amazon-s3 doing some backpressure

https://github.com/onyx-platform/onyx-amazon-s3/blob/0.12.x/src/onyx/plugin/s3_output.clj#L88 only take max-concurrent-uploads at a time, and return false to stay at this state in the state machine https://github.com/onyx-platform/onyx-amazon-s3/blob/0.12.x/src/onyx/plugin/s3_output.clj#L103

by returning false it’ll call write-batch over and over, heartbeating in between

thank you!