Clojurians Log v2

does anyone know what might be the reason for:

2017-02-20T08:40:28.953+0000 cantona-api-383746225-whp97 ERROR [aleph.http.server:0] - error in HTTP handler
                               java.lang.Thread.run    Thread.java: 745
                                                ...
           manifold.executor/thread-factory/reify/f   executor.clj:  44
           io.aleph.dirigiste.Executor$Worker$1.run  Executor.java:  62
                                                ...
      aleph.http.server/handle-request/fn/f--auto--     server.clj: 156
                        bidi.vhosts/make-handler/fn     vhosts.clj: 199
                                     bidi.ring/fn/G       ring.clj:  12
                                       yada.bidi/fn       bidi.clj:  75
                        yada.handler/handle-request    handler.clj: 169
yada.handler/handle-request-with-maybe-subresources    handler.clj: 120
                                 clojure.core/apply       core.clj: 648
                                                ...
                            manifold.deferred/chain   deferred.clj: 909
                            manifold.deferred/chain   deferred.clj: 933
                                                ...
                                 clojure.core/apply       core.clj: 641
                                 clojure.core/apply       core.clj: 654
                                                ...
                        manifold.deferred/fn/chain-   deferred.clj: 888
              manifold.deferred.Deferred/onRealized   deferred.clj: 417
                    manifold.deferred/add-listener!   deferred.clj: 262
             manifold.deferred.Deferred/addListener   deferred.clj: 384
                io.aleph.dirigiste.Executor.execute  Executor.java: 332
java.util.concurrent.RejectedExecutionException:

your thread pool doesn’t accept new tasks

it’s full

ok, I should probably setup a stats logger

it started happening when the server was under heavy load, but it doesn't 'repair' itself when the load lowers

I'm using the default executor

hm

I’m not sure what’s the default in your stack but the default Manifold execute-pool allows an Integer/MAX_VALUE number of threads

so either the pool you’re using has lower limits, or something was hogging resources for a long time

also I’m not sure what’s the behaviour in case the pool can’t create a new Thread - can this only happen due to OOM - then an OOM would’ve been thrown I think

also might have happened if the executor was in process of being shut down

@stijn could that be the case?

I’d bet you got this exception during the shutdown process

I will scan the logs to try and find the root cause

it might be that resources were exhausted

well, I’ve seen this too. Do you, by chance, make requests to other services with aleph.http?

no, we've deliberately swapped this out for http-kit because of https://github.com/ztellman/aleph/issues/217

Yes, that what I had in mind