Clojurians Log v2

I probably have some fundamental misunderstanding of how go is supposed to work. There's an underlying fixed thread pool of size 8 by default. I'm trying to execute thousands of AWS commands using aws-api's async interface. And I end up seeing a bunch of

[warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 512k, guardsize: 0k, detached

WARN  org.eclipse.jetty.util.thread.strategy.EatWhatYouKill [qtp523638263-108] - 
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
	at java.base/java.lang.Thread.start0(Native Method)
	at java.base/java.lang.Thread.start(Thread.java:798)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.startThread(QueuedThreadPool.java:660)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:547)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.execute(EatWhatYouKill.java:373)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:308)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
	at java.base/java.lang.Thread.run(Thread.java:829)

do you have a sense of which library is potentially creating too many threads?

if you can reproduce the issue locally, I've used jvisualvm for simple jvm inspection

otherwise, it's possible to get a list of all threads with https://docs.oracle.com/javase/7/docs/api/java/lang/Thread.html#getAllStackTraces(), which you could periodically poll to try and see where the threads are coming from

from the stack trace, it looks like jetty might be the culprit. Are you spawning aws requests from a webserver or similar?

aws-api uses the jetty http client

Ah, in that case, I would consider the following options: • increase the OS thread limit, https://stackoverflow.com/questions/344203/maximum-number-of-threads-per-process-in-linux • rate limit in flight aws requests • the aws sdk for java docs say you can provide a custom ExecutorService, https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/basics-async.html • make sure you don't have long running blocking code executed on any thread pool threads

@smith.adriane aws-api != AWS SDK. It's the Cognitect library that uses core.async. But my question is not really about aws-api at its root, but rather about core.async. So I have this code:

(dotimes [_ 1000000]
    (a/go (a/<! (a/timeout 1000000))))

-XX:ThreadStackSize=1048576 -Xmx512m

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "async-dispatch-7"

Adding back pressure seems like a good answer. It's not clear what environment you're running in, but I can think of at least two options that might alleviate the memory pressure: • increasing the memory limit and add a swapfile. • If tasks are too big, you could store the task descriptions on disk and just enqueue the filename that stores the task info

Task descriptions are small - it's just a list of strings. It's the tasks themselves, I think, that are big. So basically for each string I create an a/go block via aws-api, which schedules a significantly sized task for the executor. I'll try to batch the inputs before they get sent to aws-api.