Clojurians Log v2

@alexmiller I was wondering, why is pipeline not creating/blocking a thread later in the process, right before running a task. Right now it will create/block-on n threads even if nothing happens. I think that basically creates higher memory usage for nothing.

it could be like this instead no? https://github.com/mpenet/core.async/commit/e9bb61d (edited to latest version)

that one would cost more in context switches potentially, but at least, when the input chan is empty, 0 thread usage and low memory

I guess that patch could create some more work if the chan always contains values (async/thread call, since it's from a cached thread pool), but it's probably negligible given the usage context

I think with your change, it's no longer bounded by n at all

There would never be more than n process calls parallel.

Since we take the result before recur

Actually no, you re right I think

Need less kids around to be able to play in repl, you put some doubt. But I think since every loop parks on process, every loop never runs more than one task at a time. At least it was the idea

not to mention I based that on a 6yo version of master....

more up to date https://github.com/mpenet/core.async/commit/e9bb61d

a quick unscientific sprinkling of prn statements over an atom seems to show it's indeed limited to n

tested with that horror in the same ns: https://gist.github.com/mpenet/1c3d9e052543ab1285ddb5ea9fc8285a

@alexmiller should I create a Jira with proper patch & test?

what problem are we trying to solve here?

using fewer threads?

Yes, if you don't have n+ items to consume it will use less threads, potentially 0 if you are waiting on values to be sent to the chan, or when you are dealing with bursty loads.

at the cost of spinning up those threads when needed

It doesn't change behavior if your chan producers are ahead of the pipeline "runner"

Yes

But you never have more than n threads running tasks at a time

you can file something. to really evaluate I would need to dig out the original design stuff for this

I don't think that's going to happen soon

It's possible to emulate this pattern with pipeline-async + thread on the current version. It's a bit ugly but it saves resources when idling

Alright, will do

Thanks

well, I'd lead with the problem

Lastly, since iirc we have a cached thread pool behind thread the price to "acquire" new threads is low, there's a lot of reuse going on

cached thread pool threads die, so hard to say

They remain for 60s, it's quite a lot

That is what i remember, i could be wrong

that's right