Clojurians Log v2

If I'm using pipeline-blocking and I don't care about the result of my transformation function, what should I put on the 'to' channel?

why are you using pipeline-blocking?

your pipe is leaking on the floor... :)

Maybe I shouldn't be... I have a feeling I'm in over my head

I assume your transformation function is doing some potentially blocking i/o?

Yes it is

do you want it to be parallelized? do you want bounded parallelism? do you want back pressure?

I want it to be parallelized, it doesn't have to be bounded although I assume that's the practical thing to do, and the back pressure question is one that I'm still thinking about

I mean, generally you want bounded (unbounded things only stop working at 2 am when you're on call)

I wanted to write a writer library for an existing java service that reads json and then in a very blocking way writes into a Tinkerpop graph database.

pipeline-blocking will always write a result into an output channel (that will accumulate)

The service currently takes ~4 hours to run, so I thought that by parallelizing and batching the writes we could achieve a speed up. But I'm not sure how backpressure works in a scenario like this

if you want parallel, bounded, with backpressure but don't care about the results you could give it channel with a dropping buffer that would just immediately drop the "results" on the floor

that's a solution

What's a typical n value to use with pipeline-blocking?

you could also just have a go-loop feeding a queue/channel and N threads reading from the queue and pushing stuff into the db

pipeline-blocking is doing a bunch of work to order the results that you presumably don't care about

Yeah I don't really care about the ordering of the results

I should just write it in java anyways

My team doesn't know clojure

Thanks for your help

how would you do this in java?

I'd have to do a little investigation on the best way to do it. Streams are new to me but I could probably make a streams-oriented solution