Clojurians Log v2

Good Morning!

Morning

✌️

Good moriningnging!

moin moin

Hello Cleveland!

morning

@borkdude that transit flush issue is interesting. I'm sidestepping it by having a small number or large objects I'm writing so I'm dodging that issue

I suppose the question is "what disk format should I be using?"

I wasn't happy with nippy or csv

If csv works as a format I could go for arrow and http://tech.ml.dataset, but it would be weird to then drop the dataset aspects of it

@otfrom I'm side-stepping it by first writing into an in-memory bytearray

at least your solution is on purpose rather than by accident like mine 🙂

Can you migitate it by wrapping the actual writer in a buffering writer?

@ordnungswidrig I'm not currently having a problem with it, but I seem to be hitting the sweet spot of only writing a few big objects to a file

I'm using this in clj-kondo where the files I'm writing aren't that big, so all in memory is fine

@ordnungswidrig nope. Buffering output streams follow flushing rules too.

That would rewuire a custom stream that reconciles the flashes?

@ordnungswidrig Yep. I tried to write one, it's very difficult because you need to sometimes flush in order to actually stream things to the browser to reduce overall latency (otherwise you're basically just doing a ByteArrayOutputStream).

it’s an interesting question and I think the “flush management” must be a separate concern than the wire encoding. Acutally you need to combination of time and size based decision. But honestly I do not understand how this can have an impact on chunking, which is a transport encoding.

@ordnungswidrig Well, when you flush, that causes a cascade of flushes which means the gzipoutputstream just flushes out the chunk it has right now, which the http chunking output stream then delivers.

Transit shouldn't be calling flush, period.