Clojurians Log v2

The lack of error propagation in manifold streams makes me less confident in writing servers that use stream conversion to handle incoming data. For example, I would like to write a handler that takes an InputStream, converts it into a stream of chunks (easy enough with byte-streams), then upload each of those chunks to S3 in a multipart upload. But what if the S3 API throws an error when uploading one of the chunks? The stream consumer will silently fail, but won't stop the upload. Any way to reconcile this lack of error propagation?

Yeah, I’ve registered an issue https://github.com/ztellman/manifold/issues/95 a while ago

it even references a commit with an error-propagation implementation

@ztellman said that his experience with Lamina proved that attaching error handling to streams is more trouble than it’s worth.

I disagree - for the same reasons you have, @aengelberg

🙂

I'm not convinced it's an easy problem to solve, either 🙂 In general, you'd ideally like to be able to both propagate errors forward and backward. e.g. If something hands you a stream, you'd like to know when the upstream processor stops due to failure. And if you hand something a stream, you'd like to know when the downstream processor fails.

well, I’ve implemented propagation downstream

upstream is a harder problem as sources don’t usually care about consumers

I don’t think you can really get by without the feature - in many parts of my code I’ve resorted to passing tuples of (Stream, Deferred[Error]) through combinators

Rx* libs have this too

another stream library I like a lot, most.js, provides a recoverWith method: https://github.com/cujojs/most/blob/master/docs/api.md#recoverwith

it's an interesting solution, in that the error handler function is expected to provide a replacement stream

in their example, you fall back to default data if you can't fetch data from a server