Clojurians Log v2
aleph
sadly the wrapping is not open-source atm @guswill - for no particular reason other than it's in a monorepo and it would be work to extract an oss version and maintain it
what the wrapper does is define an error container record, transparently wrap consumer fns in exception handling which outputs an error container, and provide modified versions of the stream processing fns like map, transform and reduce so that errors are propagated - i.e. preserved in a container on the stream, and returned when a reduce happens
Coincidentally I've found this while looking for error handling strategies: https://github.com/yapsterapp/er-cassandra/blob/5be28c1ed868c87f5d344f21880345dd65326651/src/er_cassandra/util/stream.clj
haha, well i wrote that, but it's quite old... things have moved on a long way since then 🙂
that's probably around the time we were realising we had a serious issue with manifold error handling - we had some cases in prod where streams were closed early with no errors, leading to downstream processes thinking that some records were gone from the input and then deleting other records to maintain consistency
Almost exactly the problem we're having now...
@guswill i don't have time to do a proper job and extract our promise/stream stuff to a library ... but maybe this will help - here's our wrapper code, and the tests for it :
in our code we replace all manifold.stream usage with the equivalent prpr.stream wrapper fn
errors will be copied to the new stream for fns like map, filter, mapcat, transform while reduce will return the first error it encounters in an errored promise
Sorry, I missed your messages! These are very useful, thanks a lot for sharing them!
Wow, you should totally publish this as a wrapper lib; it makes manifold errors way more tractable.
even given that, i prefer the manifold model of streams+promises over core.async's approach. and as you say, core.async's error handling story isn't great either
given manifold's state of abandonment, and lack of support for cljs, i've started work on a replacement - it's looking promising, but it won't be ready for a while yet