Clojurians Log v2

i use both of them - i like core.async for stream-like things (e.g. a stream of events, or a stream of file buffers), but i find it awkward for promise-like things (e.g. a single http request)

basically i really like the manifold programming model, which uses both promises (aka Deferred and Stream together - and it's fine to use core.async and promesa together to similar effect in cljs, although you won't get the automatic glue that manifold gives you (e.g. a reduce of a Stream returns a Deferred)

i'm currently working on a thing which will give you a manifold-like model, but cross-platform and with contextual programming benefits, but it's not going to be ready for a while yet

for file-handling my ideal API would be based around promises, and where content streaming is required, promises of core.async chans

e.g. file/exists? returns promise<bool>, file/content-stream returns promise<chan<buffer>>

Ah, that might not be a good fit then, the files in question are file objects that you would get from within a browser context of an upload or drag-n-drop event, so perhaps core.async might be a better fit?

Those events give you individual files or an array of files

not sure without more details - but if you've got more detail i'm happy to give you my take

Sure 😃... Let me know what details you need and if the explanation that I give below is insufficient 😃... I've got lots of Files[0] and a stack of transforms that I run which I'm then using to create blobs[1]. I don't quite know what transforms I'm running until later, but sometimes I need information about the file contents to work that out. Then once I build the appropriate transform stack I then run all the transforms on the file contents to build a blob which I then use to trigger a download. My current approach is to just read the file contents and then use that to work everything out, but of course for large files you can't just do that because you'll crash the browser tab with an OOM. So I'm thinking can I build a xform of any transformations I want to run on the file and then apply them to the file in chunks, using slice, or just a stream. One potential bit of fiddliness is that I might have to do this more than once, as some intermediate steps towards building the xform may require me to run the current xform state across the whole file to work out what the next step is.:thinking_face: - [0]: https://developer.mozilla.org/en-US/docs/Web/API/File - [1]: https://developer.mozilla.org/en-US/docs/Web/API/Blob

well, those Blobs have a .stream() method which returns a ReadableStream - which is a stream of values very much like a core.async chan, so i would convert that ReadableStream into a core.async chan, and go from there with your xform stack ... the principle challenge will probably be in dealing with errors - iirc the ReadableStream has an error state which core.async chans don't - so you'll have to map the ReadableStream error-state to a distinguished error value on the chan

Thanks for that, I'll give that a go 😃, do you mind if I follow up at some point later if I hit an issue?

not at all 🙂

i've not worked with long-running CPU intensive processes in js-land ... done lots on the back-end though - the approach that has stuck best is to divide work into chunks and put messages describing those chunks onto a stream/chan, then xform the stream/chan (wherein the work is done) until finally reducing it for a result

we also use re-frame - i wouldn't use the re-frame event system for scheduling work though - maybe step outside to core.async and dispatch a re-frame event when you are done

and if you need some introspection into what's going on inside of a core.async process, define a re-frame cofx to fetch that info

https://clojurians.slack.com/archives/C064BA6G2/p1617798039147000?thread_ts=1613754317.404500&cid=C064BA6G2 Is there an example of this you can point to? I'm not really familiar with using cofx's as I've not really grokked what they're for and how they fit with everything else 😳...

https://clojurians.slack.com/archives/C064BA6G2/p1617797946146800?thread_ts=1613754317.404500&cid=C064BA6G2 I'm assuming with this you mean dispatch a reframe event that starts some long running work in core.async and then fire an event when it's done. Wouldn't there be contention between core.async and reframe? Not super sure how I can also track progress other than dispatching a progress update event to re-frame as well...

cofx are inputs to a re-frame event handler from outside of re-frame - they are input side-effects in the pure world of re-frame (while fx are output side-effects). if you need your re-frame event handlers to consider any data which isn't in the event then you want a cofx (it's quite consistent - the app-db itself is a cofx, while any modification to the app-db is an fx)

you could, of course, just dereference any old atom in your event-handler - but then your event-handler isn't pure and is hard to test, so instead you do the atom deref in a cofx and now your event-handler is pure and easy to test and the impurity is isolated in the cofx

so if you do some async stateful stuff outside of re-frame and you want to look in on it then you can either use a cofx (e.g. with an event dispatched on a timer) or you can dispatch re-frame events from your process

yes, there could be contention btw re-frame and whatever work you are doing - if that's going to be a serious issue, then you might want to look at web workers e.g. https://shadow-cljs.github.io/docs/UsersGuide.html#_web_workers

As far as I can tell, the only way to get introspection into the core.async process (ie tracking how much progress into a long running task has been completed) is for it to fire an re-frame dispatch event as it goes writing to the app db where it is, and then having the application display that progress marker...

alternatively, keep an atom with progress/state in, and fetch from that state in a cofx, maybe from an event dispatched by a timer