Clojurians Log v2

Hi all. Is there a recommended way to manage an expiring or max-size-based cache in the app-db? I am fetching against a server and it would be nice to cache some results as they are bound to be used again and again, but I could see that a long-running session could accidentally cache too much data? Or am I better using a separate caching mechanism outside of the app-db, but then I lose the benefit of subscriptions.

How do you plan to check whether a cached value is still in use or not?

I’d simply send an event to refetch in the event of a cache miss. So it could be time-based or LRU etc.

I don't think that can work. Suppose you have a cached value. You use subscribe to get that value in a view. That subscribe is called at time T. You have TTL set to 10 minutes. At T+10 minutes some periodic process or whatever removes that entry, because there's no way to check that some subscribe uses it. That reaction returned by that call to subscribe gets re-evaluated and now returns nil. There were 0 user actions, but the UI has changed because some cache entry has expired.

Thank you - yes that makes sense. I guess I was planning to write a component that knows that if the result is nil to show a busy spinner until it isn’t nil anymore. These would be parameterised subscriptions with a unique identifier for the ‘thing’.

But that wouldn't solve the problem. What will re-fetch the data? Otherwise, the spinner will be there forever. You can't solve it without knowing exactly what data is used. Regular caches are plain data storages. app-db is, on the other hand, a data storage wrapped in a Reagent atom, which makes any change to it to automatically propagate everywhere. That's not the case for regular caches, so regular caching approaches simply won't work.

You can create some soft of a countdown latch for each and every entry, where the number shows how many clients that particular entry has. When the counter reaches 0, the entry is removed. But it will require quite some work, and will require you to make subscriptions that alter app-db, which is essentially discouraged.

I did something similar a few years ago. But it deals with normalized data with a well-defined schema with an additional constraint that IDs of the items that are actually used are stored in a predefined location. With that, I was able to periodically remove everything from the storage whose ID is not in the list. But such an approach won't work in situations where you need to subscribe to random data that's not defined exactly by an ID an that list.

Hmmm thank you - that’s really helpful and I appreciate it. Perhaps I’ll just track the size and wipe out the app-db if it exceeds a pre-defined maximum - and the way I already have this configured is to automatically trigger a fetch event if a defined entry is not available anyway. The view components are designed to cope with an entry not being available and can trigger a refetch.

Or maybe not bother and see if it is genuinely a real problem!

Thank you. Really helpful to talk it through.

No problem. Another suggestion - to prevent the view from being in the "loading" state periodically, instead of wiping, re-fetching, and placing data in app-db just fetch and swap the data.

Yes. Actually bandwidth being what it is, that is an even better idea. Thank you.

you could tie the data to the lifetime of the subscription

well, actually, re-frame caches those so maybe not.

if you used a normal ratom you definitely could

Yes - I had originally wondered whether I could hook into the subscription lifecycle or just do it independently of the app-db. Thank you

I would try it out

Reagent reactions have a notion of "disposal" where once reagent's runtime detects that no one is listening to it, it will remove itself from listening to other reactions/ratoms and optionally run an on-dispose function if present

it's not super well documented, and I'm not sure how it would work with with re-frame's subscriptions due to what I mentioned above about how re-frame caches reactions created by subscriptions

see make-reaction here: http://reagent-project.github.io/docs/master/reagent.ratom.html#var-make-reaction

Ahhh! That looks very helpful! Thank you very much - I’ll have a look!

question about async-flow. I'm seeing a situation where two async flows - concurrent "instances" of the same async flow - are interfering with each other. We perform the initial event twice, but then perform the secondary event just once, since the loser of the race sees that the winner already did it. (And the rule says to halt on that event.) One solution could be to thread in a uuid and match on that - to make the :seen? events distinct and only halt if we've seen "our" event. Another approach could be to unwind the async flow rules into callbacks. Any one have an opinion on this? Or see an option I've missed? I'm wondering in particular if async-flow lib already has a way to distinguish one instance of a flow from another.

i do see [:async-flow/id-n ] events coming through, where n identifies one instance of the flow from another. can i tap into that?