Clojurians Log v2

👋 Does anyone have experience optimizing re-frame for systems that are based on data streaming? Basically when granular updates are being streamed to a client

I'm observing a problem where an atomic update causes a chain of recalculations in re-frame subscriptions. This gets slow when parts of the chain are doing any kind of collection transformations (indexing, mapping etc). Right now I'm experimenting with streaming subscriptions, basically applying transducing approach to updates propagation in re-frame subscriptions. Curious to learn if anyone is doing something in that area as well.

Do you use layer 3 subscriptions?

Sure, that's what I meant by "a chain of subscriptions"

Perhaps, that chain needs to be reorganized. Layer 2 subs should never do any heavy lifting in general, it should be left to layer 3. If all is organized correctly, you will see recomputation only of the subs that actually do have their inputs changed.

another approach here could be to fuse together all data transformations into a single level-3 subscription, but that has to be measured first to prove efficiency

If those transformations must always happen together - yeah, sure. If not, then you will likely see unnecessary computation.

of course, the issue comes from a chain of level-3 subscriptions where on every step data is being extracted, indexed etc there are a couple of things I have in mind to solve the local problem, but here I'm more interested in more generic approach, if it exists

Working on a similar app and the way we’re dealing with the problem is to do nothing to the data. No indexing no mapping. I guess it depends on how you use the data, but for my use case I rather loop and map on the fly when I actually need to consume some state than running many transformations many times a second

Not sure if it will help for you, but I wrote https://github.com/riverford/compound in part to address this problem. Instead of doing indexing in subscriptions, data is indexed as it goes into the re-frame db, subscriptions then tend to be simple gets.

With the right sub tree, you won't see any unnecessary transformations - only the subs that are in use will call their handler fn.

@lucio Thats a good point. In my case the state is consumed (reflected in UI) constantly, meaning that every granular update causes an update on a screen.

Ah, and yes, I also index the data right when it gets into the DB. But I guess this approach might or might not be worth it depending on how much data there is.

@roman01la If all else fails, perhaps you could group all necessary transformations together and run them in a web worker.

lol no, de/serializing EDN between UI process and a web worker would be more expensive

Oh, EDN is infamously slow as well. :) JSON-based protocol, like Transit, would absolutely perform better.

unfortunately it's still slow, depending on how much data is being processed of course

I have screens that needs to react on the current time, with a precision of a second. I want to be able to register subscriptions based on the current second. I can write the current time to the db every second, but wonder if that is really a wise thing to do. Thoughts? Questions?

If you really want to use a sub, then writing to the app-db is the right thing. You can still use regular ratoms though. Whether it's something you want to do depends on whether you consider the current second a part of your app's state. If you're worried about performance implications, make sure to use layer 3 subs for anything that does anything more complicated than (get-in db [:a :b :c]).

Thanks. So a subscription such at :current-time, then the components would use subs that in turn use that sub?

I am using a ratom now, but find that I recreate a parallell subscription model with the way the different components rely on this ratom and logic using it….

Yes, or you may use :current-time directly in a view if it e.g. has to display the time.

So that is still layer 3, or just that it is worth the performance cost there?

Not sure I understand your question. Layer 2 is a sub that gets app-db as its input. Layer 3 is a sub that gets other subs as an input. Layer 2 subs get recomputed on every single app-db change. That's why they should stay as simple as possible. Layer 3 subs get recomputed on every change of their layer 2 or layer 3 input subscriptions, also called input signals. Which usually happens much less often than a change of app-db.

could you use the current-time ratom as an input to your subscriptions?

using the re-frame app-db could introduce a lot of delay to your timer

so if you need to be accurate, I would eschew using events/app-db. I might even eschew using reagent, since rendering is also async and buffered

Technically you could. But it doesn't sound right (especially if you think of time as a part of your app's state) and it might introduce some issues with re-frame-10x or similar things. > using the re-frame app-db could introduce a lot of delay to your timer How exactly? And how lot is "a lot"?

@p-himik, I don’t think I know enough about this to ask coherent questions… 😃 Anyway, we have quite a lot of layer 2 subscriptions in the app. That will become extra problematic if I would write to the db every second, is that correctly understood of me from your advice?

It depends on what those layer 2 subs are doing and what you mean by "problematic". If it all fits in a single animation frame then it will all be fine - no matter whether it takes 1ms or ten times as much.

@lilactown precision is not important here. I need to update things when the user expects it to happen, and the Ux can handle missing even several seconds, I think.

I think most of our subscriptions are cheap, @p-himik. I guess I need to try it and see if I run into troubles or not.

re-frame-10x is out of my reach anyway. This is a React Native app.

I think, however, that I think of time as part of the app state, so that is a very good point.

I think you should be able to use re-frame-10x via https://github.com/flexsurfer/re-frisk-remote

I’ll check that out!

> How exactly? And how lot is "a lot"? if you're dispatching a re-frame event on every tick, then you have to wait for it to go through re-frame's event queue (non-deterministic depending on how busy the event queue and main thread are), then you're going to update a large map (slower than updating an atom), then you're going to go through reagent's async render queue (non-deterministic again), then it's handed to React...

it probably wouldn't be much delay on my MBP but on an old machine or smart phone, you might see ticks w/ a noticeable delay

because of this, you I would suggest different solutions based on whether you're just updating a value every ~second, or you're showing a value based on the current time

if you actually care about the time, you probably don't want to store the time in the app-db or a ratom at all; rather, you want to trigger a render and in the render read the current time. a counter would be better

I'm not prescribing, but describing some complexity I see here. I've tried to implement e.g. timers and when you're on a busy page, seeing each "seconds" tick take 1.3-1.8s to render was jarring for the user

we went with tracking a number of ticks and reading the current time during render, so that we could show a more accurate time

Thanks!

@pez I would start with a simple update to app-db and see how you go. I suspect it will be fine. And it is very simple. If you do discover you have a performance problem, then you could do something more exotic like ... subscriptions can be based on an atom. So don't put the time updates into app-db, and instead put it into a separate atom, but deliver it via a subscription. Assuming you have this which you update every second via a setInterval (def the-atom-into-which-you-put-current-time (r/atom "")) You could:

(def-sub 
  :current-time 
  (fn []
     the-atom-into-which-you-put-your-time)
  (fn [t]
    t))

@mikethompson A sub fn is in a reaction anyway, so you can use it directly, without any extra signals.

I was thinking more that you can initially write the code in terms of a subscription to app-db, and then, later, fairly easily tweak that subscription, so the views are none-the-wiser about the change in source.

Ie. it would be a simple change later ... but only if it was proved necessary

Sure, I just meant that :current-time could be written as

(def-sub :current-time
  (fn [_ _]
    @the-atom-into-which-you-put-your-time))

Oh, right. I see what you mean now. That works too,

except that subscription will now run on each change to app-db

Because although that first fn arg is written as _, it is computationally @app-db

Ah, right.

But depending on the application context, this may end up being a distinction without a difference.