👋 Does anyone have experience optimizing re-frame for systems that are based on data streaming? Basically when granular updates are being streamed to a client


I'm observing a problem where an atomic update causes a chain of recalculations in re-frame subscriptions. This gets slow when parts of the chain are doing any kind of collection transformations (indexing, mapping etc). Right now I'm experimenting with streaming subscriptions, basically applying transducing approach to updates propagation in re-frame subscriptions. Curious to learn if anyone is doing something in that area as well.

p-himik 2021-02-26T12:19:17.007Z

Do you use layer 3 subscriptions?


Sure, that's what I meant by "a chain of subscriptions"

p-himik 2021-02-26T12:23:28.011100Z

Perhaps, that chain needs to be reorganized. Layer 2 subs should never do any heavy lifting in general, it should be left to layer 3. If all is organized correctly, you will see recomputation only of the subs that actually do have their inputs changed.


another approach here could be to fuse together all data transformations into a single level-3 subscription, but that has to be measured first to prove efficiency

p-himik 2021-02-26T12:24:49.012300Z

If those transformations must always happen together - yeah, sure. If not, then you will likely see unnecessary computation.


of course, the issue comes from a chain of level-3 subscriptions where on every step data is being extracted, indexed etc there are a couple of things I have in mind to solve the local problem, but here I'm more interested in more generic approach, if it exists

Lu 2021-02-26T12:26:08.014300Z

Working on a similar app and the way we’re dealing with the problem is to do nothing to the data. No indexing no mapping. I guess it depends on how you use the data, but for my use case I rather loop and map on the fly when I actually need to consume some state than running many transformations many times a second

danielneal 2021-02-26T12:27:34.014500Z

Not sure if it will help for you, but I wrote in part to address this problem. Instead of doing indexing in subscriptions, data is indexed as it goes into the re-frame db, subscriptions then tend to be simple gets.

p-himik 2021-02-26T12:27:35.014700Z

With the right sub tree, you won't see any unnecessary transformations - only the subs that are in use will call their handler fn.


@lucio Thats a good point. In my case the state is consumed (reflected in UI) constantly, meaning that every granular update causes an update on a screen.

p-himik 2021-02-26T12:28:17.015200Z

Ah, and yes, I also index the data right when it gets into the DB. But I guess this approach might or might not be worth it depending on how much data there is.

p-himik 2021-02-26T12:29:10.015500Z

@roman01la If all else fails, perhaps you could group all necessary transformations together and run them in a web worker.


lol no, de/serializing EDN between UI process and a web worker would be more expensive

p-himik 2021-02-26T12:32:01.016100Z

Oh, EDN is infamously slow as well. :) JSON-based protocol, like Transit, would absolutely perform better.


unfortunately it's still slow, depending on how much data is being processed of course

pez 2021-02-26T15:22:36.019800Z

I have screens that needs to react on the current time, with a precision of a second. I want to be able to register subscriptions based on the current second. I can write the current time to the db every second, but wonder if that is really a wise thing to do. Thoughts? Questions?

pez 2021-02-27T08:28:12.029400Z

Thanks for all these inputs and elaborations, all of you. This is great!

pez 2021-02-27T08:46:25.029900Z

@lilactown in my use case I have components that should change their style, content, and behaviour at certain times of the day. So it is not really about an update every second. I want the updates to happen on even minutes (not every minute), because it felt weird when the clock on the phone switched minute and the changes happened up to a minute later.

pez 2021-03-02T11:14:27.117800Z

Hmmm. I was very happy with how the app behaved until I saw this in the log:

Warning: Please report: Excessive number of pending callbacks: 501. Some pending callbacks that might have leaked by never being called from native code: {"471":{"module":"UIManager","method":"configureNextLayoutAnimation"},"665":{"module":"UIManager","method":"configureNextLayoutAnimation"},"1192":{"module":"UIManager","method":"configureNextLayoutAnimation"},"1553":{"module":"UIManager","method":"configureNextLayoutAnimation"},"1896":{"module":"UIManager","method":"configureNextLayoutAnimation"},"1897":{"module":"UIManager","method":"configureNextLayoutAnimation"},"1923":{},"1925":{"module":"UIManager","method":"configureNextLayoutAnimation"},"1934":{"module":"UIManager","method":"configureNextLayoutAnimation"},"1936":...
Started a thread in #reagent about it, since it doesn’t matter if I use a re-frame sub or react on the ratom directly.

p-himik 2021-02-26T15:26:12.019900Z

If you really want to use a sub, then writing to the app-db is the right thing. You can still use regular ratoms though. Whether it's something you want to do depends on whether you consider the current second a part of your app's state. If you're worried about performance implications, make sure to use layer 3 subs for anything that does anything more complicated than (get-in db [:a :b :c]).

pez 2021-02-26T15:31:27.020100Z

Thanks. So a subscription such at :current-time, then the components would use subs that in turn use that sub?

pez 2021-02-26T15:33:11.020300Z

I am using a ratom now, but find that I recreate a parallell subscription model with the way the different components rely on this ratom and logic using it….

p-himik 2021-02-26T15:35:02.020500Z

Yes, or you may use :current-time directly in a view if it e.g. has to display the time.

pez 2021-02-26T15:40:16.020700Z

So that is still layer 3, or just that it is worth the performance cost there?

p-himik 2021-02-26T15:53:08.020900Z

Not sure I understand your question. Layer 2 is a sub that gets app-db as its input. Layer 3 is a sub that gets other subs as an input. Layer 2 subs get recomputed on every single app-db change. That's why they should stay as simple as possible. Layer 3 subs get recomputed on every change of their layer 2 or layer 3 input subscriptions, also called input signals. Which usually happens much less often than a change of app-db.

lilactown 2021-02-26T17:22:31.021100Z

could you use the current-time ratom as an input to your subscriptions?

lilactown 2021-02-26T17:23:54.021300Z

using the re-frame app-db could introduce a lot of delay to your timer

lilactown 2021-02-26T17:25:07.021500Z

so if you need to be accurate, I would eschew using events/app-db. I might even eschew using reagent, since rendering is also async and buffered

p-himik 2021-02-26T17:28:59.021800Z

Technically you could. But it doesn't sound right (especially if you think of time as a part of your app's state) and it might introduce some issues with re-frame-10x or similar things. > using the re-frame app-db could introduce a lot of delay to your timer How exactly? And how lot is "a lot"?

pez 2021-02-26T17:29:01.022Z

@p-himik, I don’t think I know enough about this to ask coherent questions… 😃 Anyway, we have quite a lot of layer 2 subscriptions in the app. That will become extra problematic if I would write to the db every second, is that correctly understood of me from your advice?

p-himik 2021-02-26T17:30:00.022200Z

It depends on what those layer 2 subs are doing and what you mean by "problematic". If it all fits in a single animation frame then it will all be fine - no matter whether it takes 1ms or ten times as much.

pez 2021-02-26T17:32:21.022500Z

@lilactown precision is not important here. I need to update things when the user expects it to happen, and the Ux can handle missing even several seconds, I think.

pez 2021-02-26T17:33:43.022700Z

I think most of our subscriptions are cheap, @p-himik. I guess I need to try it and see if I run into troubles or not.

pez 2021-02-26T17:34:43.022900Z

re-frame-10x is out of my reach anyway. This is a React Native app.

pez 2021-02-26T17:35:34.023100Z

I think, however, that I think of time as part of the app state, so that is a very good point.

p-himik 2021-02-26T17:45:44.023300Z

I think you should be able to use re-frame-10x via

pez 2021-02-26T17:48:35.023600Z

I’ll check that out!

lilactown 2021-02-26T18:20:17.023800Z

> How exactly? And how lot is "a lot"? if you're dispatching a re-frame event on every tick, then you have to wait for it to go through re-frame's event queue (non-deterministic depending on how busy the event queue and main thread are), then you're going to update a large map (slower than updating an atom), then you're going to go through reagent's async render queue (non-deterministic again), then it's handed to React...

lilactown 2021-02-26T18:21:08.024Z

it probably wouldn't be much delay on my MBP but on an old machine or smart phone, you might see ticks w/ a noticeable delay

lilactown 2021-02-26T18:24:37.024200Z

because of this, you I would suggest different solutions based on whether you're just updating a value every ~second, or you're showing a value based on the current time

lilactown 2021-02-26T18:25:32.024400Z

if you actually care about the time, you probably don't want to store the time in the app-db or a ratom at all; rather, you want to trigger a render and in the render read the current time. a counter would be better

lilactown 2021-02-26T18:27:05.024600Z

I'm not prescribing, but describing some complexity I see here. I've tried to implement e.g. timers and when you're on a busy page, seeing each "seconds" tick take 1.3-1.8s to render was jarring for the user

lilactown 2021-02-26T18:28:00.024800Z

we went with tracking a number of ticks and reading the current time during render, so that we could show a more accurate time

p-himik 2021-02-26T18:46:53.025Z



@pez I would start with a simple update to app-db and see how you go. I suspect it will be fine. And it is very simple. If you do discover you have a performance problem, then you could do something more exotic like ... subscriptions can be based on an atom. So don't put the time updates into app-db, and instead put it into a separate atom, but deliver it via a subscription. Assuming you have this which you update every second via a setInterval (def the-atom-into-which-you-put-current-time (r/atom "")) You could:

  (fn []
  (fn [t]

p-himik 2021-02-26T23:16:08.025700Z

@mikethompson A sub fn is in a reaction anyway, so you can use it directly, without any extra signals.


I was thinking more that you can initially write the code in terms of a subscription to app-db, and then, later, fairly easily tweak that subscription, so the views are none-the-wiser about the change in source.


Ie. it would be a simple change later ... but only if it was proved necessary

p-himik 2021-02-26T23:23:35.026800Z

Sure, I just meant that :current-time could be written as

(def-sub :current-time
  (fn [_ _]


Oh, right. I see what you mean now. That works too,


except that subscription will now run on each change to app-db


Because although that first fn arg is written as _, it is computationally @app-db

p-himik 2021-02-26T23:28:35.027900Z

Ah, right.


But depending on the application context, this may end up being a distinction without a difference.