Clojurians Log v2

has anyone ever implemented this benchmark for re-frame? https://github.com/krausest/js-framework-benchmark

I'm kinda interested to see how the subscribe stuff performs in benchmark conditions

not interested in the rendering/react aspects at all but I'm interested to see what the performance of mounting/unmounting queries looks like

@thheller Not the same, but there was this (out of date) analysis: https://www.freecodecamp.org/news/a-real-world-comparison-of-front-end-frameworks-with-benchmarks-e1cb62fd526c/

@jacek.schae updates A RealWorld Comparison of Front-End Frameworks every year, and may have done other benchmarks, or would like to.

yeah I saw that one but I was looking specifically for this benchmark

@mikethompson https://medium.com/dailyjs/a-realworld-comparison-of-front-end-frameworks-2020-4e50655fe4c1

mostly because its an easy setup to see some common situations in webapps in benchmark conditions

same thing only newer

(eg. no app should have 1000 items rendered in a list but in a benchmark its useful to see)

@huxley thanks

the real world bench is useful for other stuff and probably more useful overall but I'm specifically looking for examples that stress the subscribe mechanisms

I can set write it myself, just wondering if someone else did before me

shouldn't the results be essentially identical to the reagent?

the rendering yes but thats not what I'm interested in

FWIW I did this benchmark for the stuff I'm working on and have two variants. one that would be similar to re-frame (full) and the other pure reagent (light). both perform more or less the same overall but full performs better in updates (eg. select row) but light performs better in mount/unmount (create/clear rows). I would expect re-frame to be somewhat similar in results when compared to reagent. https://github.com/thheller/js-framework-shadow-grove

At some point, I'll free re-frame subscriptions from using reagent reactions

that would be neato. then I could just plug it into my stuff 🙂

I'll be moving to using one map for all the subscriptions. The keys in this map will be the subscription vectors And the values in the map will be a map representing the subscription, including a dirty? flag, a set of dependent (subscriptions, identified by their subscription vectors) s, etc

So simple

yeah. as far as I understand one "common" bottleneck for re-frame apps is when one modification results in too many subscription updates

eg. in a map of a thousand items you update one and 1000 subscriptions run to figure out if they are dirty. which is fast as long as its just a (get db thing) and compares identical? for 999 items

but if you actually compute something that becomes a bottleneck

Yep

We've never found it to be much of an issue

Because the identical? test shortcuts early

And because we don't display 1000 things (elements of a vector?)

My proposed rework of subscriptions is more about clean up and doing it a much simpler way

Hey, while you are here ... I know you have been working on a framework ...

... Zach Oaks has convinced me that using a map for app-db is ultimaitely a bad idea ... flat data is better

maybe the stuff I tried to describe here with transacted observed would also benefit the re-frame model (it should) https://github.com/thheller/shadow-experiments/blob/master/doc/what-the-heck-just-happened.md

I assume you are referring to his o'doyle rules thing?

Indeed

yeah that looks very interesting but seems impossible to scale this to what an webapp would need (adding/removing rules seems rather expensive)

That was the reason I didn't use DataScript originally. We had too much data. And it took too long to import. But ... The problem with using a map is that it is very "placeful"

I didn't realise quite how placeful until I started writing a tutorial on writing reusable components for re-frame

It struck me a little hard. The difficult part of creating a reusable component in re-frame is that the component's subscriptions and event handlers have to know "where" in pp-db they need to access

placeful

That means it is hard to take components from one app to another

There are things you can do to mitigate this

You can load data into well known places within app-db

well you can always make things reusable by separating that out and just having a render function that takes the data and renders it

and the other one that gets it from wherever and calls it

Yes, correct

These are all suggestions I came up with. BUT ... I was quite struck by this placefulness thing It means the subscriptions and event handlers had to get parameterised (by place)

but I agree that flat is better. my normalized db thingy is one big map with one level of nesting.

yeah but how else would you solve that?

Anyway ... just a thought

Datascript is flat

but then you have to pay everwhere for turning it into maps so you can actually work with the data again

(update thing :foo inc) becomes rather difficult too

Hmm.

There were some libraries which stored "entities" in app-db in a very regular way. I can't find them offhand.

They might yet be a middle ground

Anyway, I don't have an answer yet, but it is on my mind.

yeah I do think so. normalizing the data to one level to avoid duplication of entities seems to be enough. making it flatter to EAV tuples means you need to reconstruct maps all the time where you really just want to do (get db id) (which datascript has a wrapper for to emulate but thats not the same thing)

Ah, one of them was https://github.com/den1k/subgraph

Hmm. I'll have to read up again.

Night

ah that looks good

gn8

https://github.com/noprompt/meander/blob/epsilon/examples/datascript.cljc

currently it looks like everyone is trying to solve the same problem in a similar way

https://github.com/lilactown/autonormal/

oh thanks. didn't see that one yet.

I am currently trying to solve the same problem myself as a hobby ; )

in production we use rum + patched datascript, but it is very slow and tedious

I think I'm struggling with a similar problem atm, where one thing in a collection changes and that causes recalculations in multiple subscription chains

Data is preprocessed first, then indexed in various ways and postprocessed at leafs. We have something that implements incremental indexing at the root, but further change propagation requires calculations from scratch in every subscription. My current idea is to have "streaming subscriptions" that would propagate changes only, similar to the idea behind transducers

The normalized flat db is great, but there is a problem similar to datascript, all subscriptions are recalculated with every change in the db. It is possible to do something like in posh, check each transaction and see if it matches the query. I have no idea how efficient this would be though, but I'm going to test and find out 😉

I'm currently reading the penpot code from cover to cover, they have a slightly different approach to state management there.

exactly based on streaming

Penpot is UXbox right? I think posh's change inference wasn't always correct at the time, curious if that's changed

yes, uxbox is now a penpot

and yes, posh sometimes returned incorrect results, especially when retracting from db

by the way, datascript is not fast at any time, in fact performance is poor

with and without posh

That's my experience with datascript as well

simple flat db in the form of a {?id ?map} searched with meander is just as fast for query and many times faster when it comes to transactions or pull

A small correction - all level 2 subscriptions are recalculated on each app-db change.

Just some potentially useful musings. I ended up rewriting subgraph to accommodate my needs and I'm still quite happy with the outcome. Although, I use it only for the domain data that comes from an RDBMS in an already normalized form. One thing I have found quite helpful is having multiple indices for every foreign key attribute and being able to query based on that. "Give me all items by a coll of IDs" is useful, but sometimes "give me all items that reference that other item" is even better.

Another feature that's very useful for my use cases is to be able to store user-caused changes separately and query the changed state and the original state separately. Sort of like a transaction. It facilitation creation of complex and reusable components where you can enter a bunch of data, see what exactly you've changed, check validation results, and only then save the data.

Just in case - there are also https://github.com/riverford/compound and https://github.com/threatgrid/asami

if it's interesting to y'all, I was inspired by trying to use datascript + re-frame to author this lib: https://github.com/lilactown/autonormal

it normalizes your data into a relatively flat, simple map. and gives an API to easily pull data out using EQL; though of course you can just get-in to get specific data out too

it's not the power of datalog but I found that datalog wasn't what I wanted most of the time anyway when relying on datascript to store my app's entities; it wasn't performant enough

somewhere here I posted autonormal as an example 😉

I see that now 😄 skimmed too fast

i'm hoping to eventually base a "pathom-client" lib on it but in the meantime, I find it moderately useful just for getting out of the "place-oriented" ness that y'all have been talking about

it still relies on parameterizing your subs/events to talk about specific entities tho. which is why i typically only use app-db for domain data and let UI state live in local component state

btw I'm basically just waiting for someone to ask for custom schemas in autonormal. happy to riff on what a good API for that would look like if anyone is interested

I recently wrote my thoughts on pretty much the same topic. Also, pretty much everyone here wrote something that influenced how I think about user interfaces! Thanks! (Tony Kay from fulcro and the minds behind hoplon were also big influences). > An entity can use three main techniques to refer to another entity: nesting, identifiers, and stateful references. > - Clojure Applied Chapter 1 The main difference between using datascript and subscriptions isn't flat vs nested, it's the type of reference that is used. Applications that use datascript as a data model tend to use identifiers as references whereas re-frame uses stateful references. What makes a component "reusable" isn't the type of reference that is used, it's whether or not the information needed to produce a reference is passed as an argument to the component or is hard coded in the component. When using identifiers, passing all the necessary information to build a reference is trivial (it's just the identifier). For data models that use nesting or stateful references, it's less straightforward. The tricky part with users interfaces is that there tends to be a lot of incidental state. Given the choice between implicit state handling that's less reusable and explicit state handling that's more reusable, developers tend to prefer implicit state handling. Ideally, incidental state should be handled implicitly and essential state should be handled explicitly which would make it easier to write reusable components by default. Here's the long version: https://blog.phronemophobic.com/reusable-ui-components.html

Creating components is one thing

Using re-frame, always at some point I have problems with data denormalization and map fatigue in general. On the other hand, when e.g. I use datascript, or other ideas for flat normalized db, the problem is performance and synchronization with backend and/or local-storage. Some data should be excluded from storage, what is trivial in case of nested maps, but more complicated in case of flat structure.

every solution has its weaknesses

@smith.adriane paths are identities in re-frame (paths within app-db)

so is it more correct to say re-frame uses both nesting and stateful references or just paths/nesting?

I mention this only because I'm not sure what "stateful references" are. I know, I know, I should read your page.

I don't actually explain "stateful references"

Ah

Then I'm off the hook :-)

See http://day8.github.io/re-frame/reusable-components/

I just kind of ignore them, but I give re-frame as an example of a library that uses stateful references

so I can update it if I'm wrong

See the link provided above

I actually cite that page in my post, https://day8.github.io/re-frame/reusable-components/#implications

I have actually rewritten that page at some time, but never published the rewrite. I'm stuck on the final part of how best to get around placefulness

it's a tough problem.

Tradeoffs all the way down.

I've been using macros to "track" usage of data derived from props to automatically produce references based on paths/nesting, but I'm not sure it's the right solution

I can remember once spending an entire weekend trying to make Datascript faster, by turning it into a column store database and using the GPU. (GPUs are very fast with vectors) I failed. :-)

And when I say I failed, I mean I failed at the very first hurdle: using the GPU.

Back to identity http://day8.github.io/re-frame/reusable-components/#providing-identity

the other approaches I've looked at: • Om (uses proxies). After using it, it was ok, but had too many caveats for me to want to try again • macros (my current approach) • fulcro has ui components define queries as part of the component definition. I think that approach could also work, but I'd rather the queries were automatically written for me.

Sorry, it was the section above the link I supplied

http://day8.github.io/re-frame/reusable-components/#what-is-identity

I like Clojure Applied's definition: > An entity can use three main techniques to refer to another entity: nesting, identifiers, and stateful references.

I'm not reading that and feeling wiser :-)

path's within an app-db being the same as "nesting"

It's been tough finding good resources on the subject. Re-frame's docs are one of the best resources I could

It has been interesting comparing different approaches. Re-frame mainly focuses on path/nesting references and fulcro focuses on identifers as references

An identity is something which identifies an entity. And it is very context specific.

In C we used pointers

In a database, it is a unique key

In Clojure it is typically a keyword

Or a path of keywords and integers (a path)

I think it's possible to write UI components in a way such that they're independent of which reference type is required, but it's a challenge to design it in a way that it's not an abstract mess.

The identity depends on the application's data model, not the language (although languages will usually have a lot to say about what kind of data model is idiomatic).

Yeah, that's a distinction without a difference for me. Practically speaking. But maybe I'm missing something.

I have to go but I present this challenge ... What is the identity of a certain collection of customers (the big ones)

So just to be clear, I'm not talking about individual customers

I'm talking about the collection of these customers

is it a subset or the full collection?

Because I might need to distinguish good customers from bad ones

Yeah, so how do I identify the two collections.

Now I want to show the user this collection, now I want to show the other collection

It depends on the implementation. A single customer can be ID'ed by [:customer 7], for example. A collection of customers by [:customer [1 2 3]], for example. Of course, this precludes having a single customer having an ID that's a collection, but I think there can be reasonable limitations. Another way would be something like [[:customer 1] [:customer 2] [:customer 3]], similar limitations apply. And there are many other ways - again, depends on the implementation. And perhaps the particular set of limitations/idiosyncrasies you're willing to deal with.

;; uses nesting and identifiers
(def data
  {:customers {0 {:name "Bob"}
               1 {:name "Mary"}
               2 {:name "Sue"}}
   :lists {0 {:name "good"
              :customers [[:customers 0]
                          [:customers 2]]}
           1 {:name "bad"
              :customers [[:customers 1]]}}})

;; reference is [:lists 0] or [:lists 1]

;; alternative data model
(def data
  {"b4931d86-efca-4e86-b397-e54315673d32" {:name "Bob"}
   "133e9085-9481-4a64-befd-ce3b071a4c8a" {:name "Mary"}
   "f3c630f5-35e2-4600-8bf6-dc6fdbe76dae" {:name "Sue"}
   
   "b97c51b5-1078-45a9-a325-b8cb71f0959d" {:name "good"
                                           :customers ["b4931d86-efca-4e86-b397-e54315673d32"
                                                       "f3c630f5-35e2-4600-8bf6-dc6fdbe76dae"]}
   "51bf61bf-5163-4fd9-a52a-060de35ab7f7" {:name "bad"
                                           :customers ["133e9085-9481-4a64-befd-ce3b071a4c8a"]}})

;; references are "b97c51b5-1078-45a9-a325-b8cb71f0959d" and "51bf61bf-5163-4fd9-a52a-060de35ab7f7"

;; stateful references
;; not encouraged!
(let [bob (ref {:name "bob"})
      mary (ref {:name "Mary"})
      sue (ref {:name "Sue"})]
  (def customers [bob mary sue])
  (def good-customers (ref [bob sue]))
  (def bad-customers (ref [mary])))

;; references are good-customers and bad-customers

@p-himik yeah I was imagining that there might be two collections at different points within app-db. I was attempting to draw out that path is identity in re-frame, collections have identity. So too do the customer entities with them.