Clojurians Log v2

@martinklepsch I’ve been working on https://github.com/lilactown/flex/tree/main/packages/flex-core when I have time which is v advanced, v alpha but sounds more like what you’re looking for. it might inspire you

mostly it is v slow 😅

@lilactown interesting, thanks! that looks very similar to derivatives in that they’re both attempting to implement some kind of incremental dataflow. I haven’t thought about scheduling much at all tbh but a lot of this makes sense to me.

What makes it so slow? the basic design would make me think it should be fast?

The https://github.com/martinklepsch/derivatives/blob/master/src/org/martinklepsch/derivatives.cljc is 120 lines if you ignore some of the fringes but it’s all synchronous(?) via add-watch! and I am not doing anything special to order/batch updates. I think the ordering should mostly work though since the low level computations are updated first, then the ones that depend on those etc.

There are multiple slow parts I think. The part that stands out to me, if memory serves, is the calculate! function used to calculate and set the new state and dependents of each computation

that and the heap algorithm I use to order updates

the heap is important for ordering because even if you do what you’re saying (breadth first) you can end up in diamond dependencies that require extra work

there’s an inherent cost to the way flex stores state in a dynamic variable that will always be slower than storing the state on the computation object itself

i.e. every dereference requires a lookup in this dynamic var

my hypothesis is I can get it Fast Enough:tm: that the flexibility of being able to use multiple environments for e.g. testing, or reusing computations between different parts of a system, is worth it

at Pitch we are using re-frame, but it shows its weakness due to the nature of dataflow in the app: since Pitch is collaborative realtime app we are sending data events between clients, the whole state of user’s workspace is calculated out of atomic objects (folder, presentation, user, permission, font, style, etc. records). When there’s a new object in memory re-frame has to recalculate related indexes from scratch. While that still works to prevent change propagation in subscriptions graph, recalculating indexes is still quite expensive.

in our case ideally we want some sort of incrementality when calculating derived state

we are also exploring a completely diff approach: event based state updates, basically Kafka on the frontend where instead of randomly writing into an app db and relying on subs graph to figure out what changed, we would update state atomically based on events signalling what exactly is about to change (what type of an object is added/updated/removed)

my conclusion so far is that change inference via subs graph doesn’t scale well in terms of performance, but it’s very easy to use

The Kafka/change feed approach is also something I’ve been thinking about

One problem with it is a bit that it’s nice for lists of similar entities but probably a bit trickier for “streams” that combine data from multiple sources. Like ad-hoc subscriptions become harder

the tricky part of using an event ledger is that your events need to be fine grained enough to allow change detection to be precise

Would be very interested to have a chat about this with some folks some time if you’re down 🙂

ah I think we just said similar thing :D

We can upload it to youtube afterwards or something 😄

I wish I had time today

I think the timely-dataflow / differential-dataflow stuff is the state of the art here

IIRC it combines a log of changes a la kafka with a dataflow graph to allow far more efficient (CPU time) of incremental computations at the cost of additional memory

it’s been a little bit since I read the paper

If I was to start a multi-user UI collaboration project, I would have a hard look at https://croquet.io/sdk/docs - it use to be a 3D peer-to-peer world implementation in Smalltalk - https://web.archive.org/web/20090220083519/http://www.croquetconsortium.org/images/2/2b/2003_Croquet_Collab_Arch.pdf

“Croquet is a synchronization system for multiuser digital experiences. It allows multiple users to work or play together within a single shared distributed environment, and it guarantees that this distributed environment will remain bit-identical for every user.”

Yay, finding myself in a new/old rabbit hole: https://www.youtube.com/watch?v=9xWDTwdqfno