Clojurians Log v2

@tonsky What do you consider state of the art these days for reactive UIs? I found an old tweet about precept, which you seemed to suggest at that time was better than things built on datascript. Anything new?

I’m trying to get ahead of optimizations I know I’ll want to do later… is there an understood way to detect whether a particular query has changed based on tx-data returned from a transaction?

@lilactown Currently looking into similar things. You might find https://github.com/comnik/clj-3df interesting, as well as https://github.com/CoNarrative/precept

More reading: https://timelydataflow.github.io/differential-dataflow/

This is actually what motivated my above question too.

precept does look cool but I’m looking for something uncoupled from reagent

I don’t follow the field of reactive datalog, unfortunately. Can’t recommend anything

yeah I’m not even really interested in the reactive part, more just trying to see if there’s a way to detect a query has changed that’s cheaper than executing the query itself

pull-based is a-ok for my use case

I’m just worried that as my app grows and runs over time, the DB will grow and number of queries will start to slow down my app

There is also https://github.com/mpdairy/posh which claims it now works with multiple frontends.

"Posh is also very fast because the in-component data queries only run when the database is updated with relevant data (found by pattern matching on the tx report)."

Yeah I actually rediscovered an issue I had filed like 6 months ago 😛 https://github.com/mpdairy/posh/issues/35

it’s made more complicated by the fact that React really doesn’t want to let be in control of when computation should occur

^ Yes, however, Posh is doing rather primitive pattern matching and gets things wrong sometimes. Also, when a new datom matches a query, it still has to rerun the query to get complete results. So, better than nothing, but not ideal. 3df is very promising, and ultimately differential dataflow is probably the right solution to the problem (this was our leaning for datsync, before seeing the 3df talk at the conj this year). However, the down side there is that since its implemented in rust, you can't run clojure functions in reactive queries. Apparently, the rust part has been compiled to web assembly, but I don't know what the front-end deployment scenario looks like there; likely somewhat nontrivial.

Would be great to have our own solution in cljc, but means implementing the underlying differential dataflow logic, which is not trivial.

Precept is a rules engine, so a little different in scope, and RETE also doesn't support the full range of expressiveness as datalog (recursive rules, for example).

yeah I’m worried that in order to get a truly performant + ergonomic solution I would have to build something that integrates a little more tightly with React

which I’m not smart enough to do yet

Nah; I don't think that's necessary. Are you a Reagent user? Reactions should be all you need. Honestly, if we're able to get 3df running on the browser, we may be good with a sort of hybrid solution where simpler queries that work on 3df get run through that, but those that need clj functions go through something more heuristic, like Posh (or just get rerun every time).

I'm not. I'm using just React and Hooks

Well, there's your problem right there 😉

j/k; Posh is designed to be fairly modular, and bindings have been written for both Reagent and Rum. I'm guessing someone could create a minimal set of bindings directly upon React + Hooks.

hehe

I’ll have to look deeper. my concern is that I’m trying to use the new React Concurrent features, which makes relying on external subscriptions problematic

Ah; interesting. Thanks for sharing that context.

I think ideally I would have just some API that I can call:

(if (query-changed? tx-info query)
  ;; execute new query
  (q new-db query)
  ;; use cached
  cached-value)

this way I could store the DB in React state, and components that queried it could check to see if any relevant datoms updated / were added

this might not be feasible tho 😕

how does posh determine a query has changed? you said it does some kind of pattern matching?

Datascript has a tx-listener/callback feature. So basically, every time a tx hits the db, it fires any attached listeners. Posh attaches such a listener that looks at the datoms to see if any of them match patterns associated with each of the queries to which you're subscribed. If there's a match, the queries are rerun, and if there's a change, (in the case of the reagent plugin) an r/atom representing the result is updated, and everything flows from there according to the standard Reagent machinery.

The problem with this is that the pattern matching that Posh is doing is 1/2 way there towards a 1/2 baked implementation of a Datalog engine (Datalog is, after all, a pattern matching language). So you can pretty quickly see how much nicer it would be if everything was just operating in a real incremental/differential update flow.

Re: 3df Just to be certain, the software we need to run in the browser is the 3df server, right? The clj-3df lib doesn’t do anything for us? Or am I misunderstanding?

You're more or less right

The server is what does the actual differential dataflow evaluation

And is the part that's in rust

The clj-3df lib is just making calls out to that

so it sounds like if I were to do a similar solution as posh (not fully incremental) that would work with Concurrent React, I would want to get rid of the subscription mechanism. I want something that I can synchronously transact, match, and query.

@niko963 - Am I recalling correctly about the feasibility of getting the server part running on the browser? I seem to remember you saying that someone had already tried compiling to web assembly, but that there would be more work needed for a smooth deploy path. Do I have that right, and is there anything else you can add about the path forward there?

the listener/subscription sounds like basically a way of optimizing the computation of what has changed for the (possibly many) listeners on one query

I might be able to get back some of that performance using memoization

You could probably use Posh as a lib, and run the checks against it that way. You wouldn't have to reimplement it necessarily.

that would be what I want yes 😅

I need to have time to sit and look at posh’s source

the differential data flow stuff sounds really interesting, but I don’t really get it… yet. do you think it would also work in this synchronous + immutable way?

actually if it’s wasm, it would I think be running on another worker so it would have to be calling me back anyway

Yup; 3df will have to be async I'm afraid.

that would probably be OK for my use case. I’m slowly giving up on the idea of having “one DB to rule them all” when it comes to React applications

Is there not a way to plug in asynchronous updates via React Hooks? I thought that was part of the whole point? (Disclaimer: I know very little about this newfangled Hooks business)

there is, you just lose out on certain things