Clojurians Log v2
datomic
Hey team, are there any resources, or have there been any previous attempts at turning datalog queries into realtime queries? Problem to figure out what queries changed based on new fact seems daunting, but potentially doable with the right hacks. If someone has thoughts/ resources, would love to hear!
I tried playing with the declarative-dataflow stuff a couple times a while back, but it's become a bit outdated now, hard to build.
Isn’t this straightforward in principle?
Go over the txn log. Get an as-of db for each txn. Run each query. Compare to the as-of db for the next txn.
Should even be parallelizable. (partition 2 1 dbs) and run each pair of queries in a thread.
The technique is called differential dataflow: https://www.nikolasgoebel.com/2018/09/13/incremental-datalog.html
This is one impl for datomic using some Rust parts
The differential dataflow library and a server based on it are written in Rust
Ohmagad. Very exciting, thanks team will dive deeper (re: running each query—initially assumed this would be too expensive, but looks like this essay has an implementation to do this incrementally.)!
@stopachka I’d love to know what you mean by “real-time queries” and the capability you’d like to have in your system to solve some real problem. And what the problem is :)
Hey Joe, I was thinking along the lines of https://tonsky.me/blog/the-web-after-tomorrow/ , and wanted to see what would be possible to hack together today. May dive deeper this week!