datomic

Ask questions on the official Q&A site at https://ask.datomic.com!
2021-04-25T16:44:31.389400Z

Hey team, are there any resources, or have there been any previous attempts at turning datalog queries into realtime queries? Problem to figure out what queries changed based on new fact seems daunting, but potentially doable with the right hacks. If someone has thoughts/ resources, would love to hear!

2021-04-27T06:49:49.413400Z

I tried playing with the declarative-dataflow stuff a couple times a while back, but it's become a bit outdated now, hard to build.

1👍
2021-04-25T16:45:32.389500Z

Isn’t this straightforward in principle?

2021-04-25T16:47:37.389700Z

Go over the txn log. Get an as-of db for each txn. Run each query. Compare to the as-of db for the next txn.

2021-04-25T16:48:20.389900Z

Should even be parallelizable. (partition 2 1 dbs) and run each pair of queries in a thread.

favila 2021-04-25T16:55:09.391Z

The technique is called differential dataflow: https://www.nikolasgoebel.com/2018/09/13/incremental-datalog.html

11👆
favila 2021-04-25T16:56:00.392Z

This is one impl for datomic using some Rust parts

favila 2021-04-25T16:57:05.393Z

The differential dataflow library and a server based on it are written in Rust

2021-04-25T17:02:26.393700Z

Ohmagad. Very exciting, thanks team will dive deeper (re: running each query—initially assumed this would be too expensive, but looks like this essay has an implementation to do this incrementally.)!

Joe Lane 2021-04-25T18:16:36.396800Z

@stopachka I’d love to know what you mean by “real-time queries” and the capability you’d like to have in your system to solve some real problem. And what the problem is :)

2021-04-25T18:33:36.398700Z

Hey Joe, I was thinking along the lines of https://tonsky.me/blog/the-web-after-tomorrow/ , and wanted to see what would be possible to hack together today. May dive deeper this week!

kenny 2021-04-25T21:26:52.399300Z

For those interested, we switched to on-demand mode 4 days ago and all the DDB provisioned throughput problems have gone away 🙂 As an added bonus, our ddb costs dropped ~30%.

1👏
kenny 2021-04-25T21:33:10.399600Z

Thought I'd persist this question on ask.datomic so others can reference it and add their opinion: https://ask.datomic.com/index.php/606/recommended-way-to-handle-db-ident-enums-when-used-with-pull

1
Joe Lane 2021-04-25T21:44:34.400300Z

Great to hear Kenny !