datahike

https://datahike.io/, Join the conversation at https://discord.com/invite/kEBzMvb, history for this channel is available at https://clojurians.zulipchat.com/#narrow/stream/180378-slack-archive/topic/datahike
whilo 2020-11-11T18:52:12.061200Z

@geodrome Do you mean with tx the datoms you submit to transact? As @timok mentioned, the hitchhiker-tree actually already incorporates a decoupled flushing mechanism and buffering. We already have a design to do a distributed transactor which does what you want. The concept is very simple, we either just constantly stream the hitchhiker-tree root nodes to all peers (push) or you fetch it when you need it (pull) (which can also be done from the underlying shared memory, i.e. kv-store). That way you can always locally dereference the current db without an additional roundtrip to the transactor and you just access all referenced immutable tree fragments starting from these hitchhiker-tree root nodes. Caching happens locally on the query engine side, so in case where only the buffer of the root node is changed, all your caches stay valid, in case additional nodes are changed you will fetch them from the key value store. We have focused on a baseline datahike REST server as transactor now to be available in all environments, even ones where the query engine cannot run locally in process, but you dispatch also via REST. But maybe we have overlooked some aspect and could do caching better, do you see problems with our approach?