Clojurians Log v2

May I ask what backend you’re targeting?

@adam622 I'm targeting firebase.

Still a work in progress https://github.com/alekcz/konserve-fire

That’s awesome! I’ve been spending time lately looking into using Datahike on a new project, exploring how to implement konserve storage adapters, with Firestore and FoundationDB in mind. Was recently reading konserve storage protocol implementations from datahike and/or a konserve storage adapter, trying to understand how stored data is organized and mapped to the semantics of the storage. Am I correct that batches of what Datahike sees as a single key-value are serialized and stored in a single key in the storage?

@adam622 yeah it looks like it. I only got it working late last night.

I'm now wondering what happens when the value in the kv is at capacity

Yup. FoundationDB also has byte-size limits for values and the transaction as a whole.

@adam622 Yes, the underlying hitchhiker tree stores tree fragments under single key-value pairs. @alekcz360 That is super cool. I have hit this -update-in also in the past, but decided to simplify the API by having just one protocol instead of multiple arities. I would still like to know what the problem is. It seems to work in some setups.

We are having a group call on Friday discussing a metadata extension to the konserve protocols to support our concurrent GC (basically we need to store last edit time of key value pairs). Current backend implementations should need minor changes, but we also need to write migration code. I would recommend getting a backend to run now anyway and we can help you to update it to the new protocol.

The work you are doing is super awesome. Thank you!

@whilo I cloned konserve-pg and was building off that. Turns konserve-pg was using an outdated version of konserve before -update-in in the current was in the protocol. So the error was justified.

@whilo I'm going to do a pull request on it. I also noticed on konserve-pg that in -get-in the read is potentially blocking.

I'm actually thinking it may be worth while for me to put together a template repo for backends with lots of notes robust test. Would love you to review it afterwards.

Ok, sure. Given your level of interest you could also join our call on Friday on the new protocol and the filestore implementation.

I'm keen @whilo. What time is the call?

@alekcz360 If you do the template repo thing, please post it to #datahike ...would be fantastic learning tool. I'm interested in seeing if Datahike can be combined with CRDTs...so achieving an understanding where the layer boundaries by seeing just the "scaffolding" would be very helpful.