datahike

https://datahike.io/, Join the conversation at https://discord.com/invite/kEBzMvb, history for this channel is available at https://clojurians.zulipchat.com/#narrow/stream/180378-slack-archive/topic/datahike
whilo 2020-05-21T09:16:56.196400Z

@alekcz360 That is super cool. Can you encode the version of each value in its first byte? I think we can probably generalize the serialization between the filestore and the couchdb and redis backend, because they write the metadata and value in one underlying value in the key-value store.

alekcz 2020-05-21T09:19:10.197800Z

@whilo I can. The example in the filestore was only pushed recently. So I was a bit blind.

alekcz 2020-05-21T09:19:33.198300Z

I'll add it to the two and then I can update the template repo too

alekcz 2020-05-21T09:21:22.199900Z

@whilo quick question so the version will be 1 right?

whilo 2020-05-21T09:23:54.200300Z

yes, you can define a constant for it in the namespace

whilo 2020-05-21T09:29:14.201400Z

The idea is that you can just read the first byte and then decide whether you need to trigger a migration or can read the file right away.

alekcz 2020-05-21T09:39:00.202300Z

Yeah that makes sense. That would be separate to get-version right?

whilo 2020-05-21T18:54:35.203700Z

get-version on the store level is not strictly necessary, or would you see a use case for it?

whilo 2020-05-21T19:21:06.205100Z

It would basically do the same thing, but it is enough for it to do it on your byte representation directly before you read it. It does not need to know the full store.

alekcz 2020-05-21T19:34:12.208200Z

The best case scenario would be storing the version separately to the data. Then access is inexpensive. Otherwise you need to download the object

{:meta {:key ....}
 :data <object>
 :version 1}

alekcz 2020-05-21T20:04:49.209300Z

This would be possible in couch, postgres, firebase

whilo 2020-05-21T22:20:02.210500Z

Yes, but in general you already know that you want to access the object somehow when you read the version, .e.g. for get-meta or get. It is important that you do not have to fetch the value for get-meta.