datahike

https://datahike.io/, Join the conversation at https://discord.com/invite/kEBzMvb, history for this channel is available at https://clojurians.zulipchat.com/#narrow/stream/180378-slack-archive/topic/datahike
whilo 2019-10-31T01:13:03.054500Z

Just to give some perspective. @cjsauer Is any of that helpful for your framework?

cjsauer 2019-10-31T16:20:39.075300Z

@whilo yes absolutely, very helpful, thank you. I’ll be watching that video soon and will return with questions :) >I designed them to be UUIDs because they should not accidentally clash globally Makes sense. I’ve been devouring any replikativ material I can find, and the concepts are starting to become clearer. > hasch.core/uuid can map values to unique UUIDs. What is the difference here from normal UUID generation? Does this mean that the resulting UUID is somehow content-addressable? > user-id is supposed to refer to an external identity system that can be used to authenticate users Authentication is still where my understanding is lacking. I’ve been reading through kabel-auth source, and am having trouble fitting my understanding with this distributed paradigm. Does kabel-auth still imply a central server authority, correct? The server is “special” in the sense that it can hand out auth tokens, and is in control of certain data mutations. How does authorization work in these systems? For example, what prevents me from seeing/mutating someone’s potentially private data? >Are you aware of datsync? Yes! I stumbled across datsync a while back, and it has been great inspiration. Tightrope attempted a slightly different approach, specifically around entity IDs and schema sharing (datsync is a bit more sophisticated here). >We are currently working on import utilities from Datomic to datahike This would be great to avoid “lock in”, but sounds non-trivial. One of the hard parts about these “datom-based” dbs that I’ve encountered is that they’re all slightly incompatible in nuanced ways. For example, I recently contributed unique-identity-cardinality-many attributes to datascript in 0.18.7, but the discussion to do the same in Datomic was mostly ignored. So now the two diverge even further…I would love to see a shared protocol for these databases that we could all adhere to, but that seems very unlikely given datomic’s proprietary model. Datahike being forked from datascript tho is very promising. It’d be great to see them retain equivalent semantics. >And ncg is working on 3DF Ah yes, I saw Nikolas’s talk at Conj 2018 about 3DF. Incredible stuff. > A problem is to prioritize things and getting some help I’m certainly interested in contributing where I can. Mainly I’d love to see these ideas/experiments all aggregated into an off-the-shelf Clojure app framework. I think that could do really great things for Clojure’s popularity. I think too that it could really help developers transition into this more distributed architecture/paradigm, which admittedly still hasn’t “sunk in” for me personally (hence my questions above).

cjsauer 2019-11-01T14:31:55.092400Z

> Build reasonable small support libraries to partition application data for efficient client side consumption, Datomic and Datascript. Look into datsync etc. Found this in the roadmap of replikativ. I’m going to try out some experiments. Will report back.

whilo 2019-10-31T18:11:48.077Z

Yes, exactly it is content addressable. The idea was to have public private key authentication and link that to other identifiers, e.g. by accepted authorities or trust networks.

whilo 2019-10-31T18:12:29.077200Z

Yes we would like to stay compatible with DataScript. So far we have not had any discussion with Nikita yet though.

whilo 2019-10-31T18:14:00.077400Z

👍 for a unified framework, I would say a unified, reactive datalog programming model would be ground breaking.

➕ 2
cjsauer 2019-10-31T18:40:22.077600Z

> The idea was to have public private key authentication and link that to other identifiers, e.g. by accepted authorities or trust networks. I just finished watching the talk you linked. Thanks for that. The piece regarding authentication that stuck out to me is when you mentioned “a single CRDT for app state”; does this necessitate some form of consensus, or am I thinking about that incorrectly? If I were to build a more traditional web stack with replikativ today, and wanted to authorize data access (and mutation), it seems I would have to split the app state into several CRDTs, and then guard against unauthorized subscriptions/changes to them.

cjsauer 2019-10-31T18:48:55.078100Z

“Consensus” meaning something like Tendermint, as you mentioned in your talk. Reading up on that now…

cjsauer 2019-10-31T19:26:50.078300Z

Looking into swarm.js, it seems they do indeed fracture the app state up into “nodes”, each being a CRDT

2019-10-31T19:58:18.080100Z

> For example, I recently contributed unique-identity-cardinality-many attributes to datascript in 0.18.7, but the discussion to do the same in Datomic was mostly ignored. @cjsauer This is already possible in Datomic. In fact, this is how you can "alias/rename" an attribute; It's possible to have multiple :db/ident assignments for a given entity.

cjsauer 2019-10-31T20:03:32.083400Z

@metasoarous interesting. Is :db/ident a special built-in in some sense? I’m referring specifically to the constraint that :db.unique/identity attributes must also be :db.cardinality/one. Datomic’s documentation states this explicitly. What’s interesting tho is that it works exactly as one would anticipate, but this is apparently an “accident” and not officially supported. There’s a very brief forum post about this here: https://forum.datomic.com/t/unique-cardinality-many-attributes-why-not/1208

cjsauer 2019-10-31T20:05:49.085200Z

Datascript also had this constraint, and so when attempting to upsert entities using a unique-card-many attribute whose value was a collection, you’d get an exception. That is no longer the case. Datascript will now reduce over that collection and ensure that each value does indeed resolve to the same entity, otherwise it results in a conflict.

cjsauer 2019-10-31T20:09:17.086500Z

It was actually datascript’s exception that brought this to my attention in the first place…I was using unique-card-many attributes in datomic without issue, and only when syncing them to datascript did things go wrong. I’ve since refactored away from using them so I can stay on the supported path, begrudgingly 😉

cjsauer 2019-10-31T20:11:05.087700Z

I’m curious tho if datahike would consider merging that PR. I still believe the semantics are perfectly valid.

2019-10-31T20:32:34.088300Z

I just tested this on Datomic Cloud and it worked

cjsauer 2019-10-31T20:36:09.089400Z

Yeah, I was testing on Cloud as well. Apparently tho it’s not actually supported. When I brought it up in #datomic, Marshal was actually surprised that it works at all. They’re considering adding a schema exception/warning if you attempt to assert unique-card-many attributes.

cjsauer 2019-10-31T20:36:45.089700Z

So, it works today, but I suppose it might not work tomorrow

2019-10-31T21:14:54.089900Z

Ah; Interesting.

2019-10-31T21:15:18.090400Z

I wonder how :db/ident and aliasing will get handled then...

cjsauer 2019-10-31T21:24:54.091400Z

Not sure. Could be that datomic handles those cases explicitly for the purposes of aliasing? Datascript doesn’t actually assert the cardinality of :db/ident: https://github.com/tonsky/datascript/blob/184665afeb9c969662ef74f312c9ebce3cfad255/src/datascript/db.cljc#L23 I think in that case it acts like card-one.