Clojurians Log v2

Yes, every transaction is written to durable storage. However, Cloud's use of DDB is opaque unless accessed from Datomic. So, I am not sure what insight you would get by reviewing the stream of writes other than "writes are occurring," which you could also see by monitoring with metrics in CloudWatch (txdatoms, txbatchbytes) or by reviewing the Cloud Dashboard (txes,Txbytes DDB usage, write count). I am interested in hearing more about what you're envisioning with DBstreams, maybe there is something here we should look at as a feature for Datomic. 🙂

I was thinking of the tx-report-queue functionality in the peer library and how that might be possible in Cloud.

fwiw, ident is also in the dictionary, it just means identification, certainly I see this choice a much better name than name which is such an overloaded name for names that trying to google anything with it would be extremely frustrating. Definition of identification a: an act of identifying : the state of being identified b: evidence of identity

Ah, but the identity of :db/ident is 10. :db/ident is just an enum, an alias, an aka. A good counterargument here is @favila's point that idents survive as external references where the numerid ID does not, but that is just an example of the power of idents as implemented by Datomic and given some operation on a database. (What would alter entity-ids of the "same" entity?) If one looks inside Datomic, one would see that the true identity of :db/ident is 10. 10 gets linked to :db/ident by that :db/ident being the value where the attribute is :db/ident and --wait for it -- the entity-id is 10. :db/ident, after all, is just a namespaced keyword. This brings us to that other quagmire, tempid. With tempid we see we can have the numeric entity-id absent a :db/ident. And not the other way around. Do we sense the walls closing in on :db/ident? :)

I think this is confusing two different concerns. an entity-id’s only purpose is to join facts asserted about the same “thing”. In that sense it is an “identity”, but a very weak one that isn’t aware of the meaning of the data. It’s also weak because it’s kind of an implementation detail of datomic: the only guarantee is that they will be referentially consistent, not that their values will be stable. In the google dictionary, this is the second meaning of the word “identity”

in the data modeling domain, “identity” is the assertion that makes an entity “be” that thing. so :db/ident’s identity is not 10, 10's identity is :db/ident

note also the attribute schema for these : :db/unique :db.unique/identity, i.e., this is an attribute that, when asserted, gives an entity an identity

note also the attribute schema for these : :db/unique :db.unique/identity, i.e., this is an attribute that, when asserted, gives an entity an identity

> What would alter entity-ids of the “same” entity? These are rare or hypothetical in practice, but: 1. cognitect has said in the past that it couldn’t rule out entity renumbering in future versions of datomic. This is actually useful for performance because you can rearrange commonly-accessed values to be together in the datom sort order. (You can do this manually with on-prem using partitions, which are the top 20ish bits of an entity id. cloud stopped exposing partition control.)

2. “decanting”, which is essentially a “git rebase”-like operation. You run through the transaction log of a db, and reapply the transactions with transformations to a second db. At Clubhouse we did this in order to renumber entity ids with partitions for performance. A key property of this operation is that entity ids are not guaranteed the same between the two dbs.

3. base schema version changes. Datomic at some point introduced new base schema attributes (the version that introduced tuples.) To do this you need to install new attributes with the d/administer-system function. The entity id of these new attributes depends on what transaction in your particular db performed the upgrade--they are not the same on all dbs like the older attributes are.

Good points all. I am going to save all these for my Datomic tutorial before they scroll off the Clojurian history.

One perhaps acceptable result I am seeing, where tempids are involved, is that datomic identities can be idendtity-less. I just transacted this twice: `

[:db/add "Hi, Mom!" :movie/title "Harvey"]

sure, but I still think using the word “identity” to think about entity or temp ids is going to be confusing in the long run. Identity is from/for humans, ids are for machines

a tempid is a degenerate case of an entity id: it’s an entity id that is referentially consistent, but scoped to a particular transaction instead of a particular db, so it’s even more short-lived

if necessary to preserve references in the db, the tempid will be “upgraded” to a newly minted entity id that has a longer lifetime; or if it is involved in an assertion about an identity attribute, datomic will ensure that all other facts in the transaction about that tempid will be asserted on the same entity that has the same identity

in an attribute/assertion-centric data model, entities are very intangible

an entity id is even less than an autoincrement column in a relational db, which is already not very much

maybe another angle on this: an entity is a map-projection of all the facts asserted about one thing. that thing may not have an identifier--the only thing that identifies it is the collection of assertions about it

(or the entity id--if you peek under the covers)

this is another way of saying “entity ids are only for joining facts” and don’t really grant an entity identity

If this were pure mathematics I’m sure they’d find a way to do this without an entity id

"that thing may not have an identifier--the only thing that identifies it is the collection of assertions about it" and "this is another way of saying “entity ids are only for joining facts” and don’t really grant an entity identity". But how do you have the "it" in "collection of assertions about it" without sth designating object identity? Put another way, the "only" in "only for joining facts" seems unjust. 🙂 Joining the facts is where object identity begins, it seems.

yes, that’s the epistemological model. there is no “there” there but what is said of it

here’s a thought question which may clarify: when does an entity exist?

can an entity id be said to exist or not exist? If so, what makes it exist or not exist?

If you can show it, it exists. Also, ident is not identity nor id. It's evidence of identification (or encoding for a process of identification). If you have an ident, you still need a db and additional work to get an identity. (hopefully I didn't stretch the analogy too far :D)

"when does an entity exist?" An entity exists by definition. "When" depends on the frame of reference. Our frame of reference is a datomic database. There, when we assert a fact we get an entity ID, whether we like it or not. And we get an entity, but perhaps a ghost if we use a tempid with no :db/ident attribute. Ghosts thus are on the user. So I agree that ghosts and the edge cases of re-partitioning and decanting mean entity IDs are not part of object identity: that must be arranged by the user, via db/ident, a great, well, name for it!

I think the question is a trick question. Entities (and entity ids) don’t meaningfully exist or not-exist the same way a row in a sql table exists. Only assertions about entities (i.e. datoms) exist. if nothing is said about an entity, you can still pull that entity id and see nothing. I think you can even do this if the entity id has never been used before. You can also use an entity id only as the object (ref value) of a datom--so now you have something which is the object of assertions but never the subject of any assertions. when you write an application, “does this record exist” isn’t usefully answered by looking at entity ids but by a looking for a datom with a unique-value or unique-identity attribute and matching value.

'by definition'? where is the definition stored then? 🙂 how do you access it?

"You can also use an entity id only as the object (ref value) of a datom--so now you have something which is the object of assertions but never the subject of any assertions." I think I tried using 42 as an entity-id (not a ref) and got yelled at. I should think I would at least to have created a :db/entity with a nice name and used the resulting eid. In which case there would be one fact: the :db/ident value, yes?

I don't think you should care so much about entity ids. And no, I don't think an entity can exist just with one datom, but I could be wrong, haven't checked it.

@hiskennyness : > Good points all. I am going to save all these for my Datomic tutorial before they scroll off the Clojurian history. This channel is archived to Zulip as https://clojurians.zulipchat.com/#narrow/stream/180378-slack-archive/topic/datomic which is a full, searchable history back to whenever the @zulip-mirror-bot was added to this channel.

(I figured this was a good opportunity to remind folks of the free, searchable archive of many channels here!)

Here is that video: https://www.youtube.com/watch?v=7lm3K8zVOdY

Thanks ill give it a watch.

For development we have a custom implementation of d/q-protocol which sends queries to tap>. it enables us to see what performance is of specific queries. we just wrap the existing connection in our system-map.