Clojurians Log v2

I posted a question on the dev forum, but because it's on an old thread, I wondering if it'll draw any attention, so I was wondering if there's anyone here in Slack who might be able to provide some insight into the behaviour of unique composite-tuples we are seeing? https://forum.datomic.com/t/upsert-behavior-with-composite-tuple-key/1075?u=jcarnegie

This is a caveat of unique-identity composite ref tuples

I would even say a “gotcha”

however, I’m not sure there’s an easy fix. “upsertion” works by looking for a to-be-applied assertion with a tempid and an upserting attr and resolving the tempid to an existing id if the value matches an existing id

composite tuples need to look at a just-completed transaction, see which composite component attributes were “touched”, and adding an additional datom to update the composite

having upsertion resolve to a composite tuple would create a cycle here

instead of two simple phases, it would become a constraint problem

Yeah, I see what you mean. It's just that this is the sort of thing that transaction isolation could give us, right?

I’m not sure what you mean?

is your scenario combining upserting of the components of the ref also with upserting of the composite ref itself?

I’m trying to imagine why you don’t either have an entity id already, or know you are creating the entity and thus cannot conflict

Well, yeah, this can be solved by issuing multiple transactions, but I'm trying to avoid that. The system itself receives events (the entities) from different sources at different times/orders with partial data - enough to create id's of (potentially) new entities that are required refs of other entities. So in general, we can't know, without issuing queries, if a particular entity already exists. So we want to upsert all the time, and we need a single transaction. Like I said in my post, we've solved this by generating unique id fields from our own external definition of composite-ids - and this is a bit of a pain (we have to manage the lifecycle of this schema and related code between clients). I'm wondering how folk are using these (unique) composite-tuples in the real world given how they currently work.

> I’m not sure what you mean? What I mean is, it seems feasible that this constraint could be solved within the transaction if the datomic team wanted to implement this. I understand that it's currently not the case.

> is your scenario combining upserting of the components of the ref also with upserting of the composite ref itself? I'm still trying to grok this 🙂 but I think so. I can post a little schema and transaction if you're interested?

yeah, I am

[
     ;; commit entity
     {:db/ident :kipz.commit/sha
      :db/cardinality :db.cardinality/one
      :db/valueType :db.type/string}

     {:db/ident :kipz/repo
      :db/cardinality :db.cardinality/one
      :db/valueType :db.type/ref}

     {:db/ident :kipz.commit/id
      :db/valueType :db.type/tuple
      :db/unique :db.unique/identity
      :db/tupleAttrs [:kipz.commit/sha
                      :kipz/repo]
      :db/cardinality :db.cardinality/one}

     ;; repo entity
     {:db/ident :kipz.repo/id
      :db/cardinality :db.cardinality/one
      :db/unique :db.unique/identity
      :db/valueType :db.type/string}

     {:db/ident :kipz.repo/name
      :db/cardinality :db.cardinality/one
      :db/valueType :db.type/string}

     {:db/ident :kipz.repo/owner
      :db/cardinality :db.cardinality/one
      :db/valueType :db.type/string}]

[[:db/add "r1" :kipz.repo/id "repo-id-1"]
 [:db/add "r1" :kipz.repo/name "repo-name-1"]
 [:db/add "r1" :kipz.repo/owner "repo-owner-1"]
 [:db/add "c1" :kipz.commit/sha "commit-sha-1"]
 ;; always fails without this, fails after first time with it
 ;; this is the line that the docs says we should never do, but only works
 ;; with specific known "r1" eid
 [:db/add "c1" :kipz.commit/id ["commit-sha-1" "r1"]]
 [:db/add "c1" :kipz/repo "r1"]]

:kipz.repo/id has been made a scalar to help show the issue

Datomic beginner here. I have a question about schema evolution from initial experience. Developing a simple web app, I began with the following query to populate a form:

(defn find-nutrient [eid]
  (d/q '[:find ?eid ?name ?grams-in-stock ?purchase-url ?note 
         :keys eid name grams-in-stock purchase-url note 
         :in $ ?eid
         :where [?eid :nutrient/name ?name]
         [?eid :nutrient/grams-in-stock ?grams-in-stock]
         [?eid :nutrient/purchase-url ?purchase-url]
         [?eid :nutrient/note ?note]
       (d/db conn) eid))

(defn find-nutrient [eid]
  (d/q '[:find ?eid ?name ?grams-in-stock ?purchase-url ?note ?category-eid
         :keys eid name grams-in-stock purchase-url note category-eid
         :in $ ?eid
         :where [?eid :nutrient/name ?name]
         [?eid :nutrient/grams-in-stock ?grams-in-stock]
         [?eid :nutrient/purchase-url ?purchase-url]
         [?eid :nutrient/note ?note]
         [?eid :nutrient/category ?category-eid]]
       (d/db conn) eid))

have you read about the pull api?

sounds more like a job for it

Ok, I will look into it.

yes, did you add the new attribute to existing entities?

I'm thinking ahead if this could be a potential issue with the evolution of a production app.

There are only a few entities, so one has and the others don't.

I'll work on modifying the function to use the pull api and see what happens.

yes, the pull api is designed for this

Ok, got it working. Is there a handy way to specify the keys used in the map that is returned using the pull api? I didn't find one in a scan of the docs.

like this? https://docs.datomic.com/cloud/query/query-pull.html#as-option

clojure has select-keys

{:db/id 4611681620380876878,
 :nutrient/name "Vitamin A",
 :nutrient/grams-in-stock 40,
 :nutrient/purchase-url "<http://www.link.com>",
 :nutrient/note "beta carotene and palmitate",
 :nutrient/category #:db{:id 96757023244374}}

Maybe I should be using the fully qualified keys in my web forms?

don't see why not, as long as you are inside the same process one should rely on them IMO

sometimes they aren't pretty to work with though

How would I flatten :nutrient/category #:db{:id 96757023244374}}

I'm not yet familar with what #: designates

I should just try to do it myself ...

@nando That is just clojure shorthand for :nutrient/category {:db/id 96757023244374} .

Oh! That helps a lot! Then a get-in should do it.

Well, wait, what is the type of ref that :nutrient/category is pointing at?

I'm sufficiently sorted out. Thanks very much @kaxaw75836 & @lanejo01

@lanejo01 It's essentially a string, a category name

{:db/ident :nutrient/category
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/one}
{:db/ident :category/name
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/unique :db.unique/identity
:db/doc "Nutrient category"}

Cool. If you're using d/pull or using pull in a query you can supply a pull pattern like this.

(d/pull db '[:nutrient/name 
             :nutrient/grams-in-stock 
             :nutrient/purchase-url 
             :nutrient/note
             {:nutrient/category [:category/name]}] eid)

Ok, got it!

Have fun, reach out if you have more questions!

This is fun!

Are you using dev-local?

Yes, I am.

Cool, I would love to hear some feedback on your experience.

If you were interested, of course 🙂

Either way, glad to hear you think it's fun!

I've always wanted to use datomic, for years, but it was difficult to find a sensible path in as a solo developer. For me, datomic justified learning clojure. Anyway, some weeks back i decided to bite the bullet and dive into developing an app to learn Clojure. I thought I was going to use next.jdbc and a relational database. Well, I had trouble getting the mysql driver to work with the mysql version installed on my dev laptop ... and the next day I decided to hell with it, I'm going to find a way to use datomic instead! So I started looking around and found Stu's message here that dev-local had been released the day before. 🙂