datomic

Ask questions on the official Q&A site at https://ask.datomic.com!
seancorfield 2020-09-09T00:12:48.345900Z

LOL! Sounds like me... I've been meaning to learn Datomic for "ages" but now that dev-local is available, I think I actually might.

nando 2020-09-09T00:20:13.350100Z

I'm not yet at the stage of working with anything that complex, but so far I'm very happy, probably just because it all makes so much sense ...

John Leidegren 2020-09-09T05:07:15.352300Z

How can I move the identity :test/id from the entity 17592186045418 to the new entity (referenced by :test/ref). Do I have to do this in two separate transactions? All I want to do is move the identity to a new entity in a single transaction. I understand why the temp ID resolution is taking place and resolving the temp ID to a conflict but how can I avoid it. How can I force a new entity here?

cmdrdats 2020-09-09T07:01:28.355700Z

I imagine the same ref-resolution phase code applies.. I don't know the exact implementation details of course, but that's the picture I have in my head xD It would basically handle every datom against the immutable value prior to the transaction

cmdrdats 2020-09-09T07:04:39.355900Z

interestingly, this implementation also implies that you can't provide two values of a :cardinality/one field in the same transaction

cmdrdats 2020-09-09T07:04:44.356100Z

(d/transact (:conn datomic)
      [[:db/add "temp" :ent/displayname "hello"]
       [:db/add "temp" :ent/displayname "bye"]]
      )

cmdrdats 2020-09-09T07:05:06.356300Z

:db.error/datoms-conflict Two datoms in the same transaction conflict
                             {:d1 [17592186045457 :ent/displayname \"hello\" 13194139534352 true],
                              :d2 [17592186045457 :ent/displayname \"bye\" 13194139534352 true]}

cmdrdats 2020-09-09T07:05:23.356500Z

since it can't imply the db/retract for the "hello" value

cmdrdats 2020-09-09T07:05:52.356700Z

/s/field/attribute, of course

John Leidegren 2020-09-09T07:11:54.356900Z

Yeah, the application of transactions is unordered, so if you say add twice for the same attribute of cardinality one it cannot know which one you meant so it rejects the transaction.

cmdrdats 2020-09-09T07:17:14.357100Z

ah, I see - so by that constraint, the same applies for retracting and to re-using identity on a new entity

marshall 2020-09-09T11:57:17.357400Z

what version of datomic are you using?

marshall 2020-09-09T11:57:22.357600Z

and cloud or on-prem?

John Leidegren 2020-09-09T12:24:40.357800Z

@marshall It's actually datomic-free-0.9.5703.21 so maybe this isn't a problem elsewhere

marshall 2020-09-09T13:16:02.358Z

i believe this was fixed in https://docs.datomic.com/on-prem/changes.html#0.9.5390

marshall 2020-09-09T13:16:08.358200Z

but its possible this is unrelated

John Leidegren 2020-09-09T13:16:52.358400Z

hehe, that description seems to fit my problem very well. oh well. Thanks for letting me know.

marshall 2020-09-09T13:43:52.358800Z

@john.leidegren do you have a starter license? can you try it in starter and/or with Cloud?

marshall 2020-09-09T13:44:02.359Z

i can also look at trying to reproduce

John Leidegren 2020-09-09T14:16:21.359400Z

Thanks but I'm just fooling around. As long as I know this isn't the intended behavior that's fine. I know what to do now.

marshall 2020-09-09T14:29:56.359600Z

:thumbsup: we’ll look into it anyway

1❤️
jaret 2020-09-10T15:10:02.380Z

Hey @john.leidegren, Marshall tasked me with looking into this and I wanted to clarify that this is indeed intended behavior and not related to the fix Marshall described. You already have the rough reason here: > Yeah, the application of transactions is unordered, so if you say add twice for the same attribute of cardinality one it cannot know which one you meant so it rejects the transaction. You cannot transact on the same datom twice and have it mean separate things in the same transaction. You have to split the transactions up to retract the entity then assert the new identity. Ultimately what you're doing here is cleaning up a modeling decision and in addition to separating your retraction and add transactions you could alternatively model a new identity and use that identity going forward, preserving the initial decision.

jaret 2020-09-10T15:10:36.380200Z

I know you were already past this problem, but I hope that clears things up.

John Leidegren 2020-09-11T10:44:38.392400Z

@jaret Oh, thanks for getting back to me. I really appreciate it.

cmdrdats 2020-09-09T05:31:19.352900Z

Interesting problem! I'm interested to see the solution for this.. I expect you'd have to split to two transactions since you're working with db.unique/identity though

John Leidegren 2020-09-09T06:43:37.353100Z

Yeah. That's what I did. I don't like it because now there's a point in time where the database sort of has an inconsistent state. It's not the end of the world but I really want it to commit as a single transaction. For this to actually go though, the transactor, would have to somehow react to the fact that the identity is being retracted during the transaction and because of that, it mustn't be allowed to partake in temp ID resolution. (either that, or you tag the temp ID as unresolvable to force a new entity...)

cmdrdats 2020-09-09T06:45:01.353300Z

it seems like a modelling problem if you need to get to a state where an entity has an 'identity', then loses it and gives it to another entity - so I could see why this could be unsupported behaviour

John Leidegren 2020-09-09T06:46:57.353700Z

I'm fixing a data problem or rather I'm doing this because I'm revising the data model. I ran into this as part of a upgrade script I wrote.

John Leidegren 2020-09-09T06:48:45.353900Z

I know Marshal has commented in the past on some of these transactional isolation behaviours. As to why it might need to work this way but I'm curious to what the reasoning for it is. As I can see a way to program around it but I can also understand that you might not want to just do that.

John Leidegren 2020-09-09T06:54:16.354300Z

You could argue that I'm https://blog.datomic.com/2017/01/the-ten-rules-of-schema-growth.html or some such.

cmdrdats 2020-09-09T06:54:55.354500Z

I suspect that it's a bit of a trade-off - if you have this behaviour, it's simpler to reason about transactions, since it's likely implemented in a ref-resolution phase, then an actual write phase

cmdrdats 2020-09-09T06:55:31.354700Z

but if you have clever transactions where you effectively mutate the state for each fact, then things get trickier to accurately reason about

cmdrdats 2020-09-09T06:56:51.354900Z

I had this kind of issue for new schema, and schema that used the new schema:

[{:db/ident :ent/displayname}
 {:db/ident :ent/something
  :ent/displayname "Hello"}]
would complain that :ent/displayname is not part of the schema yet

cmdrdats 2020-09-09T06:57:26.355100Z

so I had to write a function that checks existence of the properties and then split the schema assertion into multiple phases

John Leidegren 2020-09-09T06:58:54.355300Z

Yeah, so this rule applies to attributes. Which I sort of understand. You cannot refer to schema before it exists but for data though. Are the same constraints equally valid?

cmdrdats 2020-09-09T07:01:28.355700Z

I imagine the same ref-resolution phase code applies.. I don't know the exact implementation details of course, but that's the picture I have in my head xD It would basically handle every datom against the immutable value prior to the transaction

cmdrdats 2020-09-09T07:04:39.355900Z

interestingly, this implementation also implies that you can't provide two values of a :cardinality/one field in the same transaction

cmdrdats 2020-09-09T07:04:44.356100Z

(d/transact (:conn datomic)
      [[:db/add "temp" :ent/displayname "hello"]
       [:db/add "temp" :ent/displayname "bye"]]
      )

cmdrdats 2020-09-09T07:05:06.356300Z

:db.error/datoms-conflict Two datoms in the same transaction conflict
                             {:d1 [17592186045457 :ent/displayname \"hello\" 13194139534352 true],
                              :d2 [17592186045457 :ent/displayname \"bye\" 13194139534352 true]}

cmdrdats 2020-09-09T07:05:23.356500Z

since it can't imply the db/retract for the "hello" value

cmdrdats 2020-09-09T07:05:52.356700Z

/s/field/attribute, of course

John Leidegren 2020-09-09T07:11:54.356900Z

Yeah, the application of transactions is unordered, so if you say add twice for the same attribute of cardinality one it cannot know which one you meant so it rejects the transaction.

cmdrdats 2020-09-09T07:17:14.357100Z

ah, I see - so by that constraint, the same applies for retracting and to re-using identity on a new entity

marshall 2020-09-09T11:57:17.357400Z

what version of datomic are you using?

marshall 2020-09-09T11:57:22.357600Z

and cloud or on-prem?

John Leidegren 2020-09-09T12:24:40.357800Z

@marshall It's actually datomic-free-0.9.5703.21 so maybe this isn't a problem elsewhere

marshall 2020-09-09T13:16:02.358Z

i believe this was fixed in https://docs.datomic.com/on-prem/changes.html#0.9.5390

marshall 2020-09-09T13:16:08.358200Z

but its possible this is unrelated

John Leidegren 2020-09-09T13:16:52.358400Z

hehe, that description seems to fit my problem very well. oh well. Thanks for letting me know.

marshall 2020-09-09T13:43:52.358800Z

@john.leidegren do you have a starter license? can you try it in starter and/or with Cloud?

marshall 2020-09-09T13:44:02.359Z

i can also look at trying to reproduce

John Leidegren 2020-09-09T14:16:21.359400Z

Thanks but I'm just fooling around. As long as I know this isn't the intended behavior that's fine. I know what to do now.

marshall 2020-09-09T14:29:56.359600Z

:thumbsup: we’ll look into it anyway

1❤️
Abe 2020-09-09T17:32:58.364600Z

Hello, I'm new to Clojure and Datomic. I'm using the min aggregate to find the lowest-priced product, but can't seem to figure out how to get the entity ID of the product along with it -

;; schema
(def product-offer-schema
  [{:db/ident :product-offer/product
    :db/valueType :db.type/ref
    :db/cardinality :db.cardinality/one}
   {:db/ident :product-offer/vendor
    :db/valueType :db.type/ref
    :db/cardinality :db.cardinality/one}
   {:db/ident :product-offer/price
    :db/valueType :db.type/long
    :db/cardinality :db.cardinality/one}
   {:db/ident :product-offer/stock-quantity
    :db/valueType :db.type/long
    :db/cardinality :db.cardinality/one}
  ])
(d/transact conn product-offer-schema)

;; add data
(d/transact conn
  [{:db/ident :vendor/Alice}
   {:db/ident :vendor/Bob}
   {:db/ident :product/BunnyBoots}
   {:db/ident :product/Gum}
  ])
(d/transact conn
  [{:product-offer/vendor  :vendor/Alice
    :product-offer/product :product/BunnyBoots
    :product-offer/price   9981 ;; $99.81
    :product-offer/stock-quantity 78
   }
   {:product-offer/vendor  :vendor/Alice
    :product-offer/product :product/Gum
    :product-offer/price   200 ;; $2.00
    :product-offer/stock-quantity 500
   }
   {:product-offer/vendor  :vendor/Bob
    :product-offer/product :product/BunnyBoots
    :product-offer/price   9000 ;; $90.00
    :product-offer/stock-quantity 15
   }
  ])

;; This returns the lowest price for bunny boots as expected, $90:
(def cheapest-boots-q '[:find (min ?p) .
                        :where
                        [?e :product-offer/product :product/BunnyBoots]
                        [?e :product-offer/price ?p]
                       ])
(d/q cheapest-boots-q db)
;; => 9000

;; However I also need the entity ID for the lowest-priced offer, and
;; when I try adding it, I get the $99.81 boots:
(def cheapest-boots-q '[:find [?e (min ?p)]
                        :where
                        [?e :product-offer/product :product/BunnyBoots]
                        [?e :product-offer/price ?p]
                       ])
(d/q cheapest-boots-q db)
;; => [17592186045423 9981]
I think I might see what's going on - it's grouping on entity ID, and returning a (min ?p) aggregate for each one (so basically useless). But I'm not sure how else to get the entity ID in the result tuple... should I not be using an aggregate at all for this?

favila 2020-09-09T17:48:17.364700Z

datalog doesn’t support this kind of aggregation (neither does sql!)

favila 2020-09-09T17:48:54.364900Z

you can do this with a subquery that finds the max, then find the e with a matching max in the outer query; or, do it in clojure

1👍
favila 2020-09-09T17:50:38.365100Z

:find ?e ?p then (apply max-key peek results) (for example)

favila 2020-09-09T17:51:41.365300Z

the reason datalog and sql don’t do this is because the aggregation is uncorrelated: suppose multiple ?e values have the same max value: which ?e is selected? the aggregation demands only one row for the grouping

favila 2020-09-09T17:52:15.365500Z

(you still have that problem BTW--you may need to add some other selection criteria)

favila 2020-09-09T17:58:23.365700Z

what was it before?

favila 2020-09-09T18:01:03.365900Z

> is your scenario combining upserting of the components of the ref also with upserting of the composite ref itself? That’s indeed what is happening. you are trying to allow repo upserting (one of the components of the ref) while also allowing the commit entity to upsert (the composite ref itself)

favila 2020-09-09T18:05:39.366100Z

I think this may be a complecting in :db.unique/identity itself. To identify an entity you generally can’t use refs--refs are internal identifiers, but :db/unique is to mark external identifiers

favila 2020-09-09T18:05:54.366300Z

but :db.unique/identity also has this upserting behavior you want

favila 2020-09-09T18:06:03.366500Z

which I’m guessing you want to use here for deduplication

favila 2020-09-09T18:07:57.366700Z

so, if the repository id never changes, and the commit->repo reference never changes, and repo id is always available to the application at tx time (I don’t see how it couldn’t be with this schema design) consider denormalizing by putting the repo id on the commit entity

favila 2020-09-09T18:08:42.366900Z

you can do this a few ways

favila 2020-09-09T18:09:28.367100Z

1. add :commit/repo-id, and make the :commit/id use that as one of its components (I suggest putting repo id first for better indexing)

favila 2020-09-09T18:09:54.367300Z

2. just write :commit/id as a tuple with those two values (don’t use a composite, just a tuple). This has the advantage of not adding a datom, but the disadvantage of being less clear

favila 2020-09-09T18:12:09.367700Z

both these have the advantage that you can now produce lookup refs for commits without a db: [:kipz.commit/id [commit-sha-string repo-id-string]]

favila 2020-09-09T18:12:41.367900Z

this is not possible for ref composites generally--it’s that notion of external identity again

favila 2020-09-09T18:14:04.368100Z

alternatively, if you just want to enforce uniqueness, consider not using upserting or a composite attribute at all. You can query first and speculatively create entities if they’re not found, and use :db/ensure to allow the transaction to fail if you violate the constraint.

favila 2020-09-09T18:14:11.368300Z

(i.e. optimistic commit style)

favila 2020-09-09T18:14:47.368500Z

you can use some, none, or all indexes according to your preference and the concurrency of the workload

favila 2020-09-09T18:17:04.368700Z

e.g. here, you could look up the repo and use that id; and if not found create a repo but allow the tx to fail (using :db.unique/value instead of identity) if someone else made the same repo in the meantime. you can recalculate and reissue the tx

favila 2020-09-09T18:17:17.368900Z

(that doesn’t actually need :db/ensure at all)

favila 2020-09-09T18:17:46.369100Z

anyway, those are just some ideas

favila 2020-09-09T18:18:37.369300Z

I don’t think expecting composite tuple upserting constraint resolution is a realistic expectation because of performance: the transactor has a global write lock on the db (essentially) while it’s doing all this tempid resolution and composite tuple maintenance, so it has to be as fast as possible

favila 2020-09-09T18:21:14.369500Z

that said, you can always write a transaction function that does what you want. it would take the repo and the commits plus some DSL for your own tempid replacement for the other assertions you want to make on those entities, do the lookup-or-create, then replace your tempid and return the expanded transaction. Essentially implementing the upserting logic yourself before the transactor does tempid resolution

Abe 2020-09-09T18:41:00.369900Z

Ah I see, thank you!

v 2020-09-09T20:00:38.371300Z

Hello I am playing with dev-local datomic. When I try to create a database I get error.

java.nio.file.NoSuchFileException: "/resources/dev/quizzer/db.log"
....
Here is the full code
(ns quizzer.core
  (:require
   [datomic.client.api :as d]))

(def client (d/client {:server-type :dev-local
                       :storage-dir "/resources"
                       :system "dev"}))

;; Creating a database
(defn make-conn [db-name]
  (d/create-database client {:db-name db-name})
  (d/connect client {:db-name db-name}))

(comment
  (d/create-database client {:db-name "quizzer"}))
Any ideas? 🙂

alexmiller 2020-09-09T20:02:05.371500Z

does /resources/dev/quizzer exist?

alexmiller 2020-09-09T20:02:35.371800Z

or more simply, does /resources exist?

v 2020-09-09T20:06:16.372900Z

I placed it in the root directory. Here is project structure

.
├── README.md
├── deps.edn
├── resources
│   └── dev
│       └── quizzer
│           └── db.log
├── src
│   └── quizzer
│       └── core.clj
└── test
    └── quizzer
        └── core_test.clj

7 directories, 5 files

v 2020-09-09T20:06:51.373400Z

And the deps.edn structure

{:paths ["src" "resources" "test"]
 :deps {org.clojure/clojure              {:mvn/version "1.10.1"}
        com.datomic/dev-local            {:mvn/version "0.9.195"}}
 :aliases {:server {:main-opts ["-m" "quizzer.core"]}
           :test {:extra-paths ["test/quizzer"]
                  :extra-deps  {lambdaisland/kaocha {:mvn/version "0.0-529"}
                                lambdaisland/kaocha-cloverage {:mvn/version "1.0.63"}}
                  :main-opts   ["-m" "kaocha.runner"]}}}

alexmiller 2020-09-09T20:09:05.373600Z

"/resources" is an absolute path

alexmiller 2020-09-09T20:09:41.374Z

I assume that's in your ~/.datomic/dev-local.edn

1😮
v 2020-09-09T20:29:28.374400Z

Absolute path was the problem, thank you @alexmiller

Jake Shelby 2020-09-09T21:34:05.377400Z

I have a datomic cloud production topology, which shows the correct number of datoms in the corresponding CloudWatch dashboard panel..... however, the datoms panel for my other solo topology never shows any datoms, no matter how many I transact into the system

kenny 2020-09-09T21:48:44.377700Z

I know solo reports a subset of the metrics, but according to https://docs.datomic.com/cloud/operation/monitoring.html#metrics solo should report that datoms metric. > Note In order to reduce cost, the Solo Topology reports only a small subset of the metrics listed above: Alerts, Datoms, HttpEndpointOpsPending, JvmFreeMb, and HttpEndpointThrottled. Not sure what's going on. I'm seeing the same on our solo stacks though @jake.shelby.

kenny 2020-09-09T21:49:31.377900Z

Even the solo https://docs.datomic.com/cloud/operation/monitoring.html#dashboardsshows the datoms metric.

Jake Shelby 2020-09-09T21:52:23.378200Z

thanks for checking your system @kenny, what version is yours? (I just launched mine last week, so it's the latest version

▶ datomic cloud list-systems                                                                                           
[{"name":"core-dev", "storage-cft-version":"704", "topology":"solo"},
 {"name":"core-prod",
  "storage-cft-version":"704",
  "topology":"production"},

kenny 2020-09-09T22:01:40.378400Z

Same version

v 2020-09-09T23:42:11.379400Z

Does anyone have an example on how tx-report-queue is used