datomic

Ask questions on the official Q&A site at https://ask.datomic.com!
2020-08-06T10:54:19.106800Z

Yeah that's the big question with that approach. Although we could potentially host our read clients inside AWS meaning that only the writes would originate in GCP which seems likely to work for our usecase. Not ideal to have to run a service away from our others but we need to weigh that up against running a database. Usually running services is easier than databases!

onetom 2020-08-06T17:30:00.107300Z

im also just about to explore this topic in our current project. what's your current approach?

onetom 2020-08-06T17:31:54.107500Z

the hong kong region would be a welcome addition to that list. i have the gut feeling that it supports all the required aws features already.

onetom 2020-08-06T17:39:45.112900Z

i've just started to use the client api recently (after a few years of dealing only with the peer api). when i tried to test something simple, i was getting this error:

Execution error (ExceptionInfo) at datomic.client.api.impl/incorrect (impl.clj:43).
Query args must include a database
on a closer look, i can provoke this error by trying to run some of the official examples from the cloud docs (https://docs.datomic.com/cloud/query/query-data-reference.html#calling-static-methods), against a com.datomic/client-pro 0.9.63 backed by a datomic-pro peer-server & transactor 1.0.6165 (and i also have a com.datomic/dev-local 0.9.183 loaded into the same process too):
(dc/q '[:find ?k ?v
        :where [(System/getProperties) [[?k ?v]]]])
or this even simpler query:
(dc/q '[:find [?b]
        :in [?tup]
        :where [[(untuple ?tup) [?a ?b]]]]
      [1 2])
is that expected or a bug?

onetom 2020-08-06T17:41:14.113500Z

for the 1st case, where the :in clause is omitted, i might understand the error, but for the second case, i definitely don't expect it

marshall 2020-08-06T17:41:15.113700Z

client query must take a db, can’t be run against a collection the way peer query can

marshall 2020-08-06T17:41:37.113900Z

@onetom ^

onetom 2020-08-06T17:42:04.114Z

so it's a "bug" in the documentation then, isn't it?

marshall 2020-08-06T17:42:28.114200Z

i can look at the docs, if they are client-specific docs then yes

onetom 2020-08-06T17:43:02.114400Z

the cloud docs - i linked - is all client api specific, no?

marshall 2020-08-06T17:43:21.114600Z

yes, that example should take a db

onetom 2020-08-06T17:44:39.114800Z

but now that we are talking about it, of course it needs a "db", since that's how the query itself can reach the query engine over the network...

marshall 2020-08-06T17:45:02.115200Z

right 🙂

onetom 2020-08-06T17:48:07.117800Z

i keep finding myself reaching for the peer api functions, like entid, ident etc. the https://docs.datomic.com/on-prem/clients-and-peers.html#peer-only section of the docs makes it clear that we should use the pull api instead of these functions, but that's just much more verbose. same issue with the lack of the find-coll and find-scalar from find-spec. is there any official or popular compatibility lib which fills this gap?

onetom 2020-08-06T17:49:10.118700Z

or is there any good examples how to sturcture an app in a way that it's concise to test?

onetom 2020-08-06T17:52:02.120600Z

for example, given this function:

(defn by-name
  [db merchant-name]
  (-> {:query '{:find  [?merchant]
                :in    [$ ?merchant-name]
                :where [(or [?merchant :merchant/name ?merchant-name]
                            [?merchant :merchant/name-en ?merchant-name])]}
       :args  [db merchant-name]}
      (dc/q)
      (ffirst)))
my test would look like this:
(deftest by-name-test
  (testing "exact match"
    (let [db (db-of [{:db/ident      :some-merchant
                      :merchant/name "<merchant name in any language>"}])]
      (is (match?
            (->> :some-merchant (dc/pull db [:db/id]) :db/id)
            (merchant/by-name db "<merchant name in any language>"))))))
where db-of is just a with-db with some schema, made from a dev-local test db. that (dc/pull db [:db/id]) :db/id is the annoying part and it's even more annoying if im expecting multiple values.

onetom 2020-08-06T17:54:15.121100Z

the benefit of operating with idents is that the test failure messages are symbolic and i don't have to muck around with destructuring string temp-ids, potentially across multiple transactions

onetom 2020-08-06T18:02:21.125100Z

i can understand that the client api doesn't want to provide find-scalar and find-coll and ident, entid, so the interface size is small, which helps providing alternative implementations, like the dev-local one, but these functions are just too useful for REPL work and automated tests. i can also understand how they might seep into application code, promoting inefficient code, but that's not a strong reason for not providing them officially.

1👍
onetom 2020-08-06T19:07:06.126900Z

for now, i made a custom matcher, which results in tests like this:

(is (match?
      (idents-in db
                 :matching-merchant-1
                 :matching-merchant-2)
      (merchant/named-like db "matching")))
where idents-in looks like this:
(ns ...
  (:require
    [matcher-combinators.core :refer [Matcher]] ...))

(defrecord MatchIdents [db expected-idents]
  Matcher
  (-matcher-for [this] this)
  (-matcher-for [this _] this)
  (-match [_this actual-entity-refs]
    (if-let [issue (#'matcher-combinators.core/validate-input
                     expected-idents
                     actual-entity-refs
                     sequential? 'in-any-order "sequential")]
      issue
      (#'matcher-combinators.core/match-any-order
        expected-idents
        (mapv (comp :db/ident (partial dc/pull db [:db/ident]))
              actual-entity-refs)
        false))))

(defn idents-in [db & entity-idents]
  (->MatchIdents db entity-idents))

dregre 2020-08-06T19:35:30.127Z

My approach has been to use an in-memory database loaded with the right schema and mock data (fixtures) and then run queries against them — but I wonder if there’s a better approach.

dregre 2020-08-06T19:37:17.127200Z

I’m also interested in finding a query profiler or explainer, if anyone’s come across any.

kschltz 2020-08-06T23:37:49.127800Z

I have an incident where datomic cloud started giving me busy indexing errors

kschltz 2020-08-06T23:38:17.128400Z

I supposed I exceeded my node limit, the stop transacting entirely

kschltz 2020-08-06T23:38:33.128800Z

but after 40min it still gives me busy indexing errors

favila 2020-08-07T15:02:07.130300Z

Sorry, no, I don’t have any production experience with cloud

marciol 2020-08-07T19:29:07.132700Z

Yes, in the end we restarted all machines from the primary computational group.

kschltz 2020-08-07T19:58:47.132900Z

as it turns out we largely exceeded our nodes capacity believing it would scale automatically

1👍