@robert-stuttaford any chance the attribute has :db/noHistory set to true?
@robert-stuttaford second thought, you’re getting the history db
from the db
you see the datom in? If so, that sounds like something we would want to investigate. Would you be able to give us a small repro or better yet, a backup that shows this behavior?
that's right - db
and (d/history db)
@jaret it's in our prod db, which has all our PII in it, started circa 2012 🙂
perhaps we could arrange a zoom and i could show you via screen share, and then we can see about next steps from there?
Hey there, we are using datomic and are currently diagnosing a performance issue related to a recursive rule. We have a tree structure in datomic, that for each node, links a parent (or not), so a schema like this:
(def schema
[;; Additional tree attributes omitted
{:db/ident :node/parent
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/one}])
Now, we sometimes look through the tree to find, say a descendant of a node. To do so, we have a recursive descendants-of?
rule that binds this collection of descendants to ?node
:
(def descendants-of?
'[[(descendants-of? ?root ?node)
[?node :node/parent ?root]]
[(descendants-of? ?root ?node)
[?node :node/parent ?intermediate]
(descendants-of? ?root ?intermediate)]])
So far so good, we can do queries that reason about descendants, for example finding all descendants for a root:
(d/q '[:find ?node
:in $ ?e %
:where
(descendants-of? ?e ?node)]
(d/db (conn))
root-eid
descendants-of?])
Now, sometimes we have a candidate set of nodes, and of those candidates, we need to find the descendants, say like this:
(d/q '[:find ?node
:in $ ?name %
:where
;; assuming that `?name` only exists in one tree
[?e :node/name ?name]
(descendants-of? ?e ?node)]
(d/db (conn))
“name”
descendants-of?])
In the most pathological case, where all nodes are named ?name
, we will be binding all nodes in a tree to ?e
, and then find the descendants of those ?e
s and bind those to ?node
. The result will be the same as the query above: all nodes but the root.
However, this query appears to be much slower. I think that makes sense intuitively if we assume that descendants-of?
is kinda expanded per ?e
. For each member of ?e
, we can potentially redo descendant seeking, if those descendants are also in ?e
.
Is there a way to optimise here, if there are potential queries that bind descendants and ancestors to ?e
?Using :as
with :db/id
does not appear to have any effect. Is this expected?
(d/q
'[:find (pull ?e [(:db/id :as "foo")])
:where
[?e :db/ident :db/cardinality]]
(d/db conn))
=> [[#:db{:id 41, :ident :db/cardinality}]]
Other db/* attrs work:
(d/pull (d/db conn)
'[(:db/id :as "foo")
(:db/ident :as "ident")]
:db/cardinality)
=> {:db/id 41, :db/ident :db/cardinality, "ident" :db/cardinality}
This one is strange since :db/ident
is included twice.
:db/doc
is not included twice.
(d/pull (d/db conn)
'[(:db/id :as "foo")
(:db/ident :as "ident")
(:db/doc :as "doc")]
:db/cardinality)
=>
{:db/id 41,
:db/ident :db/cardinality,
"ident" :db/cardinality,
"doc" "Property of an attribute. Two possible values: :db.cardinality/one for single-valued attributes, and :db.cardinality/many for many-valued attributes. Defaults to :db.cardinality/one."}
@kenny :db/id
isn't an attribute. where is no Datom[e :db/id v]
. It's a "special thing" that d/pull
assoc into it's response.
So? I don't think pull requires the selection to be attributes. From the doc "Pull is a declarative way to make hierarchical (and possibly nested) selections of information about entities."
Even if that was a requirement, I don't think it makes sense for it to behave differently than everything else.
I agree that it's a bug
Why not:
'[[(descendants-of? ?root ?node)
[?node :node/parent ?root]]
[(descendants-of? ?root ?node)
[?intermediate :node/parent ?root]
(descendants-of? ?intermediate ?node)]]
?The second implementation of decendents-of? necessarily scans all :node/parent if ?node is unbound
if you know ?root is bound, you can start from that
in fact, datomic has a syntax for ensuring a rule input var is bound:
'[[(descendants-of? [?root] ?node)
[?node :node/parent ?root]]
[(descendants-of? [?root] ?node)
[?intermediate :node/parent ?root]
(descendants-of? ?intermediate ?node)]]
Ah let me take a look, I may have simplified the example a bit too much when lifting it from our codebase
Opened a support ticket for this https://support.cognitect.com/hc/en-us/requests/2797
Oh you are on to something! Thank you so much