asami

Asami, the graph database https://github.com/threatgrid/asami
quoll 2021-03-10T00:35:00.200400Z

Asami 2.0.0 alpha2 is now up.

šŸ‘ 1
1
šŸš€ 5
quoll 2021-03-10T00:36:27.201900Z

This fixes a few things from Alpha 1, including a new serializable datatype for entity nodes that doesnā€™t use keywords

quoll 2021-03-10T00:37:57.202300Z

Again, Iā€™d love any feedback people have. Thank you!

quoll 2021-03-10T02:53:06.203400Z

Iā€™d like to comment that the reason itā€™s called an ā€œalphaā€ is because Iā€™m following the mantra ā€œrelease early, release oftenā€.

quoll 2021-03-10T02:53:34.204Z

So there will be things to fix! But any and all feedback will be helpful. Thank you

Andrea 2021-03-10T15:44:24.207800Z

hello asamists šŸ‘‹:skin-tone-2:, Iā€™m somewhat confused by the semantic of retractions in Asami:

(def id
  (let [{:keys [tempids db-after]}
        @(d/transact conn
           [{:db/id -1
             :person/name "MsX"
             :person/address "10999"}])
        id (get tempids -1)]
    (println (d/entity db-after id))
    id))

(let [{:keys [db-after]}
      @(d/transact conn
         [[:db/retract id :person/address "10999"]])]
  (println (d/entity db-after id))) 
Iā€™m seeing :person/address in the second call to entity but I wouldnā€™t expect that. Maybe Iā€™m using wrong transaction data for the retraction? Tested this code with Asami 2.0.0-alpha2 (hereā€™s a notebook to reproduce https://nextjournal.com/a/PDfq1fshbQsrL4dv6GDn6y/edit?token=X6uXajPPMadokXn62oh5Se)

quoll 2021-03-10T17:00:10.208600Z

That confuses me too. I have a few things happening today look into this as soon as I can

Andrea 2021-03-11T09:58:58.210900Z

> and then using that in subsequentĀ `:db/retract`Ā statements. yes exactly. I wonder if in the prior example giving a :db/ident say ā€œidā€ instead of the tempid -1, then a retraction statement like [:db/retract "id" :person/address "10999"] would have worked. I tend to think about db/idents as https://docs.datomic.com/on-prem/schema/identity.html#lookup-refs but maybe I should not. Do Asami tx data in the vector form only support actual db/ids? (in other words node-1 mentioned in the wiki https://github.com/threatgrid/asami/wiki/Transactions#tx-data has to be a db/id)

quoll 2021-03-11T12:33:20.220800Z

When I found the bug I did notice that it does not support this. I thought that maybe it should, but then realized that this sort of statement means that you are explicitly trying to remove statements during the same transaction that youā€™re trying to add them. This doesnā€™t make sense (in Datomic it causes an exception). Itā€™s also difficult, because transactions are performed by performing deletions, followed by assertions. To make it work, these deletions would have to occur after the assertions that autogenerate that ID. Otherwise theyā€™d delete a non-existing statement (a no-op) before inserting it, meaning that the statement would still be created.

quoll 2021-03-11T12:34:56.222700Z

I could make it work, but I see no reason why it should. Unless thereā€™s a use-case Iā€™m not seeing?

mkvlr 2021-03-11T18:06:41.222900Z

I think @andrea712 is talking about "id" as a :db/ident , not a temp id

mkvlr 2021-03-11T18:07:12.223100Z

is it possible to perform a retraction using an :db/ident?

mkvlr 2021-03-11T18:07:40.223300Z

asami does not support datomic style lookup refs, e.g. [:db/ident "id"] correct?

Andrea 2021-03-11T18:08:21.223500Z

no, sorry for the confusion, itā€™s me not understanding well enough the :db/ident construct: I thought, since they can be used to lookup an entity which was asserted at any time via (asami.core/entity db my-ident) I thought that values of :db/ident could be used in vector statements for retractions in tx-data in 2nd position: eg [:db/retract my-ident attr val]

quoll 2021-03-11T18:10:08.223700Z

Asami doesnā€™t have that lookup styleā€¦ no. I made a ticket for it (and it wonā€™t be hard), but it hasnā€™t been a priority.

mkvlr 2021-03-11T18:11:34.223900Z

ok, so if you want to retract an attribute using a :db/ident you need to do a query for its :db/id first, correct?

šŸ‘ 1
Andrea 2021-03-11T18:11:51.224200Z

okay, weā€™re currently resolving :db/idents -> :db/id in retraction statements before transactions and it works well enough for the moment, I think

Andrea 2021-03-11T18:12:01.224400Z

yes @mkvlr doing that currently

quoll 2021-03-11T18:12:08.224600Z

idents are used for identifying entities, and we (as inā€¦ my team) only really wants to delete things from entities when updating a property (meaning, delete the existing statement, and insert a new one), then weā€™ve used the entity update annotation for that

quoll 2021-03-11T18:13:28.224900Z

Internally, all the entity code does those lookups for you. Which is to say that thereā€™s no big deal moving it to other places as well.

mkvlr 2021-03-11T18:13:53.225100Z

understood, supporting [:db/retract [:db/ident my-ident] attr val] would be useful for us, but of course also easy for us to do it before we submit the tx

quoll 2021-03-11T18:15:30.225300Z

So if I have the entity:

{:db/ident "Mary"
 :name "Mary Smith"
 :age 22}
Then we can update by saying:
{:db/ident "Mary"
 :age' 23}
That does the lookup, retracts the [id :age 22] and assert [id :age 23]

quoll 2021-03-11T18:15:37.225500Z

Sorryā€¦ I have to leave for an hour

mkvlr 2021-03-11T18:20:33.225700Z

yes, so given that entity with a :db/id :

{:db/id 123
 :db/ident "Mary"
 :name "Mary Smith"
 :age 22}
you can update like youā€™ve written but if I want to retract the :age itā€™s only possible (without a query) using [:db/retract 123 :age 22] and it might be nice to also support [:db/retract [:db/ident "Mary"] :age 22] like in datomic

mkvlr 2021-03-11T18:21:13.225900Z

also happy to look at that (probably tomorrow) if youā€™d accept a PR for this

šŸ‘ 1
quoll 2021-03-11T20:47:28.226100Z

If so, then it would go in src/asami/entities.cljc:133 and would probably do something like what line 60 does in that file

quoll 2021-03-11T20:50:52.226300Z

(let [node-ref (if id
                             (and (seq (gr/resolve-triple graph id '?a '?v)) id)
                             (ffirst (gr/resolve-triple graph '?r :db/ident ident)))
This goes straight to the graph (skipping the query API with the associated costs), and checks if the thing being looked for is valid as an entity node (i.e. it appears in entity-attribute-value tuples in the entity position), and if that doesnā€™t work out, then it checks to see if the value appears in the value position for a statement where the property is :db/ident.

quoll 2021-03-11T20:54:35.226500Z

the resolve-triple function returns a seq of bindings values. A bindings values is a vector containing position dependent bindings for the requested variables. So the first one will return a seq of vectors that have a count of 2 (representing values for the ?a and ?v), and the second will return a seq containing (hopefully) 1 vector, and that vector will have a count of 1.

quoll 2021-03-11T20:55:33.226700Z

The (if id ā€¦ part means that if a :db/id is provided, then that is considered canonical, and ignore the :db/ident

quoll 2021-03-11T20:57:26.227Z

These notes are in case you want to go ahead and do it. Otherwise, theyā€™re a prompt for me to get it done šŸ˜‰

šŸ™ 1
quoll 2021-03-11T20:58:19.227200Z

(ohā€¦ when I said line 133, that was for the deletions. Assertions will need to be treated the same way)

mkvlr 2021-03-12T13:24:41.227500Z

thanks for those pointers, Iā€™ll open a PR shortly

mkvlr 2021-03-12T13:32:47.227700Z

https://github.com/threatgrid/asami/pull/118

quoll 2021-03-12T14:42:14.228Z

Thank you @mkvlr. Iā€™ve responded to this

šŸ‘€ 1
quoll 2021-03-12T18:57:04.228300Z

I also updated Zuko (which does the deconstruction of entities into triples) to include the lookup refs

quoll 2021-03-12T18:57:23.228500Z

Itā€™s in main, and will come out in the next alpha

quoll 2021-03-12T18:57:34.228700Z

thank you again

mkvlr 2021-03-12T19:05:45.229100Z

thank you!

quoll 2021-03-10T17:34:26.208700Z

@andrea712 OKā€¦ itā€™s a bug. Iā€™ve fixed it and am just writing a test for it now. Somewhat embarrassingly, no one I work with retracts statements this way, and I didnā€™t have full test coverage, so it was overlooked. Thank you very much!

šŸ™‚ 1
quoll 2021-03-10T18:08:49.208900Z

Itā€™s up on github. Iā€™ll make an Alpha 3 when I get back later today

Andrea 2021-03-10T19:08:13.209300Z

oh, thank you so much! Iā€™ll check that tomorrow.

Andrea 2021-03-10T19:10:03.209500Z

I have more examples of retracting by db/idents which were not working as expected, will check those against alpha3...

quoll 2021-03-10T20:04:46.209700Z

Iā€™m not sure what ā€œretracting by db/identsā€ might mean. ā€œRetractionsā€ are a statement describing an entity-attribute-value tuple to be removed, while a :db/ident is a way to identify an entity that is stored as multiple tuples. The only thing I can think of is what you provided in your example, where you used :db/ident to identify the node that represents the entity, and then using that in subsequent :db/retract statements.

quoll 2021-03-10T20:49:27.210400Z

OKā€¦ this problem is addressed in 1.2.16 and 2.0.0-alpha3

šŸ™ 3
1