datomic

Ask questions on the official Q&A site at https://ask.datomic.com!
adamtait 2020-08-27T02:42:12.071200Z

I have a query on a Datomic Ion that consistently returns partially different results when run remotely vs on the Ion. The query returns 9 results (both remotely & on-ion) but remotely returns correct entity-ids and on-ion returns 3 correct entity-ids and 6 empty/new entity-ids. I think this will be hard to reproduce b/c I can’t even reproduce it on a 2nd Ion with the same schema (it works correctly there). Here’s the query in case anyone can see something obviously wrong:

(d/q
    '[:find ?mp
      :in $ ?low ?high
      :where
      [?mp :membership-period/end ?end]
      [(< ?low ?end)]
      [(< ?end ?high)]]
    db inst-low inst-high)
Has anyone experienced this strange inconsistent behavior before? I’d love to hear from you.

joshkh 2020-08-27T08:01:03.073900Z

i've seen inconsistencies when using > < operators on dates. try using [(.after ?date1 ?date2)]

Joe Lane 2020-08-27T13:05:13.085900Z

Are you using (Date.) in the ion? I would confirm time zones aren’t messing this up.

adamtait 2020-08-27T22:36:38.092400Z

Thanks both of you! I’m using the standard Java (Date.) objects in the comparison. I also thought it might be a timezone issue, so I ran the same query with wider date ranges and got similarly strange results. I was able to get the query working consistently using [(.before ?low ^java.util.Date ?end)] . I’m still not sure why Datomic was returning :db/ids for non-referenced entities in the results but at least I can move on… Thanks again!

cmdrdats 2020-08-27T07:58:54.073600Z

Good morning! I remember seeing/hearing somewhere that :db/index is now superfluous, or, at least, we should actually always make it :db/index true , since it's basically doing that under the hood in Datomic now anyhow... I can't find the source now, but the docs still state :db/index defaults to false - so is this something we still need to consider for our schema?

cmdrdats 2020-09-01T13:49:00.171300Z

ah, cool, thanks!

cmdrdats 2020-08-27T07:59:14.073800Z

(I'm using on-prem datomic)

joshkh 2020-08-27T08:01:03.073900Z

i've seen inconsistencies when using > < operators on dates. try using [(.after ?date1 ?date2)]

Ben Sless 2020-08-27T09:38:35.076300Z

Hi all, working through the https://docs.datomic.com/on-prem/tutorial.html I find myself missing a slightly more complex use case. For an example of order items, what if I wanted to edit an order and retract a specific item or update an item's amount? I find it a bit complicated when dealing with componenets

Ben Sless 2020-08-28T08:33:45.097200Z

yeah, it looks like to be safe and atomic you must use a tx fn

cjsauer 2020-08-28T13:31:46.099700Z

Somewhat out of date now, but Valentin’s Datofu library had some helpers for resetting to-many relations as a tx fn. It might be worth studying the code for ideas: https://github.com/vvvvalvalval/datofu#resetting-to-many-relationships

2👍
favila 2020-08-27T10:30:23.078100Z

You are probably thinking of datomic cloud: it makes a value index for everything so it doesn’t have a :db/index attribute at all

favila 2020-08-27T10:30:47.079Z

For on prem nothing has changed, and you should still consider whether to index or not

marshall 2020-08-27T11:37:01.079200Z

No, import cloud is only for dev local

v 2020-08-27T12:16:51.083100Z

What’s so complicated about that ?

v 2020-08-27T12:17:12.084Z

Do you have a code that you would like to share?

Kevin Mungai 2020-08-27T12:18:29.084700Z

Hi, while choosing an AWS region for Datomic Cloud I found out that https://aws.amazon.com/local/africa/ is https://aws.amazon.com/marketplace/pp/prodview-otb76awcrb7aa?qid=1598530512130&sr=0-1&ref_=srh_res_product_title. How do I go about choosing the Africa Region (Capetown)? Also, is it possible to migrate from one AWS region using Datomic Cloud to another AWS Region? e.g. AWS EU (London) to AWS Africa Region (Capetown). Thanks in advance for your answers.

Joe Lane 2020-08-27T13:05:13.085900Z

Are you using (Date.) in the ion? I would confirm time zones aren’t messing this up.

souenzzo 2020-08-27T13:20:35.086100Z

About move across instances, Once you do a datomic backup, um can restore it in any other instance, including a local instance in your machine.

marshall 2020-08-27T13:34:33.086300Z

@souenzzo That is true for Datomic On-Prem, not for Datomic Cloud

1😱
marshall 2020-08-27T13:37:09.086500Z

@mungaikamau7 Datomic Cloud is not currently available in the South Africa region. I can investigate whether we could add support for that region in a future release. We don't currently have a built-in mechanism for transferring a Datomic Cloud database from one AWS region to another. However, because Datomic keeps the history of your entire database (the log) you could write a tool that reads the log of an existing database and transacts the same data into a new database in your new region.

Kevin Mungai 2020-08-27T13:57:52.086900Z

Looking forward to it. I am from Nairobi, Kenya and the nearest AWS Region is South Africa, followed by UAE, India and then the EU. The EU regions are not bad because African fibre optic cables connect to the EU countries and many African companies use cloud providers that are thousands of miles away from their customers. Adding more regions would certainly help with latency. Thanks.

marshall 2020-08-27T14:15:29.087100Z

We do have some India regions available for now I investigate the Cape Town region

Kevin Mungai 2020-08-27T14:28:40.087300Z

Sorry I didn't find the India Region. Maybe am restricted from choosing the India Region due to my location.

marshall 2020-08-27T14:53:21.087500Z

My apologies - I believe we're working on the Mumbai region support now

marshall 2020-08-27T14:53:25.087700Z

it should be available in a future release

Kevin Mungai 2020-08-27T14:56:09.087900Z

Thanks. Looking forward to it.

Ben Sless 2020-08-27T15:05:47.088100Z

@vishal.gautam sure

(def schema
  [{:db/ident :product/name
    :db/cardinality :db.cardinality/one
    :db/unique :db.unique/identity
    :db/valueType :db.type/string}

   {:db/ident :order/items
    :db/valueType :db.type/ref
    :db/cardinality :db.cardinality/many
    :db/isComponent true}

   {:db/ident :order/id
    :db/valueType :db.type/long
    :db/cardinality :db.cardinality/one
    :db/unique :db.unique/identity}

   {:db/ident :item/product
    :db/valueType :db.type/ref
    :db/cardinality :db.cardinality/one}

   {:db/ident :item/amount
    :db/valueType :db.type/long
    :db/cardinality :db.cardinality/one}])

Ben Sless 2020-08-27T15:05:57.088300Z

@(d/transact  conn schema)
@(d/transact conn [{:product/name "coke"} {:product/name "pizza"}])

Ben Sless 2020-08-27T15:06:07.088500Z

Then a simple order

@(d/transact
  conn
  [{:order/id 1
    :order/items
    [{:item/product [:product/name "pizza"] :item/amount 3}
     {:item/product [:product/name "coke"] :item/amount 1}]}])

Ben Sless 2020-08-27T15:07:31.088700Z

Now assuming there was a mistake and the client ordered just one pizza, can I write it with reference only to the order/id? Because if I do the following it just adds another item

@(d/transact
  conn
  [{:order/id 1
    :order/items [{:item/product [:product/name "pizza"] :item/amount 1}]}])

Ben Sless 2020-08-27T15:07:49.088900Z

Or do I have to refer to the item's eid?

kardan 2020-08-27T15:08:06.089100Z

I guess we’re all interested in different regions, I been looking out for Stockholm 🙂

Kevin Mungai 2020-08-27T15:12:23.089300Z

Awesome to see Datomic Cloud has interest from different places in the world. Maybe Datomic Cloud should be available in all AWS regions.

Ben Sless 2020-08-27T15:16:52.089500Z

I'm exploring the best model for my domain. It looks like I have entities which can have optional components, and I might get an update like "These were the components previously, these are the components now". There might be commonalities between them, but no guarantee of that.

kenny 2020-08-27T15:26:14.089700Z

Is there a different way to copy data from one Datomic system to another? The use case being to copy production data into a non-prod system for deep analysis, transformation, and testing.

marshall 2020-08-27T16:10:46.089900Z

@kardan we attempted to launch in Stockholm region earlier this year, but AWS Marketplace didnt have the correct support there yet

2
marshall 2020-08-27T16:11:20.090100Z

You could write a program to read the log and transact the data into another system

marshall 2020-08-27T16:11:41.090300Z

there are some community blogs/etc about that approach

marshall 2020-08-27T16:12:10.090500Z

it requires some work because you have to maintain a mapping of entity IDs, as there will new EIDs in the new database

kenny 2020-08-27T16:14:15.090700Z

Right. Curious, is there a technical reason import-cloud couldn't connect to a separate Datomic system?

marshall 2020-08-27T16:19:27.090900Z

i dont know 🙂

1
v 2020-08-27T17:26:33.091200Z

@ben.sless you will need the item id to update the amount. So here is the code that does that

(defn update-product [product-name amount]
  (let [item-id (d/q '[:find ?eid .
                       :in $ ?product
                       :where
                       [?eid :item/product ?product]]
                     (d/db conn) [:product/name product-name])]
    (d/transact conn [{:db/id item-id
                       :item/amount amount}])))

(update-product "pizza" 1)

(d/q '[:find ?count .
       :where
       [?id :item/product [:product/name "pizza"]]
       [?id :item/amount ?count]]
    (d/db conn))

Ben Sless 2020-08-27T17:43:49.091400Z

I see. No way around it, then. Thanks

kardan 2020-08-27T18:21:09.091700Z

That became an odd emoji.. 🙂

Jake Shelby 2020-08-27T22:11:03.091900Z

just to push on this use case some more (because I've had similar wishes to @ben.sless 's) ... @vishal.gautam since you are getting the item id first, and then transacting it, it's not concurrent safe, especially since Ben is talking about possible retracting the sub-entities (like replacing) ... I wonder if there is a way to do this in one transaction, maybe using a TX fn??

Jake Shelby 2020-08-27T22:12:39.092100Z

(another similar use case to this, is a many cardinality attr, and you want to replace the current set of values with new values, but it's important to do it in one transaction for concurrency concerns)

adamtait 2020-08-27T22:36:38.092400Z

Thanks both of you! I’m using the standard Java (Date.) objects in the comparison. I also thought it might be a timezone issue, so I ran the same query with wider date ranges and got similarly strange results. I was able to get the query working consistently using [(.before ?low ^java.util.Date ?end)] . I’m still not sure why Datomic was returning :db/ids for non-referenced entities in the results but at least I can move on… Thanks again!