datomic

Ask questions on the official Q&A site at https://ask.datomic.com!
shields 2021-01-22T00:07:07.009300Z

I'm using a universal login, nothing to do with Cognito. Context is that I'm doing a refactor and the current stack had Auth0 default authorizer integration but it's only available using HTTP, not REST. But I think I found a solution here in the forums. Just need to pass the token through the header from the request and then have a middleware verify on the backend. https://forum.datomic.com/t/where-can-i-find-cognito-or-iam-details-from-api-gateway-when-using-http-direct/1675/7

pyry 2021-01-22T05:40:11.009800Z

Hello! We're running Datomic Pro 1.0.6165 on-prem and are a bit puzzled as to why certain queries we think ought to be fast, well, aren't.

favila 2021-01-22T10:33:01.015Z

I expect your datoms version to be the fastest. The query versions will be retaining all records in memory at once. There’s no way in datomic to avoid visiting every item in that index. Datomic doesn’t have index metadata of eg cardinality info or set members

favila 2021-01-22T10:36:51.020900Z

You need to cache more (larger object cache, valcache or memcached secondary cache) or a faster storage; the first query issuance fill be slow but subsequent ones will be faster (assuming the same peer performs the query each time). If that’s still not good enough, consider keeping the counts precomputed. You can perform the query then have something listen to txs using tx-report-queue or tx-range polling to keep the count up to date

pyry 2021-01-22T14:41:34.021200Z

All right, thanks for this.

pyry 2021-01-22T05:40:22.009900Z

As part of a sysadmin's view to our system, we provide a page listing counts of entities per domain type stored in our datomic instance. There are roughly 10 million such entities at the moment, spread quite unevenly across 80+ domain types.

pyry 2021-01-22T05:40:38.010100Z

Rendering this page is at the moment slow, as getting the counts from datomic typically takes tens of seconds.

pyry 2021-01-22T05:41:47.010500Z

What we're doing at the moment to get the counts is roughly the following:

(->> (d/datoms db :aevt ::object/type)
       (map :v)
       (frequencies))

pyry 2021-01-22T05:43:10.010800Z

As mentioned, this doesn't perform too well on our data. A natural alternative would of course be to query the counts instead, which I think was what we did earlier. If memory serves me correctly, this however didn't perform too well either..

pyry 2021-01-22T05:43:42.011Z

Nonetheless, I think it makes sense to again try writing the query as something like so:

(d/q '[:find ?t (count ?e)
       :in $
       :where [?e ::object/type ?t]]
       db)

pyry 2021-01-22T05:45:24.011300Z

Question: Should datomic be able to get the (count ?e) above efficiently from eg. some index metadata (if that's a thing) or will it have to essentially traverse the entire index to calculate the counts?

pyry 2021-01-22T05:46:27.011600Z

Additionally, I'm wondering if I should expect a call to qseq instead of q to perform better with the above query?

favila 2021-01-22T10:33:01.015Z

I expect your datoms version to be the fastest. The query versions will be retaining all records in memory at once. There’s no way in datomic to avoid visiting every item in that index. Datomic doesn’t have index metadata of eg cardinality info or set members

favila 2021-01-22T10:36:51.020900Z

You need to cache more (larger object cache, valcache or memcached secondary cache) or a faster storage; the first query issuance fill be slow but subsequent ones will be faster (assuming the same peer performs the query each time). If that’s still not good enough, consider keeping the counts precomputed. You can perform the query then have something listen to txs using tx-report-queue or tx-range polling to keep the count up to date

pyry 2021-01-22T14:41:34.021200Z

All right, thanks for this.

2021-01-22T23:10:16.023200Z

Are :limit and :offset supported in all Datomic versions with index-pull? or were those params added in more recent versions? The index-pull doc string doesn't show those options, but this page demonstrates their use https://docs.datomic.com/on-prem/index-pull.html

2021-01-22T23:10:46.023300Z

I'm using on-prem, not client