Ask questions on the official Q&A site at https://ask.datomic.com!
kenny 2021-05-14T14:37:23.102200Z

DDB can return Internal Server Errors (500). Datomic will occasionally get these and return them as an anomaly that looks like this.

{:datomic.client-spi/context-id "95591228-605e-4a62-aa0d-e4a9c9c83906", :cognitect.anomalies/category :cognitect.anomalies/fault, :datomic.client-spi/exception com.amazonaws.services.dynamodbv2.model.InternalServerErrorException, :datomic.client-spi/root-exception com.amazonaws.services.dynamodbv2.model.InternalServerErrorException, :cognitect.anomalies/message "Internal server error (Service: AmazonDynamoDBv2; Status Code: 500; Error Code: InternalServerError; Request ID: 71DOKMAO3VJQ82UHRMU4MMB7HVVV4KQNSO5AEMVJF66Q9ASUAAJG; Proxy: null)", :dbs [{:database-id "f3253b1f-f5d1-4abd-8c8e-91f50033f6d9", :t 90311936, :next-t 90311937, :history false}]}
Now the weird part is according to the DDB docs on "https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Programming.Errors.html#Programming.Errors.MessagesAndCodes.http5xx", you should retry these. > You might encounter internal server errors while working with items. These are expected during the lifetime of a table. Any failed requests can be retried immediately. Does the scenario "working with items" apply for how Datomic is using DDB? i.e., Should I be retrying these anomalies?

ghadi 2021-05-14T14:38:12.102800Z

AFAIK datomic does already retry these, but perhaps the retry policy was exhausted?

kenny 2021-05-14T14:38:37.103100Z

Datomic's retry?

ghadi 2021-05-14T14:40:13.104100Z

Yeah, internally.

ghadi 2021-05-14T14:40:14.104200Z

500s can happen during dynamo partition scaling operations

ghadi 2021-05-14T14:40:14.104300Z

not uncommon

kenny 2021-05-14T14:40:51.104900Z

Huh, ok. Should I be retrying those anomalies from my end then?

ghadi 2021-05-14T14:42:24.105900Z

Someone from datomic team can confirm, perhaps the anomaly is misclassified as a fault (unretriable)

kenny 2021-05-14T14:44:00.106100Z

In your experience, what is the scale of the duration this event may take? Seconds, minutes?

kenny 2021-05-14T15:02:20.107400Z

On the topic of miscategorized anomalies, I may have another one. Our system clearly had a fun time last night... A query timeout elapsing should be interrupted, not incorrect.

{:cognitect.anomalies/category :cognitect.anomalies/incorrect, :cognitect.anomalies/message "processing clause: [?cloud-acct :cs.model.monitored.cloud-acct/mode :cs.model.monitored.cloud-acct/mode-fetch-all], message: java.util.concurrent.TimeoutException: Query canceled: timeout elapsed", :dbs [{:database-id "493d38a5-5434-4c1d-81c6-c1412460540b", :t 2916826, :next-t 2916827, :history false}]}
 at datomic.client.api.async$ares.invokeStatic (async.clj:58)
    datomic.client.api.async$ares.invoke (async.clj:54)
    datomic.client.api.sync$unchunk.invokeStatic (sync.clj:48)
    datomic.client.api.sync$unchunk.invoke (sync.clj:46)
    datomic.client.api.sync$eval2238$fn__2261.invoke (sync.clj:123)
    datomic.client.api.impl$fn__11642$G__11635__11649.invoke (impl.clj:41)
    datomic.client.api.impl$call_q.invokeStatic (impl.clj:150)
    datomic.client.api.impl$call_q.invoke (impl.clj:147)
    datomic.client.api$q.invokeStatic (api.clj:393)
    datomic.client.api$q.invoke (api.clj:365)

Puneet Arora 2021-05-14T21:05:58.109600Z

Hello. In datomic, is there a way to register a listener for when there's a new basisTs? (I'm accessing dataomic through a java client)