DDB can return Internal Server Errors (500). Datomic will occasionally get these and return them as an anomaly that looks like this.
{:datomic.client-spi/context-id "95591228-605e-4a62-aa0d-e4a9c9c83906", :cognitect.anomalies/category :cognitect.anomalies/fault, :datomic.client-spi/exception com.amazonaws.services.dynamodbv2.model.InternalServerErrorException, :datomic.client-spi/root-exception com.amazonaws.services.dynamodbv2.model.InternalServerErrorException, :cognitect.anomalies/message "Internal server error (Service: AmazonDynamoDBv2; Status Code: 500; Error Code: InternalServerError; Request ID: 71DOKMAO3VJQ82UHRMU4MMB7HVVV4KQNSO5AEMVJF66Q9ASUAAJG; Proxy: null)", :dbs [{:database-id "f3253b1f-f5d1-4abd-8c8e-91f50033f6d9", :t 90311936, :next-t 90311937, :history false}]}
Now the weird part is according to the DDB docs on "https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Programming.Errors.html#Programming.Errors.MessagesAndCodes.http5xx", you should retry these.
> You might encounter internal server errors while working with items. These are expected during the lifetime of a table. Any failed requests can be retried immediately.
Does the scenario "working with items" apply for how Datomic is using DDB? i.e., Should I be retrying these anomalies?AFAIK datomic does already retry these, but perhaps the retry policy was exhausted?
Datomic's retry?
Yeah, internally.
500s can happen during dynamo partition scaling operations
not uncommon
Huh, ok. Should I be retrying those anomalies from my end then?
Someone from datomic team can confirm, perhaps the anomaly is misclassified as a fault (unretriable)
In your experience, what is the scale of the duration this event may take? Seconds, minutes?
On the topic of miscategorized anomalies, I may have another one. Our system clearly had a fun time last night... A query timeout elapsing should be interrupted, not incorrect.
{:cognitect.anomalies/category :cognitect.anomalies/incorrect, :cognitect.anomalies/message "processing clause: [?cloud-acct :cs.model.monitored.cloud-acct/mode :cs.model.monitored.cloud-acct/mode-fetch-all], message: java.util.concurrent.TimeoutException: Query canceled: timeout elapsed", :dbs [{:database-id "493d38a5-5434-4c1d-81c6-c1412460540b", :t 2916826, :next-t 2916827, :history false}]}
at datomic.client.api.async$ares.invokeStatic (async.clj:58)
datomic.client.api.async$ares.invoke (async.clj:54)
datomic.client.api.sync$unchunk.invokeStatic (sync.clj:48)
datomic.client.api.sync$unchunk.invoke (sync.clj:46)
datomic.client.api.sync$eval2238$fn__2261.invoke (sync.clj:123)
datomic.client.api.impl$fn__11642$G__11635__11649.invoke (impl.clj:41)
datomic.client.api.impl$call_q.invokeStatic (impl.clj:150)
datomic.client.api.impl$call_q.invoke (impl.clj:147)
datomic.client.api$q.invokeStatic (api.clj:393)
datomic.client.api$q.invoke (api.clj:365)
Hello. In datomic, is there a way to register a listener for when there's a new basisTs? (I'm accessing dataomic through a java client)