datomic

Ask questions on the official Q&A site at https://ask.datomic.com!
twashing 2020-09-24T00:19:24.017600Z

Ok, nevermind. Sorted it!

steveb8n 2020-09-24T02:29:49.017800Z

exactly. almost anything can be generalised with this design. It’s non-trivial but worth it imho

tatut 2020-09-24T06:22:56.019300Z

any recommendations for backup/restore in datomic cloud? the feature request hasn’t seen activity in a long time. we’ve been rolling our own application level export/import but that unfortunately makes every entity id change

steveb8n 2020-09-24T09:23:10.000300Z

basically there isn’t a solution except to build your own. the (un)official answer seems to be that backups are not required because s3 is so reliable and, because there’s no excision, no data is ever lost

1😮
steveb8n 2020-09-24T09:26:34.000500Z

I can see the sense in that response but I think it doesn’t account for our customers who don’t understand our powerful new toolset. it forces us to take our customers out of their comfort zone which is not great for conservative (i.e. many enterprise) customers

2➕
steveb8n 2020-09-24T09:28:07.000700Z

the new local dev client supports importing cloud data to local if that’s one of your use-cases

steveb8n 2020-09-24T09:28:21.000900Z

but for migration, you have to roll your own

steveb8n 2020-09-24T09:30:03.001100Z

I’ll be happy to be corrected on any of these interpretations. FWIW it doesn’t change the fact that I really like the cloud managed service.

steveb8n 2020-09-24T10:12:01.001300Z

I have a uuid attr on every entity. If you have this, you don't care about entity IDs changing. Is that something you have tried?

kennytilton 2020-09-24T11:47:51.001700Z

Not sure if you have the luxury of time, but from what little I know it would be a good idea to do what it takes to abide entity-ids changing. My understanding is that backup/restore will not guarantee entity-ids being unchanged.

joshkh 2020-09-24T21:36:21.003600Z

i am in the exact same boat right now, and am finding it challenging to justify to our enterprise customers that we can't "simply" backup/restore a db from storage to meet their (i.e. not our) DR requirements. and unfortunately for us, dev-local is not yet an option because we have string values that exceed dev-local's max character limit. that being said, i did a small test with dev-local to replay the demo mbrainz db transaction log into a new db and it worked well, but the t values are of course different which is a real shame.

steveb8n 2020-09-25T02:03:34.005900Z

@joshkh exactly! despite the fact that we might not need backup/recovery, in an enterprise sales cycle this can be a real problem. Even worse if it’s an RFP situation because the prospect writing the RFP might consider common DR techniques to be a must have. It doesn’t matter that we can explain it away, the internal politicians in the customer can simply use this as a battering ram to avoid choosing our product. It’s not always a technical question: I hope one day Cognitect will provide an answer for export so that Datomic Cloud can be used without this risk in the sales cycle. @marshall any comments on this?

steveb8n 2020-09-25T02:06:27.006100Z

what’s interesting is that Datomic provides DR features that other dbs cannot e.g. you can recover a single tenants data to any point in time, even in a multi-tenant system. Update in-place dbs cannot do this. So we are technically superior for DR. But that doesn’t always work in enterprise sales.

Nassin 2020-09-25T02:09:44.006300Z

Curious, why does your application relies on entiti ids ?

tatut 2020-09-25T05:00:31.006500Z

DR is also to protect against human error, customers want to have a backup file stored in a completely different AWS account S3 bucket… or even have it downloaded to their own infastructure

tatut 2020-09-25T05:01:27.006700Z

and when you have backups you need to be able to restore from them… now we’ve rolled our own and fixed some mistakes we’ve made in relying too much on :db/id values

steveb8n 2020-09-25T06:22:22.007Z

that’s true. you could accidentally delete your Datomic s3 bucket and then you’d be finished! goodbye biz 😞

steveb8n 2020-09-25T06:23:40.007200Z

I wonder if some kind of s3 level backup would be supported by Datomic to guard against this?

stuarthalloway 2020-09-25T19:31:29.007600Z

@joshkh @steveb8n We hear you and working on things in this area.

2👍1🙏
joshkh 2020-09-25T19:33:03.008100Z

cheers. i know that we're in good hands. 🙂

daniel.spaniel 2020-09-27T18:23:29.029100Z

This is mega important, not just for enterprise, but for any datomic cloud user who wants to preserve their sanity. The sooner the better Stuart. This is huge deal.

tatut 2020-09-24T06:23:30.019400Z

mainly for disaster recovery or just migrating the database to a new environment

steveb8n 2020-09-24T09:23:10.000300Z

basically there isn’t a solution except to build your own. the (un)official answer seems to be that backups are not required because s3 is so reliable and, because there’s no excision, no data is ever lost

1😮
steveb8n 2020-09-24T09:26:34.000500Z

I can see the sense in that response but I think it doesn’t account for our customers who don’t understand our powerful new toolset. it forces us to take our customers out of their comfort zone which is not great for conservative (i.e. many enterprise) customers

2➕
steveb8n 2020-09-24T09:28:07.000700Z

the new local dev client supports importing cloud data to local if that’s one of your use-cases

steveb8n 2020-09-24T09:28:21.000900Z

but for migration, you have to roll your own

steveb8n 2020-09-24T09:30:03.001100Z

I’ll be happy to be corrected on any of these interpretations. FWIW it doesn’t change the fact that I really like the cloud managed service.

steveb8n 2020-09-24T10:12:01.001300Z

I have a uuid attr on every entity. If you have this, you don't care about entity IDs changing. Is that something you have tried?

arohner 2020-09-24T11:02:20.001500Z

Where are the API docs for the proxy? I’m not finding anything

kennytilton 2020-09-24T11:47:51.001700Z

Not sure if you have the luxury of time, but from what little I know it would be a good idea to do what it takes to abide entity-ids changing. My understanding is that backup/restore will not guarantee entity-ids being unchanged.

joshkh 2020-09-24T21:36:21.003600Z

i am in the exact same boat right now, and am finding it challenging to justify to our enterprise customers that we can't "simply" backup/restore a db from storage to meet their (i.e. not our) DR requirements. and unfortunately for us, dev-local is not yet an option because we have string values that exceed dev-local's max character limit. that being said, i did a small test with dev-local to replay the demo mbrainz db transaction log into a new db and it worked well, but the t values are of course different which is a real shame.

joshkh 2020-09-24T21:47:41.004900Z

for the sake of testing query performance, is there a way to flush/bust the query cache in Datomic Cloud other than by renaming bound variables?

steveb8n 2020-09-24T22:54:44.005300Z

There aren’t any docs. This technique relies upon undocumented (i.e. unsupported) use of the api client protocols. you can see an example of this here https://github.com/ComputeSoftware/datomic-client-memdb/blob/master/src/compute/datomic_client_memdb/core.clj

steveb8n 2020-09-24T23:13:27.005600Z

in the (unlikely) event that Cognitect changes these protocols, you can always refactor using this technique (which is where I first tried the middleware idea) https://github.com/stevebuik/ns-clone