datomic

Ask questions on the official Q&A site at https://ask.datomic.com!
2020-10-23T02:44:35.127800Z

thanks @lanejo01, I'll check it out

cmdrdats 2020-10-23T11:58:55.129900Z

hi - we're trying to restore a backup of our datomic database (it's a tiny 9mb db, as a test), and it just seems to be hanging forever.. how do we go about figuring out what we're doing wrong? We've set the log level in the logback.xml for datomic to TRACE - and theres a tiny bit of logging (I'll attach that in thread), but nothing else

cmdrdats 2020-10-23T11:59:06.130Z

2020-10-23T14:33:52.132300Z

I am trying to get started with datomic in a dev-local setup. Any idea why my dev-local db is not found? I can see can see db.log and log.idx files were created.

(ns flow.db
  (:require [datomic.client.api :as d]))

(let [dir     (str (System/getProperty "user.dir") "/var/datomic")
      db-name "flow"
      client  (d/client {:server-type :dev-local
                         :storage-dir dir
                         :system      "dev"})
      _       (d/create-database client db-name)
      conn    (d/connect client {:db-name db-name})]
  conn)

;; Unhandled clojure.lang.ExceptionInfo
;; Db not found: flow
;; #:cognitect.anomalies{:category :cognitect.anomalies/not-found,
;;                       :message "Db not found: flow"}

jaret 2020-10-24T14:05:22.150200Z

Totally agree this seems to be a bug in using dev-local client and create DB. It should throw an error like in cloud for expected map. I've logged a bug and we'll look at fixing!

jaret 2020-10-24T14:06:07.150400Z

;Cloud client
(d/create-database client "testing")
Execution error (ExceptionInfo) at datomic.client.impl.shared/api->client-req (shared.clj:258).
Expected a map

;Dev local client
(d/create-database client "testing")
=> true

jaret 2020-10-24T14:12:12.150600Z

in either case no DB is created, but we should throw an error in both cases!

2020-10-24T15:48:36.156400Z

Thanks for all the help. Working as expected now.πŸ˜€

jaret 2020-10-23T14:48:10.132500Z

Can you try testing with the absolute path of dir and just execute d/client without the let? Also does the absolute path already have a "dev" system folder?

jaret 2020-10-23T14:48:45.132700Z

@cmdrdats Can you share what command you're using to run backup/restore?

jaret 2020-10-23T14:49:04.132900Z

If you'd like feel free to log a case to me by e-mailing <mailto:support@cognitect.com|support@cognitect.com> and we can dive further.

jaret 2020-10-23T14:49:44.133100Z

then list-dbs on the client

jaret 2020-10-23T14:50:29.133300Z

(d/list-databases client {})

2020-10-23T14:56:17.133600Z

Sure.

(def client (d/client {:server-type :dev-local
                       :storage-dir "/home/jam/src/flow/var/datomic"
                       :system      "dev"}))
(d/create-database client "flow")  ; =&gt; true
(d/list-databases client {})   ; =&gt; []

2020-10-23T14:57:27.133800Z

file system looks like this:

$ tree /home/jam/src/flow
/home/jam/src/flow
β”œβ”€β”€ deps.edn
β”œβ”€β”€ <http://FLOW.org|FLOW.org>
β”œβ”€β”€ src
β”‚Β Β  └── flow
β”‚Β Β      └── db.clj
└── var
    └── datomic
        └── dev
            β”œβ”€β”€ db.log
            └── log.idx

2020-10-23T15:00:37.134Z

I created var/datomic. dev/ gets created by create-database fn

2020-10-23T15:17:12.134200Z

I tried setting :storage-dir in ~/.datomic/dev-local.edn and it has the same problem. Creates some files but no db found.

jaret 2020-10-23T15:30:42.134800Z

What version of Dev-local are you using?

jaret 2020-10-23T15:31:52.135Z

Can you share the .datomic/dev-local.edn file in your home directory?

jaret 2020-10-23T15:32:24.135200Z

Might be best to just share your deps.edn and I will try to re-create. So I can see version of client and dev-local.

2020-10-23T15:37:51.135600Z

jam@stick:~/.datomic$ cat dev-local.edn 
{:storage-dir "/home/jam/var/datomic"}

2020-10-23T15:39:00.136Z

jam@stick:~/opt$ java -version
openjdk version "11.0.8" 2020-07-14

jaret 2020-10-23T16:02:53.136200Z

@millettjon I don't see a version of client in your deps? Do you have datomic client in your .clojure/deps.edn? Could you include com.datomic/client-cloud "0.8.102"

jaret 2020-10-23T16:07:33.136800Z

Also worth testing with the latest dev-local and use 0.9.225

2020-10-23T16:38:49.137100Z

Ok. I was following instructions here: https://docs.datomic.com/cloud/dev-local.html and didn't know about that additional dep. Unfortunately, adding it didn't make any difference. I will try updating to 0.9.225.

jaret 2020-10-23T17:41:26.137500Z

@millettjon I can't reproduce is there anything else I could be missing? Are you starting a new repl to do this? Can you try making a new system name to confirm you are local when a new dir is made? What OS are you using on your system?

2020-10-23T17:52:58.139200Z

What's the best way to report/ask about questions on the official datomic docs? e.g The docs for import-cloud on the docs: https://docs.datomic.com/cloud/dev-local.html#import-cloud Don't match the docstring. I assume the docstring is correct as the documentation page lists the same value for source and dest.

2020-10-24T18:06:31.158700Z

@jaret thanks for fixing it. Do you know if there is a light weight way to get datomic cloud consulting? I run into little blockers here and there and I would rather shell some money then get stuck for a day off someone can easily help me trouble shoot things.

jaret 2020-10-24T18:12:28.158900Z

@drewverlee I am going to tag @marshall on this. You can also e-mail him directly at <mailto:marshall@cognitect.com|marshall@cognitect.com>. He is running point on our Datomic consulting services along with a few other folks at Cognitect. I'll bring this up with him on Monday if you two don't connect here.

dpsutton 2020-10-23T17:54:07.139900Z

There’s a new ask.datomic site that probably fits the bill

3πŸ‘
joshkh 2020-10-23T17:58:44.141300Z

should one be alarmed when periodically seeing an Unable to load index root ref &lt;uuid&gt; exception appear in Datomic Cloud logs? (from com.amazonaws.services.dynamodbv2.model.InternalServerErrorException )

joshkh 2020-10-26T12:40:14.170300Z

thanks, Jaret. we haven't noticed any performance issues related to the exception, so i was mostly just curious. but since you mentioned that it could be related to often deleting DBs, which we very rarely do, then i'll just mention it in some future support ticket. it's a low priority for us. πŸ™‚

cmdrdats 2020-10-23T19:10:55.141500Z

@jaret we're doing this:

/storage/datomic/current/bin/datomic -Xmx1g -Xms1g restore-db "file:/storage/datomic/archive/restore/" "datomic:sql://...?jdbc:<mysql://192.169.56.1:3305/datomic?user=...&amp;password=>..."
if there's nothing we can really self diagnose at a high level, then sure, I'll send a mail on monday πŸ™‚

ghadi 2020-10-23T19:16:28.141800Z

look at the args for create-database

ghadi 2020-10-23T19:16:58.142Z

needs to be {:db-name "flow"} not "flow"

1πŸ‘
ghadi 2020-10-23T19:17:31.142300Z

@millettjon @jaret

jaret 2020-10-23T19:25:27.142700Z

!!! Ghadi, great catch!

jaret 2020-10-23T19:25:30.142900Z

That's it

ghadi 2020-10-23T19:26:00.143100Z

calling it with a raw string probably shouldn't return true

jaret 2020-10-23T19:26:15.143300Z

No it should not!

jaret 2020-10-23T19:27:32.143500Z

Try restoring locally using :dev protocol, just to see if it's underlying storage related. But if it couldn't write to storage due to perms I would expect an error not a hang.

2020-10-23T19:56:37.143800Z

Thanks!

jaret 2020-10-23T20:07:47.144Z

I'll fix that!

jaret 2020-10-23T20:07:58.144200Z

Thanks for catching it.

jaret 2020-10-23T20:14:55.144400Z

Do you often delete DBs? In general, you can see this error if a client or node is asking for a deleted db. The error would also correlate to an outage but these calls are retried so if you don't see this error often or repeatedly it's probably not a major issue.

jaret 2020-10-23T20:15:53.144600Z

As always, it is my support-person duty to recommend upgrading to the latest Cloud release CFT and if you'd like me to look more closely at your system logs I'd be happy to poke around with a Read-Only CloudWatch account. If you want to go down that path, log me a case at <mailto:support@cognitect.com|support@cognitect.com> and I can take a look.

2020-10-23T22:51:58.146300Z

When i try to use dev-tools to import a cloud locally it complains that it can't use the endpoint to connect/"name or service is not known". I'm connected and can query the databases though so i'm not sure what the issue is.