datomic

Ask questions on the official Q&A site at https://ask.datomic.com!
helios 2021-03-09T08:47:04.132300Z

Hey folks, I want your opinion. I need to put in my datomic schema an ordered list of refs to other entities. Since datomic doesn't support that natively, how would you do that? The difficult part is probably how to make sure the ordering stays consistent. Are composite tuples a good way of modeling that? :thinking_face:

helios 2021-03-09T08:49:44.132500Z

What I had in mind was something like suggested here: https://forum.datomic.com/t/handling-ordered-lists/305/5

2021-03-09T09:08:52.134200Z

Hey! Our team is using Datomic Analytics for some dashboards. One metric is based on the depth of trees in our system. I noticed the query sometimes breaks, I think in datomic-land. The result of the query is:

java.sql.SQLException: Query failed (#20210309_090647_00001_tab9g): bytes is negative

2021-03-09T09:09:32.134300Z

WITH RECURSIVE t(lvl, parent, db__id) AS (
  SELECT 1, parent, db__id
  FROM mydb.node
  WHERE parent IS NULL
  UNION ALL
  SELECT lvl + 1, t.parent, t.db__id
  FROM mydb.node
  JOIN t
    ON mydb.node.parent = t.db__id
)
SELECT * FROM t LIMIT 1;
This is the query

2021-03-09T09:09:50.134500Z

The node table is simply db__id, parent, and name. where parent refers to another node (or nil)

2021-03-09T09:10:52.134800Z

It sometimes happens. So running the query multiple times yields different results. We're currently on datomic 1.0.6222.

2021-03-09T10:18:56.135700Z

Spotted another odity; binding the same attribute multiple times doesn't work (in datomic analytics):

java.sql.SQLException: Query failed (#20210309_101720_00072_tab9g): Symbol name already has assignment "name_20", while adding "name_24"

2021-03-09T10:19:28.135800Z

WITH RECURSIVE t(lvl, parent, db__id, name, root_name) AS (
  SELECT 1, r.parent, r.db__id, r.name, r.name
  FROM mydb.node AS r
  WHERE r.parent IS NULL
  UNION ALL
  SELECT lvl + 1, c.parent, c.db__id, c.name, root_name
  FROM mydb.node as c
  JOIN t
    ON c.parent = t.db__id
)

SELECT * FROM t ORDER BY t.lvl
Given this query. The need for 2 name-bindings is so the leaf and the root are shown

danm 2021-03-09T10:47:42.137800Z

https://docs.datomic.com/cloud/client/client-api.html#timeouts says that timeouts are normally returned as ::anom/unavailable, but we seem to get a lot of ::anom/interrupted exceptions where the text content is Datomic Client Timeout. Is there some bad way of interacting we're likely to be doing that would cause us to get interrupted rather than unavailable?

2021-03-09T11:12:54.138400Z

If I have a to-many ref attribute, is there an easy way to query for that being empty?

2021-03-09T11:17:08.138800Z

I suppose maybe missing? is the easiest way.

1
2021-03-09T11:27:44.140100Z

I'm not quite sure how to use missing? in an or clause, though, because of the unification requirement. Hmm.

Joe Lane 2021-03-09T16:36:35.142100Z

@jeroen.dejong how consistently can you reproduce? Once you can consistently, can you upgrade to the latest release and then try to repro again?

Joe Lane 2021-03-09T16:37:34.143100Z

This ball is entirely in prestosql’s court.

2021-03-09T16:52:21.144200Z

Fair enough! It was hard to judge this one. I simply wrapped the value in a array, which did the trick.

2021-03-09T16:55:37.147500Z

I could not repro consistently. But I noticed an error in my query. Its recurring until OOM. Sometimes returning "out of resource " (expected). Sometimes returning the bytes-error. I'll upgrade the cluster later this week anyway and see if I can reproduce.

Joe Lane 2021-03-09T17:01:37.147700Z

What is the size of your analytics gateway and is it pointed at a query group?

2021-03-09T17:24:58.150300Z

I'm using on prem, right now it's a test setup with 10GB ram per query and 16GB max object heap. As a reference; when I fixed the query it consumed 1.5GB, the entire datomic DB is under a gig

Joe Lane 2021-03-09T17:26:26.150500Z

Ahh, I was thinking you were on cloud.

2021-03-09T18:33:22.150700Z

I think that discussion is pretty sound. you end up with a "table" of [order, ref] pairs. there's not really a better way to do it and keep it queryable. abusing tuples for a dynamically-sized list is not likely to end well.

jaret 2021-03-10T14:20:17.168900Z

Thanks! I forgot to update that link to reflect the new doc org

1πŸ‘
ghadi 2021-03-09T22:09:11.151500Z

Both are retriable...

2021-03-09T22:39:55.153800Z

Hi. I'm trying to follow the Datomic Ions tutorial, but am stuck on the https://docs.datomic.com/cloud/ions/ions-tutorial.html#configure-connection I'm not sure how to determine the :endpoint : I have hacked up

:endpoint "<http://entry.mbrainz-stu.us-east-1.datomic.net:8182>"
to look more like my system, but I assume I must have got it wrong Can I find the correct endpoint in the CloudFormation logs?

jaret 2021-03-10T13:57:25.168700Z

@ben.hammond sorry about the frustrating start. Our hands are a bit tied on the master stack creation adding UUIDs for the nested stacks. We need to overhaul the tutorial as well, but I would like to make a recommendation for your next stack. Now that you are subscribed from Marketplace start your new system with split stacks. (launch storage and then compute). You can get the tempaltes from our https://docs.datomic.com/cloud/releases.html#current. And you can follow the split stack instructions here: https://docs.datomic.com/cloud/operation/split-stacks.html#howto

1πŸ‘
2021-03-10T17:17:42.169500Z

ok will try that. thanks

2021-03-13T21:14:18.197400Z

Hi @jaret. I followed the split stack instructions, and can access datomic from the REPL. However I am not able to successfully deploy; the deploy status returns as

{:deploy-status "FAILED", :code-deploy-status "FAILED"}
When I look into the CodeDeploy events I see that is is complaining DownloadBundle
Error code
UnknownError
Script name
Message
Access Denied
I can successfully download the pushed zip file from S3...

2021-03-13T21:14:49.197600Z

am wondering if this is some side-effect of the split-stack deletion/recreations?

2021-03-13T21:15:43.197800Z

I found https://stackoverflow.com/questions/54342398/how-to-troubleshoot-access-denied-in-code-deploy-for-downloadbundle-stage, just trying to figure how that relates... if it relates

2021-03-13T21:32:43.198100Z

oh, the datomic-code-eu-west-1 was permitting ListCodeBucket/ReadCodeBucket to a previous s3 bucket. thats strange

2021-03-13T21:32:57.198300Z

well, updated it and looks like its working

2021-03-09T22:41:26.154Z

datomic -r eu-west-1 client access 
seems to have started the SSH tunnel without error

2021-03-09T22:43:53.154200Z

Ah found it. It was in the Outputs of the Compute Stack

1πŸ‘1
2021-03-09T23:03:39.154600Z

hmmm how do I specify an aws region in

clojure -A:ion-dev '{:op :push}'

2021-03-09T23:05:35.154800Z

{:command-failed "{:op :push}",
 :causes
 ({:message
   "Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.",
   :class SdkClientException})}

2021-03-09T23:12:12.155Z

a somewhat low-tech

export AWS_REGION=eu-west-1
does the trick... sure there must be a nicer way though

kenny 2021-03-09T23:14:13.155200Z

@benha See push docs: https://docs.datomic.com/cloud/ions/ions-reference.html#push πŸ™‚

kenny 2021-03-09T23:14:40.155400Z

Add :region "my-region-1"

1πŸ‘
2021-03-09T23:19:47.155700Z

this seems an unfortunate error message upon the deploy

{:command-failed
 "{:op :deploy, :group vorpal-ion-starter-Compute-A2UL4PZOZHGU, :uname \"benjy-1\"}",
 :causes
 ({:message
   "A Lambda function name, which is the concatenation of the query group and the function name specified in the ion-config.edn file must be &lt;= 64 characters.\nThe following names are too long for Lambda: vorpal-ion-starter-Compute-A2UL4PZOZHGU-get-items-by-type-lambda-proxy",
   :class RuntimeException})}

2021-03-09T23:20:53.155900Z

I have control of the first 18 chars of that compute group name but not the last 20

kenny 2021-03-09T23:21:16.156200Z

Yeah: https://docs.datomic.com/cloud/ions/ions-reference.html#lambda-config. I've hit this before too 😞

kenny 2021-03-09T23:23:26.156400Z

Perhaps they should add a warning note to the tutorial on this point.

2021-03-09T23:23:38.156600Z

so the CodeDeployApplicationName should be no more than 11 characters long

2021-03-09T23:23:55.156800Z

do I have to tear the whole stack down and recreate it?

2021-03-09T23:24:21.157Z

that really ought to be in the tutorial if so

kenny 2021-03-09T23:25:40.157200Z

I don't believe you can change the name without recreating it.

kenny 2021-03-09T23:25:59.157400Z

In https://docs.datomic.com/cloud/operation/planning.html#naming, they recommend keeping names under 24 characters.

2021-03-09T23:27:22.157900Z

yeah just saw that. Ah well, thats a sign to pack it in for the evening

2021-03-09T23:27:26.158100Z

thanks for your help

kenny 2021-03-09T23:28:30.159Z

Sure thing. Sorry the result isn’t ideal 😞

2021-03-09T23:28:51.159200Z

its all about the journey