datomic

Ask questions on the official Q&A site at https://ask.datomic.com!
zendevil 2021-03-19T02:36:51.019600Z

@lanejo01 I was able to deploy on beanstalk, but when I use the following line:

(d/client {
                     :server-type :ion
                     :region "us-east-1" ;; e.g. us-east-1
                     :system "humboi-march-2021"
                     :creds-profile "humboi"
                     :endpoint "<http://entry.humboi-march-2021.us-east-1.datomic.net:8182>"
                     :proxy-port 8182
                     })
I get “Unable to connect to localhost:8182” in the beanstalk logs

zendevil 2021-03-19T02:38:52.020200Z

Here are the logs. Notice line 363.

zendevil 2021-03-19T02:44:49.021100Z

I’m guessing that’s because the following command isn’t run on beanstalk:

./datomic client access humboi-march-2021 -p humboi -r us-east-1
But then how can one run this command on beanstalk?

onetom 2021-03-19T03:49:28.023900Z

datomic solo up &lt;system name&gt; --wait fails with

Upping &lt;system name&gt;-TxAutoScalingGroup-89CAXXFQ591C
Upping &lt;system name&gt;-BastionAutoScalingGroup-128DD1GX31OU7
Waiting for gateway to start.
Execution error (ExceptionInfo) at datomic.tools.ops.process/sh! (process.clj:64).
Shell command failed
it doesn't throw this error without the --wait option though, so at least it's usable. anyone with the same error or anyone who is using it successfully? couldn't find an posts on http://forum.datomic.com about this issue, so i will make one eventually.

onetom 2021-03-19T03:53:11.024Z

ehhhh, it's shelling out to the aws cli command, which i don't have in my specific environment:

[{:type clojure.lang.ExceptionInfo,
    :message "Shell command failed",
    :data
    {:args
     ("aws"
      "ec2"
      "wait"
      "instance-running"
      "--filters"
      "Name=tag:Name,Values=xxx-datomic-system-bastion"
      "Name=tag:datomic:system,Values=xxx-datomic-system"
      "Name=instance-state-name,Values=running"),

onetom 2021-03-19T04:02:14.024300Z

i got this error from the stack trace which was saved into /var/folders/dm/bjgtcwgx7nqfh3flbpq7m0qc0000gn/T/clojure-927549602744154670.edn such trace file paths are always printed at the end of clojure cli errors, but i've noticed that ppl often forget to look inside them.

tatut 2021-03-19T07:57:53.025600Z

anyone using divert-system? I’m not really sure what it means actually… I’ve only used local dev with locally created database in file storage. Now I’m looking into having test envs that have a copy of a cloud database as basis

tatut 2021-03-19T07:59:05.025700Z

what does divert actually do? do queries copy data from the diverted system

alexmiller 2021-03-19T08:04:12.025900Z

you use import-cloud to import (a subset of your) data from your cloud system to your local storage, then divert-system will direct queries to be answered via the local storage instead of prod

alexmiller 2021-03-19T08:06:22.026100Z

see https://docs.datomic.com/cloud/dev-local.html

tatut 2021-03-19T08:14:16.026300Z

yeah, I read that page but it wasn’t clear to me what it does… ok so import-cloud is the one I need

Joe Lane 2021-03-19T14:02:19.030900Z

Let me know if you run into problems with that.

kenny 2021-03-19T15:51:11.038Z

The DesiredCapacity knob is a bit strange when deploying a query group. We have our query groups deployed with auto scaling, so the ASG is "managing" the desired count. If I change a parameter (e.g., MaxSize) and have DesiredCapacity set, CloudFormation will actually set the DesiredCapacity to the value passed in. This is pretty nasty, particularly in the situation where you'd be increasing the MaxSize (e.g., DesiredCapacity is set to 2 and MaxCapacity is set to 4. You're currently running at MaxCapacity and need to immediately increase MaxCapacity to meet current demand. You set MaxCapacity to 6 and update the CF script. CF will set MaxCapacity and DesiredCapacity to 2. This makes the high demand situation worse!). The safest workaround seems to be to always have DesiredCapacity set to MaxCapacity. Does the Datomic team have any advice on how to handle a situation like this? FWIW, we deploy our services onto Fargate via Pulumi. Pulumi has the ability to set "ignoreChanges" on certain properties (e.g., ignoreChanges: ["desiredCount"]). This lets me set an initial desiredCount for my Fargate service and ignore any changes that have occurred since initialization. It would see a similar knob for query groups would solve this issue.

ghadi 2021-03-19T15:53:06.038500Z

@kenny are you making use of drift detection?

kenny 2021-03-19T15:54:24.038600Z

No. I've seen it in the console before but never used it and I am not familiar with it. Does it help in this situation?

ghadi 2021-03-19T15:56:05.038900Z

yes it helps identify drift between reality and the CF template

ghadi 2021-03-19T15:56:16.039100Z

CF manages the ASG which manages the instances

ghadi 2021-03-19T15:56:37.039300Z

the ASG is not the head honcho

ghadi 2021-03-19T15:57:12.039500Z

I would recommend consulting the drift detection before making manual changes to resources

kenny 2021-03-19T15:57:21.039700Z

In the case of DesiredCapacity, why do I care about "drift"?

kenny 2021-03-19T15:58:00.039900Z

A change in DesiredCapacity doesn't seem like drift.

kenny 2021-03-19T15:58:38.040100Z

Drift sounds like something unexpected. I fully expect DesiredCapacity to change 🙂

ghadi 2021-03-19T15:59:34.040400Z

desired capacity is an ASG parameter

ghadi 2021-03-19T15:59:51.040600Z

it's part of the ASG API, right?

ghadi 2021-03-19T16:00:15.040800Z

I get that it's the thing most likely to be manually tweaked outside of source control

ghadi 2021-03-19T16:00:27.041Z

bbiab

kenny 2021-03-19T16:00:29.041200Z

I read https://aws.amazon.com/blogs/aws/new-cloudformation-drift-detection/ on drift detection. It sounds like a heck of a lot of extra work. When updating an ASG, you don't need to set desired capacity again.

kenny 2021-03-19T16:05:51.041600Z

Drift detection also doesn't solve the problem. Say I run the drift detection and see the actual ASG desired capacity is different than the capacity set in the CF parameters. What action am I supposed to take? Change my CF DesiredCapacity param update to match the current state? That's not what I actually want. I want to update MaxCapacity and leave DesiredCapacity unchanged. I don't want all future updates to set DesiredCapacity to the value I am forced to set it to for this particular update. Worse, DesiredCapacity could have changed from the time I ran the drift detect and the time I run the CF update.

ghadi 2021-03-19T16:57:49.041800Z

drift detection is just a tool that exposes changes made to resources under management by a cloudformation stack Is there a particular reason you don't want to change Min/Max/Desired via CloudFormation?

ghadi 2021-03-19T16:58:12.042100Z

just trying to understand, not suggest anything specific

mikejcusack 2021-03-19T17:30:47.042300Z

If you are importing an existing db. If you just want to create a local test db divert-system will direct calls to local.

Joe Lane 2021-03-19T17:34:05.045200Z

@kenny this is happening in what context? A datomic version upgrade A datomic parameter update Something else?

jaret 2021-03-20T13:34:02.072300Z

Kenny, I think it's required because we are setting up an ASG with the CFT (per AWS requirements) when you launch the CFT. I will double check that with the team.

kenny 2021-03-22T18:54:58.086500Z

From the doc ^ > If you do not specify a desired capacity when creating the stack, the default is the minimum size of the group.

kenny 2021-03-24T17:24:58.101900Z

Since this conversation is likely to fall off of the Slack retention window, I have added a question here: https://ask.datomic.com/index.php/603/desiredcapacity-optional-parameter-query-group-template

kenny 2021-03-19T17:58:00.045300Z

I am changing the Max via CF. I'm saying, perhaps poorly 🙂, that changing via CF has undesirable side effects.

kenny 2021-03-19T17:58:24.045500Z

Changing MaxCount from 4 -> 6.

kenny 2021-03-19T17:58:39.045700Z

The example I gave is the exact thing that happened to us 🙂

Joe Lane 2021-03-19T18:00:29.045900Z

What does this have to do with datomic cloud? Are you reporting a bug or bemoaning how CF works?

kenny 2021-03-19T18:01:02.046100Z

With the exception of DesiredCapacity, all query group CF parameters are managed (updated/changed) via a CF update. The DesiredCapacity is only ever controlled by the ASG scaling policy.

Joe Lane 2021-03-19T18:01:07.046300Z

(I'm not saying it doesn't have anything to do with Datomic cloud, I just don't understand in what usage scenario you're running into this.)

Joe Lane 2021-03-19T18:02:24.046700Z

So, is this a feature request?

kenny 2021-03-19T18:05:12.046900Z

I think Datomic's CF implementation leads to undesirable results (though, I am not a CF expert so it could be a problem with CF itself). If DesiredCount were an optional parameter, I think this would not be an issue.

kenny 2021-03-19T18:05:40.047100Z

Or bug report. Was trying to start from the problem to ensure that the problem was actually a problem.

ghadi 2021-03-19T18:08:03.048300Z

I thought Desired was part of the cf stack params

ghadi 2021-03-19T18:08:21.049500Z

That would explain our misunderstandings

Joe Lane 2021-03-19T18:09:06.050700Z

So you're saying that the QueryGroup CF template always sets the DesiredCapacity and the MaxCapacity equal to the same value? • When upgrading a stack to a new version? • When updating the stack for some reason? • Something else is happening?

kenny 2021-03-19T18:09:16.050900Z

The problem is that CF is trying to manage the DesiredCount parameter (ensure actual DesiredCount matches the DesiredCount set in the params). This is problematic because the DesiredCount is entirely managed by the ASG.

kenny 2021-03-19T18:10:12.052200Z

Not exactly. The problem is that the CF template is always setting the DesiredCount param.

kenny 2021-03-19T18:10:42.052500Z

It is.

kenny 2021-03-19T18:11:00.052900Z

It’s a required param.

kenny 2021-03-19T18:12:12.053600Z

Besides the very first run, I never want CF to touch the desired count parameter.

Joe Lane 2021-03-19T18:12:24.053800Z

And it shouldn't because....?

Joe Lane 2021-03-19T18:12:27.054Z

Ok

Joe Lane 2021-03-19T18:13:17.054200Z

And so it's "Always setting the DesiredCount param" • When you're upgrading a version? • Updating a stack with some new config value (like an env-var)? • Something else?

kenny 2021-03-19T18:13:23.054400Z

This is the situation: DesiredCapacity is set to 2 and MaxCapacity is set to 4. You're currently running at MaxCapacity and need to immediately increase MaxCapacity to meet current demand. You set MaxCapacity to 6 and update the CF script. CF will set MaxCapacity and DesiredCapacity to 2. This makes the high demand situation worse

kenny 2021-03-19T18:13:36.054600Z

Updating MaxCount.

kenny 2021-03-19T18:13:56.054800Z

I'm not sure on the other situations.

kenny 2021-03-19T18:14:55.055Z

Intuitively I would expect the same behavior (i.e., DesiredCount gets set to the CF param). Would need to test it to verify ofc.

Joe Lane 2021-03-19T18:16:44.055200Z

> DesiredCapacity is set to 2 and MaxCapacity is set to 4. In the ASG or CF? > You set MaxCapacity to 6 and update the CF script Again, ASG or CF?

Joe Lane 2021-03-19T18:18:36.055400Z

What is "CF script"? Are you referring to your scripts or the QG CF template parameters? This is an update right?

kenny 2021-03-19T18:19:27.056300Z

Oh, sorry. In all cases I mean query group CF template.

kenny 2021-03-19T18:19:57.056800Z

And yes, an update.

Joe Lane 2021-03-19T18:22:53.057Z

So, In the template you have the DesiredCount at 2, MaxCount at 4. You're at your max, because you manually changed the ASG DesiredCount to 4 (to match your load). The problem is that when you update your MaxCount from 4 to 6 but leave the DesiredCount at 2, you think it should keep the 4 you manually set in the ASG console? Am I close? If I'm wrong, can you copy the above prose and edit it, then paste it back here in this thread?

kenny 2021-03-19T18:33:57.058Z

So, In the template you have the DesiredCount at 2, MaxCount at 4. You're at your max, because the ASG scaling policy scaled up the group DesiredCount to 4 (to match your load). The problem is that when you update your MaxCount from 4 to 6 via a CF template update but leave the DesiredCount at 2, I think it should keep the 4 the ASG scaling policy set.

Joe Lane 2021-03-19T18:37:58.058200Z

What is the ASG scaling policy based upon? Why has it scaled up to 4 machines? CPU, mem?

kenny 2021-03-19T18:38:08.058500Z

Cpu

kenny 2021-03-19T18:38:28.058900Z

Target track 50%.

kenny 2021-03-19T19:33:48.060Z

Worth opening an ask.Datomic topic on this?

em 2021-03-19T19:46:44.060200Z

> You set MaxCapacity to 6 and update the CF script. CF will set MaxCapacity and DesiredCapacity to 2. This makes the high demand situation worse Curious, if you set MaxCapacity to 6, why would CF set MaxCapacity to 2? The problem makes sense though, as a dynamic parameter DesiredCapacity is modified by ASG to control behavior. But CF template updates want to set it as a default param. Maybe an optional tick box in the template would solve the issue without breaking changes?

jaret 2021-03-19T19:52:15.060400Z

@kenny how are you updating? from CFT? CloudFormation does not change your ASG settings when you update or upgrade. It reads your ASG settings. Maybe I am missing something here, but adding machines is going to potentially bounce the process monitoring CPU or lower the CPU average, right?

kenny 2021-03-19T19:57:24.062700Z

Yes, the objective would be lower cpu. Since I’m not fiddling things in the console directly, there could be something in our deployment process affecting this. Let me create a minimal repro and get back to you in a couple hours.

jaret 2021-03-19T20:04:17.062900Z

@kenny what do you have set for your warmup time?

jaret 2021-03-19T20:04:23.063100Z

kenny 2021-03-19T20:06:30.063600Z

300

jaret 2021-03-19T20:06:49.063800Z

So the default, then

jaret 2021-03-19T20:06:58.064Z

and did you disable scale in?

kenny 2021-03-19T20:07:20.064200Z

Joe Lane 2021-03-19T20:10:08.064600Z

And you're using i3.xlarges?

kenny 2021-03-19T20:10:35.064900Z

m5

Joe Lane 2021-03-19T20:12:46.065700Z

Have you timed how long they take to come up and start accepting traffic? If you start reporting metrics before that, they may start reporting CPU metrics against your ASG policy, lowering the utilization, and killing instances.

kenny 2021-03-19T20:16:12.068600Z

I have not. I do not think that was what happened. I recall checking in the ASG activity log and saw a DesiredCount change due to CF (I think). Need to double check. Not at a computer atm and AWS mobile console is hard to navigate.

jaret 2021-03-19T20:26:11.068800Z

@kenny I think I understand your issue, let me know if this clarifies it for you. I think the missing piece is that AWS CF looks at parameters it sets. It only knows about parameters it has set. If you have changed your ASG settings outside of CF (i.e. your policy changed your settings) the CF defaults the parameter to the last CF-set parameter. Does that make sense? So you need to manually set these parameters or update your script to query for the currently set "DesiredCount" in you asg and inject that into your DesiredCount CloudFormation parameter.

esp1 2021-03-19T20:40:33.069Z

Thanks @lanejo01! This is helpful, but I am actually interested in getting users that are logged in to our corporate AWS cloud through VPN access to the Datomic VPC - so they would be accessing it from outside the Datomic VPC, but inside our corporate AWS network. The two options I was exploring were: • Setting up peering/routing to the Datomic VPC directly • Using a private API Gateway endpoint

esp1 2021-03-19T20:42:13.069300Z

I can go the peering/routing way, but that would involve making changes to the Datomic VPC, and I was concerned that if I needed to update Datomic VPC via the CF templates those changes might be lost.

kenny 2021-03-19T20:44:05.070400Z

Yes, I think that’s exactly what happened. I don’t think those extra steps are necessary though. Why is DesiredCount a required parameter?

esp1 2021-03-19T20:44:20.070600Z

The private API GW endpoint seemed like it would be a solution that could be managed independently from the Datomic CF stacks, but I haven't set up one of these before and have been struggling with how to craft an appropriate resource policy to make it work.

kenny 2021-03-19T20:45:29.071300Z

(Your solution is inherently racy 🙂)

jaret 2021-03-19T20:56:44.071500Z

We should work together on this in a support case ^ but I would recommend an API GW.

jaret 2021-03-19T20:57:24.071700Z

Can you throw me an e-mail at <mailto:support@cognitect.com|support@cognitect.com> and I will get together a recommended policy.

jaret 2021-03-19T20:57:58.071900Z

perhaps after I can add this to the docs.

esp1 2021-03-19T21:30:09.072100Z

Thanks @jaret, will do