datomic

Ask questions on the official Q&A site at https://ask.datomic.com!
tatut 2021-03-25T08:17:26.104300Z

we haven’t had need for stopping the background stuff

tatut 2021-03-25T11:37:54.104600Z

an outage? what does it mean “cluster node goes down”?

Joe Lane 2021-03-25T13:38:42.106100Z

If for some reason a machine needs to be replaced (an auto-scaling event, for example), it wont be.

pvillegas12 2021-03-25T13:47:48.107300Z

Hello! We are getting timeouts in this step of codedeploy consistently

aws s3 cp <s3://datomic-code-92caa718-792a-4ced-9b36-16d4c09dabd4/datomic/libs/mvn> /home/datomic/.cognitect-s3-libs/.m2/repository --recursive --only-show-errors --exclude * --include crisptrutski/boot-cljs-test/0.2.2-20160402.204547-3.zip
Is this a known problem?

ccortes 2021-03-25T13:58:09.109400Z

What's you aproach to doing analytics? Is the out-of-the-box support good enough in your experience or do you use any other ETL tools? Mainly I'm looking for a way to work with datomic data on python

Joe Lane 2021-03-25T14:12:54.109600Z

@cuaucortes (assuming Cloud) have you looked at https://docs.datomic.com/cloud/analytics/analytics-jupyter.html ? (there is a similar entry for on-prem)

1👍
Joe Lane 2021-03-25T14:15:21.109800Z

FWIW, it looks like AWS pushed the https://docs.aws.amazon.com/lambda/latest/dg/runtime-support-policy.html from the end of March.

furkan3ayraktar 2021-03-25T14:46:52.110100Z

@lanejo01 Do you have any best practices / solutions in this topic?

Joe Lane 2021-03-25T14:53:12.110300Z

What actual problem are you trying to solve @furkan3ayraktar? • "Having background threads do work" isn't a problem, it's a capability. • "I have to process a lot of data and I need to apply back-pressure to ensure I don't overwhelm my system" is closer to a problem. I could tell you all sorts of things about lambdas, coordination, orchestration of stateful things in a distributed system, but I'd rather have the concrete scenario.

furkan3ayraktar 2021-03-25T15:16:53.110500Z

Thanks! I’m trying to figure out the best way to implement this and it would be very helpful if you can direct me to the right direction. Here is a concrete scenario. Let’s say I have a SQS queue and have a query group that is named worker-query-group. The each node in the worker-query-group is tasked to have a background thread which will poll from SQS continuously and process the messages received. I have this setup: 1. Ion Deploy worker-query-group 2. A new deployment is created in CodeDeploy 3. CodeDeploy deployment is successful 4. An EventBridge event is triggered after successful deployment 5. EventBridge event triggers a Lambda Ion, named sqs-poll-controller 6. sqs-poll-controller Lambda Ion is executed in one of the nodes within the worker-query-group 7. Polling from SQS is started I can also call sqs-poll-controller Lambda Ion manually to start/stop the background thread for polling from SQS. I have a problem when I have more than one nodes in the worker-query-group. The Lambda Ion (sqs-poll-controller) executes only on one of the instances and I’m in the search of figuring out how I can control all of the nodes within the query group with a Lambda Ion or any other way that is recommended. I got this idea of controlling background threads via a Lambda Ion from a Datomic team member’s https://forum.datomic.com/t/message-listeners-in-ions/860/2 and other linked https://forum.datomic.com/t/kafka-consumer-as-an-ion/823/4 of him on the forum.

jaret 2021-03-25T15:24:12.111Z

Update. This issue is resolved inhttps://docs.datomic.com/cloud/releases.html#ion-dev-282 > Improvement: Limit how long to wait for a cluster node to gracefully shutdown.

Joe Lane 2021-03-25T15:36:08.111200Z

Is this actually your use-case? Do you actually have an SQS queue of work?

Joe Lane 2021-03-25T15:39:29.111400Z

> I have a problem when I have more than one nodes in the worker-query-group. This is a problem with one of many possible SOLUTIONS to problem X. What is problem X?

furkan3ayraktar 2021-03-25T15:51:19.111600Z

Yes, I’m not creating a non-existing issue. I have an SQS queue and a query-group dedicated for consuming messages from that SQS queue. Since there are many messages pushed to the queue, in order to increase the capacity, we wanted to add more nodes by increasing the number of instances in the auto-scaling group for the query-group. I agree, there might be different solutions to this problem. What I’m trying to learn is what is the best practice. I’m open to the ideas how to overcome the problem where there is one queue full of messages that needs to be read and processed.

furkan3ayraktar 2021-03-25T15:52:48.111800Z

And, the problem you quoted above comes from the fact that I’m trying to control (start/stop) background threads in the nodes via a Lambda Ion. I got that idea from the forum, but you can direct me a totally different approach to the root problem I’m trying to solve, which is, having a queue full of messages and needing nodes to read those messages and process.

Joe Lane 2021-03-25T16:04:18.112100Z

Instead of using a pull based integration (workers polling) you should assess a push based model combining https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html . You would need to enhance one of the roles we create for an ion with an additional policy (documented in the link above) to allow SQS to invoke the lambda. Technically this isn't something we officially support, but I've seen it done successfully. A downside with this approach is that you run the risk of overwhelming the primary group with transactions from your autoscaling query group if you're not careful. The upside is that you don't have any state to manage in a distributed system.

Joe Lane 2021-03-25T16:28:35.112400Z

Using Step Functions with Ions is another approach, depending on the needs of the business problem. (e.g., it is a long running process that requires human approval steps)

futuro 2021-03-25T18:43:11.112900Z

Are the docs (https://docs.datomic.com/cloud/index.html) down for anyone else?

futuro 2021-03-25T18:43:41.113200Z

I get the following when I visit that

Joe Lane 2021-03-25T18:46:23.114300Z

@futuro Thanks for the heads up, we'll check it out.

futuro 2021-03-25T18:46:45.114700Z

👍:skin-tone-2:

Joe Lane 2021-03-25T19:03:31.114900Z

@futuro Fixed

futuro 2021-03-25T19:03:46.115100Z

Fantastic, thank you 🙂

2021-03-25T22:27:30.115300Z

Hi there. Can anyone confirm if Datomic will run on Azure and/or any docs/blogs. I've done some googling but didn't find much.

2021-03-26T20:03:33.130600Z

Can confirm. We run Datomic on-prem in Azure. Both peers and transactor as Azure Container Instances, with Postgresql as backend. Setup running in production since September 2020.

2021-03-28T13:54:17.143900Z

Thanks @ornulf.risnes

furkan3ayraktar 2021-03-25T23:41:07.115500Z

I’ve implemented polling SQS via Lambda in another project in the past, however, Lambda polling SQS has some issues. You can see a glimpse of it here in this https://zaccharles.medium.com/lambda-concurrency-limits-and-sqs-triggers-dont-mix-well-sometimes-eb23d90122e0. For that reason, I prefer polling from the query-group rather than relying on a Lambda. I’m having hard time imagining a solution with Step Functions to this problem. Also, both cases will incur additional costs and complexity.  Anyway, my understanding is that there is no best practice around communicating to all of the nodes within the Datomic query groups. Something like a special kind of Lambda Ion which could trigger a Clojure function in all of the nodes of a query-group would be a very nice to have in order to communicate with the running nodes easily.