Clojurians Log v2
aws
Hello, a more generic AWS question here, but is I have a python script that I want to offload to AWS, because it needs way too much RAM and runs for many hours to generate a model (similar to ML but for Optimisation). Is AWS Lambda or Fargate what I need?
The alternative to have run the script on a server instead
Lambdas cannot run for longer than 15 minutes as a hard limit, so that might eliminate Lambda for your use case
EC2 would be the most akin to running your script the way it’s already running; EC2 just gives you a virtual machine to run your program on, you can install python and run your script
I don’t think you need fargate
Right, I just looked at https://greenm.io/aws-lambda-or-aws-fargate/ and seems like fargate might fit quite well. Or do you still recommend EC2? I agree that EC2 is most akin to doing so. But would serverless be a better way since it is like a one time use thing?
Maybe Ill just go with EC2 after all, there is no option to use fargate with aws educate it seems
The most flexible approach would be to containerise your app. you can start with running it on EC2 and when comfortable migrate to Fargate. you can also consider Sagemaker that is geared towards ML, but it's more expensive
I would say go with EC2 to start. It’s the “simplest” in a way, and things like Lambda, etc are more akin to cost optimization strategies.
Right, that makes sense! Will go with that! Think that workflow might work out better after all