Clojurians Log v2

What is a good workflow with portkey ? With ring handlers, I can: 1. write some code in emacs 2. update function in ring app-handler via repl 3. hit refresh on browser pointed a local host 4. see changes with portkey it's 1. write some code 2. can't really test it locally 3. need to pk/mount! (generally takes 3-4 seconds on my machine) 4. warm for vm to load new code 5. testit is there a better way to test portkey code ?

So I think there are some options. I was thinking that with ring, one could do say (http-kit/run-server #'handler) and (pk/mount! handler :live true) to get both to refresh on the background

supposing that handler knows how to differentiate between Lambda and repl runtime. With a HTTP accessible database like DynamoDB, this could done by having a way to pass configuration (maybe the endpoint, say a local DynamoDB in repl, real thing in the Lambda). With a DB with a more persistent connection, like RDS, this could be a (def db (connect (fetch-connection-parameters)))where fetch-connection-parameters would know how to choose between repl and Lambda runtime.

on the speed front, Christophe proposed to have a delta redeploy, where after initial packaging, we would transfer only changed classes: https://github.com/portkey-cloud/portkey/issues/10

This would mean using S3 directly, something that would be beneficial to implement now even (https://github.com/portkey-cloud/portkey/issues/9)

on the delta redeploy, we could then load changed classes on every Lambda invocation

[will return to workflow question in a bit]

For accessing AWS APIs via portkey, what is recommended? Amazonica ?

when we get around to this, I think it would make operating in the repl quite fast. I think that AWS doesn’t provide guarantees on how long the Lamba containers are alive (http://docs.aws.amazon.com/lambda/latest/dg/lambda-introduction.html) but have seen 4 hours after initial deploy been mentioned

there’s another super alpha library for AWS access 🙂 aws-clj-sdk: https://github.com/portkey-cloud/aws-clj-sdk

For now yes but the plan is to use aws-clj-sdk as it matures (we use it internally)

@viesti: my current approach is: 1. setup VPN,, connect to live VPC, so that anything accessible in prod runtime, my local dev machine can also access 2. make server side code work both (1) inside amazon vpc and (2) on local machine (via iam) 3. as you suggested above, mount handlers both remotely and in local ring 4. then local env should mirror prod env

sounds legit

or the afternoon coffee wasn’t strong enough

amazonica is amazing

when building "server api" lambda functions in portkey, is it better to: 1. have lots of separate lambda functions OR 2. have one uber lambda function, and have it dispatch what to do on the :tag field of the data it receives ?

[ personally, I'm inclined to follow approach 2, but I'm not sure ]

don’t have experience but I think I’d have a lambda per separate piece of functionality and dispatch based on the route they are mounted to (so that API gateway does the dispatch)

one route/lambda might contain a number of closely related functionality though

I'm going to define 'business logic' api as everything that: 1. is stateless 2. takes input directly from client 3. directly modifies the DB 4. sends output directly back to client I'm now of the belief that all 'business logic api' functions should be ONE SINGLE UBER LAMBDA ... for the following reasons: a. default *.jar fileds is 4.5MB, putting all lambdas in one is not going to make it larger b. load balancing / auto scaling works better when there's ONE MASSIVE LAMBDA instead of 20 small separate lambdas c. I think it actually makes better sense, I don't want lambdas spread all over the place, I want a single place which says "here is all the public facing API calls, and here's how we handle them" d. if I had a 30 different lambdas, "updating the lambdas" is a stateful -- I'm overwriting one o the 30 different lambdas, I have no clue what the value of the other 29 currently are e. if everything is one big uber lambda, I'm saying "okay, I made this change, I'm now declaring what the ENTIRE PUBLIC API is" f. now, "business logic api" and "triggers on dynamodb" should be separate lambdas, -- but I'm only talking aboutcombining all public APIs into a single lambda

this is my current belief, having barely gotten portkey working a few days ago -- if anyone can tear this to shreds, I would greatly appreciate hearing it now (instead of runninginto issues a few weeks later)

@qqq Here is my opinion, based on having thought a lot about Lambda and Serverless over the last two years but in the main not having deployed much because my use case is always 20 minutes into the future (e.g., I needed VPC affinity before it landed, then I moved on and needed PCI-DSS certification before that landed, then I moved on again and now I need, or want, Cognitect’s Vase to support Datomic-Client before I can really use Lambda for what I need):

One of the primary advantages the Serverless paradigm buys you is that your functions can scale, not just automatically, but independently and you lose that advantage when you glom all your logic into one Lambda

My intuition is that you want many small, small fns that do exactly one thing in your business logic, and then one fn that does listen to the /* endpoint of your API and performs routing by dispatching to those small fns

Remember that you can call a Lambda from a Lambda directly, though you can also use Kinesis Streams or SQS as “plumbing” - which gives you extra places to stick backpressure, cross-region failover, observability, etc. alongside whatever you add to your business logic to address those concerns

I agree that "daemons" and "janitorial work lambdas" should be their own lambdas. However, as for the main business logic api -- I never get code right the first time. I can't imagine trying to debug code when I don't even know what versino f what function is loaded. Whereas, if there is one 'uber business logic api', then any time when I load, I know exactly what version each API endpoint is running.

was thinking about independently deployable functions, minimizing cold start (allowing separate teams to work on parts of the API). API gateway has stages which have deployment history and rollback possibility, don't know if it's possible to attach metadata even (like git commit). Maybe lambdas could even attach metadata like git commit in http headers. Just thoughts, I haven't yet used Lambda for a web service in anger :). Thusfar I've used lambdas for ETL like data gathering, no webapps yet.

ah; I should clarify: I'm a solo engineer building a webapp. I'm not doing any ETL. However, I can definitely see that in ETL environments, having each "lambda" be like a grep/awk/sed/.... part of a "unix pipe | ... | .... | ... | ... " would definitely suggest a "small lambda" approach.