portkey

Portkey: from REPL to Serverless in one call
qqq 2017-10-22T03:33:08.000057Z

What is a good workflow with portkey ? With ring handlers, I can: 1. write some code in emacs 2. update function in ring app-handler via repl 3. hit refresh on browser pointed a local host 4. see changes with portkey it's 1. write some code 2. can't really test it locally 3. need to pk/mount! (generally takes 3-4 seconds on my machine) 4. warm for vm to load new code 5. testit is there a better way to test portkey code ?

viesti 2017-10-22T12:22:50.000054Z

So I think there are some options. I was thinking that with ring, one could do say (http-kit/run-server #'handler) and (pk/mount! handler :live true) to get both to refresh on the background

viesti 2017-10-22T12:27:49.000078Z

supposing that handler knows how to differentiate between Lambda and repl runtime. With a HTTP accessible database like DynamoDB, this could done by having a way to pass configuration (maybe the endpoint, say a local DynamoDB in repl, real thing in the Lambda). With a DB with a more persistent connection, like RDS, this could be a (def db (connect (fetch-connection-parameters)))where fetch-connection-parameters would know how to choose between repl and Lambda runtime.

viesti 2017-10-22T12:28:47.000006Z

on the speed front, Christophe proposed to have a delta redeploy, where after initial packaging, we would transfer only changed classes: https://github.com/portkey-cloud/portkey/issues/10

viesti 2017-10-22T12:29:54.000059Z

This would mean using S3 directly, something that would be beneficial to implement now even (https://github.com/portkey-cloud/portkey/issues/9)

viesti 2017-10-22T12:33:32.000071Z

on the delta redeploy, we could then load changed classes on every Lambda invocation

qqq 2017-10-22T12:34:56.000038Z

[will return to workflow question in a bit]

qqq 2017-10-22T12:35:14.000038Z

For accessing AWS APIs via portkey, what is recommended? Amazonica ?

viesti 2017-10-22T12:37:11.000003Z

when we get around to this, I think it would make operating in the repl quite fast. I think that AWS doesn’t provide guarantees on how long the Lamba containers are alive (http://docs.aws.amazon.com/lambda/latest/dg/lambda-introduction.html) but have seen 4 hours after initial deploy been mentioned

viesti 2017-10-22T12:37:49.000067Z

there’s another super alpha library for AWS access 🙂 aws-clj-sdk: https://github.com/portkey-cloud/aws-clj-sdk

cgrand 2017-10-22T12:38:00.000022Z

For now yes but the plan is to use aws-clj-sdk as it matures (we use it internally)

qqq 2017-10-22T12:38:26.000010Z

@viesti: my current approach is: 1. setup VPN,, connect to live VPC, so that anything accessible in prod runtime, my local dev machine can also access 2. make server side code work both (1) inside amazon vpc and (2) on local machine (via iam) 3. as you suggested above, mount handlers both remotely and in local ring 4. then local env should mirror prod env

viesti 2017-10-22T12:40:00.000094Z

sounds legit

viesti 2017-10-22T12:40:48.000023Z

or the afternoon coffee wasn’t strong enough

qqq 2017-10-22T12:56:04.000045Z

amazonica is amazing

qqq 2017-10-22T13:14:27.000068Z

when building "server api" lambda functions in portkey, is it better to: 1. have lots of separate lambda functions OR 2. have one uber lambda function, and have it dispatch what to do on the :tag field of the data it receives ?

qqq 2017-10-22T13:14:39.000042Z

[ personally, I'm inclined to follow approach 2, but I'm not sure ]

viesti 2017-10-22T13:22:16.000081Z

don’t have experience but I think I’d have a lambda per separate piece of functionality and dispatch based on the route they are mounted to (so that API gateway does the dispatch)

viesti 2017-10-22T13:23:30.000082Z

one route/lambda might contain a number of closely related functionality though

qqq 2017-10-22T13:27:00.000001Z

I'm going to define 'business logic' api as everything that: 1. is stateless 2. takes input directly from client 3. directly modifies the DB 4. sends output directly back to client I'm now of the belief that all 'business logic api' functions should be ONE SINGLE UBER LAMBDA ... for the following reasons: a. default *.jar fileds is 4.5MB, putting all lambdas in one is not going to make it larger b. load balancing / auto scaling works better when there's ONE MASSIVE LAMBDA instead of 20 small separate lambdas c. I think it actually makes better sense, I don't want lambdas spread all over the place, I want a single place which says "here is all the public facing API calls, and here's how we handle them" d. if I had a 30 different lambdas, "updating the lambdas" is a stateful -- I'm overwriting one o the 30 different lambdas, I have no clue what the value of the other 29 currently are e. if everything is one big uber lambda, I'm saying "okay, I made this change, I'm now declaring what the ENTIRE PUBLIC API is" f. now, "business logic api" and "triggers on dynamodb" should be separate lambdas, -- but I'm only talking aboutcombining all public APIs into a single lambda

qqq 2017-10-22T13:28:09.000025Z

this is my current belief, having barely gotten portkey working a few days ago -- if anyone can tear this to shreds, I would greatly appreciate hearing it now (instead of runninginto issues a few weeks later)

chris_johnson 2017-10-22T13:46:41.000072Z

@qqq Here is my opinion, based on having thought a lot about Lambda and Serverless over the last two years but in the main not having deployed much because my use case is always 20 minutes into the future (e.g., I needed VPC affinity before it landed, then I moved on and needed PCI-DSS certification before that landed, then I moved on again and now I need, or want, Cognitect’s Vase to support Datomic-Client before I can really use Lambda for what I need):

chris_johnson 2017-10-22T13:47:38.000008Z

One of the primary advantages the Serverless paradigm buys you is that your functions can scale, not just automatically, but independently and you lose that advantage when you glom all your logic into one Lambda

chris_johnson 2017-10-22T13:48:59.000105Z

My intuition is that you want many small, small fns that do exactly one thing in your business logic, and then one fn that does listen to the /* endpoint of your API and performs routing by dispatching to those small fns

chris_johnson 2017-10-22T13:50:23.000105Z

Remember that you can call a Lambda from a Lambda directly, though you can also use Kinesis Streams or SQS as “plumbing” - which gives you extra places to stick backpressure, cross-region failover, observability, etc. alongside whatever you add to your business logic to address those concerns

qqq 2017-10-22T13:56:02.000110Z

I agree that "daemons" and "janitorial work lambdas" should be their own lambdas. However, as for the main business logic api -- I never get code right the first time. I can't imagine trying to debug code when I don't even know what versino f what function is loaded. Whereas, if there is one 'uber business logic api', then any time when I load, I know exactly what version each API endpoint is running.

viesti 2017-10-22T15:00:57.000026Z

was thinking about independently deployable functions, minimizing cold start (allowing separate teams to work on parts of the API). API gateway has stages which have deployment history and rollback possibility, don't know if it's possible to attach metadata even (like git commit). Maybe lambdas could even attach metadata like git commit in http headers. Just thoughts, I haven't yet used Lambda for a web service in anger :). Thusfar I've used lambdas for ETL like data gathering, no webapps yet.

qqq 2017-10-22T15:31:35.000103Z

ah; I should clarify: I'm a solo engineer building a webapp. I'm not doing any ETL. However, I can definitely see that in ETL environments, having each "lambda" be like a grep/awk/sed/.... part of a "unix pipe | ... | .... | ... | ... " would definitely suggest a "small lambda" approach.