Clojurians Log v2

If I'm using a shared http-client with Cognitect's aws-api, is it ok to create many client records without calling stop?

yeah if you're making many aws-api clients, calling stop will bork you

we're working on this problem actively

we're trying to slowly divorce ourselves from the current http-client, and have a BYO model

Sorry, so it's ok to do as I said?

yup

if you are sharing http-clients, you should avoid calling stop

Got it. With that BYO model, seems like this lib could be easily used in CLJS too.

slow your roll 😉

there are other concerns for CLJS, like signing routines

porting to CLJS has not been fully evaluated

Mostly interested in use in AWS Lambda.

there is a ticket if you'd like to register your interest or a motivating case

Cool, I suspected so. Is that because of startup time?

Yes

what would be an acceptable startup time for CLJ lambda?

For this particular app, under 10s is ok. Cold starts result in 20-30s execution time with production deps.

Almost everything is written in vanilla clojure so getting it to work in Node would be somewhat easy.

yeah we can get there, my goal is under 2s for clojure + core.async + aws-api

can't vouch for external deps, most are pigs

might need a custom lambda runtime + JDK11

2s would be fantastic. 10s is really pushing the limit for us but would likely be acceptable for a while.

what are the deps you're pulling in, if you can share?

Pretty much everything in there is either cljc or has a JS client available. Largest missing piece is Datomic. Hoping they publish a protocol definition for the client API so we can have JS clients.

that is a ton

Yep, tis' the world we live in haha.

the #1 thing you can do for startup time is to load less code

datomic in a lambda is super weird

if you need to connect to Datomic, consider running in an Ion -- free lambda

I actually don't think any one of those deps could be removed unfortunately. The only alternative is removing deps. Removing deps means writing the code ourselves which is sometimes hard to justify the engineering time.

I just want to paint a realistic image that it is impossible to get fast startup time with bloated deps

Ions are too restrictive. There's no great way to instrument code in them. You're stuck with CW logs (unfortunately same applies with Lambda though). We have several extremely expensive, somewhat long-running ops that would almost certainly have a large impact on the Datomic instances.

long-running ops -> not a good use case for lambdas either impact on datomic instances -> use a query group

can you elaborate on code instrumentation?

using honeycomb?

Long running = 5-10 min highly CPU intensive task. Yes

seems really appropriate for an Ion, IMHO

you can use honeycomb within an Ion

Query Groups allow you to isolate the CPU intensive part so that it's not running on the Primary Group

but you get the benefit of any queries happening locally instead of across the wire

@ghadi Sorry, had to step out for a while. Couple things there: - We're actually using Datadog APM right now, not honeycomb, but I assumed the usage was similar -- you need to start the jvm process with a java agent. I don't think Ions allow you to change the JVM args. - Running in an Ion requires you to run infrastructure 24/7. A huge benefit to running in Lambda is you only pay for execution time. - The Ion deployment model does not fit in very well with existing infrastructure tools (e.g., Terraform, Pulumi, AWS CDK, etc).

honeycomb doesn’t use an agent, but agents are possible with Datomic Zion’s

But I get you, there’s a ton of crap to consider

Oh they are? I didn't realize you could use Java agents with Ions.