Clojurians Log v2

I’ve done some catching up with reading. I think i understand the idea behind bulkheads now 🙂

i think both core.async and manifold are suitable as they both allow you to run things on dedicated threadpools

I don’t think erlang is supporting the bulkheads out of the box. As the actors itself might restart, but with erlang you can also have actors in a distributed deadlock if you are not careful

I guess I don't see how you can implement the "generic" bulkheads - don't find the definition that useful

if by generic you mean can never fail, I agree I don’t see how you can implement that either

in Hystrix a bulkhead just limits the amount of concurrent calls to the component

i thought in Hystrix you had a thread (pool) per component ?

that's one way to limit the amount of calls

the other is using semaphores

anyway, I don't see it as an implementable pattern, but rather an implementation approach in a more general sense

but this obviously depends on what you take as the definition

this book, preview, original probably better, has some useful context too https://www.nginx.com/wp-content/uploads/2015/01/Building_Microservices_Nginx.pdf

But we don’t really do microservice or SOA so most of the things do not apply to my application. Though general principles, especially in the Nygard book are useful as a rule of thumb to prevent cascading failures

What I find appealing at Hystrix at the moment is it’s out of the box monitoring and latency control. And because I don’t want to be trapped in a Java like architecture and I want some custom controls I’m trying to build my own version

I’m planning to use this for latency control too (if I get a patch in): https://github.com/ztellman/aleph/issues/273

We had some queueing issues 🙂

did you try using executor stats?

no I didn’t know about those before. However, i think those are aggregates? I need to know on a per request level

yeah, those will be aggregate

hm

Since we have only 100ms or so, if you are already waiting for 75ms it might not be worth it to continue to take the offering into account

https://github.com/ztellman/dirigiste/blob/master/src/io/aleph/dirigiste/Stats.java

public enum Metric {
        QUEUE_LENGTH,
        QUEUE_LATENCY,
        TASK_LATENCY,
        TASK_ARRIVAL_RATE,
        TASK_COMPLETION_RATE,
        TASK_REJECTION_RATE,
        UTILIZATION
    }

Thanks, looks useful to add to our monitoring as well

How do you handle errors in a websocket/sse stream? I have an error somewhere and cannot find it easily. I would like to wrap the stream with some error logging. manifold.deferred/catch doesn’t seem to work in this case

hehe

https://github.com/ztellman/manifold/issues/95

ah thanks

thought already i misunderstood the examples

from some manually testing it seams you have to wrap the put! call with error handling. Hmm will test a bit more

there's a bunch of cases where an error can happen inside some callback and the stream gets closed + the error gets logged

you can't get the error in any way

like the consume example

or zip

hm, maybe not zip exactly, as it can't really fail

if your xform throws

in this case I would be happy with a log, but haven't seen that either

i’m actually not sure what happens

ah yes configuring logging correctly helps 🙂

Would be nice though to have some way to control where the exception goes when the unexpected happens

you can upvote the issue 🙂

even better, write your experiences

done https://github.com/ztellman/manifold/issues/95#issuecomment-251946186 🙂

exception management and async is always a gnarly issue. i think in manifold’s case, simply modularizing an error handler would be good enough indeed

@jeroenvandijk i found out yesterday that next week’s meetup is going to be about onyx, so that’s gonna be interesting!

Yes indeed 🙂

We are using it pre-production. What about you?

i haven’t been able to find a good enough reason to use it 😕

we’re mostly focused on AWS, and as such use Redshift for our data warehouse, and for the more complex processing we use Spark

both hosted by AWS

How do you get the data to S3/Redshift?

We use it to move data from our bidder cluster via kafka to s3 and other services

But it’s quite complicated and i wouldn’t recommend it if you have something simple working

we use Kinesis Firehose

which is like kafka

ah i see, so they manage the connection to Redshift i guess

sounds good

we need to send data to other services too

yes, basically it caches on S3 until it reaches a certain threshold (every 900 seconds or 64MB is what we use) and then imports it into redshift

I would like to try Redshift too on our data

we then use AWS DataPipeline to query redshift

in fact, i don’t think anyone ever directly connects to redshift 🙂

(over here in our org)

no? not through a jdbc connection?

nop

i hope to be able to have a SQL interface on our data via either redshift, drill or bigquery sometime

https://aws.amazon.com/blogs/aws/autheos-at-the-nexus-of-marketing-and-e-commerce/

that was an article i wrote 6 months ago about our architecture

it’s pretty interesting how much you can do with the cloud nowadays

nice featured post 😎

that's interesting

i wouldn’t mind getting rid of kafka if something with similar performance would be offered on AWS, but i think kinesis is still a bit different. But it is amazing how well it all scales

it’s fascinating indeed

AWS is good for the start when your do not (yet) need very complex configurations, and you’re not into high volume yet

you do have to take into account that “eventual consistency” is very real

especially with things like S3

yeah and different in some regions too 🙂

but i think now they have fixed that

@jeroenvandijk: looks very sweet!