Clojurians Log v2

anyone here that has some experience with cqrs event modeling ? i'm a bit stuck finding good resources on how to actually model events, specifically: how to choose between low level event types ("job started", "job stopped", etc) versus higher level events ("report requested", "report finished"), etc. in one hand, "report requested" has more semantical meaning, and thus leads to more "natural" events that are closer to the business domain. on the other hand, maybe you have a service that doesn't just look for report types, but logs all job start/stop results. in this case, you may not want this service to be aware whether a job is for a report or something else; you want to keep that domain logic out of there. then, naturally, the question comes up: maybe one event ("job completed") can trigger another event ("report finished") ? but this then slowly goes into the territory of complex event processing... are there any rules of thumb on how to approach this?

disclaimer: I don't have the answer. However, I find the question super interesting, and at the core of it I see the essence of how we manage complexity and many of the questions that sprung from this are questions that I often make to myself (for example, the obvious similar-question is "which things do I log", do I log the arguments of an "add-two-numbers" function?) I would also really love input and ideas from others in here. Last but not least, my answer is going to be a bit abstract.. The key here, I think, is domain modelling (see DDD). One needs to clearly write down all concepts and entities from the business domain, and all the processes that are taking place (along with the entities they involve). Having that initial high-level view, one should write down a business-level description/analysis for every process. What you end up with are the business-level steps that together compose a recipe for a business process to complete. These map one-to-one to the events we want to model. Anything else is tooling and implementation details. So, to answer my own question about the "add-two-numbers" function, we only need have a log entry on that function if our domain is "math" or "addition" is a business concept. If that is not the field we are working in, then the addition is a detail. Now, this means that you need to separate events/logs that are about the domain from the ones that are about the technical details/implementation/infrastructure. Whether you actually make use of the later is your choice and it depends. The former is the basis upon which you invoke new processes, and it is those events/logs that should be used for replay, to rerun a process after you have made a bugfix for example. The later cannot be used in that way. So, another guideline is that the business events should be able to drive the whole system, while the technical logs are mostly about debugging and monitoring. You cannot replay the later, as that would mean that all events that had been generated must still be there and be processed or ignored and this will be non-flexible. So, as everything really, it depends, and this time it depends on the domain you are modelling and how the events you are talking about are actually relevant to the domain.

right, this is a very interesting observation: the business domain is all that matters, and technical events have different semantics. they can live next to each other, but should be treated differently.

“job stopped” is a technical event, but when translating it from the technical domain to the business domain, a “report finished” event can be derived

where this translation takes place is an implementation detail I suppose

(semi-relevant) This site has techniques to help with performing and documenting your event modeling https://eventmodeling.org/about/

thanks

very interesting

The Azure docs also have some nicely-written articles on CQRS and event sourcing, with links to additional resources. This is an overview https://docs.microsoft.com/en-us/azure/architecture/patterns/cqrs. And this is a guide to the modeling process (caveat: published circa 2012): http://aka.ms/cqrs

Historical note: CQRS and event sourcing somewhat grew out of the C# community (Greg Young etc)

A few more links I’ve found helpful: • https://www.confluent.io/designing-event-driven-systems/ • https://leanpub.com/cqrs • https://leanpub.com/esversioning • (published this month) https://pragprog.com/book/egmicro/practical-microservices • I have previously dug through lots of HN comments in search of links to good resources https://news.ycombinator.com/item?id=19978091 • Elixir’s Commanded is a fairly mature CQRS/ES toolkit. Its docs link out to lots of goodness https://github.com/commanded/commanded

Adam has a nice approach on DDD and "event modelling" is his thing 😉

(that’s Adam of http://eventmodeling.org, not me 😉)

> then, naturally, the question comes up: maybe one event (“job completed”) can trigger another event (“report finished”) ? but this then slowly goes into the territory of complex event processing... What @lmergen wrote sounds like the saga pattern https://microservices.io/patterns/data/saga.html. Which I don’t think necessarily turns into complex event processing (CEP), if your sagas are more “when this then that”. https://zeebe.io is a nice, new event-driven orchestration and modeling tool. I think it would be a nice building block, both within a service and across services

thanks a lot, this is all very interesting reading material

Glad to continue chatting about it all! A challenge for me has been figuring out what combination of Clojure tooling + infrastructure (DB, queue, etc) to use to implement CQRS/ES My domain is a B2B SaaS CRM.

I've never liked Sagas, they seem such a contraption e.g. what if your undo fails? or undo-of-undo? More than plausible in a distributed setting I've long thought that "distributed transaction" should be regarded as an oxymoron. Two services behaving like one, in sync? Either make them one (i.e. consolidate services) or accept inconsistency

If I understand correctly, sagas can be async, in addition to working synchronously.