Clojurians Log v2

I know it's in the midst of a major hype cycle, but anyone here attempt any ethereum development with clojure?

I was hoping to use the collective knowledge in this channel to find some inspiration. I’m looking for some platform or framework (doesn’t have to be in Clojure) similar to workflow engines such as Apache Airflow or Azkaban, but subtly different. The workflow engines I mentioned seem really focussed on batch processing, whereas I am looking for something more “reactive”. A bit more context: • We are developing an application where we receive events about new/updted/deleted “records” from somewhere upstream. • When we receive such an event we (usually) have to update our own data based on a complex set of rules and dependencies: • Say just the title of a “record” has changed upstream, then we just have to update our own database and search index. • But if an URL has changed, we will probably initiate a download, after which several new jobs might start that need the downloaded file. These completed jobs might kick-off even more processes. In essence our job is to keep a dataset that is in some way “derived” from the upstream data up to date. All upstream events will trigger some process in our application and the faster our derived data reflects that, the better. That’s why the batch-based workflow engines I find seem like a bad fit. Maybe I’m wrong and these existing workflow engines can be adapted to suit our needs, maybe a more suitable platform already exists but I just haven’t found it yet. Or maybe we will have to build it ourselves. Would be great to hear your thoughts or ideas about this!

have you seen aws step functions? big disclaimer I've never used them for work, but I am a fan, and they are sort of my go to example for workflow kinds of things. one of the things I like about it is they have a spec for the json dsl they use for defining workflows (https://states-language.net/spec.html), which if you decide nothing existing fits your needs may be a good jumping off point for writing your own workflow system

but structurally, re-reading your description, it sounds like you need a state machine per record, not a state machine per task

so like, if a given record is already doing a download, and a new event comes in for the record, it seems unlikely you want to spin up another concurrent task for that record

Have you looked into Materialize? I think what you’re trying to do sounds exactly up their alley: https://materialize.com/about/

There is also a rust library that does this sort of MV maintenance: https://github.com/TimelyDataflow/timely-dataflow

@louiedinh thanks for the tips, I will look at them!

You’re right about needing a state machine per record I think

Haven’t seen them yet, will take a look thanks!

I’m now reading the step functions docs, but those would be state machines per task I guess…

in my mind a state machine per record is much more like an actor based system. each record has an inbox of events and a little loop (a state machine) processing that inbox

(vs. a workflow parallel task kind of system)

A bit like an agent in clojure?

Is this issue with Tether falsely claiming USD parity going to affect the BTC price?

sure

There's a #bitcoin channel...

Man, the rain here is like sideways, noah get the boat, apocalyptically heavy. There are 3 directions the wind can blow, and I'm fine. BUt one direction if it blows water comes into my home....

kafka streams... although it might be overkill

In essence our job is to keep a dataset that is in some way "derived" from the upstream data up to date. <-- kafka streams (and maybe kafka connect) are built for this purpose

@jlmr, check out https://github.com/oakes/odoyle-rules. it sounds pretty close to your description

@louiedinh Have you used Materialize? I just read about it for the first time. Sounds interesting... differential dataflow compatible with Postgresql