Clojurians Log v2

@val_waeselynck Yeah we’re already experimenting with that, and maybe it’s good enough. But if those tests take 1 second each because of setup/teardown of Datomic databases, that’s too long for me. For unit tests, I prefer to keep things as lean as possible.

@stefan.van.den.oord forking solves that problem

Put a populated db value in some Var, and then create a Datomock connection from it in each test, there's virtually no overhead to this

Sounds good, will definitely try, thanks! 🙂

We use the transaction report queue to push data into a kinesis stream, then run lambdas on those events

Triggering side effects on Dynamo db writes is likely not what you want since datomic is writing full blocks to storage (not a datom at a time)

@bhurlow when running on multiple/scalled instances, how do you manage the tx-report-queue?

We run a single, global process which just subscribes to the queue and pushes events to kinesis

other Datomic traffic is scaled horizontally but doesn't invoke the queue

Kinesis -> Lambda integration works reasonably well

one bonus is you can do one queue to many lambda consumers

@bhurlow can you share which instance size you use for this report-queue?

subscribing to the tx report queue and putting into lambda is not a very intensive process

t3.large would be fine imo

tnks @bhurlow

hmm, i don't suppose you know if something like the "transaction report queue" is available on Datomic Cloud, do you? i have often been in need of exactly what souenzzo mentioned, but instead settled for querying / sipping the transaction log on a timer

I'm not sure about cloud, have only used the above in on-prem

I'd assume it's inside the system but possibly not exposed

Clients don't have a txReportQueue indeed. Polling the Log is usually fine IMO (and having a machine dedicated solely to pushing events seems wasteful, and it's also fragile as it creates a SPoF).

I work with datomic cloud and datomic on-prem (on different products) IMHO, datomic on-prem still way easier/flexible then cloud Cloud has to many limitations. You can't edit IAM for example, and if you edit, you break any future updates.

thanks guys

One interesting construction might be using AWS Step Functions + Lambda for polling the Datomic Log into Kinesis, using the Step Functions state to keep track of where you are in consuming the Log