Clojurians Log v2

thanks @lmergen , that helped a lot! one other question, it appears the the onyx GC fn is being called on every replica in the cluster. is that the correct approach? From the docs, it sounds like you want to call the fn on one replica and it would trigger GC on all the peers. I also see some errors in our logs originating from the GC fn in the stack trace related to ZooKeeper

i believe the GC is mostly about cleaning up the distributed log -- since all peers should have the same "view" of the log, it makes sense that it's executed on all peers

what - i think - actually happens is that a simple entry is written in the log, "perform gc" -- all peers read this, and since GC is completely deterministic, at the end of the operations they all still have one consistent view of the (now GC'ed) log

this is a bit of a consequence of the masterless design

I think I get it, it would get executed on all peers by onyx itself, but should only be triggered once in the cluster, is that right? I think we may be writing the special “perform gc” to the log several times, if that makes sense

@lmergen appreciate the help. we are scaling up and this system that has been somewhat reliable for about a year is starting to have issues as we scale.

yes that's correct

i don't think it's too much of a problem to write too much GC, apart from wasted CPU cycles

ok. thanks. Either way, we have an easy way to get that to run on one replica, so might as well get eliminate that as a cause.