Clojurians Log v2

Hey. Has anyone ever tried to excise large amounts of data? We currently have 15B datoms, and are thinking of dropping roughly 1/3 - 1/2 of them. Doing it in one go clearly would not hold, so just wondering whether there are any guidelines/benchmarks than could help us tackle this

you might want to ask on https://ask.datomic.com

May I ask what is driving your desire to remove 1/3 to half your DB? Is there an underlying problem causing you to consider this option? In general, excision was not designed to cleanup old data. https://docs.datomic.com/on-prem/reference/excision.html#performance It was specifically designed to meet https://docs.datomic.com/on-prem/reference/excision.html#motivationand puts substantial burden on indexing. If you were to create a too large excision job without proper batching/testing and understanding you can potentially render the system unusable while indexing finishes. That being said, we have greatly improved the performance of excision in the most recent release and it may be possible to design a batched excision job to address. If you want to discuss further, I'd encourage you to open a support case and perhaps we can meet to chat about this topic. <mailto:support@cognitect.com|support@cognitect.com> or the website https://www.datomic.com/support.html