Clojurians Log v2

So I fired up visualVM

and it seems to be spending all the time in clara.rules.engine$retract_accumulated.invoke()

which lines up with the theory that implicit retractions are causing this

I can share the snapshot if that would be interesting for people

I'll spend some time trimming the example down to a bare bones app I can put in a gist

the gist can be found here: https://gist.github.com/wdullaer/cecf88b3266ba0ac90b4f060eefe5208

when making this I noticed that the 2 million case is not exactly the same as 2x 1 million, because of how the attributes are randomly generated

2x 1 million will generate much more consent for a given personId

(giving the accumulator more work)

2x 1 million goes fast enough if I make sure they generate distinct sets

however 1x 2 million tweaked to have the same overlap in consent as 2x 1 million still goes an order of magnitude faster

I know clara doesn't deduplicate facts, but I think in this case that would prevent me from running into this scenario

@wdullaer thanks for the details

I will take a look. It may be about an hour or so though. Doing a few other things at the moment.

no worries, take all the time you need

I can supply the vm snapshot as well (just need to find a quick place to upload it)

Yeah, if you have one you can share, that’d be good to take a glance at.

How to share hmm

I'll put it into dropbox, just a sec

https://www.dropbox.com/s/no6h9pwa03hzt7w/clara-perf.nps?dl=0

ok

@wdullaer I’ve only got to look at this briefly so far today

It seems like these collections created via acc/distinct would be fairly large. Also, there are many of them after the first 1mil facts. There’d be a big acc/distinct collection associated with each personId + purpose grouped Consent facts. When the next 1mil facts come in, I believe they will contribute to many (or perhaps even all) of those same big collections.

The work from the first 1 mil facts is stored in working memory. Clara builds the new collections on the 2nd wave of inserts. It then must remove the old accumulated work it did from working memory - in order to replace it with the updates.

I think the fact that there are many of these large collections in working memory is making it get really expensive to remove them all from working memory. Clara tries to be efficient in finding and removing things in memory. I think huge collections may be a pitfall.

I’d like to look a little deeper at it, but haven’t been able to yet. I’m not sure what a reasonable workaround may be, other than inserting everything in one batch (as you’ve found).

It’s an interesting case to explore some more for sure.

thanks for the input

I can see how this is a worst case scenario

we probably won’t see this amount of massive updates in a real life scenario, and if it does (say a big data migration), we can always consider starting from scratch

it was a bit surprising that updating these structures is a lot more expensive than creating them

if I can help in any way here, just let me know

I have a few other benchmarks I’m going to run over the coming days as well

I need to look deeper at what is happening to have a stronger sense of what could be done

I’ve logged https://github.com/cerner/clara-rules/issues/385 for this @mikerod @wdullaer , I’d suggest that we post findings there when we have them

From the snapshot it looks like the memory is the bottleneck, which isn’t too surprising. From my first glance over it I suspect there are some performance optimizations we can make for cases like this, but it may be a bit before I have time to write my thoughts down in a sufficiently articulate form. The benchmark and snapshot are really helpful and make diagnosing things like this much easier - thanks for that. It is useful to have benchmarks of realistic rule patterns that stress-test Clara.