Clojurians Log v2

Hi all, I'm back 🙂 . Sorry for the several questions and thank you very much for all your prompt and very helpful responses; it's been very enjoyable working with Clara. I am progressing very well in building up the ruling system in Clara however I am getting some OOMs when I surpass a certain number of facts in my application. My application has around 15 rules, some of which are fired by others, and the facts may well range between 300k - 2m, but when I go closer to the upper end I get OOMs. So, I thought that it may be reasonable to run a few benchmark tests (and maybe start profiling?). I tried the clara-benchmark repository and experimented a bit with the AccumInsert test; in this I've played with the number of Customers, setting it from 250k to 10m. I noticed that on my machine, Clara threw an OOM above 5m facts. I was wondering whether you have a direction to suggest on methods to cut down such costs? In my application the first try was to chop the initial facts only to the absolutely necessary fields required (I am using clojure maps as extracted from a DB) but without luck. Thank you all in advance!

@konstantinos562 2m facts themself would be quite a memory burden in the jvm I think. If it’s just memory issues is the heap space you’re giving enough?

You can use a profiler to look more at the memory layout but Clara will certainly take more space than those 2m facts since it has working memory structure it uses on top of what they already take alone. And if you have rules inserting facts based on them.

The benchmark stuff would be more aimed at execution time. Which isn’t too useful if you’re running out of memory. If it’s slow it’s probably due to GC thrashing before OOM

Or at least that’d hide any other perf issues.

Hi @mikerod, yes that's true, the heap space is set to 4G. I have tried to trim down the facts' fields to the absolute necessary, and now I am considering of using a strategy to reduce the size of facts sent to Clara

@konstantinos562 it’d be good to know how much of the 4G you are using before running the rules

To gauge how much clara is adding relative to that.

Hmmm good question, I haven't noted this down, although I think it's relatively low as I start from a clean repl.

But the diagnosing the issue will also come down to what you’re rules are doing. If they match too many facts in ways where they are producing factors like n^2 or more rule activations and new fact insertions - I think that already is putting you at quite a lot of objects in memory

but will note this with VisualVM

Visualvm can be good to get some info for this.

Yes, that is more close to what happening

Also you can try to just jump heap to like 6G then 8G to get a sense of how much memory it really needs. Of course this may not be acceptable in your production, but for testing

Also Clara shouldn’t hold references to facts that can never match any rules.

I think I will try another pass on the rules in an effort to downsize the domain

So in some situations, if you actually can filter out a bunch of stuff early, you could do some sort of “stream of facts” in a sense. But your calling code would also have to not hold references past inserting them to Clara and firing rules.

I see what you mean

I am currently retrieving all facts from the DB, which maybe streaming this in Clara without holding a head would make a difference... thanks for the suggestion

Yeah to not have them all in memory at once I think you’d have to be a bit careful.

Not sure something like insert-all would be careful enough within Clara. Would probably have to like near N number at a time, then fire rules. Then repeat the next batch

But if they are mostly matching or even “partial matching” rules they will be held in memory. So it can be brittle

A fact is held in memory if it can even “potentially” be used to satisfy a rule later. That’s the “partial match”

(I was just checking insert-all's source, as this is what I am using to insert facts - but will need to check a bit more carefully on the DB side too)

Which intuitively makes sense. Since it may be needed later when new facts come.

Makes sense yeah

But yeah my first thought would be the original stuff I said. Understand memory use a bit more.

Thanks a bunch @mikerod

hi @mikerod, I am trying to run the tests from clara repository and I keep receiving this error:

clojure.lang.ExceptionInfo: failed compiling file:src/test/common/clara/test_common.cljc {:file #object[<http://java.io|java.io>.File 0x1085a882 "src/test/common/clara/test_common.cljc"]}

there is any setup I need to do when adding a new test namespace?

Caused by: java.lang.RuntimeException: Unable to resolve symbol: test-rule in this context

@iagwanderson are you working with clj or cljs for this? And explain what you’re doing a bit more. You added new test ns? How are you trying to run them?

I am working with clj, but as I see it should work with cljs as well. I created a file called test_blacklist.cljc inside the test/common/clara directory

and I tried to run using a simple lein test which runs fine if I remove my new namespace

I tried to run the lein test again only with this new empty namespace and I got this error msg already

@iagwanderson I’d have to see your changes I think at this point to help

This repository has the changes and the test file: https://github.com/wandersoncferreira/clara-rules

I changed the compiler.clj file

https://github.com/wandersoncferreira/clara-rules/blob/ede84e40a0f9671544ce4ab349e7e6735ab151e0/src/main/clojure/clara/rules/compiler.clj#L2040

and my test file: https://github.com/wandersoncferreira/clara-rules/blob/ede84e40a0f9671544ce4ab349e7e6735ab151e0/src/test/common/clara/test_blacklist.cljc

Thanks. I’ll take a look in a few.

Ok, thanks. There is no clear relationship between my implementation and the error messages

@iagwanderson you need to use def-rules-test for cross platform “common” tests

That may be your issue

Look at another test ns in that dir to see it used.

It’s used in all the rule sort of tests there.

Uhmm... Ok. I will make this change.

Can’t say it’s certainly the issue. But likely. I can’t remember without looking a bit more

There are differences in how you would set this up in clj vs cljs.

So the test helper makes that not an issue

For portable test

I never did cljc before, so I was betting on something related to that

I will make the test work and open a PR for discussions around the implementation

@mikerod looks like the Compiling ["resources/public/js/simple.js"] failed. because in the cljs_deps.js does not have definitions for my new file added

fixed, lein do clean, compile 😃

Yeah. I have a long standing issue to improve the cljs setup and build stuff. But it is involved. So for now have to work with this stuff as it is.