Clojurians Log v2

oh, hello 👋

Hi Alan!

just took a cruise through the asami query engine, very nice 👏

i like the subgraph discovery piece that avoids unnecessary cross joins

Aww, thank you.

Do you mean the analytics namespace, or are you referring to the aggregates part? (I’m unhappy with aggregate queries right now)

oh, i was referring to paths i think the function was called

Ah!

finding the route thruogh the tuples for the least amount of tuple-scooping

That’s the planner

err, the rules rather

Yup. It helps a lot!

I can’t say that I did a GREAT job of it, but I did it. 😊

Working code beats perfect proposals

i like it and it's small

i have been on a personal quest lately to write the smallest viable tuple store/datalog in javascript

eschewing optimizations when possible etc

i got kind of lost in the literature about bottom-up vs top-down evaluation, magic sets, etc

Oh... going back to Naga in late 2016 should provide a good Cljs example. Easily portable to JS, I think

The “paths” function wasn’t as good, but the query engine was definitely minimum viable!

On my phone, but let me look...

https://gist.github.com/alandipert/4e390808b5dcee9020b6622c0a24023e#file-tinylog-js-L54-L60 is my vision for the DSL, tho the engine on that gist is too naive/broken

Try the files here: https://github.com/threatgrid/naga/tree/cd4313229b0eec5d2e5d729e05770a733e0edbdb/src/naga/storage/memory

The only supported operation is inner joins, but it works. The index namespace is tiny as well

cool. yeah for indexes i'll just be doing JS arrays + binary search i think

That version of the planner used dynamic programming, so definitely don’t touch that old planner

Yes, that will pack in far better, and give you other benefits as well

Hashmaps are expensive

The durable storage does arrays, though they’re broken into chunks and accessed via a tree

But I’m familiar with using arrays 🙂

i prefer linked lists of course, but i'll deal 😄

i'd be curious to hear your reaction to my general ideas around using stuff like this for UI maintenance

I think it’s extremely flexible

I was storing lots of things in this sort of format without a query engine, because of the flexibility it provides

And it can be used to represent shallow maps trivially. Deeper structures take more effort, but are possible

Do you have specific ideas in terms of UI?

a general feeling, punctuated with concrete ideas

the general UI problem is continuously/efficiently synchronizing/managing UI primitives as business data underneath them changes

since most business data is hierarchical, most UI toolkits and UI data models propose hierarchical definitions of equality (and thus, change)

but in practice state used "internally" (for composition of UI primitives) gets muddled with business data

anyway, i have a good feeling about little datalog databases wired up to one another, each with independent ideas of what constitutes a "change", based on which queries are installed

my current direction is, make minimum viable database and query, and then build "subscriptions" on that. just run subscribed queries every time a new fact goes in, and if the result is different, fire a callback with the differences

When the data is quickly accessible in memory, then even without a query engine, I think it offers a lot. It’s easy to build any kind of structure in there. And you can store the universe in there without messing things up. You can also link things (like business data and UI elements) arbitrarily. It’s easy to get anything out of it. And it’s efficient.

the difference set can be interpreted at the DOM node level to inform adding/removing/changing DOM nodes etc

It’s relatively easy to find out if a fact can affect a subscribed query too. Naga does this

ah cool, i've noodled about that. i was thinking about how you could transform a query into a function that returned whether or not a fact could possibly affect it... and then only actually run the query again if there's a chance it would

that line of thinking got me back into rete

eg. If your query has: [?e :type ?t] [?e :name ?n] Then the output isn’t affected if you insert: [:entity124 :value 13]

Did you ever see my 2016 talk at the Conj?

Because I was discussing exactly this

no, but a watching is imminent

And pointing you at that is much easier than typing on my phone :rolling_on_the_floor_laughing:

It takes a shortcut. It assumes that data is only added, rather than allowing modifications. This is how Description Logic systems usually work. And you can emulate modification detection by having a system with time stamped add/delete statements. Or you can just save query results and compare them. It depends on your scalability needs

If it’s in pure JS, then comparing results (as opposed to result-counts) is probably ideal

that's what i'm thinking... data inherently small... and timestamp for handling deletions not required

Naga’s engine would be ideal, I think

It’s in 2 phases. The first finds dependencies between rules (this is run once, but you would run it after each insert) and then the engine looks for changes via counts, though you would look for data changes instead

i will watch the talk, study naga, and emit JS codes. thanks much for the info and pointers!

The code that does what I just said is all in here: https://github.com/threatgrid/naga/blob/main/src/naga/engine.cljc