Clojurians Log v2

Alpha5 is now done. Same as Alpha4, but queries are significantly faster on large datasets

putting Alpha5 thru some paces

Well, it’s alpha so we can find the big problems and address them before it’s called a “release” 🙂

gosh I'm rusty at clojure 8^)

also, do not buy Brach's 24 Flavours tiny Jelly Beans

their attempt at Jelly Belly knockoffs... It's like they didn't realize that the flavors must harmonize when you shovel a handful into your maw

ok, think I broked it

For anyone wondering, Craig is allowed to push me around in here. He’s no longer at Cisco, but it was his bright idea that I write my own graph database.

Yup? What’s happened?

Locked up importing a few thousand objects

I'll break up the txn

and we'll see what's happening. I must first eliminate my own stupidity...

Actually, breaking up the transaction is a bad thing to do. How big is the file that you’re importing?

(bad, because you end up expanding the indexes significantly)

Also, if you’ve made a mistake, I’d like to know about that too. I should document gotchas, and mitigate some of the more obvious ones

yah, so it's not locked up, but just slow. Mind you I'm throwing a lot of large complex objects at it

I'll get data to you shortly

“large complex” is going to be an issue. Zuko (the module that breaks it up into triples) is now faster than it used to be, but there’s still a lot of work for it to do

Yah, I'm thinking it's a chance to intrument the whole thing with metrics data

Importing o1 into asami Importing to asami "Elapsed time: 53402.866849 msecs" Imported 4503 Importing o2 into asami Importing to asami "Elapsed time: 644524.77233 msecs" Imported 41484

FAIL in (load-test) (core_test.clj:21) Test loading and parsing expected: (= (count o1) (count (:tempids tx1))) actual: (not (= 271 4503)) lein test :only netgrok.core-test/load-test FAIL in (load-test) (core_test.clj:22) Test loading and parsing expected: (= (count o2) (count (:tempids tx2))) actual: (not (= 2570 41484))

So the failures are me expecting the entity count to be the input object count. The difference tells you just how complex some of the objects are, with many nested entities...

It creates lots of temporary IDs, but unless you ask, I would generally think they’d match the provided objects. :thinking_face:

I’m assuming that some or all of this data can be shared?

not sure. it's packet dumps form my home network

I will find some representative data

The :tempids in the tx would include the nested objects right?

I didn’t think so (unless you asked it to). So unless I’ve forgotten something it’s a problem

It creates lots of IDs, but that map is supposed to just be for the top level entities, and things you’ve provided your own temporary IDs to

yah, so that seems wrong then

I provided no temp IDs for anything

So, I ran the same thing with in mem db

lein test netgrok.core-test Preparing o1 "Elapsed time: 0.005525 msecs" Preparing o2 "Elapsed time: 8.82E-4 msecs" Importing o1 into asami Importing to asami "Elapsed time: 262.289293 msecs" Imported 4503 Importing o2 into asami Importing to asami "Elapsed time: 2622.369329 msecs" Imported 41484

Is zuko involved in that too?

yes

It’s a library that pulls entities apart into triples

ok, so it's not in zuko then eh

So I am coercing string keys in JSON to keywords... OUt of.. habit?

Would I be violating any assumptions of Asami if I did not do that?

no

or… I hope not 🙂

I need to check my undertanding here:

[:tg/node-929806 "ip.flags" "0x00000040"]
 [:tg/node-623767 "layers" :tg/node-623768]
 [:tg/node-623767 :db/ident :tg/node-623767]
 [:tg/node-623767 :tg/entity true]

netgrok.core> (d/entity (d/db (conn)) :tg/node-623767)
{}

I would not expect that to be an empty entity

the triples are from: (d/export-data (d/db (conn)))

this is asami alpha5 running in memory

(conn) is just (d/db-connect URI) ...

so I'm making a new connection using the DB uri, and a new DB...

Ah, ok, if I don't coerce keys to keywords in the structs, entiy loading fails

So I think that's a boog?

could be

interesting!

this is in memory DB

I’m working with the cleaned up data right now, so the attributes are all keywords. I’ll try with the strings shortly

Oh, that’s interesting too

yah, since the entity func is basically per storage...

export-data is my new pal

yah, so the file I sent you, I beleive has string keys

it does, yes

:smiling_face_with_3_hearts: this is a pleasant way to explore data

BTW, the large number of entities in tempids was expected, but I’m revisiting it, and I think they should not be included.

So I’m going to update Zuko to remove them

ok, gotten more familiar with the query language. Able to identify all the devices on my network, and start digging into their behavior

The query language is impressie Paula

calling it a day tho, so I stop obsessing over it