Clojurians Log v2

I have some [x y] location tuples in datomic, If I want to do a bounding box query (xmin,ymin - xmax,ymax) range… I think I need to untuple those and use numeric comparisons… I’m guessing it would be more efficient to model x and y as separate attributes? or can tuples be efficiently pulled from index

Tuples can be pulled from index, but you’re going to get them collated by x then y

yeah, seems logical

Something like this seems most efficient, but it may potentially read a lot of unnecessary datoms:

(->> (d/seek-datoms db :avet :location [xmin, ymin])
     (take-while (fn [{[x _y] :v}] (<= x xmax)))
     (filter (fn [{[_x y] :v}] (<= y ymax))))

I think I’ll change to separate x/y attributes as either one (x or y range) could be the that filters out the most

You might also consider projecting them into a secondary database that supports an r-tree index natively

you could read the tx queue and just pump simple [attribute value entity time] rows into e.g. sqlite

for the coordinate-valued attributes

that has more moving parts and I don’t want to do it unless I need to

that’s fair

Hi all. I'm having trouble debugging an issue with my datomic installation. When trying to commit a transaction, I get the following exception:

Caused by: clojure.lang.ExceptionInfo: Missing keys {:missing #{:key :rev :id}}
        at datomic.common$require_keys.invokeStatic(common.clj:224)
        at datomic.common$require_keys.invoke(common.clj:218)
        at datomic.kv_cluster$same_ref_QMARK_.invokeStatic(kv_cluster.clj:131)
        at datomic.kv_cluster$same_ref_QMARK_.invoke(kv_cluster.clj:128)
        at datomic.kv_cluster.KVCluster$fn__17383$fn__17387.invoke(kv_cluster.clj:227)
        at datomic.kv_cluster.KVCluster$fn__17383.invoke(kv_cluster.clj:215)
        at clojure.core$binding_conveyor_fn$fn__5754.invoke(core.clj:2030)
        at clojure.lang.AFn.call(AFn.java:18)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        ... 1 more

Hi @adriaan.callaerts, what does the "reset environment" script do? If it is deleting the underlying storage (SQL) for Datomic then you have corrupted the DB. Do you have a datomic level backup for this system?

No I don't. Mind you we've only tried this on a staging environment so far, so if anything is lost it's not a disaster. The goal for us is to be able to, on regular intervals, create a "fresh slate" for the entire system. I figured I could do that by dropping and recreating the database. Is there any state persisted elsewhere (in peers/transactor) that I should also be aware of and should be cleared?

No, Datomic uses the underlying storage as a KV store. Deleting the underlying storage will corrupt the Datomic DB. I would be surprised if the transactor would even start against such a DB.

May I recommend that you run backup at the Datomic level on your production system. https://docs.datomic.com/on-prem/operation/backup.html

Regarding your desire for a fresh state, you could also create a solution using Datomic backup/restore to create such environments

My goal is not to restore a backup here. It's to recreate the "new environment"-scenario. Shouldn't a transactor act as if it's being started for the first time when it encounters an empty (but validly structured) database?

Ah, in that case you will want to follow these steps: https://docs.datomic.com/on-prem/overview/storage.html#sql-database

I actually walked through provisioning a new system on an old blog post here https://jaretbinford.github.io/SQL-Storage/

@adriaan.callaerts are you turning off all peers and transactors before calling your script?

Additionally, what specifically does the script do?

no, I might still have the transactor running. So that's probably what's causing the issue...

Yes, absolutely. I'd also ask, could you perhaps utilize delete-database for this solution? (note: if you do go this route you will want to delete the DB and then create a new uniquely named DB. Using the same DB quickly before all resources are cleaned up can lead to an issue.)

I am uncomfortable with anything that alters underlying storage and prefer to work at the Datomic API level if possible 🙂 (but then again, I work for the Datomic team so I am biased)

I wanted to add that I managed to get things working the way I wanted, thanks to your help. Restarting the transactor and peers did the trick!

Is there a dependency tree for the Datomic Cloud primary compute group?

What are you trying to do @futuro?

I recently deployed my first ion (Yay!) and ran into some dependency difficulties along the way. We're using metosin/jsonista for json parsing, and it relies on a com.fasterxml.jackson.core/jackson-core (plus a couple others) which are newer than what's in the Cloud system. Overriding just jackson-core will result in something like a Class not found exception on load, because two other fasterxml libs are expecting a class from a newer version of jackson-core. If I update all of the dependencies the load errors go away.

That's the context.

What I'd like is to either learn about these conflicts without having to push (and without having to commit anything, so I can tweak and re-tweak as I learn more.

I'm also curious if there's a way to only need to consider my projects dependencies, and not the Datomic Cloud dependencies (maybe a query group with a minimal set? I'm not sure.)

1. You can push an unreproducible build without committing to git and it will return a (hopefully empty) set of dependency conflicts. 2. Your ion runs inside datomic's jvm and shares a classpath so there isn't currently a way to isolate the dependencies. 3. We are unlikely to address the problem by making a QG with minimum dep set. 4. We usually update dependency versions every release. 5. Explicitly matching the same jackson dep version to other jackson deps seems to have addressed your problem, correct?

1. Can I re-use the unreproducible build name for rapid iteration, or do I need to make it unique each time (presuming I don't intend to deploy that build). 2. Yep! I didn't mention it cause I didn't want to propose solutions. 3. Hmm...ok 4. That's what I was hoping for, and why I updated our stack yesterday, but the version of jackson-core is 2.10.1, from Nov 2019, and the latest is 2.12.3 from April. Without knowing what requires this version, or why, I can't say whether going to the latest version makes "sense", but it's definitely old. 5. It has solved the issue where loading throws a Class Not Found exception, but I'm not sure how jsonista relies on the three jackson deps I've overridden, so it's plausible there are issues waiting for me down the road.

jackson is probably a transitive dep (via something like transit-clj -> transit-java) so it may be that datomic is taking newest version of top-level deps (but not transitive)

I know for instance that transit-java is way behind because it is not an easy upgrade to latest jackson there

That makes sense, and mirrors my experience trying to update jackson in the past.

The unreproducible build policy is "latest-push-wins" aka overwrites all prior ones. So your unrepro name can be futuro-awesome-unrepro and you can bang on it all day long 🙂

Huzzah! That definitely helps the fast-iteration on dependency conflicts, thanks Joe!

FWIW, we return a dependency map for you to add to your deps.edn so there are no more conflicts. Is that not working as expected for you @futuro?

Two questions in response to that: 1. Is there a roadmap/plan for updating those dependencies? (This isn't about specific timelines, but instead of prioritization and whether that it's currently being planned for or not) 2. To help improve my thinking on this, I'd love to hear how you all consider and ward against bugs caused by dependency conflicts. Very thorough unit tests? My thinking currently is to start adding tests against my dependencies, alongside my project's tests, but that doesn't quite seem right.

Specifically for overriding top-level deps, yes. With the jackson-core dependency I have it did not, because of three other transitive jackson dependencies that weren't overridden as well.

My approach so far has been to review my dependency tree to see which of my deps has a dep that's being overridden, to better understand what kind of bugs might be introduced by it being overridden, or if I can downgrade that library to where it relied on the overridden dep at the version that's in the cloud system.

1. See 4. above 2. Your deps will never conflict with our deps. We enforce that. If you need to test against different dep versions (of dubious value IMO) then you need to use our version of the dep in your project. A better strategy might be avoid libraries with big dep graphs.

1. I hear that, though also the system is using, somehow, a version of jackson-core from 2019.

2. Ah, my question may be better worded "I'm trying to figure out how to consider and ward against dep conflicts; how do you all think of this problem (so I may improve my thinking)?"

I've updated the question

"I'm adding a new piece of equipment to my lightweight airplane and it needs to be mounted on the left side of the body. I bought these nuts from the supplier at a different time than these bolts. They are the right thread size, material and spec, but just to be sure they work, I'm considering pouring concrete around the fasteners, just to be safe" <- What I'm hearing 🙂 I realize that's a cheeky response (glad you're my friend @futuro ), but I hope it illustrates the benefit/cost ratio of YOU writing tests for your deps. I'd rather just avoid the dep all together and stay light.

I think that's a mostly accurate read on the situation, and I'd also like to stay light (I'm currently working on adjusting how I deploy the ion to keep the json handling code elsewhere), but I'm not certain it's always going to be possible.

It sounds like the issue here for you is jsonista depending on jackson. Can you just use a different json library or the version of clojure.data.json that cloud depends on?

I'd say the more accurate cheeky response is "I'm adding a new piece of equipment to my lightweight airplane and it needs to be mounted on the left side of the body. I bought these nuts from the supplier at a different time than these bolts. I think they're the right thread size, material, and spec, but I'm not sure because I'm not actually working with nuts/bolts and it's more complicated than that."

Possibly yeah, though I need to a bit more research.

I believe there's a way forward for this specific instance (complicated a bit by using a polylith architecture, which I otherwise quite enjoy), and now I'm trying to understand the issue in a more general sense to navigate this should it come up again in the future.

Partially for my own benefit, and partially because I'm the Datomic Cloud evangelist/lead on my team and want to be able to properly represent the trade-offs/risks with dependency conflicts, and how it relates to Datomic On-Prem (which I don't particularly want to spend time setting up or operating when there's Datomic Cloud around).

I think this issue (in the context of Datomic) is specific to jackson and the other handful of deps that cloud depends on that could conflict with user-space. I'm not sure you need to generalize, and I'm also not sure I can help give general advice 🙂

That's legit.

On-prem has the same situation though with the peer library. This is just normal java classpath dependency conflicts, we just happen to tell you that we overrode your deps in cloud (as opposed to spring boot or other jvm container solutions )

That is a very helpful piece of information, thank you.

It's possible the "solution" is to have the Cloud deps tree and assess whether those libraries are in our critical path or not. Whether that's something y'all are open to sharing, I have no idea. And it's also possible that the solution is to build smaller ions and, if push comes to shove, use ECS/EC2/etc to run whatever our critical-path-can't-override-deps code is and have the ions talk to it.

> It's possible the "solution" is to have the Cloud deps tree and assess whether those libraries are in our critical path or not. A solution needs a problem. > Whether that's something y'all are open to sharing, I have no idea. Probably not. > And it's also possible that the solution is to build smaller ions and, if push comes to shove, use ECS/EC2/etc to run whatever our critical-path-can't-override-deps code is and have the ions talk to it. Again, what problem does "build smaller ions" solve? "Smaller" is a characteristic, I could have 10000 deps of 1kb in size. I highly doubt that the "optimal" solution will ever be don't run in ions because of the problem "my dep conflicts with cloud's dep" .

"Smaller" in terms of the deps tree. The "problem" being a dependency conflict that breaks a needed code path in a project dependency, which can't be resolved by downgrading or upgrading that particular dependency.

Do you actually have that problem or are you hypothesizing? If you actually have that problem you should contact support and we can help figure out what to do 🙂

I have worked around the current manifestation of this problem, and no longer have that problem (could have sworn I said as much ;) ). If I hit that problem, though, I'll reach out to support 🙂.

Thanks for hashing this out with me @lanejo01 🙂

Can't wait til this whole thing blows over and we can grab a pint @futuro, always fun chatting with you!

Omg, for real. 😅