datomic

Ask questions on the official Q&A site at https://ask.datomic.com!
joshkh 2021-05-25T08:41:25.050900Z

we tend to explicitly add dates because the historical data is not accessible to our external integrations via Datomic Analytics. also, if you ever replay your tx log from one database to another then the dates of the transactions will differ

tatut 2021-05-25T11:44:37.052800Z

I have some [x y] location tuples in datomic, If I want to do a bounding box query (xmin,ymin - xmax,ymax) range… I think I need to untuple those and use numeric comparisons… I’m guessing it would be more efficient to model x and y as separate attributes? or can tuples be efficiently pulled from index

favila 2021-05-25T12:27:58.052900Z

@joshkh “if you ever replay your tx log from one database to another then the dates of the transactions will differ” that’s not completely correct. the :db/txInstant assertion is in the tx log, so it will copy over unless you filter it out

favila 2021-05-25T12:29:11.053100Z

The use case for allowing this is to back-date data, but the tx/Instant of any new transaction must be >= the previous one, so this technique is limited to fresh databases

favila 2021-05-25T12:30:19.053400Z

Tuples can be pulled from index, but you’re going to get them collated by x then y

joshkh 2021-05-25T12:30:39.053600Z

that's news to me, thanks for the correction @favila

tatut 2021-05-25T12:35:10.054100Z

yeah, seems logical

favila 2021-05-25T12:35:37.054300Z

Something like this seems most efficient, but it may potentially read a lot of unnecessary datoms:

favila 2021-05-25T12:35:38.054500Z

(->> (d/seek-datoms db :avet :location [xmin, ymin])
     (take-while (fn [{[x _y] :v}] (<= x xmax)))
     (filter (fn [{[_x y] :v}] (<= y ymax))))

tatut 2021-05-25T12:36:09.054700Z

I think I’ll change to separate x/y attributes as either one (x or y range) could be the that filters out the most

favila 2021-05-25T12:36:49.054900Z

You might also consider projecting them into a secondary database that supports an r-tree index natively

favila 2021-05-25T12:37:30.055100Z

you could read the tx queue and just pump simple [attribute value entity time] rows into e.g. sqlite

favila 2021-05-25T12:37:51.055300Z

for the coordinate-valued attributes

tatut 2021-05-25T12:38:07.055500Z

that has more moving parts and I don’t want to do it unless I need to

favila 2021-05-25T12:38:24.055700Z

that’s fair

Adriaan Callaerts 2021-05-25T13:20:13.058Z

Hi all. I'm having trouble debugging an issue with my datomic installation. When trying to commit a transaction, I get the following exception:

Caused by: clojure.lang.ExceptionInfo: Missing keys {:missing #{:key :rev :id}}
        at datomic.common$require_keys.invokeStatic(common.clj:224)
        at datomic.common$require_keys.invoke(common.clj:218)
        at datomic.kv_cluster$same_ref_QMARK_.invokeStatic(kv_cluster.clj:131)
        at datomic.kv_cluster$same_ref_QMARK_.invoke(kv_cluster.clj:128)
        at datomic.kv_cluster.KVCluster$fn__17383$fn__17387.invoke(kv_cluster.clj:227)
        at datomic.kv_cluster.KVCluster$fn__17383.invoke(kv_cluster.clj:215)
        at clojure.core$binding_conveyor_fn$fn__5754.invoke(core.clj:2030)
        at clojure.lang.AFn.call(AFn.java:18)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        ... 1 more
we haven't encountered this error before, but what changed recently was: we upgraded to datomic v1.0.6269 and we're trying a "reset environment"-script which clears the storage (MySQL in our case) and then inserts new data through the normal peer

jaret 2021-05-25T13:25:28.059700Z

Hi @adriaan.callaerts, what does the "reset environment" script do? If it is deleting the underlying storage (SQL) for Datomic then you have corrupted the DB. Do you have a datomic level backup for this system?

Adriaan Callaerts 2021-05-25T13:27:22.061700Z

No I don't. Mind you we've only tried this on a staging environment so far, so if anything is lost it's not a disaster. The goal for us is to be able to, on regular intervals, create a "fresh slate" for the entire system. I figured I could do that by dropping and recreating the database. Is there any state persisted elsewhere (in peers/transactor) that I should also be aware of and should be cleared?

jaret 2021-05-25T13:29:55.063300Z

No, Datomic uses the underlying storage as a KV store. Deleting the underlying storage will corrupt the Datomic DB. I would be surprised if the transactor would even start against such a DB.

jaret 2021-05-25T13:30:27.063900Z

May I recommend that you run backup at the Datomic level on your production system. https://docs.datomic.com/on-prem/operation/backup.html

jaret 2021-05-25T13:31:32.065900Z

Regarding your desire for a fresh state, you could also create a solution using Datomic backup/restore to create such environments

Adriaan Callaerts 2021-05-25T13:31:41.066300Z

My goal is not to restore a backup here. It's to recreate the "new environment"-scenario. Shouldn't a transactor act as if it's being started for the first time when it encounters an empty (but validly structured) database?

jaret 2021-05-25T13:32:36.067400Z

Ah, in that case you will want to follow these steps: https://docs.datomic.com/on-prem/overview/storage.html#sql-database

jaret 2021-05-25T13:34:24.068800Z

I actually walked through provisioning a new system on an old blog post here https://jaretbinford.github.io/SQL-Storage/

jaret 2021-05-25T13:37:13.069600Z

@adriaan.callaerts are you turning off all peers and transactors before calling your script?

jaret 2021-05-25T13:37:27.070200Z

Additionally, what specifically does the script do?

Adriaan Callaerts 2021-05-25T13:37:39.070600Z

no, I might still have the transactor running. So that's probably what's causing the issue...

1➕
jaret 2021-05-25T13:39:37.073100Z

Yes, absolutely. I'd also ask, could you perhaps utilize delete-database for this solution? (note: if you do go this route you will want to delete the DB and then create a new uniquely named DB. Using the same DB quickly before all resources are cleaned up can lead to an issue.)

jaret 2021-05-25T13:40:21.074Z

I am uncomfortable with anything that alters underlying storage and prefer to work at the Datomic API level if possible 🙂 (but then again, I work for the Datomic team so I am biased)

1😁
Adriaan Callaerts 2021-05-25T14:36:29.074900Z

I wanted to add that I managed to get things working the way I wanted, thanks to your help. Restarting the transactor and peers did the trick!

2💯
futuro 2021-05-25T14:41:11.076Z

Is there a dependency tree for the Datomic Cloud primary compute group?

Joe Lane 2021-05-25T14:48:37.076400Z

What are you trying to do @futuro?

futuro 2021-05-25T14:53:07.080800Z

I recently deployed my first ion (Yay!) and ran into some dependency difficulties along the way. We're using metosin/jsonista for json parsing, and it relies on a com.fasterxml.jackson.core/jackson-core (plus a couple others) which are newer than what's in the Cloud system. Overriding just jackson-core will result in something like a Class not found exception on load, because two other fasterxml libs are expecting a class from a newer version of jackson-core. If I update all of the dependencies the load errors go away.

futuro 2021-05-25T14:53:12.081Z

That's the context.

futuro 2021-05-25T14:53:39.081700Z

What I'd like is to either learn about these conflicts without having to push (and without having to commit anything, so I can tweak and re-tweak as I learn more.

futuro 2021-05-25T14:55:02.083300Z

I'm also curious if there's a way to only need to consider my projects dependencies, and not the Datomic Cloud dependencies (maybe a query group with a minimal set? I'm not sure.)

Joe Lane 2021-05-25T15:11:26.087900Z

1. You can push an unreproducible build without committing to git and it will return a (hopefully empty) set of dependency conflicts. 2. Your ion runs inside datomic's jvm and shares a classpath so there isn't currently a way to isolate the dependencies. 3. We are unlikely to address the problem by making a QG with minimum dep set. 4. We usually update dependency versions every release. 5. Explicitly matching the same jackson dep version to other jackson deps seems to have addressed your problem, correct?

cjsauer 2021-05-25T15:16:23.091800Z

@kenny do you happen to use this technique to inject authorization constraints onto queries as well? Authorization seems to touch every query in my system, and it occurred to me I might be able to append those rules onto queries using generic helper functions..

futuro 2021-05-25T15:17:42.093300Z

1. Can I re-use the unreproducible build name for rapid iteration, or do I need to make it unique each time (presuming I don't intend to deploy that build). 2. Yep! I didn't mention it cause I didn't want to propose solutions. 3. Hmm...ok 4. That's what I was hoping for, and why I updated our stack yesterday, but the version of jackson-core is 2.10.1, from Nov 2019, and the latest is 2.12.3 from April. Without knowing what requires this version, or why, I can't say whether going to the latest version makes "sense", but it's definitely old. 5. It has solved the issue where loading throws a Class Not Found exception, but I'm not sure how jsonista relies on the three jackson deps I've overridden, so it's plausible there are issues waiting for me down the road.

alexmiller 2021-05-25T15:19:14.094100Z

jackson is probably a transitive dep (via something like transit-clj -> transit-java) so it may be that datomic is taking newest version of top-level deps (but not transitive)

alexmiller 2021-05-25T15:19:54.094700Z

I know for instance that transit-java is way behind because it is not an easy upgrade to latest jackson there

futuro 2021-05-25T15:21:34.095Z

That makes sense, and mirrors my experience trying to update jackson in the past.

Joe Lane 2021-05-25T15:24:07.095600Z

The unreproducible build policy is "latest-push-wins" aka overwrites all prior ones. So your unrepro name can be futuro-awesome-unrepro and you can bang on it all day long 🙂

futuro 2021-05-25T15:36:18.095800Z

Huzzah! That definitely helps the fast-iteration on dependency conflicts, thanks Joe!

Joe Lane 2021-05-25T15:37:12.096600Z

FWIW, we return a dependency map for you to add to your deps.edn so there are no more conflicts. Is that not working as expected for you @futuro?

futuro 2021-05-25T15:43:04.101600Z

Two questions in response to that: 1. Is there a roadmap/plan for updating those dependencies? (This isn't about specific timelines, but instead of prioritization and whether that it's currently being planned for or not) 2. To help improve my thinking on this, I'd love to hear how you all consider and ward against bugs caused by dependency conflicts. Very thorough unit tests? My thinking currently is to start adding tests against my dependencies, alongside my project's tests, but that doesn't quite seem right.

futuro 2021-05-25T15:44:24.101700Z

Specifically for overriding top-level deps, yes. With the jackson-core dependency I have it did not, because of three other transitive jackson dependencies that weren't overridden as well.

futuro 2021-05-25T15:45:41.101900Z

My approach so far has been to review my dependency tree to see which of my deps has a dep that's being overridden, to better understand what kind of bugs might be introduced by it being overridden, or if I can downgrade that library to where it relied on the overridden dep at the version that's in the cloud system.

Joe Lane 2021-05-25T15:47:13.102100Z

1. See 4. above 2. Your deps will never conflict with our deps. We enforce that. If you need to test against different dep versions (of dubious value IMO) then you need to use our version of the dep in your project. A better strategy might be avoid libraries with big dep graphs.

futuro 2021-05-25T15:51:22.102300Z

1. I hear that, though also the system is using, somehow, a version of jackson-core from 2019.

futuro 2021-05-25T15:52:15.102500Z

2. Ah, my question may be better worded "I'm trying to figure out how to consider and ward against dep conflicts; how do you all think of this problem (so I may improve my thinking)?"

futuro 2021-05-25T15:52:52.102800Z

I've updated the question

Joe Lane 2021-05-25T16:01:32.103Z

"I'm adding a new piece of equipment to my lightweight airplane and it needs to be mounted on the left side of the body. I bought these nuts from the supplier at a different time than these bolts. They are the right thread size, material and spec, but just to be sure they work, I'm considering pouring concrete around the fasteners, just to be safe" <- What I'm hearing 🙂 I realize that's a cheeky response (glad you're my friend @futuro ), but I hope it illustrates the benefit/cost ratio of YOU writing tests for your deps. I'd rather just avoid the dep all together and stay light.

1😂
futuro 2021-05-25T16:09:02.103300Z

I think that's a mostly accurate read on the situation, and I'd also like to stay light (I'm currently working on adjusting how I deploy the ion to keep the json handling code elsewhere), but I'm not certain it's always going to be possible.

Joe Lane 2021-05-25T16:10:53.103500Z

It sounds like the issue here for you is jsonista depending on jackson. Can you just use a different json library or the version of clojure.data.json that cloud depends on?

futuro 2021-05-25T16:11:20.103700Z

I'd say the more accurate cheeky response is "I'm adding a new piece of equipment to my lightweight airplane and it needs to be mounted on the left side of the body. I bought these nuts from the supplier at a different time than these bolts. I think they're the right thread size, material, and spec, but I'm not sure because I'm not actually working with nuts/bolts and it's more complicated than that."

1🙂
futuro 2021-05-25T16:12:09.104Z

Possibly yeah, though I need to a bit more research.

futuro 2021-05-25T16:13:02.104200Z

I believe there's a way forward for this specific instance (complicated a bit by using a polylith architecture, which I otherwise quite enjoy), and now I'm trying to understand the issue in a more general sense to navigate this should it come up again in the future.

futuro 2021-05-25T16:15:15.104400Z

Partially for my own benefit, and partially because I'm the Datomic Cloud evangelist/lead on my team and want to be able to properly represent the trade-offs/risks with dependency conflicts, and how it relates to Datomic On-Prem (which I don't particularly want to spend time setting up or operating when there's Datomic Cloud around).

Joe Lane 2021-05-25T16:15:19.104600Z

I think this issue (in the context of Datomic) is specific to jackson and the other handful of deps that cloud depends on that could conflict with user-space. I'm not sure you need to generalize, and I'm also not sure I can help give general advice 🙂

futuro 2021-05-25T16:16:09.104800Z

That's legit.

Joe Lane 2021-05-25T16:16:34.105Z

On-prem has the same situation though with the peer library. This is just normal java classpath dependency conflicts, we just happen to tell you that we overrode your deps in cloud (as opposed to spring boot or other jvm container solutions )

futuro 2021-05-25T16:16:54.105200Z

That is a very helpful piece of information, thank you.

futuro 2021-05-25T16:19:30.105500Z

It's possible the "solution" is to have the Cloud deps tree and assess whether those libraries are in our critical path or not. Whether that's something y'all are open to sharing, I have no idea. And it's also possible that the solution is to build smaller ions and, if push comes to shove, use ECS/EC2/etc to run whatever our critical-path-can't-override-deps code is and have the ions talk to it.

Joe Lane 2021-05-25T16:26:11.105700Z

> It's possible the "solution" is to have the Cloud deps tree and assess whether those libraries are in our critical path or not. A solution needs a problem. > Whether that's something y'all are open to sharing, I have no idea. Probably not. > And it's also possible that the solution is to build smaller ions and, if push comes to shove, use ECS/EC2/etc to run whatever our critical-path-can't-override-deps code is and have the ions talk to it. Again, what problem does "build smaller ions" solve? "Smaller" is a characteristic, I could have 10000 deps of 1kb in size. I highly doubt that the "optimal" solution will ever be don't run in ions because of the problem "my dep conflicts with cloud's dep" .

futuro 2021-05-25T16:30:13.105900Z

"Smaller" in terms of the deps tree. The "problem" being a dependency conflict that breaks a needed code path in a project dependency, which can't be resolved by downgrading or upgrading that particular dependency.

Joe Lane 2021-05-25T16:31:35.106100Z

Do you actually have that problem or are you hypothesizing? If you actually have that problem you should contact support and we can help figure out what to do 🙂

futuro 2021-05-25T16:34:30.106300Z

I have worked around the current manifestation of this problem, and no longer have that problem (could have sworn I said as much ;) ). If I hit that problem, though, I'll reach out to support 🙂.

futuro 2021-05-25T16:34:53.106500Z

Thanks for hashing this out with me @lanejo01 🙂

Joe Lane 2021-05-25T16:35:48.106700Z

Can't wait til this whole thing blows over and we can grab a pint @futuro, always fun chatting with you!

futuro 2021-05-25T16:37:00.106900Z

Omg, for real. 😅

kenny 2021-05-25T18:06:57.107100Z

We do not, sorry.