Ask questions on the official Q&A site at https://ask.datomic.com!
Leah Neukirchen 2021-07-01T06:47:18.346900Z

Yeah it needs a withEndpointConfiguration call to the builder... I'm not sure why they don't offer it, there are many other S3 providers. 😞

pedrorgirardi 2021-07-01T07:01:01.347100Z

That’s great @jacek.schae! I’m keen to check it out once it’s available.

rap1ds 2021-07-01T07:33:49.347600Z

I've understood that composite tuples can't be deregistered, is that right? I mean, if I add a composite tuple a+b, but later notice that I need to add a new tuple (or well, triple) a+b+c and the tuple a+b is becomes useless, Datomic still keeps updating the a+b tuple and it can't be deregistered. I assume there's some performance penalty to have unused composite tuples. Because of this, we have avoided using composite tuples if we have a doubt that we might need to change it in the future. Is that a valid reason to avoid them? Or am I overestimating the performance penalty of updating composite tuples?

favila 2021-07-01T11:18:18.349200Z

The performance penalty is a la carte—when you add a composite tuple it’s up to you to “touch” all entities that don’t have it yet to populate it

favila 2021-07-01T11:21:43.353800Z

The reason you cant change it probably just comes down to the inherent complexity of schema changes in a temporal db (what happens to the history view? To all old TX records?) and the philosophical stance Rich has against making the same name mean something different over time. His view: just use a new name and deprecate / remove the old one.

favila 2021-07-01T11:24:07.356400Z

Notice that no schema changes which change type are allowed—tuples are not unique in that way

favila 2021-07-01T11:28:17.359900Z

I think you may be thinking of composite tuples as a pure ephemeral projection of "real” data like an index in a relational db. That’s not really how it’s implemented in datomic—it’s more like actual data that the transactor automatically updates when it notices it’s components change

favila 2021-07-01T11:29:13.361400Z

It doesn’t eagerly project it, it can’t repopulate it for you, and the data is in the same covering indexes as all other data

rap1ds 2021-07-01T11:29:59.362200Z

Thanks for the answer! But I'm still wondering, isn't there performance penalty in the "just use a new and and deprecate the old" strategy? I mean, if I have attributes :a, :b and :c, and a composite tuple a+b, which I then decide to deprecate in favor of a new composite tuple a+b+c, then whenever I'm changing the attribute :a or :b, Datomic will update the composite tuple a+b, even though it's deprecated.

rap1ds 2021-07-01T11:31:05.362600Z

> I think you may be thinking of composite tuples as a pure ephemeral projection of "real” data like an index in a relational db. That’s not really how it’s implemented in datomic—it’s more like actual data that the transactor automatically updates when it notices it’s components change Right... so it's not really a performance penalty, but penalty in storage?

favila 2021-07-01T11:34:57.363Z


favila 2021-07-01T11:35:30.364100Z

Which you can mitigate by eg adding noHistory to the attr and removing any value indexes if you have it

favila 2021-07-01T11:36:21.365100Z

If you really want it gone you need to create new component attrs also

Boyan 2021-07-01T12:45:35.365600Z

Would you say then @favila that the storage cost of old unnecessary composite tuples shouldn't really be much of a factor in deciding between composites vs the other types of tuples, when a composite would otherwise work?

favila 2021-07-01T12:47:54.365800Z

I would say that it’s rare that storage cost is a factor

favila 2021-07-01T12:48:36.366Z

I also wish you could “turn off” a composite tuple--i.e. signal to the transaction processor that it should stop updating it

favila 2021-07-01T12:50:29.366200Z

composite tuples do something no other tuple can do: they know the effective value of the db at the moment right before committing the transaction datoms, so they can update composites to their correct value within that transaction even if the contents of the tx-data was uncoordinated

Boyan 2021-07-01T12:51:07.366400Z

Got it! Yeah, I wish we could. Maybe that will come as a feature one day. It sounds like the type of schema change that could be allowed.

favila 2021-07-01T12:51:59.366700Z

you can use a “normal” tuple and update it yourself, but you will have to be careful that you only prepare transaction data where you know what the final value will be when the tx-data arrives at the transactor, and that nothing else in the tx-data might alter that calculation.

favila 2021-07-01T12:52:14.366900Z

but if storage cost is a concern, that’s what you gotta do

favila 2021-07-01T12:52:50.367100Z

It’s not impossible--datomic didn’t have tuples of any kind for years. we were manually maintaining composite indexes as serialized strings

Boyan 2021-07-01T12:54:00.367300Z

Storage isn't really that big of a concern in our case, I think. It was more like the bad aftertaste of having unused and unnecessary attributes getting asserted perpetually.

favila 2021-07-01T12:55:16.367700Z

datomic doesn’t let you remove the cognitive burden of past mistakes. I think that’s the unspoken downside to the “no breaking changes” mantra

Boyan 2021-07-01T12:56:22.369Z

Yeah, though there's a difference between old/deprecated attributes in the schema and having values for them asserted on entities.


my understanding is that the lambda produced when deploying an Ion is really just a proxy to code running on the compute or query group. does that mean that memory allocated to the lambda via the lambda configuration is less consequential than a typical lambda?


very interesting, thanks again

favila 2021-07-01T13:01:11.370600Z

because of the history db and tx log, there’s always an assertion somewhere

Joe Lane 2021-07-01T16:13:18.372600Z



thanks Joe! does the code that is proxied to run in its own memory space? in other words, if my long running http-direct process has some state, say via mount, then there's no reason to expect that the proxied-to function can access that state, right?


(we tested this for fun and ruled it out, but i thought i'd ask anyway)

luiseugenio 2021-07-01T20:53:32.374200Z

Hi. Is there a Datomic Connector (Source and Sink) for Kafka?

refset 2021-07-02T19:31:34.388800Z

Hi 🙂 in lieu of a more relevant response from someone else, you may be able to borrow and adapt some code from Crux https://github.com/juxt/crux/tree/master/crux-kafka-connect

luiseugenio 2021-07-03T09:18:51.389100Z

Hi, that’s the plan 🙂 (if “someone else” doesn’t show up) haha

refset 2021-07-04T14:04:07.389400Z

Awesome 😄