Clojurians Log v2

Is it a known bug that when there's a bunch of datums that get transacted simultaneously, it can randomly cause a :db.error/tempid-not-an-entity tempid '17503138' used only as value in transaction error?

It is often caused by one single entry that is the same structure as many others. Everything is fine, but for some reason, Datomic doesn't like it. Removing that one entry solves the problem.

Ordering of the entries in the transaction vector doesn't seem to matter either

The two datums causing problems:

[{:account/photo
   "REDACTED",
   :account/first-name "REDACTED",
   :account/bio
   "REDACTED",
   :account/email-verified? false,
   :account/location 2643743,
   :account/vendor-skills [17592186045491],
   :account/id #uuid "dd33747e-5c13-4779-8c23-9042460eb3f3",
   :account/vendor-industry-experiences [],
   :account/languages [17592186045618 17592186045620],
   :account/vendor-specialism 17592186045640,
   :account/links
   [{:db/id "REDACTED",
     :link/id #uuid "ea51184c-d027-44d0-8f20-df222e58daf3",
     :link/type :link-type/twitter,
     :link/url "REDACTED"}
    {:db/id
     "REDACTED",
     :link/id #uuid "c9577ca4-332d-41f0-b617-c00e89fc94b4",
     :link/type :link-type/linkedin,
     :link/url
     "REDACTED"}],
   :account/last-name "REDACTED",
   :account/email "REDACTED",
   :account/vendor-geo-expertises
   [17592186045655 17592186045740 17592186045648],
   :db/id "17503138",
   :account/vendor-type 17592186045484,
   :account/roles [:account.role/vendor-admin],
   :account/job-title "Investor"}]

[{:account/primary-account "17503138",
   :company/headline "REDACTED",
   :account/accounts ["17503138"],
   :tenant/tenants [[:tenant/name "REDACTED"]],
   :company/name "REDACTED",
   :company/types [:company.type/contact],
   :db/id "REDACTED",
   :company/id #uuid "ee26b11f-53ba-43f9-a59b-f7ad1a408d41",
   :company/domain "REDACTED"}]

The meaning of this error is that the string “17503138” is used as a tempid that is the value of an assertion, but there is no place where the tempid is used as the entityid of an assertion; the latter is necessary for datomic to decide whether to mint a new entity id or resolve it to an existing one

Well, as you can see in the actual datums I posted, it clearly is being used as :db/id.

I had my program dump all datums into a file before transacting, and I copied the two that refer to this string over into here

In your example, I see the second item says :account/accounts “17503138”. Are both these maps together in the same transaction?

(Btw a map is not a datum but syntax sugar for many assertions—it’s a bit confusing to call it that)

Yes, they are both together in the same transaction. True, I mixed up the terminology... Entity would be more fitting

If they are indeed both in the same tx I would call that a bug. Can you reproduce?

Why is each map in its own vector?

Yes, reliably, every time with the same dataset. Both locally with a dev database as well as on our staging server using PostgreSQL.

Conformity wants it that way, for some reason

Conformity for data?

I had that same issue a while back in a normal transaction without conformity as well though

Separate vectors in conformity means separate transactions...

The migration library called conformity

I’ve only ever used conformity for schema migrations; using it for data seems novel; but I’m suspicious that these are really not in the same transaction

See if you can get it to dump the full transaction that fails and make sure both maps mentioning that tempid are in the same transaction

During a meetup recording that I haven't uploaded yet I recorded my own maven private token from https://cognitect.com/dev-tools/view-creds.html is there a way I can regenerate that token?

Hey, I just missing something and can't figure out what. I am calling tx on datomic:

(defn add-source [conn {:keys [id name]
                        :or {id (d/squuid)}}]
  (let [tx {;; Source initial state
            :db/id                    (d/tempid :db.part/user)
            :source/id                id
            :source/storage-type      :source.storage-type/disk
            :source/job-status        :source.job-status/dispatched
            :source/created           (java.util.Date.)
            :source/name              name}]
    @(d/transact conn [tx])))

;; and then later API will call
(add-source conn entity-data)

tempids resolve to existing entities if you assert a :db.unique/identity attribute value on them that already exists. Are any of these attributes :db.unique/identity? are you sure you are not supplying an id argument to your function?

(btw I would separate transaction data creation into a separate function so it’s easier to inspect)

{:db/doc                "Source ID"
           :db/ident              :source/id
           :db/valueType          :db.type/uuid
           :db/cardinality        :db.cardinality/one
           :db/id                 #db/id [:db.part/db]
           :db.install/_attribute :db.part/db}

this is schema for source/id, I am not using :db.unique/identity

And I agree with separation tx and creation but first I would like to get it work

Id I removed :db/id from transaction, I shoud still be able to create new entity, right? But everytime first one is rewritten

can you give a clearer get/expect case? maybe a repl console?

something that shows you calling add-source twice with the returned tx data, and pointing out what you think is wrong with the result of the second call?

Ok I had unique on other parameter:

{:db/doc "Source name"
 :db/ident :source/name
 :db/unique :db.unique/identity
 :db/valueType :db.type/string
 :db/cardinality :db.cardinality/one
 :db/id #db/id [:db.part/db]
 :db.install/_attribute :db.part/db}

Can you send an email to <mailto:support@cognitect.com|support@cognitect.com> and we will help with this?

Well, I guess I am going to do my migrations using a home-made solution now. I just lost all trust in Conformity. It doesn't write anything to the database most of the time I noticed.

Or are there alternatives?

can you describe your problem with conformity in more detail?

heya, coming here for a question about datomic cloud. I've noticed that while developing on a repl, I get exceptions as described in the datomic.api.client api:

All errors are reported via ex-info exceptions, with map contents
as specified by cognitect.anomalies.
See <https://github.com/cognitect-labs/anomalies>.

{:cognitect.anomalies/category :cognitect.anomalies/incorrect, :cognitect.anomalies/message \"Query is referencing unbound variables: #{?string}\", :variables #{?string}, :db/error :db.error/unbound-query-variables, :dbs [{:database-id \"48e8dd4d-84bb-4216-a9d7-4b4d17867050\", :t 97901, :next-t 97902, :history false}]}

@filipematossilva are you using the same API (sync or async) in both cases?

think so, yeah

have a ion handling http requests directly, and the repl is calling the handler that's registered on the ion

so it should be the same code running

we can see on the aws logs that the error is of a different shape

let me dig it up

on the aws logs, logging the exception, shows this

{
    "Msg": "Alpha API Failed",
    "Ex": {
        "Via": [
            {
                "Type": "com.google.common.util.concurrent.UncheckedExecutionException",
                "Message": "clojure.lang.ExceptionInfo: :db.error/not-a-binding-form Invalid binding form: :entity/graph {:cognitect.anomalies/category :cognitect.anomalies/incorrect, :cognitect.anomalies/message \"Invalid binding form: :entity/graph\", :db/error :db.error/not-a-binding-form}",
                "At": [
                    "com.google.common.cache.LocalCache$Segment",
                    "get",
                    "LocalCache.java",
                    2051
                ]
            },
            {
                "Type": "clojure.lang.ExceptionInfo",
                "Message": ":db.error/not-a-binding-form Invalid binding form: :entity/graph",
                "Data": {
                    "CognitectAnomaliesCategory": "CognitectAnomaliesIncorrect",
                    "CognitectAnomaliesMessage": "Invalid binding form: :entity/graph",
                    "DbError": "DbErrorNotABindingForm"
                },
                "At": [
                    "datomic.core.error$raise",
                    "invokeStatic",
                    "error.clj",
                    55
                ]
            }
        ],

(note: this was not the same unbound var query as above)

printing the error on the repl, we see this instead

#error {
                                      :cause "Invalid binding form: :entity/graph"
                                      :data {:cognitect.anomalies/category :cognitect.anomalies/incorrect, :cognitect.anomalies/message "Invalid binding form: :entity/graph", :db/error :db.error/not-a-binding-form, :dbs [{:database-id "48e8dd4d-84bb-4216-a9d7-4b4d17867050", :t 97058, :next-t 97059, :history false}]}
                                      :via
                                      [{:type clojure.lang.ExceptionInfo
                                        :message "Invalid binding form: :entity/graph"
                                        :data {:cognitect.anomalies/category :cognitect.anomalies/incorrect, :cognitect.anomalies/message "Invalid binding form: :entity/graph", :db/error :db.error/not-a-binding-form, :dbs [{:database-id "48e8dd4d-84bb-4216-a9d7-4b4d17867050", :t 97058, :next-t 97059, :history false}]}
                                        :at [datomic.client.api.async$ares invokeStatic "async.clj" 58]}]

that ^ is an anomaly

which is a data map

more precisely, (ex-data e) returns the anomaly inside that exception

ah, instead of ex-info ?

I imagine the datomic client wraps the exception doing something like (ex-info e anomaly cause)

we're not wrapping it on our end, just calling ex-data over it to get the anomaly

but on the live system, ex-data over the exception returns nil

which I think means it wasn't created with ex-info

I mean, I wouldn't be surprised if this is indeed intended to not leak information on the live system

that anomaly contains database ids, time info, and history info

I have a migration that is in a function. Conformity runs the function normally, but instead of transacting the data returned from it, it just discards it. The data is definitely valid; I made my migration so it also dumps the data into a file. I can load that file as EDN and transact it to the db using d/transact perfectly fine.

just wanted to make sure if it was intended or not before working around it

Conformity doesn't even give an error, it just silently discards it.

is this cloud or on prem?

@filipematossilva are you saying that you are not able to get a :cognitect.anomalies/incorrect from your failing query on the client side?

On prem, both for the dev backend and the postgresql one

not sure what to tell you. you need to analyze this further before throwing up your hands

if by client side you mean "what calls the live datomic cloud system", then yes, that's it

@filipematossilva so what's different about your "live system" vs. the repl?

clearly it's an ex-info at the repl

I really don't know, that's what prompted this question

perhaps print (class e) and (supers e) in your live system when you get the error

or (Throwable->map e)

sync api or async api?

sync

Conformity does bookkeeping to decide whether a “conform” was already run on that database. If you’re running the same key name against the same database a second time, it won’t run again. Is that what you are doing?

Conformity is really for schema management, not data imports

regarding printing the error

I'm printing the exception proper like this:

(cast/alert {:msg "Alpha API Failed"
                   :ex  e})

do you have wrappers/helpers around your query? running it in a future?

on the live system the cast prints this

https://clojurians.slack.com/archives/C03RZMDSH/p1602169996347800

oh, yeah that's a com.google.common.util.concurrent.UncheckedExecutionException at the outermost layer

then the inner exception is an ex-info

on the repl, when cast is redirected to stderr, the datomic binary shows this

thanks. @marshall ^

No, that is not what I am doing.

Well, the transaction is changing the schema, and then transforming the data that is in there.

Or at least, that is what it is supposed to be doing.

https://github.com/avescodes/conformity#norms-versioning

https://github.com/avescodes/conformity#norms-versioning

jinx

jinx

We’re pointing out a case where it may evaluate the function but not transact

you can use conforms-to? to test whether conformity thinks the db already has the norm you are trying to transact

that may help you debug

just realized that the logged response there on the live system wasn't complete, let me fetch the full thing

ok this is the full casted thing on aws logs

understood

Well, what is the second argument to conforms-to? ? It's neither the file name nor the output of c/read-resource

It wants a keyword, but what keyword?

now that I look at the full cast on life, I can definitely see the cause and data fields there

which leaves me extra confused 😐

let me clarify:

in your REPL, you are getting an exception that is: * clojure.lang.ExceptionInfo + anomaly data in your live system you are getting: * com.google.common.util.concurrent.UncheckedExecutionException * clojure.lang.ExceptionInfo + anomaly data

where the Ion has the ex-info as the cause (chained to the UEE)

make sense? seems like a bug @marshall

to work around temporarily, you can do (-> e ex-cause ex-data) to unwrap the outer layer

and access the data

I can see that via indeed shows different things, as you say

but the toplevel still shows data and cause for both situations

I imagine that data would be returned from ex-data

the keyword in the conform map

let me edit those code blocks to remove the trace, I think it's adding a lot of noise and not helping

{:name-of-norm {:txes [[…]] :requires […] :tx-fn …}}

the :name-of-norm part

that’s the “norm”

done

I think it's important to separate the exception object chain from the data that represents it (which may pull data from the root exception, not from the top exception)

Throwable->map for example pulls :cause, :data, :via from the root exception (deepest in the chain)

@alexmiller it's not clear to me what you mean by that in the current context

(besides the factual observation)

is it that you also think that the different behaviour between the repl+datomic binary and live system should be overcome by calling Throwable->map prior to extracting the data via ex-data?

root exception is the wrapped ex-info

you could do (-> e Throwable->map :data) to get at the :incorrect piece

I’m just saying that the data you’re seeing is consistent with what Ghadi is saying

Even though that may be confusing

ok I think I understand what you mean now

thank you for explaining

but the inconsistency is a bug 🙂

currently deploying your workaround, and testing

@filipematossilva this is in an Ion correct?

@marshall correct

in a handler-fn for http-direct

@ghadi I replaced my (ex-data e) with this fn

(defn error->error-data [e]
  ;; Workaround for a difference in the live datomic system where clojure exceptions
  ;; are wrapped in a com.google.common.util.concurrent.UncheckedExecutionException.
  ;; To get the ex-data on live, we must convert it to a map and access :data directly.
  (or (ex-data e)
      (-> e Throwable->map :data)))

I can confirm this gets me the anomaly for the live system

slightly different than on the repl still

live:

{:cognitect.anomalies/category :cognitect.anomalies/incorrect, :cognitect.anomalies/message "Invalid binding form: :entity/graph", :db/error :db.error/not-a-binding-form}

{:cognitect.anomalies/category :cognitect.anomalies/incorrect, :cognitect.anomalies/message \"Invalid binding form: :entity/graph\", :db/error :db.error/not-a-binding-form, :dbs [{:database-id \"48e8dd4d-84bb-4216-a9d7-4b4d17867050\", :t 97901, :next-t 97902, :history false}]}

which makes sense, because in the live exception the :dbs property just isn't there

but tbh that's the one that really shouldn't be exposed

so that's fine enough for me

thank you

is there are an official method to move data from dev-local to cloud?

the workaround is fine enough for me, but maybe you'd like more information about this?

nope, that’s enough thanks; we’ll investigate

Does anyone know how I get the t from tx (d/tx->t tx), but my tx is a map and the error in the conversion?

{:db-before datomic.db.Db@ad41827d, :db-after datomic.db.Db@d8231b67, :tx-data [#datom[13194139534369 50 #inst "2020-10-08T19:30:19.852-00:00" 13194139534369 true] #datom[277076930200610 169 #inst "2020-10-08T06:00:59.275-00:00" 13194139534369 true] #datom[277076930200610 163 17592186045452 13194139534369 true] #datom[277076930200610 165 277076930200584 13194139534369 true] #datom[277076930200610 170 17592186045454 13194139534369 true] #datom[277076930200610 162 277076930200581 13194139534369 true] #datom[277076930200610 167 #inst "2020-10-08T19:30:19.850-00:00" 13194139534369 true] #datom[277076930200610 168 17592186045432 13194139534369 true] #datom[277076930200610 166 #uuid "5f7f68cb-08f0-4cb2-964b-4e811a34a949" 13194139534369 true]], :tempids {-9223090561879066169 277076930200610}}

You need to grab the tx from a datom in :tx-data , in your case 13194139534369. I think something like (-> result :tx-data first :tx) will give you it

I think also (-> result :db-after :basisT) will give you your new t directly

thks

I’ve reproduced this behavior and will report it to the dev team

thank you, I've just sent an email over

Q: I want to store 3rd party oauth tokens in Datomic. Storing them as cleartext is not secure enough so I plan to use KMS to symmetrically encrypt them before storage. Has anyone done something like this before? If so, any advice? Or is there an alternative you would recommend?

One alternative I am considering is DynamoDB

how many oauth keys? how often they come in/change/expire?

I provide a multi-tenant SAAS so at least 1 set per tenant

Also looking at AWS Secrets Manager for this. Clearly I’m in the discovery phase 🙂

but appreciate any advice

interaction patterns within KMS are not supposed to be for encryption/decryption of fine granularity items

usually you generate key material known as a "DEK" (Data Encryption Key) using KMS

then you use the DEK to encrypt/decrypt a bunch of data

ok. I can see I’m going down the wrong path with Datomic for this data

that's not the conclusion for me

it looks like Secrets Manager with a local/client cache is the way to do

you talk to KMS when you want to encrypt/decrypt the DEK

so when you boot up, you ask KMS to decrypt the DEK, then you use the DEK to decrypt fine-grained things in the application

where to store it (Datomic / wherever) is orthogonal to how you manage keys

if you talk to KMS every time you want to decrypt a token, you'll pay a fortune and add a ton of latency

the oauth ciphertexts could very well be in datomic

if I am weighing pros/cons of DEK/Datomic vs Secrets Manager, what are the advantages of using Datomic?

secrets manager is for service level secrets

it seems like the same design i.e. cached DEK to read/write from Datomic

you could store your DEK in Secrets manager

the downside would be no excision c.f. Secrets Manager

you cannot put thousands of oauth tokens in secrets manager

excision is desirable for this kind of data

well, depending on how rich you are

I’m not rolling in money 🙂

if you need to excise, you can throw away a DEK

hmm. is 1 DEK per tenant practical?

I would google keystretching, HMAC, hierarchical keys

seems like same scale problem

you can have a root DEK, then create per tenant DEKs using HMAC

deteministically

ok. that’s an interesting idea. a mini DEK chain

tenantDEK = HMAC(rootDEK, tenantID)

then the root is stored in Secrets Manager

right

where would the tenant DEKs be stored?

need to store an identifier so that you can rorate the DEK periodically

you don't store the tenant DEKs

you derive them on the fly with HMAC

ok. I’ll start reading up on this. thank you!

sure. with HMAC you'll have to figure out a different excision scheme

you could throw away the ciphertext instead of the DEK

because you can't throw away the DEK (you can re-gen it!)

etc.

but yeah db storage isn't your issue :)

key mgmt is

interesting. that means Datomic is no good for this i.e. no excision

or am I missing a step?

are you using cloud or onprem?

cloud / prod topo

stay tuned

now that’s just not fair 🙂

I will indeed

how often does a tenant's 3p oauth token change?

It’s a Salesforce OAuth so the refresh period is configurable I believe. would need to check

i.e. enterprise SAAS is why good design matters here

I’ll need to build a v1 of this in the coming weeks

now that I think about it, I could deliver an interim solution without this for a couple of months and “stay tuned” for a better solution

I’ll hammock this…

🙏