datomic

Ask questions on the official Q&A site at https://ask.datomic.com!
armed 2020-09-29T06:29:34.069800Z

Hi. Is there any way to make custom transaction function to omit operaton (e.g. return nil instead of transaction operation)? I want omit insertion of data in some situations.

armed 2020-09-29T06:36:01.070100Z

I have permission entity with composite tuple (unique) on all three attributes. When I try to bulk insert list of permissions transation sometimes aborts with unique exception. I want to make something like postgres's on conflict do nothing. Here is my transaction function, which not workig obviously.

(defn try-add-permission
   [db {:keys [permission/app
               permission/user
               permission/role] :as perm}]
   (if (d/q '[:find ?p .
              :in $ ?app ?user ?role
              :where
              [?p :permission/app ?app]
              [?p :permission/user ?user]
              [?p :permission/role ?role]]
            db app user role)
     nil
     perm))

tatut 2020-09-29T07:06:28.070800Z

how about returning empty vector instead of nil?

armed 2020-09-29T07:18:12.071Z

I already tried that. Got error

{:status :failed,
         :val #error{:cause "Cannot write auth.server.cas_sync$try_add_permission@723b2a26 as tag null",
                     :via [{:type java.util.concurrent.ExecutionException,
                            :message "java.lang.IllegalArgumentException: Cannot write auth.server.cas_sync$try_add_permission@723b2a26 as tag null",
                            :at [datomic.promise$throw_executionexception_if_throwable invokeStatic "promise.clj" 10]}
                           {:type java.lang.IllegalArgumentException,
                            :message "Cannot write auth.server.cas_sync$try_add_permission@723b2a26 as tag null",
                            :at [org.fressian.handlers.WriteHandlerLookup
                                 requireWriteHandler
                                 "WriteHandlerLookup.java"
                                 48]}],
...

armed 2020-09-29T07:21:46.071700Z

@(d/transact (db/get-connection) [[auth.server.cas-sync/try-add-permission perm]])

tatut 2020-09-29T07:32:10.071900Z

you need to quote db fn name

armed 2020-09-29T07:35:36.072100Z

quoting does not help. And docs does not use quoting https://docs.datomic.com/on-prem/database-functions.html#using-transaction-functions

tatut 2020-09-29T07:36:46.072300Z

ah, it’s on-prem, don’t know about that

favila 2020-09-29T08:28:05.074100Z

This is a quoting issue. The exception is related to serializing the function, which you can’t do. Your function is not being executed yet

favila 2020-09-29T08:31:41.075600Z

In fact your transaction data hasn’t left the peer. What is the error you get when you quote the function name?

armed 2020-09-29T09:25:43.075800Z

when I quote like this:

armed 2020-09-29T09:25:46.076Z

@(d/transact
   (db/get-connection) [['auth.server.cas-sync/try-add-permission perm]])

armed 2020-09-29T09:26:14.076300Z

I get error: Could not locate auth/server/cas_sync__init.class, auth/server/cas_sync.clj or auth/server/cas_sync.cljc on classpath. Please check that namespaces with dashes use underscores in the Clojure file name.

favila 2020-09-29T09:32:27.076500Z

is this function installed on your DB?

favila 2020-09-29T09:33:05.076700Z

you’ll note in your link you need to make classpath transaction functions available on the classpath of the transactor. This error looks like it can’t find the function

1👍
favila 2020-09-29T09:35:06.077Z

actually it can’t even find the namespace

armed 2020-09-29T10:44:39.079400Z

@favila thanks, It seems that I misunderstood how transaction functions work.

tatut 2020-09-29T07:06:28.070800Z

how about returning empty vector instead of nil?

armed 2020-09-29T07:18:12.071Z

I already tried that. Got error

{:status :failed,
         :val #error{:cause "Cannot write auth.server.cas_sync$try_add_permission@723b2a26 as tag null",
                     :via [{:type java.util.concurrent.ExecutionException,
                            :message "java.lang.IllegalArgumentException: Cannot write auth.server.cas_sync$try_add_permission@723b2a26 as tag null",
                            :at [datomic.promise$throw_executionexception_if_throwable invokeStatic "promise.clj" 10]}
                           {:type java.lang.IllegalArgumentException,
                            :message "Cannot write auth.server.cas_sync$try_add_permission@723b2a26 as tag null",
                            :at [org.fressian.handlers.WriteHandlerLookup
                                 requireWriteHandler
                                 "WriteHandlerLookup.java"
                                 48]}],
...

armed 2020-09-29T07:21:46.071700Z

@(d/transact (db/get-connection) [[auth.server.cas-sync/try-add-permission perm]])

tatut 2020-09-29T07:32:10.071900Z

you need to quote db fn name

armed 2020-09-29T07:35:36.072100Z

quoting does not help. And docs does not use quoting https://docs.datomic.com/on-prem/database-functions.html#using-transaction-functions

tatut 2020-09-29T07:36:46.072300Z

ah, it’s on-prem, don’t know about that

favila 2020-09-29T08:28:05.074100Z

This is a quoting issue. The exception is related to serializing the function, which you can’t do. Your function is not being executed yet

favila 2020-09-29T08:31:41.075600Z

In fact your transaction data hasn’t left the peer. What is the error you get when you quote the function name?

armed 2020-09-29T09:25:43.075800Z

when I quote like this:

armed 2020-09-29T09:25:46.076Z

@(d/transact
   (db/get-connection) [['auth.server.cas-sync/try-add-permission perm]])

armed 2020-09-29T09:26:14.076300Z

I get error: Could not locate auth/server/cas_sync__init.class, auth/server/cas_sync.clj or auth/server/cas_sync.cljc on classpath. Please check that namespaces with dashes use underscores in the Clojure file name.

favila 2020-09-29T09:32:27.076500Z

is this function installed on your DB?

favila 2020-09-29T09:33:05.076700Z

you’ll note in your link you need to make classpath transaction functions available on the classpath of the transactor. This error looks like it can’t find the function

1👍
favila 2020-09-29T09:35:06.077Z

actually it can’t even find the namespace

tatut 2020-09-29T09:50:43.078400Z

Updating datomic cloud compute group to 2020/09/23 715-8973 release, the log shows that the compute nodes don’t seem to get up after upgrade… complaining that our application code has syntax error (which it shouldn’t as it worked in previous version)

onetom 2020-10-01T07:53:26.151100Z

has this been solved yet?

tatut 2020-10-01T11:33:49.156800Z

yes, workaround with support… it seemd you can’t have paths that point to for example `“../common/src” (like we have sharing backend and frontend cljc code)

tatut 2020-10-01T11:34:02.157Z

it worked in all previous versions, but it doesn’t anymore with the latest

tatut 2020-10-01T11:34:21.157200Z

workaround with symlinks seems ok

tatut 2020-09-29T09:51:54.078500Z

production topology

tatut 2020-09-29T09:53:40.078700Z

"Msg": ":datomic.cloud.cluster-node/-main failed: Syntax error compiling at …

tatut 2020-09-29T09:54:11.079100Z

it doesn’t seem to find a required .cljc file

armed 2020-09-29T10:44:39.079400Z

@favila thanks, It seems that I misunderstood how transaction functions work.

onetom 2020-09-29T11:49:10.089100Z

When I connect a web ion via an API Gateway proxied thru a lambda, my ion function is supposed to receive a ring-compatible map as an argument (according to the official Ion docs). However, the map I receive only contains :headers, :server-name and a :datomic.ion.edn.api-gateway/data and /json keys, so I can't just use the typical routing libs to build my web-app or http API, because those depend on the :request-method and :uri keys of the request map. Is it a know issue? Is it something related to the Lambda proxy data format version? Is it just some kind of mis-configuration?

xceno 2020-10-07T10:28:39.290800Z

Hi guys, just found this thread because I'm working on the exact same thing right now (trying to deploy an SPA as ion / lambda proxy) I got confused by the mismatch between the Ion tutorial and the API Gateway console, so just to clarify once more: What the Datomic Ion docs are talking about is now called REST API on AWS? And the HTTP API is a new thing, that is not officially supported?

jeff.terrell 2020-10-07T13:41:44.294100Z

Yes, REST is the old kind that the docs implicitly refer to. HTTP can work, but it's not what the docs describe specifically.

xceno 2020-10-07T14:00:30.299400Z

Understood, thank you!

onetom 2020-09-29T11:49:54.090100Z

here is an example request map I observed:

{:headers
                  {"accept-encoding"   "gzip, deflate",
                   "content-length"    "0",
                   "host"              "<http://8g759uq7nb.execute-api.ap-southeast-1.amazonaws.com|8g759uq7nb.execute-api.ap-southeast-1.amazonaws.com>",
                   "user-agent"        "http-kit/2.0",
                   "x-amzn-trace-id"   "Root=1-5f730b24-4ac64db84deabaf53c38af60",
                   "x-forwarded-for"   "42.200.88.157",
                   "x-forwarded-port"  "443",
                   "x-forwarded-proto" "https"},
     :server-name "<http://8g759uq7nb.execute-api.ap-southeast-1.amazonaws.com|8g759uq7nb.execute-api.ap-southeast-1.amazonaws.com>",
     :datomic.ion.edn.api-gateway/json
                  "{\"version\":\"2.0\",\"routeKey\":\"$default\",\"rawPath\":\"/\",\"rawQueryString\":\"\",\"headers\":{\"accept-encoding\":\"gzip, deflate\",\"content-length\":\"0\",\"host\":\"<http://8g759uq7nb.execute-api.ap-southeast-1.amazonaws.com|8g759uq7nb.execute-api.ap-southeast-1.amazonaws.com>\",\"user-agent\":\"http-kit/2.0\",\"x-amzn-trace-id\":\"Root=1-5f730b24-4ac64db84deabaf53c38af60\",\"x-forwarded-for\":\"42.200.88.157\",\"x-forwarded-port\":\"443\",\"x-forwarded-proto\":\"https\"},\"requestContext\":{\"accountId\":\"191560372108\",\"apiId\":\"8g759uq7nb\",\"domainName\":\"<http://8g759uq7nb.execute-api.ap-southeast-1.amazonaws.com|8g759uq7nb.execute-api.ap-southeast-1.amazonaws.com>\",\"domainPrefix\":\"8g759uq7nb\",\"http\":{\"method\":\"GET\",\"path\":\"/\",\"protocol\":\"HTTP/1.1\",\"sourceIp\":\"42.200.88.157\",\"userAgent\":\"http-kit/2.0\"},\"requestId\":\"Tn6trha8yQ0EMGg=\",\"routeKey\":\"$default\",\"stage\":\"$default\",\"time\":\"29/Sep/2020:10:23:32 +0000\",\"timeEpoch\":1601375012276},\"isBase64Encoded\":false}",
     :datomic.ion.edn.api-gateway/data
                  {:version         "2.0",
                   :routeKey        "$default",
                   :rawPath         "/",
                   :rawQueryString  "",
                   :headers
                                    {:accept-encoding   "gzip, deflate",
                                     :content-length    "0",
                                     :host              "<http://8g759uq7nb.execute-api.ap-southeast-1.amazonaws.com|8g759uq7nb.execute-api.ap-southeast-1.amazonaws.com>",
                                     :user-agent        "http-kit/2.0",
                                     :x-amzn-trace-id   "Root=1-5f730b24-4ac64db84deabaf53c38af60",
                                     :x-forwarded-for   "42.200.88.157",
                                     :x-forwarded-port  "443",
                                     :x-forwarded-proto "https"},
                   :requestContext
                                    {:routeKey     "$default",
                                     :stage        "$default",
                                     :time         "29/Sep/2020:10:23:32 +0000",
                                     :domainPrefix "8g759uq7nb",
                                     :requestId    "Tn6trha8yQ0EMGg=",
                                     :domainName   "<http://8g759uq7nb.execute-api.ap-southeast-1.amazonaws.com|8g759uq7nb.execute-api.ap-southeast-1.amazonaws.com>",
                                     :http
                                                   {:method    "GET",
                                                    :path      "/datomic",
                                                    :protocol  "HTTP/1.1",
                                                    :sourceIp  "42.200.88.157",
                                                    :userAgent "http-kit/2.0"},
                                     :accountId    "191560372108",
                                     :apiId        "8g759uq7nb",
                                     :timeEpoch    1601375012276},
                   :isBase64Encoded false},
     }

onetom 2020-09-29T11:52:07.091100Z

my ion-config.edn looks like this:

{:allow    [datomic.ion.starter.http/ionized-app]
 :lambdas  {:app
            {:fn          datomic.ion.starter.http/ionized-app
             :integration :api-gateway/proxy
             :description "return html app"}}
 ;:http-direct {:handler-fn datomic.ion.starter.http/return-something-json}
 :app-name "kyt-dev"}

onetom 2020-09-29T11:55:54.093400Z

I'm using the Solo topology (the version, which was the latest last week), otherwise I wouldn't bother with lamdba gateways if I could use the production topology.

onetom 2020-09-29T11:57:59.095200Z

the docs are mentioning these /json and /data keys in a note, but just in the table above the note, they are not namespaced: https://docs.datomic.com/cloud/ions/ions-reference.html#web-ion

jaret 2020-09-29T12:15:36.097100Z

Hi @steveb8n, have you actually seen a node go OOM or are you just noticing this in your metrics/dashboard? This small snippet matches with the expectations I have for indexing. The indexing job occurs in the background. Indexing is done in memory and then the in-memory index is merged with the persistent index and a new persistent index is written to the storage service. If you widen the time scale you should see a saw tooth pattern on your indexing line.

Joe Lane 2020-09-29T12:43:43.098100Z

Hey @onetom , have a look at https://github.com/pedestal/pedestal.ions And https://github.com/pedestal/pedestal-ions-sample

onetom 2020-09-29T15:27:04.103900Z

thanks, I had a look, but I don't see how would it deal with my situation. it does have a great example of a protocol which converts the response body into an input stream, which I still need, because the reitit.ring/create-resource-handler just returns a java.io.File as a :body and Datomic threw some ->>bbuff conversion error as a result. for now, I just have a middleware to transform the above mentioned request map to be ring compatible:

(if-let [gw-data (:datomic.ion.edn.api-gateway/data gw-req)]
       (-&gt; gw-req
           (assoc :uri (-&gt; gw-data :requestContext :http :path))
           (assoc :request-method (-&gt; gw-data :requestContext :http :method)))
       gw-req)

Petrus Theron 2020-09-29T15:29:32.106700Z

Just gotten bitten for an hour getting 401 Unauthorized for Datomic Pro due to missing XML schema in ~/.m2/settings.xml, which is not mentioned in https://my.datomic.com/account. Previously: https://clojurians-log.clojureverse.org/datomic/2019-01-30/1548890962.888000

2020-10-09T20:42:33.423500Z

I'm now debugging this 401 issue in my own case, where Datomic fails to download consistently on Github Actions for CI purposes but not locally on my machine.

Joe Lane 2020-09-29T15:33:30.108200Z

I want to make sure I understand, did you call apigw/ionize on your ring handler function? https://docs.datomic.com/cloud/ions/ions-tutorial.html#lambda-proxy

onetom 2020-09-29T15:34:06.108500Z

I have the suspicion that or ion-config.edn doesn't need the :integration :api-gateway/proxy option anymore if I use the newer style HTTP API gateway setup (as opposed to the RESTful API style), it's just hasn't been documented...

jaret 2020-09-29T15:34:32.109300Z

Sorry about that, what is missing in https://my.datomic.com/account ? I see the .m2/settings.xml described as:

;; In ~/.m2/settings.xml:
  &lt;!-- ~/.m2/settings.xml (see the Maven server settings docs) --&gt;
  &lt;servers&gt;
    …
    &lt;server&gt;
      &lt;id&gt;<http://my.datomic.com|my.datomic.com>&lt;/id&gt;
      &lt;username&gt;REDACTED&lt;/username&gt;
      &lt;password&gt;REDACTED&lt;/password&gt;
    &lt;/server&gt;
    …
  &lt;/servers&gt;

;; In deps.edn:
{:mvn/repos
 {"<http://my.datomic.com|my.datomic.com>" {:url "<https://my.datomic.com/repo>"}}
 :deps
 {com.datomic/datomic-pro {:mvn/version "${VERSION}"}}}

Petrus Theron 2020-09-29T15:34:45.109500Z

&lt;?xml version="1.0" encoding="UTF-8"?&gt;
&lt;settings xmlns=""
          xmlns:xsi=""
          ssi:schemaLocation=""&gt;

jaret 2020-09-29T15:35:36.110300Z

ack! let me talk to Alex so i fully understand and I can update our http://my.datomic.com account to reflect that!

1👍
onetom 2020-09-29T15:37:12.111300Z

@lanejo01 yes, that .../ionized-app is defined as (apigw/ionize app), where app is simply:

(fn [req]
      {:status  200
       :headers {"content-type" "text/plain"}
       :body    (with-out-str
                  (clojure.pprint/pprint req))})

onetom 2020-09-29T15:37:41.112200Z

that's how I obtained the request map I showed above

Joe Lane 2020-09-29T15:38:37.112800Z

To make sure I understand correctly, you're not using the supported integration. Is there still a problem if you use the supported one?

onetom 2020-09-29T15:39:04.113400Z

what do you mean by supported integration?

Petrus Theron 2020-09-29T15:39:04.113600Z

Also - and this is probably out of scope - but even after fixing settings.xml (and running clj from terminal), IntelliJ Cursive still reported a 401 for deps because it caches the 401 error for a given deps.edn (not sure if this is due to Maven or IntelliJ). Fixed after reordering any two items under :deps, then I could start a REPL.

onetom 2020-09-29T15:41:55.117300Z

I'm just realizing that probably the Datomic docs are taking about how to integrate a web ion with the traditional RESTful API gateway, not the new "HTTP API". I'm using this "new style" gateway, because it supports JWT authorizer's out of the box, without the need to deploy a lamdba function just for that purpose.

onetom 2020-09-29T15:43:29.118900Z

yes, the mentioned request map was observed when my ion-config.edn contained that : integration :api-gateway/proxy setting

Joe Lane 2020-09-29T15:48:48.120500Z

I'm not sure that approach is compatible with the current mechanism for making a ring compatible handler. I think you're in uncharted territory, rife with undefined behavior.

onetom 2020-09-29T15:49:54.121100Z

Probably... Thanks looking into it!

onetom 2020-09-29T15:51:38.123400Z

I will probably just transition to the production topology, though I can foresee issues with the VPC link and NLB in that case, which I have just as little experience with as I have with Cognito and JWT authorizers :)

Joe Lane 2020-09-29T15:52:22.123900Z

You will get a raw payload if you use http-direct

Joe Lane 2020-09-29T15:52:32.124300Z

it will not be a nice map like above.

onetom 2020-09-29T15:53:16.125200Z

Alternatively, I can write a replacement ionizer function for this new "HTTP API" gateway

Joe Lane 2020-09-29T15:54:08.126400Z

You're free to do that, but we won't be able to support that if there are issues.

1👍
Michael J Dorian 2020-09-29T18:52:25.130400Z

Hey! I have a transaction that always puts one entry into datomic, and I'd like to get it to return the entity id of the new entry. I notice that the returned map contains :tx-data, which has the data I need. But I'm not sure how to read the contents of the returned datum, and, indeed, if this is considered bad best practices or not. Help appreciated!

ghadi 2020-09-29T18:54:12.130900Z

the returned map also contains :tempids which is a map of tempid -> entity id

ghadi 2020-09-29T18:54:17.131200Z

@doby162

Michael J Dorian 2020-09-29T18:55:18.131900Z

I'm getting an empty map on that one, do I just need to add a temp-id to the transaction?

ghadi 2020-09-29T18:57:59.132300Z

paste your code/input

ghadi 2020-09-29T18:58:31.133100Z

if your transaction included tempids, datomic returns the resolved ids after it transacts

Michael J Dorian 2020-09-29T19:01:34.133900Z

{:tx-data [#:user{:email "e", :password "q", :name "q", :token "q"}]} ; this query is generated by (make-record :user) and executed
(def q (make-record :user "q" "q" "q" "q"))
(:tempids q)

Michael J Dorian 2020-09-29T19:06:38.135100Z

Ah, ok! Just had to add :db/id "nonsense" to my query and now the map gives me {"nonsense" id} !

Michael J Dorian 2020-09-29T19:06:43.135300Z

thanks!

ghadi 2020-09-29T19:09:27.135900Z

if :user/email is a unique attribute in your database, you can use it to lookup entities without entity ids

Michael J Dorian 2020-09-29T19:11:00.137100Z

Oh, nice

steveb8n 2020-09-29T22:20:26.137300Z

@jaret No I haven’t yet in prod but the same code running on Solo (test system) has gone OOM. That chart is 2 weeks, hence no saw tooth. Here’s the hour just gone. Saw tooth as expected

steveb8n 2020-09-29T22:21:40.137700Z

Interesting that you think this is normal. Is there some doc somewhere that describes what “normal” is for charts in the dashboard? That would help me (and others I suspect)

steveb8n 2020-09-29T22:22:23.137900Z

Whenever I deploy new code, the FreeMem line jumps back up to 10Mb and starts the slow decline