Clojurians Log v2

Hi. Is there any way to make custom transaction function to omit operaton (e.g. return nil instead of transaction operation)? I want omit insertion of data in some situations.

I have permission entity with composite tuple (unique) on all three attributes. When I try to bulk insert list of permissions transation sometimes aborts with unique exception. I want to make something like postgres's on conflict do nothing. Here is my transaction function, which not workig obviously.

(defn try-add-permission
   [db {:keys [permission/app
               permission/user
               permission/role] :as perm}]
   (if (d/q '[:find ?p .
              :in $ ?app ?user ?role
              :where
              [?p :permission/app ?app]
              [?p :permission/user ?user]
              [?p :permission/role ?role]]
            db app user role)
     nil
     perm))

Updating datomic cloud compute group to 2020/09/23 715-8973 release, the log shows that the compute nodes don’t seem to get up after upgrade… complaining that our application code has syntax error (which it shouldn’t as it worked in previous version)

production topology

"Msg": ":datomic.cloud.cluster-node/-main failed: Syntax error compiling at …

it doesn’t seem to find a required .cljc file

When I connect a web ion via an API Gateway proxied thru a lambda, my ion function is supposed to receive a ring-compatible map as an argument (according to the official Ion docs). However, the map I receive only contains :headers, :server-name and a :datomic.ion.edn.api-gateway/data and /json keys, so I can't just use the typical routing libs to build my web-app or http API, because those depend on the :request-method and :uri keys of the request map. Is it a know issue? Is it something related to the Lambda proxy data format version? Is it just some kind of mis-configuration?

here is an example request map I observed:

{:headers
                  {"accept-encoding"   "gzip, deflate",
                   "content-length"    "0",
                   "host"              "<http://8g759uq7nb.execute-api.ap-southeast-1.amazonaws.com|8g759uq7nb.execute-api.ap-southeast-1.amazonaws.com>",
                   "user-agent"        "http-kit/2.0",
                   "x-amzn-trace-id"   "Root=1-5f730b24-4ac64db84deabaf53c38af60",
                   "x-forwarded-for"   "42.200.88.157",
                   "x-forwarded-port"  "443",
                   "x-forwarded-proto" "https"},
     :server-name "<http://8g759uq7nb.execute-api.ap-southeast-1.amazonaws.com|8g759uq7nb.execute-api.ap-southeast-1.amazonaws.com>",
     :datomic.ion.edn.api-gateway/json
                  "{\"version\":\"2.0\",\"routeKey\":\"$default\",\"rawPath\":\"/\",\"rawQueryString\":\"\",\"headers\":{\"accept-encoding\":\"gzip, deflate\",\"content-length\":\"0\",\"host\":\"<http://8g759uq7nb.execute-api.ap-southeast-1.amazonaws.com|8g759uq7nb.execute-api.ap-southeast-1.amazonaws.com>\",\"user-agent\":\"http-kit/2.0\",\"x-amzn-trace-id\":\"Root=1-5f730b24-4ac64db84deabaf53c38af60\",\"x-forwarded-for\":\"42.200.88.157\",\"x-forwarded-port\":\"443\",\"x-forwarded-proto\":\"https\"},\"requestContext\":{\"accountId\":\"191560372108\",\"apiId\":\"8g759uq7nb\",\"domainName\":\"<http://8g759uq7nb.execute-api.ap-southeast-1.amazonaws.com|8g759uq7nb.execute-api.ap-southeast-1.amazonaws.com>\",\"domainPrefix\":\"8g759uq7nb\",\"http\":{\"method\":\"GET\",\"path\":\"/\",\"protocol\":\"HTTP/1.1\",\"sourceIp\":\"42.200.88.157\",\"userAgent\":\"http-kit/2.0\"},\"requestId\":\"Tn6trha8yQ0EMGg=\",\"routeKey\":\"$default\",\"stage\":\"$default\",\"time\":\"29/Sep/2020:10:23:32 +0000\",\"timeEpoch\":1601375012276},\"isBase64Encoded\":false}",
     :datomic.ion.edn.api-gateway/data
                  {:version         "2.0",
                   :routeKey        "$default",
                   :rawPath         "/",
                   :rawQueryString  "",
                   :headers
                                    {:accept-encoding   "gzip, deflate",
                                     :content-length    "0",
                                     :host              "<http://8g759uq7nb.execute-api.ap-southeast-1.amazonaws.com|8g759uq7nb.execute-api.ap-southeast-1.amazonaws.com>",
                                     :user-agent        "http-kit/2.0",
                                     :x-amzn-trace-id   "Root=1-5f730b24-4ac64db84deabaf53c38af60",
                                     :x-forwarded-for   "42.200.88.157",
                                     :x-forwarded-port  "443",
                                     :x-forwarded-proto "https"},
                   :requestContext
                                    {:routeKey     "$default",
                                     :stage        "$default",
                                     :time         "29/Sep/2020:10:23:32 +0000",
                                     :domainPrefix "8g759uq7nb",
                                     :requestId    "Tn6trha8yQ0EMGg=",
                                     :domainName   "<http://8g759uq7nb.execute-api.ap-southeast-1.amazonaws.com|8g759uq7nb.execute-api.ap-southeast-1.amazonaws.com>",
                                     :http
                                                   {:method    "GET",
                                                    :path      "/datomic",
                                                    :protocol  "HTTP/1.1",
                                                    :sourceIp  "42.200.88.157",
                                                    :userAgent "http-kit/2.0"},
                                     :accountId    "191560372108",
                                     :apiId        "8g759uq7nb",
                                     :timeEpoch    1601375012276},
                   :isBase64Encoded false},
     }

my ion-config.edn looks like this:

{:allow    [datomic.ion.starter.http/ionized-app]
 :lambdas  {:app
            {:fn          datomic.ion.starter.http/ionized-app
             :integration :api-gateway/proxy
             :description "return html app"}}
 ;:http-direct {:handler-fn datomic.ion.starter.http/return-something-json}
 :app-name "kyt-dev"}

I'm using the Solo topology (the version, which was the latest last week), otherwise I wouldn't bother with lamdba gateways if I could use the production topology.

the docs are mentioning these /json and /data keys in a note, but just in the table above the note, they are not namespaced: https://docs.datomic.com/cloud/ions/ions-reference.html#web-ion

Hey @onetom , have a look at https://github.com/pedestal/pedestal.ions And https://github.com/pedestal/pedestal-ions-sample

thanks, I had a look, but I don't see how would it deal with my situation. it does have a great example of a protocol which converts the response body into an input stream, which I still need, because the reitit.ring/create-resource-handler just returns a java.io.File as a :body and Datomic threw some ->>bbuff conversion error as a result. for now, I just have a middleware to transform the above mentioned request map to be ring compatible:

(if-let [gw-data (:datomic.ion.edn.api-gateway/data gw-req)]
       (-> gw-req
           (assoc :uri (-> gw-data :requestContext :http :path))
           (assoc :request-method (-> gw-data :requestContext :http :method)))
       gw-req)

Just gotten bitten for an hour getting 401 Unauthorized for Datomic Pro due to missing XML schema in ~/.m2/settings.xml, which is not mentioned in https://my.datomic.com/account. Previously: https://clojurians-log.clojureverse.org/datomic/2019-01-30/1548890962.888000

I want to make sure I understand, did you call apigw/ionize on your ring handler function? https://docs.datomic.com/cloud/ions/ions-tutorial.html#lambda-proxy

I have the suspicion that or ion-config.edn doesn't need the :integration :api-gateway/proxy option anymore if I use the newer style HTTP API gateway setup (as opposed to the RESTful API style), it's just hasn't been documented...

https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-vs-rest.html

Sorry about that, what is missing in https://my.datomic.com/account ? I see the .m2/settings.xml described as:

;; In ~/.m2/settings.xml:
  <!-- ~/.m2/settings.xml (see the Maven server settings docs) -->
  <servers>
    …
    <server>
      &lt;id&gt;<http://my.datomic.com|my.datomic.com>&lt;/id&gt;
      &lt;username&gt;REDACTED&lt;/username&gt;
      &lt;password&gt;REDACTED&lt;/password&gt;
    &lt;/server&gt;
    …
  &lt;/servers&gt;

;; In deps.edn:
{:mvn/repos
 {"<http://my.datomic.com|my.datomic.com>" {:url "<https://my.datomic.com/repo>"}}
 :deps
 {com.datomic/datomic-pro {:mvn/version "${VERSION}"}}}

<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns=""
          xmlns:xsi=""
          ssi:schemaLocation="">

ack! let me talk to Alex so i fully understand and I can update our http://my.datomic.com account to reflect that!

@lanejo01 yes, that .../ionized-app is defined as (apigw/ionize app), where app is simply:

(fn [req]
      {:status  200
       :headers {"content-type" "text/plain"}
       :body    (with-out-str
                  (clojure.pprint/pprint req))})

that's how I obtained the request map I showed above

To make sure I understand correctly, you're not using the supported integration. Is there still a problem if you use the supported one?

what do you mean by supported integration?

Also - and this is probably out of scope - but even after fixing settings.xml (and running clj from terminal), IntelliJ Cursive still reported a 401 for deps because it caches the 401 error for a given deps.edn (not sure if this is due to Maven or IntelliJ). Fixed after reordering any two items under :deps, then I could start a REPL.

https://clojurians.slack.com/archives/C03RZMDSH/p1601393646108500?thread_ts=1601380150.089100&cid=C03RZMDSH

I'm just realizing that probably the Datomic docs are taking about how to integrate a web ion with the traditional RESTful API gateway, not the new "HTTP API". I'm using this "new style" gateway, because it supports JWT authorizer's out of the box, without the need to deploy a lamdba function just for that purpose.

yes, the mentioned request map was observed when my ion-config.edn contained that : integration :api-gateway/proxy setting

my API gw was created my this sample CF template though: https://github.com/awsdocs/amazon-api-gateway-developer-guide/blob/master/cloudformation-templates/HTTP/http-with-jwt-auth.yaml which I found in these AWS docs: https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-jwt-authorizer.html

I'm not sure that approach is compatible with the current mechanism for making a ring compatible handler. I think you're in uncharted territory, rife with undefined behavior.

Probably... Thanks looking into it!

I will probably just transition to the production topology, though I can foresee issues with the VPC link and NLB in that case, which I have just as little experience with as I have with Cognito and JWT authorizers :)

You will get a raw payload if you use http-direct

it will not be a nice map like above.

Alternatively, I can write a replacement ionizer function for this new "HTTP API" gateway

You're free to do that, but we won't be able to support that if there are issues.

Hey! I have a transaction that always puts one entry into datomic, and I'd like to get it to return the entity id of the new entry. I notice that the returned map contains :tx-data, which has the data I need. But I'm not sure how to read the contents of the returned datum, and, indeed, if this is considered bad best practices or not. Help appreciated!

the returned map also contains :tempids which is a map of tempid -> entity id

@doby162

I'm getting an empty map on that one, do I just need to add a temp-id to the transaction?

paste your code/input

if your transaction included tempids, datomic returns the resolved ids after it transacts

{:tx-data [#:user{:email "e", :password "q", :name "q", :token "q"}]} ; this query is generated by (make-record :user) and executed
(def q (make-record :user "q" "q" "q" "q"))
(:tempids q)

Ah, ok! Just had to add :db/id "nonsense" to my query and now the map gives me {"nonsense" id} !

thanks!

if :user/email is a unique attribute in your database, you can use it to lookup entities without entity ids

Oh, nice