Clojurians Log v2

Just my two cents, but if your primary concern is financial in nature with data conflict concerns and an emphasis on immutability, Datomic would be a great fit as the sole database solution. I understand that you and your team might hear about Datomic's limited write-throughput due to global transactor in a single system, but this is again extremely relative, and is only really a problem at very high throughputs, or poor product fits like realtime applications (gaming, heavy clickstream logs, etc.). I doubt for financial data you reach anything close to troubling Datomic's write throughput, and even if someday you do, Cognitect's new backing is literally Nubank, the largest digital bank in the world, run primarily on Datomic (with advanced sharding etc., I'm sure you could get support if this ever becomes a necessity). Read latency is a more relevant problem especially if you want to force your application to use Google Cloud's stack for Cloud Functions etc., though unless this is a strict requirement of your organization, I'm not sure why AWS Lambdas alone wouldn't be good enough. AWS Cognito also works pretty nicely with lambda hooks into automatically syncing user information into Datomic with some simple setup. There's also just HTTP-direct for production query groups which is really fast and cuts out lambda latency - JWT authentication would let your mobile app directly hit this endpoint too without much fuss.

Hi! I know Datomic doesn’t support ordered lists. However, there are some 3rd party implementations of linked list or indexed list data structures. We want to support some simple tables. Let’s say we have a following schema: a simple table with 2-column rows.

[{:db/ident :table/row
  :db/valueType :db.type/ref
  :db/isComponent true
  :db/cardinality :db.cardinality/many}

 {:db/ident :row/a
  :db/valueType :db.type/long
  :db/cardinality :db.cardinality/one}

 {:db/ident :row/b
  :db/valueType :db.type/long
  :db/cardinality :db.cardinality/one}]

(d/transact conn [{:table/row [{:row/a 1 :row/b 2}
                               {:row/a 10 :row/b 2}
                               ; thousands of the other rows
                               
                               ]}])

I don’t think there’s any guarantee about :db/id monotonically increasing

best to treat them only as opaque identifiers from what I gather

Hey all! What's the simplest way to use Datomic for a hobby project? I've tried using datomic ions for a week or so, but the additional complexity around AWS resulted in spending most of time at infrastructure related stuff

Have you looked at dev-local? It seems designed for what you describe https://docs.datomic.com/cloud/dev-local.html

I've tried dev-local at first for learning datalog, but main concern I have with using it in a deployed app is durability and eventual scaling

What's been a pain point for Ions? I've found it to be really simple to use and actually abstracts away a lot of the annoying AWS complexity that you usually need to do to setup nodes, VPN, etc.

I've managed to follow the tutorial fine, but here's where I got stuck: I created a lambda proxy resource which acted as a single entry point to my system, with Cognito Authorizer. It looked something like this:

(def handler
  (ring/ring-handler
   (ring/router
    [["/api"
      ["/test" {:get (fn [_]
                       (-> (ok {:foo "bar"})
                           (res/header  "Access-Control-Allow-Origin" "*")))}]]])
   (ring/create-default-handler)))

(def handler-lambda-proxy
  (apigw/ionize handler))

So definitely not issues related to datomic, but I feel like I've added a lot of infrastructure complexity, and would like to take a step back to a more simpler system

datomic starter

@alen your problem is that your cognitive authorizor in api gateway is authenticating the options method causing the cors preflight to fail. If instead of using ANY for your http method try explicitly making get put post and delete authenticates but leave options unauthenticated.

@lanejo01 does that mean I shouldn't use a lambda proxy at all?

it seems that lambda proxy endpoint needs to have ANY

Ahh, I see, that's far more of an AWS documentation encouraging complex solutions, rather than help clarifying what's going on. Cognito works great with Datomic Ions, but as @lanejo01 is pointing out, it's more of an issue of the extra layer of authorization you're putting on top of API gateway. I really suggest not using an extra lambda step - it's entirely wasted latency that you could easily bake into the actual processing of your code, given that unless you're on production, you already have a lambda ingress

Not what I’m suggesting, lambda proxy is fine. It does NOT need to be ANY.

You could also fix the lambda proxy, but if you want to do Cognito auth, just verify the public key with one function in your handler would save you all this hassle

@eagonmeng I think for "getting started" ion lambda proxy is a fantastic fit, otherwise he will need to "get started" with a production topology to get access to http-direct, kind of a non-starter. He has ALSO already correctly configured cognito, it's just a confusing interaction between api-gateway and cognito.

@alen Make 5 explicit methods, each calling the same lambda. Then don't auth the options request and you should be good to go 👍 FWIW, I'm so confident this is the issue because I've hit it so many times. I brought it up with the product team literally this week as something we could better document.

Thanks a lot for you suggestions, I appreciate it! I'll give it another shot and try that

Yeah that's entirely fair, although even in the Ion solo configuration, the lambda ingress into the system isn't a blocker for simply doing the Cognito JWT auth in your handler. It saves a lambda's worth of latency, adds just one dependency to your code (buddy or other JWT library you like), still uses Cognito as IDP, and also saves about $10 a month since the solo lambda is so terrible in latency unless you do provisioned concurrency.

The one benefit of having an extra external step of a lambda proxy authorizer is blocking a lot of unwanted traffic without taxing your server, but this doesn't seem very applicable on a hobby or even medium startup level situation

You can set up a heartbeat on your ion so it's warm before going to provisioned concurrency. Let’s keep the costs down

@alen if you're still stuck after the second try dm me and we will get it sorted out.

@lanejo01 Yeah, you're absolutely right about the heartbeat approach, forgot about the good old cloudwatch ping since recent clients insisted on AWS guaranteed "provisioned concurrency", which is honestly basically punting on serverless entirely and going back to full instances... On that note, I've always wondered, is there a specific reason Lambda the ultimate is on the JVM? As far as I understand, it's literally just a nice proxy/entry point and doesn't need heavy computation, so a clojure script implementation on node.js with a much less punishing cold-start ceiling seems like it'd solve a lot of these headaches. It's a feature I desperately wanted for a long time and seems it'd mesh better for serverless-esque workloads

@eagonmeng If you're starting a new project, the heartbeat is sufficient for keeping costs down and avoiding coldstart. If you are scaling and need to increase the number of concurrent lambdas and worried about coldstart in this context, add a timeout to the client request and retry after you're over x% of your average latency (where you pick x). You'll likely be routed to a different lambda at that point and still finish the request faster than waiting for the coldstart.

@lanejo01 That's a cool approach! Haven't thought about the probabilistic retry method when concurrency on lambdas factors in, super helpful. Would it be fair to say that with these methods to combat cold start, the JVM lambda has better throughput considering warm instances than other platforms with less cold-start ceiling? Love Datomic, just trying to understand the design choices as a matter of curiosity and learning

Glad you're a fan! There are many other factors besides performance that factor in to the choice of JVM lambda vs "other" like maintainability, stability, ecosystem, stdlib, etc. I'm not going to make a general performance claim here, because the second I do, someone will throw a benchmark at me for a specific case where x is faster than y and call me a liar 🙂 Luckily, this question has nothing to do with Datomic and you can defer to all the other liars on the internet who made their own benchmarks and conclusions. In all seriousness, if performance characteristics are critical for your use-case, YOU need to do the measurements for your scenario and decide for yourself.