Clojurians Log v2

more specifically I’m interested how people are integrating datahike/datascript into projects? Do you run clojure on a server-side framework like pedestal and have datahike as a dependency for communicating w/ say postgres?

Trying to do research but there doesn’t seem to be a lot written about this that is accessible for someone new to the clojure ecosystem

If you’ve required next.jdbc.date-time then this protocol will be in effect: https://github.com/seancorfield/next-jdbc/blob/develop/src/next/jdbc/date_time.clj#L40-L50 and Instant will be stored via .setTimestamp. If you’re seeing that exception, That protocol is not in effect. Without seeing more of your code, it’s hard to tell what you’re doing tho’

(also, I’d have probably seen this sooner if it was posted in #sql)

Datascript is normally used in ClojureScript web-apps as an in-memory DB where you maintain the app state (like the UI/UX state). Datahike is used for very small applications, it be a good candidate for a desktop app written in Clojure. Or a client/server app, but with very small number of concurrent users. So maybe internal tooling and such. It doesn't scale to large dataset and it runs on a single machine, so it is made for client/server apps where your server is expected to run in a single node, thus non-distributed. Don't expect it to be resilient either, it won't replicate itself or anything, so have backups. In neither case is PostgressSQL involved. With Datascript, sometimes people have POstgressSQL as their backend DB to their webapp, and DataScript as the client-side in-memory DB used by the client.

I think people use either Ring or Pedestal for the most part.

Other used ones, but I'd say not as common as Ring or Pedestal are: Yada, Holplon and Duct.

Otherwise, if you need high scale, resiliency, fault tolerance, you are looking at using Datomic on the backend or Crux. Those would replace your use of PostgressSQL.

So, best to think of it this way: For server backend: - Where you'd use SQLite -> use Datahike or Datalevin - Where you'd use PostgressSQL, MongoDB, etc. -> use Datomic or Crux For client-side apps: - Where you'd use SQLite -> use Datahike or Datalevin or DataScript (where you'd save/load to/from EDN on disk) - Where you'd just store data in-memory in atoms or variables -> use DataScript or Asami

Or you can always choose to also just use SQLite, PostgressSQL, H2, and other relational databases. The above is if you'd prefer using a Datalog based more graph oriented DB instead.

@didibus this is exactly the kind of answer i was looking for! Very helpful in wrapping my head around where these tools fit into the stack.

I’m looking to try a graph oriented db for a project I’m starting. It’s a relatively standard web app with a user interface in the browser and needs auth and data store. Crux sounds like the best free option?

Well, I'd say it depends what this project is. Like if its a hobby, you can start with Datalevin or Datahike. That will give you a chance to learn the whole Datalog thing. And then if you need to scale, you could afterwards consider switching to Crux.

And if your browser user interface is a SPA, you can try using DataScript in it.

Yes it’s a hobby project with no users yet so the datahike idea might be good to try. And yes it’s an SPA in react so I think I’d like to try datascript as well.

One question about hooking the client up to the server - do you just use standard JSON over http to communicate, and then stick the parsed data into datahike/datascript?

Its more typical to use Transit when your use case is Clojure backend and ClojureScript frontend, and you don't intend other languages to be using the APIs. You should look into: https://github.com/metosin/muuntaja/ [preferred most commonly used] https://github.com/ngrunwald/ring-middleware-format [simpler, you could just use EDN directly with this one, only downside is slower than muuntaja]

The advantage of Transit over raw JSON is that Transit can encode all the Clojure standard types like keywords, where-as JSON can't.

Using EDN directly is nice, but the disadvantage is the compression and performance of serializing/deserializing JSON isn't as optimized as Transit.

A good place to start on the server would be with: ring/ring-core ring/ring-jetty-adapter metosin/reitit metosin/muuntaja

you’re a saint @didibus this is a treasure trove of info for a beginner and gives me a solid foundation and direction to work in. Many thanks! 🙂

I've never used spec. Is there a way to ensure that (during development) whenever the key :mykey is used in a map the corresponding value has to have certain properties?

As a blanket check, no - you need to either explicitly validate with a spec or instrument a spec’ed function to enable this

See https://clojure.org/guides/spec for a tutorial

What is a good and fast way to find the hash of a hash-map that will be the same between different invocations of my Clojure? And preferably between different versions of Clojure.

Like, will (hash {:a "hey"} always return the same value?

don't rely on hashes unless you've defined the hashing algo

Okay, I'll find my own then.

what goal are you trying to achieve with a stable hash?

https://github.com/arachne-framework/valuehash

I have lots of metadata about how a file was created. I'd like to turn that metadata into a path prefix so that I can easily tell whether the file was made with the current settings or needs to be recreated.

Cool!

i've been down similar roads before, and hash is very unforgiving of equivalence. if you change a representation but it "means" the same thing

I know that I will have many false negatives, but it is worth it to ensure the files from my workflows are up to date.

how can one branch out in a transducer pipeline?

(comp
 (map parse-string)
 (xif some-branching-condition
      (comp (map extra-work) (map some-work))
      (map some-work))
 (map process))

(defn xif
  [f if-xform else-xform]
  (fn [rf]
    (fn
      ([] (rf))
      ([result] (rf result))
      ([result input]
       (let [nrf (comp (if (f input)
                         if-xform
                         else-xform)
                       rf)]
         (nrf result input))))))

(comp
 (map parse-string)
 (map (fn [m]
        (if (some-branching-condition m)
          (some-work (extra-work m))
          (some-work m))))
 (map process))

(comp
 (map parse-string)
 (map (if some-branching-condition identity extra-work))
 (map some-work)
 (map process))

yes certainly, for this simple case i didn’t want to introduce any hypothetical scenario just for the sake of a solution, but a construct like

(defn xif [f if-xform else-xform])

i wonder if i am hitting the limitation of transducers here? or is this possible?

tried looking at xforms library. it didn’t have anything either

actually i wonder if it even makes sense

e.g. if it was then what would the following even do if some-branching-condition kept alternating for each element

(comp
 (map parse-string)
 (xif some-branching-condition
      (comp
       (take 10)
       (filter extra-work)
       (map some-work))
      (map some-work))
 (map process))

my bad, some-branching-condition is a function of the input

oh - I misunderstood here, deleting my irrelevant suggestions

i don’t think it makes much of a sense except for simple branches as explained here https://clojurians.slack.com/archives/C03S1KBA2/p1623687952379900?thread_ts=1623687432.378800&cid=C03S1KBA2

i was probably trying to squeeze too much out of transducers 😅

from a higher level vantage point, it's much easier to use comp / fn first then optimize with transducers later

doing the transducing part first puts more demands on your code reading ability, is much more error prone, and can't do anything functions can't, except for doing it faster

(defn if-tx [test-fn true-xf false-xf]
  (fn [rf]
    (let [t (true-xf rf)
          f (false-xf rf)]
      (fn
        ([] (rf))
        ([result] (f (t result)))
        ([result input]
         (if (test-fn input)
           (t result input)
           (f result input)))))))

(comment

  (sequence (comp (map inc)
                  (if-tx even? (map str) (map double)))
            (range 10))

  )

i certainly won’t use that 😛 but i learnt something about the inner working of transducers today (that let binding) :thanks2:

I do keep thinking that it would be nice to have more expressive ways of combining transducers together ... but I can't say for certain that I've really got that problem ... I keep having doubts that I'm just creating a big mess and just writing the straight code would be easier to read and work with ... I guess it depends on how much you end up sharing transducers across modules (or whatever)

i am actually using core async which is why I'm leveraging transducers more often. without them i end up something like https://blog.golang.org/pipelines

which is fine

anything that gets difficult via transducers can be easily plucked out into its own function that accepts from an in chan and outputs to an out chan

What is a chunked-seq? , chunk , etc. and why is there no documentation about it? In the definition of concat, why is there a check for chunked-seq? https://github.com/clojure/clojure/blob/38bafca9e76cd6625d8dce5fb6d16b87845c8b9d/src/clj/clojure/core.clj#L718

chunking is a behavior that realizes N elements of a lazy seq instead of just one at a time

I'm not sure about the documentation

concat is preserving the chunkiness of the input

@ps just subjectively I've seen little documentation about chunked lazy collections, and when people run into bugs caused by chunking the standard reply is that if realizing 32 items for one call instead of one item per call breaks your algorithm you shouldn't be using a lazy data type anyway

these functions are designed for implementors of chunked sequence functions, not for general api usage (which is why they are not part of the public docs)

Rich did a great talk about chunked seqs at the bay area clojure meetup during JavaOne in 2009. I happened to be there but I don't know that it was recorded.

hey look, I wrote stuff down... https://puredanger.github.io/tech.puredanger.com/2009/06/10/clojure-rich-hickey/

The transducer loops one element at a time and applies all transforms. So you can just do:

(sequence (comp (map inc)
                (map #(if (even? %) (str %) (double %))))
          (range 10))