Clojurians Log v2

Hi @ben.sless. There were some changes in the last release regarding the configuration and there should be no reload-config. The mention of it in the api is a leftover. We will fix this asap. The database you are using for datahike should exist in Postgres and your postgres-user should also not have the permission to create or delete databases anyway. If you need any help, let me know what your current problem is and I'll try to help.

Thanks timo, I managed to get everything working shortly after making the post, but thank you for taking the time. I'd say the biggest hurdle is the clarify of the documentation and reload-config If you'd like I can try to pinpoint all the parts I stumbled over and make suggestions. All in all datahike is pretty cool and I want to see it succeed 🙂

👍we are very much interested in your hurdles and suggestions

Alright, I'll try to compile a list of all the pain points

I am continuing to explore Datahike. Very excited about this project. I have a few questions: 1. Does Datahike maintain a live in memory index, so as to avoid flushing to storage on every transaction or does it just flush to storage on every transaction? 2. Does Datahike do any storage GC to delete outdated indexes? 3. Datomic Cloud Strings are limited to 4096 characters. Any limitations in Datahike? Any recommendations on string size to keep queries performant?

@timok another quick win "bug" in the doc is any reference to :dbname in the config map when it should in fact be :path . A more subtle one is that it must start with a /

Transaction question: I have in my schema entities with scores which can be modified by users (other entities) by increments of +1/-1 (upvote/downvote). I chose to represent it as a score integer, and a set of refs to the entities which voted +1 and -1. Is this a good model? What's the idiomatic way to go about it? I guess I have to work with db.fn/call? My thinking was, for upvote (without loss of generality): • find user in downvotes • if found, increment score by 2, remove from downvotes, add to upvotes • if not found, increment score by 1, add to upvotes Does this make sense?

Also, how would I avoid duplication, up/down vote look almost the same and also look pretty similar between the two cases

thoughts?

@timok another really weird behavior I just came across: all my tests work in REPL, all of them throw NPEs when I run them with lein test They throw at this part:

Exception in thread "async-dispatch-6" java.lang.NullPointerException
        at clojure.core$deref_future.invokeStatic(core.clj:2300)
        at clojure.core$deref.invokeStatic(core.clj:2320)
        at clojure.core$deref.invoke(core.clj:2306)
        at konserve.core$get_lock.invokeStatic(core.cljc:29)
        at konserve.core$get_lock.invoke(core.cljc:28)
        at konserve.core$assoc_in$fn__16363$state_machine__13314__auto____16370$fn__16373.invoke(core.clj 

Happy to hear that you're excited. 1. Datahike supports a volatile in-memory backend. We flush with each tx to the index and thus to the backend. 2. GC is worked on with the next version of the storage protocol konserve. 3. We haven't thought about limitations for strings yet. Thanks for that. We will add that to our benchmark suite.

thanks @ben.sless I don't really know what this second one is about. :thinking_face: the other issues I appended to an already open issue about the documentation in api namespace. we will get to it soon.

Maybe it has to do with how the db is initialized. I created this utility function

(defn- empty-db
  []
  (-> (dc/empty-db) (dc/db-with s/schema) dc/conn-from-db))

is there a chance the connection is nil during the tests?

Update with regards to the NPE, tried a different approach:

(def cfg {:backend :file :path "/tmp/hnc"})

(defn- empty-db
  []
  (sut/connect cfg))

;;; 
(defn connect
  [cfg]
  (d/delete-database cfg)
  (d/create-database cfg :initial-tx s/schema)
  (d/connect cfg))

I may be wrong about this, but I thought I remember discussion with @whilo to the effect that at least in theory the underlying hitchiker tree could be flushed on a different basis. But @konrad.kuehne can probably speak to that better than I.

Sure it can be flushed differently, but at the moment Datahike is using it only at tx time.

@ben.sless I would probably just store votes as {:vote/user ... :vote/post ... :vote/score ...} and compute the total :post/score using an aggregate (sum) query.

Ensuring unique votes per post+user is a little bit more work, and this is where Datomic's composite tuples as unique/identity attributes come in handy.

But you could use tx fns to ensure that

A big problem with :post/downvotes and :post/upvotes as you have it here is that you're going to blow up your indices

Which is what my ungainly solution uses

ah

Yeah, but it would be a different tx function with the data modelling approach I'm describing

I'm not sure I'll make the switch because this is just an exercise, but thanks for the advice, I didn't think about storing the votes as their own entities

The index problem is that you'd end up with very big blocks in your primary EAV index, killing performance on any queries that end up needing to scane over those entities (potentially for other attributes,)

@ben.sless Gotcha

Do I have to keep an index for :post/upvotes?

No

Manual indexing is just for performance on reverse lookups

Thanks. Your suggestion makes more and more sense, I might make the switch just so it would stop bugging me now

I'm also having trouble with the aggregate functions, as now I want to model a lookup of the top N rated posts

When does flushing happen with transact! I am using a file backend. I am creating a database and then I transact! a schema. So far so good. The file in store/data grows to 2,6kb After that I do more transact! s and the connection gets updated. But not the file.. Any idea?