Clojurians Log v2

hello, for the people using Datomic, how are you handling union queries on the server side?

@wilkerlucio we’re only using union queries for view routing on the client, so we never need to send those queries to the server. all the queries we send contain only props and joins

@ethangracer thanks, and you have thoughts on how to integrate that with pull syntax? on my previous project I was fully controlling the parsing, so was kind easy to implement, I can still go this way but I rather find a way that requires less coding

I wouldn’t describe it as “integrating with pull syntax”, but what we do is we set up all of the UI relevant concerns with InitialAppState (which include all of the union queries). Then we’ll do something like (load-data this [{:people (om/get-query Person)}]), which would build a list of people idents at :people normalized to :people/by-id. So when we need to access all people, we do so by adding [:people ‘_] to the query of the component that needs the data

So we circumvent the need to have the data placed in the app state corresponding with the UI component structure (which would require the union logic server-side that you’re talking about to return the data in the right format)

humm, my need is in the heterogeneos list style of query

I think it needs to be processed on the server side, otherwise I would be required to overfetch (and in my case that can be a disaster, some of the types requires much more data than others)

what do you mean by “heterogeneous list style?"

entities with different attributes in the same list?

exatly

gotcha

I’m not sure off the top of my head if there’s a way to do that without server-side processing

And I’m not sure how a union query would handle normalizing that data

I don't think would be possible without it, the server needs to decide the fields while fetching each entry

just because I haven’t tried it 🙂

I know how to handle the client side, I did it before, just the server is being a scratch on the head (you can see more for the client here: https://github.com/omcljs/om/wiki/Queries-With-Unions)

I suppose on the server you could write a query that pulled on set of attributes or another based on what attribute was missing from the entity that you’re pulling

i.e.

[:find ?e
 :where 
(or-join [?e]
  (and
    [(missing $ ?e :attr-type/two)]
    [?e :attr-type/one ?value])
  (and
   [(missing $ ?e :attr-type/one)]
   [?e :attr-type/two ?value]))

but that gets super ugly the more entity options there are

and it ties the query to the attributes... not very good, we need the client on this decision

yeah I would definitely vote for a post-mutation

do a separate query for each type

then manually merge the lists of idents into one list

I think doing that would be a pain to join the items in order after, would need a separated field just to keep the sorting (that might change depending on occasion)

oh, it’s order dependent!

I believe the best solution is to the server to process the union as is

somehow

In that case I’m not sure that you have an option apart from manually parsing it server-side

but you’re now out my depth!

on the other project I used something I called :union-selector, so in my manual parsing I check for this option, when it's present and the current query is an union, it would use the value of :union-selector as the key (which could be a regular attribute on the field, or a virtual one), to select which query to fetch the rest

works greatly, but then I have to re-implement most of the pull syntax myself

honestly not completely following but I understand the need to split up the union query into its constituent pull syntax fragments to get the data from datomic

@wilkerlucio with regards to the sorting, why not just do that client side? why does it need to come back from the server sorted? if you don’t need the data to be sorted by the server, then you can use the post-mutation method and just sort the list that way

@ethangracer it's possible, but I ultimaly I think the union syntax is a good query syntax, and by using it I avoid a lot of those manual transformations that will be hard to understand (and imagine a project with multiple of those across the UI)

I think solving on the server is the simplest solution

fair enough. I’d be curious to see a snippet for how you're handling that at some point if you don’t mind sharing

sure, no problem with sharing at all 🙂

@wilkerlucio Handling it on the server is why we left you to write parser emitters on the server. As long as you return your hetero list from the server, you can use the union query on the client to normalize the result.

So, perfectly normal to want to send a union query to the server, but yeah Datomic won't just "get it"

@tony.kay we, I'm getting to think that a custom pull reader is going to be required by a lot of applications, datomic pull is great, but not flexible/extensible enough, I'm porting some of the engine that I did before in Node.js to Clojure, it might get to be a lib one day 🙂

the datomic entities help a lot on that

In general, it would be nice to have something that does that repetitive work for both SQL and Datomic

with just some security plugins

@wilkerlucio we also felt datomic pull was great but not flexible/extensible enough

so we wrote this https://github.com/AdStage/pluck-api

@currentoor thanks, I'll check it out

What's the best practice to store blob files e.g. images due the lack of support for blob files in datomic? I just saw that the adstage pluck-api mention the use of a blob store in the example snippets.

At AdStage we just generate a UUID and store them as serialized bytes to postgres (using Nippy), then use the pluck api to join the data back in as if it was a key you could pull with datomic. As datomic makes indexes on values, this was the approach recommended to us in the #datomic channel, rather than trying to serialize to string and store as a value in datomic.

Sounds good for me. Thanks

(Well, to be clear, just the "don't store in datomic" part was recommended, we arrived at the "store in postgres" thing mainly because we were already setting it up as our underlying datomic data store, so easy to add an extra table somewhere...)

That's reasonable. I also have to checkout the image-library component in untangled-component

But the pluck-api/blobstore solution is more general in case for storing other types of files for e.g PDFs and so on.

@baris yeah you should be able to store anything with the pluck API and still retain the niceness of the datomic pull query

the pluck-api doesn’t yet support pluck-many, analogous to pull-many for collections, but for pull it works pretty well for us

That's pretty cool. I'll give it a try.

awesome, let me know what you think 😄

For sure

thought i’d drop a quick note in here to let people know that as much as I’ve enjoyed working on Untangled, and as much as I hope that I’ll continue to work on it, I’ve accepted a new job that isn’t working in Clojure. So I’m not sure how useful I’ll be as the framework continues to grow since I won’t be developing with it full-time or professionally. In any case, it has been a pleasure getting to know everyone here and we’ll see what happens. The rest of the team is still around and I’m still going to evangelize untangled as much as I can 🙂

@ethangracer sorry to see you go. I really appreciated your push for documentation. Good luck with the new job!

@ethangracer: All good for your future career...thanks for pushing untangled

@ethangracer good luck on your new job, and thanks for all your contributions so far 🙂

thanks everyone!

@ethangracer I just finished the basic implementation of my parser, supporting union queries, the implementation uses an idea of "portable nodes" you can say, since I started using Om.next I always though on my server as a graph, and I would like to be able to specify nodes in a way that I could re-use then at any node point on my graph, and being able to compose those, to solve this I end up with the idea of "readers", the readers can be functions (simplest one, just receives the env) maps (keys are the attribute to access, values are functions that takes env) and sequences (can express a series of readers, readers can stop there to return a value, or return ::continue to move into the next reader, like ring or something), with these ideas I got a small library for those readers, and they are the base for the rest:

then to use it:

the important bit here is read-memos, it sets the ::union-selector to use :memo/type as the field to figure which branch of union to go, the rest is generic and can be reused

@ethangracer congratulations! just out of curiosity, what are you working on now if not clojure?