Clojurians Log v2

I’m a little lost on trying to join two datasets together.

I seem to always end up messing around with rewrite 😅

Thats good!

(m/rewrite {:event event-dataset
            :venue venue-dataset}
  {:event (m/scan {:related-id id?
                   :attendance !attendance
                   :price !price})
   :venue (m/scan {:related-id id? 
                   :capacity !capacity})
  {:id ?id 
   :data [{:capacity !capacity
           :attendance !attendance 
           :price !price} ...]})

But you’re getting out :data with singleton values right?

Where I have two big lists of data that I want to merge together using a common id

Gotcha, gimme just a sec.

Umm…

Common id’s

like I have a bunch of say venues and all the events for every venue. Wanting to merge it all into one datastructure.

id? in the match pattern should be ?id 🙂

I hand typed it into slack so that is a typo haha. Kind of a mock out of what I’m trying to do.

@grounded_sage What is the kind of relationship? 1-to-1, N-to-1, etc

Hmm it’s kind of hard to describe. I’m trying to create a mock input and my desired output at the moment.

I’m hoping I haven’t simplified too much for my problem but it’s something like this.

;; Input
  {:people [{:id 1 :name "Bob"}
            {:id 2 :name "Sarah"}]
   :foods [{:id 1 :food "Apple"}
           {:id 1 :food "Banana"}
           {:id 1 :food "Olives"}
           {:id 2 :food "Pears"}
           {:id 2 :food "Grapes"}]}
  
  ;; Output
  {:id 1 {:name "Bob"
          :foods ["Apple" "Banana" "Olives"]}
   :id 2 {:name "Sarah"
          :food ["Pears" "Grapes"]}}

(me/rewrite '{:event [{:price p1, :related-id 1, :attendance a1}
                      {:price p3, :related-id 2, :attendance a1}
                      {:price p2, :related-id 1, :attendance a1}]
              :venue [{:capacity c1, :related-id 1}
                      {:capacity c2, :related-id 1}]}
  (me/with [%event {:related-id ?id
                    :attendance !attendance
                    :price !price}
            %venue {:related-id ?id
                    :capacity !capacity}]
    {:event [%event . (me/or %event !not-event) ...]
     :venue [(me/or %venue !not-venue) ...]})
  [{:id ?id 
    :data [{:capacity !capacity
            :attendance !attendance 
            :price !price} ...]}
   ;; Recursively collect the rest.
   & (me/cata {:event [!not-event ...]
               :venue [!not-venue ...]})]

  ;; Default to returning and empty vector
  _ [])

will take me a little to parse what is happening here 😛

(me/with [%event {:related-id ?id
                    :attendance !attendance
                    :price !price}
            %venue {:related-id ?id
                    :capacity !capacity}]
    {;; Find the first event and the rest of the matchings events in the vector
     ;; or collect the non-matching venues separately.
     :event [%event . (me/or %event !not-event) ...]
     ;; Find all the matching venues or collect the non-matching
     ;; venues separately.
     :venue [(me/or %venue !not-venue) ...]})

;; Rewrite as a vector with the first item holiding our "join"
  ;; object.
  [{:id ?id 
    :data [{:capacity !capacity
            :attendance !attendance 
            :price !price} ...]}
   ;; Recursively rewrite the structure with the non-matching items
   ;; and merge it into the vector.
   & (me/cata {:event [!not-event ...]
               :venue [!not-venue ...]})]

When you use cata on the right side of a rewrite rule, will apply substitution to the argument and then recurse on the rewrite.

Since both rewrite rules return vectors [p1 & (me/cata p2)] is basically (into [v1] v2) where v1 is the result of the p1 substitution and v2 is the result of the (me/cata p2) substitution.

Are the ids in people unique? Could you index by id instead?

{1 {:id 1 :name "Bob"}
2 {:id 2 :name "Sarah"}}

(Maybe not, probably spoke too soon, but something to consider)

What I should have said is can you group by food.

(def data
  {:people [{:id 1 :name "Bob"}
            {:id 2 :name "Sarah"}]
   :foods (group-by :id [{:id 1 :food "Apple"}
                         {:id 1 :food "Banana"}
                         {:id 1 :food "Olives"}
                         {:id 2 :food "Pears"}
                         {:id 2 :food "Grapes"}])})


(m/rewrites data
  {:people (m/scan {:id ?id :name ?name})
   :foods {?id [{:food !food} ...]}}
  {:id ?id 
   :name ?name
   :food [!food ...]})

IOW this is a reduction.

For the data I pass above (I forgot to paste the result):

[{:id 1,
  :data
  [{:capacity c1, :attendance a1, :price p1}
   {:capacity c2, :attendance a1, :price p2}]}
 {:id 2, :data []}]

Thanks guys! I’m still trying to grok what is happening and also finding out if my assumptions are right as well. I’ll definitely have an update tomorrow of how I go

This definitely simplifies things. I am trying to learn Meander so much I’m forgetting I have Clojure 😅

Haha, yeah it is an easy trap to fall into. Meander is fantastic, and eventually we should have good answers for things like this. But definitely don't be afraid to do a little massaging to get something in a shape that makes it easy and obvious to work with 🙂

Worked like a charm! The more I can KISS this the more valuable it is to the company because we still have to onboard people to Clojure. So going crazy with Meander syntax would be even harder for people to contribute.

That's great! Definitely understand the need to keep things familiar for new people. There is a real learning curve in thinking more symbolicly about those sorts of complicated transformations.