Clojurians Log v2

How would I capture 2 pieces of information and join them inside the match. Eg: !date and !time are captured and I can create a datetime with whatever lib I choose.

(m/rewrites {:venue venue-data
             :event (group-by :related-id event-data)}
            
            ;; Pattern
            {:venue (m/scan {:related-id ?id
                             :location (m/app str/trim ?location)})
             :event  {?id [{:date (m/pred between-x-y? !date)
                            :start-time !start-time
                            :attendance (m/pred min-size? !event-size)
                            
                            }...]}}
            ;; Action
            {:id ?id
             :name ?location
             :location [{:event-name !event-name
                         :date-time !date-time
                         :event-size !event-size} ...]})

Oh I can apply (m/app ….) on the action.

Super easy 🙂

Hit memory limits today on joining 2 datasets that weren’t particularly that large. Maybe like 50,000 and 30,000 Clojure maps with little in them. Nothing to criticise though. I think this task is better left for a database. Maybe using meander inside a transducer and process them one at a time and write to db in a stream.

Did you hit a memory limit because of meander? Or because your dataset is just too large for your heap size?

If you use something like visualvm or mission control you can inspect the memory of your application. You'd be surprised at how much memory maps can take up.

Yea and it probably holds onto the parsed CSV before it was converted to EDN because of immutability?

Actually I don’t know how I got those numbers I said earlier. It’s was more data than that :rolling_on_the_floor_laughing:

Must have been the other files I joined on

I’m thinking of bringing in datahike to write to disk and doing the transaction during the action step.