meander

All things about https://github.com/noprompt/meander Need help and no one responded? Feel free to ping @U5K8NTHEZ
noprompt 2020-03-02T04:31:25.098700Z

[meander/epsilon "0.0.402"]

aisamu 2020-03-02T22:43:24.109100Z

Thanks!

grounded_sage 2020-03-02T18:01:16.101300Z

So I have meandered on over to tech.ml.dataset for processing the columnwise data that is csv. Sorely missing the clarity of Meander patterns.

noprompt 2020-03-02T18:04:07.101800Z

I’m sorry. 🙂

grounded_sage 2020-03-02T18:33:26.104900Z

As someone with a 10,000 foot view and little idea around internals. I’m wondering if there is a new ns there for Meander to handle large column data. Or like a Meander-csv that could leverage the codebase inside of tech.ml.dataset.

timothypratley 2020-03-03T15:37:58.109500Z

Hi! Just so I can understand the desire here I’ll attempt to rephrase as:

I wish meander sequence patterns like (!xs ...) used transducers instead of memory variables.
^^ is this accurate? i.e.: The issue is that very long sequences don’t fit in memory? Or is it a different problem?

timothypratley 2020-03-03T15:49:56.111700Z

(defn unarchived' [stories]
  (remove (fn [{:keys [archived completed]}]
            (and archived (not completed)))
          stories))

(def unarchived
  (s/rewrite
    ((m/or {:archived  true
            :completed false}
           !stories) ...)
    ;;>
    (!stories ...)))
^^ for a really big CSV !stories needs to be a sequence, not an array. Conversely when do we need an array not a sequence?

grounded_sage 2020-03-03T16:55:35.111900Z

I’m still new to all of this so having some trouble keeping up. Though I am willing to dive in and contribute to this problem with a bit of guidance :)

noprompt 2020-03-03T21:13:58.112100Z

(keep
 (fn [value]
   (me/rewrite value
     {:archived false, :completed true :as ?it}
     ?it))
 '({:archived true, :completed true}
   {:archived false, :completed true}
   {:archived true, :completed false}
   {:archived false, :completed false}))
;; => 
({:archived false, :completed true})
would be decent.

noprompt 2020-03-03T21:16:19.112300Z

This also works

(me/rewrites '({:archived true, :completed true}
               {:archived false, :completed true}
               {:archived true, :completed false}
               {:archived false, :completed false})
  (me/scan {:archived false, :completed true :as ?it})
  ?it)
;; => 
({:archived false, :completed true})
but rewrites doesn’t support cata FYI.

timothypratley 2020-03-03T22:03:16.112500Z

oh good thinking.

timothypratley 2020-03-03T22:03:31.112700Z

Does that help with the original question of “Meander to handle large column data”?

noprompt 2020-03-03T22:14:59.112900Z

It can. It just depends on what you are using. If you use a single in a pattern, Meander has to apply pattern matching to everything in the collection in question. If you can rephrase the pattern in such a way that search becomes applicable its nice to go that way.

grounded_sage 2020-03-04T10:39:14.113200Z

Yea all of the interesting transformations and where meander has value for me is when I use

noprompt 2020-03-02T20:49:47.105Z

I’m hoping that zeta will have the ground work for doing that being able to deal with bytes, etc. and then building more sophisticated matching on top.

grounded_sage 2020-03-02T22:24:23.108900Z

Cool. At present I’m dealing with two CSV’s. One with 1.5m rows and another with 300k. The dataset library handles it extremely well. It’s just a little more low level than I would like to be working at having seen how nice things can be :)

noprompt 2020-03-02T23:17:44.109300Z

The goal (for me) is to be able to achieve low level performance but from the comfort of a high level. I believe we can get there via the right pattern matching primitives and pattern aliases (`defsyntax`). So, what I’m saying is, this case is motivating for me too. 🙂

👍 1