Clojurians Log v2

Hey channel, I worked on it for a while, but failed to find a solution. How could I achieve this with meander:

;; original
(def data
  {:people [{:name :john
             :age  10}
            {:name :jen
             :age  11}
            {:name :jack
             :age  12}]
   :bonus  [{:name   :john
             :amount 100}
            {:name   :jack
             :amount 200}]})

;; wanted
{:people [{:name         :john
           :bonus-amount 100
           :age          10}
          {:name :jen
           :age  11}
          {:name         :jack
           :bonus-amount 200
           :age          12}]
 :bonus  [{:name   :john
           :amount 100}
          {:name   :jack
           :amount 200}]}

one version is this, but it returns a flattened list

(me/search data
  {:people (me/scan {:name ?name :as ?person})
   :bonus (me/scan {:name ?name :amount ?amount})}
  
  {:people (assoc ?person :bonus-amount ?amount)})
;; => ({:people {:name :john, :age 10, :bonus-amount 100}}
;;     {:people {:name :jack, :age 12, :bonus-amount 200}})

And a closer but incorrect version using memory variables

(me/rewrite data
  {:people [{:name !name :as !person} ...]
   :bonus [{:name !name :amount !amount} ...]}
  
  {:people [{:bonus-amount !amount & !person} ...]})
;; => {:people
;;     [{:name :john, :age 10, :bonus-amount 100}
;;      {:name :jen, :age 11, :bonus-amount 200}]}

ok, an ugly solution of a combo of search+match, but it works

(-> (me/search data
       {:people (me/scan {:name ?name :as ?person})
        :bonus (me/scan {:name ?name :amount ?amount})}
  
       {:people (assoc ?person :bonus-amount ?amount)})
     (me/match 
       ({:people !people} ...)
       {:people !people
        :bonus (:bonus data)}))
;; => {:people
;;     [{:name :john, :age 10, :bonus-amount 100}
;;      {:name :jack, :age 12, :bonus-amount 200}],
;;     :bonus [{:name :john, :amount 100} {:name :jack, :amount 200}]}

created an issue in GH: https://github.com/noprompt/meander/issues/137

Struggling to understand this behavior:

(m/match [3 4 5 6 7 8]
  [3 4 . !xs !ys ...]
  [!xs !ys])
;; =>
[[5 7] [6 8]]

=> [5, [6, 7, 8]]

ah except maybe that would be for [3 4 . ?xs !ys ...] instead

I think it's the behavior of !xs !ys that works on every other number that is confusing.

Hey lucy. Will get back later today on this one. Sorry have been a bit busy lately

Thanks @jimmy!

The dot there is telling it where to stop the repeating. So you are asking it to repeat !xs !ys over and over again. That is why it is giving you pairs. If you moved the dot like this [3 4 !xs . !ys ...] now only the !ys are repeating and you'd get [[5], [6, 7, 8]]

yeah, my understanding is that ... looks to its left until it sees either a dot or the start of the current list/vector, and repeats everything in between

ok that makes sense. But why does it do the sliding window like that? i.e. why not [[5 6] [7 8]]? Ah, is it because it scans two by two?

yeah, it's "interleaving", so [1 !xs ...] matches [1 2 1 3 1 :foo 1 :bar]

😮 fascinating! so it's kind of like a flattened cross product. And [1 !xs] would be [[1 2] [1 3] [1 :foo] [1 :bar]]?

speaking in a regex way, it's like (1.)*

in the RHS, !xs would be [2 3 :foo :bar]

ahhh and with re-search

or perhaps re-find

but you can also use [1 !xs ...] on the RHS to expand it to [1 2 1 3 1 :foo 1 :bar], but not [[1 2] [1 3] [1 :foo] [1 :bar]]

Yeah, !xs capture one value, the !ys capture one value, so on and so forth. So you are seeing what is in the !xs and the !ys. Using rewrite you can get the back out in the same order.

(m/rewrite [3 4 5 6 7 8]
  [3 4 . !xs !ys ...]
  [!xs !ys ...])
;; =>
[5 6 7 8]

sweet! 🤓 thanks for clarifying

https://clojars.org/meander/epsilon/versions/0.0.492

This new release improves performance for scan-like patterns such as _ … p1 ,,, pN . _ ….

Significantly.

I think it would be nice to have an “unsafe” notation which tells the compiler to assume the type is known. I’m not sure what it should look like, however. Maybe something like

^::m/unsafe {:assumed-to-be-a-map? true}

You can the motivation for this idea here toward the bottom of my reply: https://github.com/noprompt/meander/issues/135#issuecomment-698215991

I'm new to this whole thing but my leaning would be towards assuming it's a map by default based on the notation? Or at least supports the associative abstraction

we're talking about this case, right?

(m/find data
     {:people (m/scan {:name ?name :id ?id :inactive true})
      :addresses (m/scan {:person-id ?id :as ?address})}
     {:name ?name
      :found 1
      :address ?address}))

The idea here is that right now the generated code explicitly checks if the data you pass in is passes the map? predicate. If it did not check this and you pass something other than a map, it could error or give weird results. But sometimes you want that extra speed and know for sure you are giving us a map.

yep -- just trying to imagine a scenario (besides a GI/GO situation) where someone would pass in something besides an associative abstraction and expect valid results without specifying the type

You might have some heterogeneous set of data and the whole thing you are doing with meander is trying to find the bits that actually match the shape you specify.

hmmm

two thoughts on that... and maybe I'm overly hostile, but first I would wonder if want to go that route is try/catch cheaper than (if (map? ...) ...) , and the second perhaps overly hostile perspective would be, isn't it on the user to make sure the sequence is full of things that support the associative abstraction?

But also we like to have safety by default. And changing that default would be a breaking change which we don't do.

aha

that's a fair point

Might be nice to just have an elide-collection-type-checks all together as well.

One of the benefits of pattern matching is that you get some of the benefits of static types without some of the downsides. In dynamic languages you can accidentally do some operation on the wrong type of thing and get weird results. (like contains? On a vec in clojure). Meander let's you be clear about those types through the data literals. If we didn't check the types what the code does now becomes unclear. But at the same time, we want people to be able to opt-in to more performant code when needed.

From an API perspective, I might think having an "unsafe" set of namespaces would be preferable to cluttering up notation with the type hinting

(that's without having a clue about the challenge of implementing that... yet!)

Well sense we use data literals you'd then have to opt-out of all safety or forgo the data literals syntax instead of being able to do it piecemeal with an existing match. Maybe you want the vector check but not the map one. We should support both.

Improving the performance of your code should not require you to go require something else. Just modify your code in place.

totally get it. In my mind I'd opt out of all safety if that were an option b/c that's just what I'm used to -- if I feed in the wrong spec it's my fault. But not sure that's a healthy attitude haha. I mean, if I had to trade safety for clean notation and speed, that would be my vote

but maybe that's cause I come from a crazy Python/cljs background where that's always the case

It’s primarily unhealthy when the assumptions unchecked code makes are broken by code changes upstream.

yes, agree -- need to maintain backwards compatibility

So, from this perspective, Meander aims to be safe by default.

yep... 100% understand that, I thought we were discussing unsafe options

But as I point out in the issue there’s a huge performance savings to be had if you know there is virtually zero risk in eliding the check.

mhm. What I'm advocating for is that for a user like me, I'd be putting {:unsafe ...} next to EVERYTHING

Oh, well, no one wants that. 🙂

But to get to where we don’t have to do that, we actually have to start from that position where we do.

ahh excellent point

Verbosity is the first step toward abstraction.

!!!

I need to meditate on that one

Does meander have transducer/xfn variations of the m/* series yet? Would that be a welcome contribution?

I think m/search could possibly return a transducer instead of results. I’ve looked into this and requires a bit of work.

It would probably just be sugar on top of something like...

(comp 
  (map (fn [data] (m/search ...))
  cat)

The transducer version of mapcat throws a bit of wrench into the mix.

is that b/c of possible thrown exceptions?

No. It’s because an infinitely deep search space can prevent the transducer from ever returning anything.

It’s a solvable problem. Just like the last patch, it probably will involve a way of indicating that a something has a particular property that is salient and compilation can be adjusted accordingly i.e.

(let [[min max] (search-space-depth-range pattern environment)]
  (infinite? max))

aha

m/search itself could be infinite depending on the pattern?

Yes. At the moment, I don’t think this is really an issue but in the future there will be several patterns that have infinite spaces. For example,

(m/sum ?a ?b)

ohhhhh nice! I didn't know that was a thing

It isn’t yet.

I’ve been hacking on it on the side though and it will appear in the next version of the library.

that's fascinating -- I've been thinking about the synergy between meander and core.logic

Along with things like

(long ?min ?max)

mhm. Without looking at internals, I suppose converting that to a lazy sequence would be quite difficult?

because then an infinite sequence is no problem

Not especially. If we know either or both of ?min and ?max matching is pretty easy. If we know neither it’s also not so bad: we simply produce the space which is all possible combinations of long values such that (<= ?min TARGET ?max) where TARGET is the value being matched.

Well that's certainly an acceptable option as well

The concepts Meander is based on are very similar to the ones found in something like core.logic but not quite.

Essentially, matching/searching is unification with the exclusions that variables cannot be unified. IOW, variables must always be unified against a ground value.

It's all some dark magic to me 😄 I'm still reworking through the end of SICP section 4 where they wire up that stuff from scratch

What I'm not understanding is (and possible it's just b/c lacking theory) how is it possible to stop at an arbitrary number and not pause/continue the computation to make it lazy

Another difference, is that Meander has been evolving (I think) a very different concept of a variable that is much more like a Clojure var than how LP normally thinks of them.

The idea is that binding is essentially a reduction which can fail.

And unbinding or yielding is unfolding which can also fail.

This process is independent from whatever it’s name happens to be.

Interesting... I have much to learn. Well I think an easy low hanging fruit would be to target things that for sure terminate, like m/match

It’s too bad this is the free Slack. I’m sure many of the long time members in this channel are like, “yeah, we’ve heard this before…” 😛

:rolling_on_the_floor_laughing:

But, I’m always happy to share my thoughts if helps/confuses/inspires. 🙂

Super inspired! I love being the new guy haha.

Where did you get the idea for this, anyway? I can't find a single precedent like it. Closest I can see is datalog and core.logic

I mean I've seen other data transformation languages but none with the declarative/data-literal syntax

Replied on the card. Hopefully there is something helpful in there.

A couple things pushed me in this direction: • Pattern matchers • Logic Programming • Math notation • Data literals • Regular expression But I think the biggest one was noticing a lot of asymmetry in how we manipulate data. We take data apart differently from how we put it together.

I thought it would be cool to see if I could make something that really emphasized symmetry and played to our mutual strengths to recognize patterns.

The other thing I noticed is that, after programming in Clojure for almost 10 years, I’m not getting better at reading arbitrary Clojure programs. I still have to slow down and mentally unpack even modestly complex code.

So, yeah, I sort of feel like I’m always haunted by this question of “shouldn’t we be able to skip all this these days and more or less draw a picture of what we want?”

I saw @jimmy’s talk and it made me realize I'm not getting better at it either. The only thing that's increased is my code reading "pain tolerance" hahaha

It’s true!

the data literal syntax is very powerful because it plugs into our visual cortex system, which is infinitely more powerful than whatever systems process functional abstractions

I was actually really surprised that I didn't find a fully defined language based on this

Pattern matching, yes... but data transformation, no

I actually think there is a lot of excellent opportunity for serious academic work here, too

I always use regular expression as a point here. If you master the language, you often do not have the problem of wondering what an arbitrary regular expression means. So I’m emphasizing semantics here. I’m not saying there aren’t exceptions to this point but, in general, I think it’s fare to say regular expression has this property of remarkably clear semantics.

We do have this trouble with code. And this makes sense because code is meant to encode semantics in a general way.

think, for instance, matrix operations. Trying to find a single matrix that is equivalent, for instance, to the multiplication of three matrices.

Actually, there is a really cool Python matcher that can do that. ☝️

I would love to steal that technology.

The same analogy could be extended to meander transformations. What single meander pattern could be compiled from 3 meander patterns?

seems like a pretty interesting study to me

what are you referring to? I can help steal/translate 🙂

https://github.com/HPAC/matchpy

Do you have a contributor doc anywhere?

I do not. Mostly that is due to the fact I have to make compromises about my time.

But, also, if anyone wants to contribute I’m much better at an organic conversation that tells someone where everything is than I am translating that into a document.

I need a spirit medium like @jimmy for that sort of thing. 🙂

Fine by me! I'm thinking as a rough mockup of a match-xfn something like this:

;; psuedocode
(defpsuedomacro matcher [{ex-handler :ex-handler alt-nil :alt-nil :as opts} prop-list]
 (comp 
   (map (fn [x]
          (try
            (m/match x ~@plist)
            (catch Exception e
              (if (some? ex-handler)
                (or (ex-handler e) alt-nil)
                alt-nil
                )))))
   cat
   (remove #{alt-nil})))

I think the best thing to do if you have ideas is to share them on Github. @timothypratley and others have shared some fantastic ideas there.

Sweet! Will do

Ideas generally take a lot of thinking about and it’s nice having several on the table to see if there are common threads among them to be addressed with a unifying solution.

For example, we’ve had a lot back and forth about aggregation and that has led to the model of binding I mentioned earlier.

It’s not ready yet but I think it would have taken longer to come up with a solution without people sharing their ideas.

I think that’s is one of the things I’m most proud of being a member in this channel is the encouragement for people to discuss their ideas, criticisms, etc.

Awesome. Just need a clojure transliteration?

Am I misusing it, or it's cata doesn't work with & ?rest on a map?

(me/match {:a 1 :b 2}
  {:a ?a & (me/cata ?rest)}
  {:aa ?a :rest ?rest}

  {:b ?b}
  {:bb ?b}) ;; => stackoverflow!

Change it to (me/some ?a)

aaah!!

it works :bananadance:

It will still match with nil even if the key doesn't exist. This is based on some of clojures behavior and something we have considered changing in zeta.

We also have thought about detecting things like this in some sort of linting like mode to help people diagnose these problems.

is it worth to contribute this case to the doc? I think it's a pretty common

Definitely :)

I've also found keeping my base case as the first match helps prevent me from doing this.

emm ... maybe not, in my example above if I reorder the two clauses, the result would be short circuited by the {:b ?b} match and never gets to the other one

Yeah not always the case. Just a general rule of thumb. In this case you definitely can't do it.

sgtm!

In this case it does avoid the stack overflow and hints at the fact that the patterns are completely overlapping.