Clojurians Log v2

So I played with this a bit but couldn't get it directly working. The reason scan doesn't work here is that it assumes all its elements are sequential.

(m/search [1 2 3 4 5 6]
  (m/scan 3 4 ?x)
  ?x)

;; =>

5

(m/search [1 2 3 4 5 6]
  (m/separated 3 4 ?x)
  ?x)

;; => 

5 6

thanks, I think I understand scan a little better now. Played around a bit more and ended up with:

(defn xml->tracks
  [xml]
  (m/search
   xml
   {:tag ::tx/tfmDataService
    :content ({:tag ::tx/fltdOutput
               :content (m/scan
                         {:tag ::fdm/fltdMessage
                          :attrs {:msgType "trackInformation"
                                  :flightRef ?flight-ref
                                  :acid ?aircraft-id
                                  :airline ?airline
                                  :depArpt ?departure-airport
                                  :arrArpt ?arrival-airport
                                  :sourceTimeStamp ?message-ts}
                          :content ({:tag ::fdm/trackInformation
                                     :content (m/scan
                                               {:tag ::nxcm/qualifiedAircraftId}
                                               {:tag ::nxcm/speed
                                                :content (?speed)}
                                               {:tag ::nxcm/reportedAltitude
                                                :content ({:tag ::nxce/assignedAltitude
                                                           :content ((m/or
                                                                      {:tag ::nxce/simpleAltitude
                                                                       :content (?altitude)}
                                                                      {:tag ::nxce/blockedAltitude
                                                                       :attrs {:min (?altitude)}}
                                                                      {:tag ::nxce/visualFlightRules
                                                                       :attrs {:altitude ?altitude}}
                                                                      {:tag ::nxce/altitudeFixAltitude
                                                                       :attrs {:preFixAltitude ?altitude}}))})}
                                               {:tag ::nxcm/position
                                                :content (m/scan
                                                          {:tag ::nxce/latitude
                                                           :content ({:tag ::nxce/latitudeDMS
                                                                      :attrs ?latitude-dms})}
                                                          {:tag ::nxce/longitude
                                                           :content ({:tag ::nxce/longitudeDMS
                                                                      :attrs ?longitude-dms})})}
                                               {:tag ::nxcm/timeAtPosition
                                                :content (?track-ts)}
                                               .
                                               (m/or
                                                {:tag ::nxcm/ncsmTrackData
                                                 :content (m/seqable
                                                           .
                                                           (m/or
                                                            {:tag ::nxcm/nextEvent
                                                             :attrs {:latitudeDecimal ?next-lat
                                                                     :longitudeDecimal ?next-long}}
                                                            (m/let [?next-lat nil ?next-long nil]))
                                                           ..1)}
                                                {:tag ::nxcm/ncsmRouteData
                                                 :content (m/scan
                                                           {:tag ::nxcm/nextPosition
                                                            :attrs {:latitudeDecimal ?next-lat
                                                                    :longitudeDecimal ?next-long}})}))})})})}
   (merge {:flight/ref (Long/parseLong ?flight-ref)
           :flight/aircraft-id ?aircraft-id
           :flight/airline ?airline
           :flight/arrival-airport ?arrival-airport
           :flight/departure-airport ?departure-airport
           :track/speed (Integer/parseInt ?speed)}
          (when (and ?next-lat ?next-long)
            {:track/next-latitude (Double/parseDouble ?next-lat)
             :track/next-longitude (Double/parseDouble ?next-long)}))))

Not especially happy with the last bit, but haven't been able to figure out a better way that doesn't generate 'extra' entries. What I'm trying to express is that the element after timeAtPosition will be either a ncsmTrackData or ncsmRouteData tag, but if it's a ncsmTrackData it may or may not contain a nextEvent with a lat/long.

@chuck.cassel I’m gonna take a swing at this. 🙂

if you haven't already I'd really appreciate a review of where I've gotten since then to see if it makes sense or there's a better/more efficient/performant/etc way to go about it. I just updated the https://gist.github.com/casselc/09b1ed46a86b500cabd2e14ada1a3719 with a copy/paste from my ns where I've been playing around with the data

thanks for taking a look

Do you have a deps.edn to go along with this?

I’m pulling together the deps myself.

{:paths ["src" "resources"]
 :deps {org.clojure/clojure {:mvn/version "1.10.1"}
        org.clojure/data.xml {:mvn/version "0.2.0-alpha6"}
        org.clojure/core.async {:mvn/version "1.3.610"}
        environ {:mvn/version "1.2.0"}
        com.solacesystems/sol-jcsmp {:mvn/version "10.9.0"}
        meander/epsilon {:mvn/version "0.0.480"}
        datascript {:mvn/version "1.0.0"}
        clojure.java-time {:mvn/version "0.3.2"}
        }
 }

This is actually pretty incredible, I must admit.

I think this is the first time I’ve seen someone use defsyntax this way.

that can't be good

No it’s great actually.

I’m just now realizing that it I need to expose something for expanding a pattern conveniently.

I initially built it up as one giant find just trying to basically transliterate the patterns from the xml until it all sort ofworked, and then wanted some way to break up the individual pieces

That makes perfect sense.

eventually I'll want to do some other message types and certain elements are reused

Yah, and that’s a totally valid use case. I’m glad you’re messing with this.

I actually have good xsd for all of this so there's probably some way to meander that into exactly what I want

I'm fairly new to clojure altogether and have just been wrangling around with this data as a way to experiment/learn with something concrete

Nice. Actually, I think you’d be the second person to join the channel in as many weeks who is both new to Clojure and to Meander. I couldn’t ask for better perspectives. 🙂

so far meander's describe the shape you have and the shape you want approach is my favorite clojure thing I've found

I’m really happy to hear that.

This example is pretty interesting. I’m going to toy around with it a little bit. I think it’s exposed some opportunities for improvement too.

what I'd like to make is something where I could essentially write the matching/extracting patterns in a less verbose/hiccup-style syntax and expand that into the actual xml pattern prior to matching

but wasn't quite sure how to get there from here and started working on something else

Do you have a sketch of what’s in your mind?

I guess I'd like to be able to write a pattern something like this to match that data:

[::fdm/fltdMessage {:msgType "trackInformation"
                    :flightRef (m/app Long/parseLong ?flight-ref)}
 [::fdm/trackInformation ; implicit separated(?) of content
  :start-of-seq ;made-up notation to anchor start of seq
  [::nxcm/qualifiedAircraftId]
  [::nxcm/speed (m/app Integer/parseInt ~speed)]
  [::nxcm/position
   [::nxcm/latitude
    [::nxcm/latitudeDMS {m/app dms->decimal ?latitude}]]]
  (m/or
   [::nxcm/ncsmTrackData ?some-data]
   [::nxcm/ncsmRouteData ?some-data])
  :? ; made-up notation for 0 or 1 of the preceding element. Implicit nil for bound vars?
  :end-of-seq]]

that conflates a few things I've been bumping into, but hopefully gets the idea across

I struggled (I still struggle, but I used to, too) quite a bit with really grokking the semantics of the sequence operators/notation and optionality. With next lat/long for instance, trying to express that if it was present it would be in one of two mutually exclusive elements, but might not be there at all.

Eventually I scrolled far enough in the cookbook to see the optional value stuff and felt less alone at least

(m/rewrite xml-edn
  {:tag ?tag, :attrs ?attrs, :content (m/seqable (m/cata !xs) ...)}
  [?tag ?attrs . !xs ...]

  ?x
  ?x)

Thanks, sorry to take so long to get back, hadn't had a chance to work on this project... Hopefully I'm not being obtuse about this. I was hoping to use the hiccup syntax as a sort of DSL to write the matching/capturing patterns. I was thinking I'd wrap the hiccup in some helper function/macro and use that wrapped form directly in the match pattern position for find/search/rewrite/etc and the helper would transform my hiccup-syntax pattern into it's map representation while preserving any logic variables, memory variables, or other operators in the pattern). i.e. I'd write something like:

(m/find example
          (from-hiccup-pattern
           [:message {:type "X"}
            [:detail {:field ?y}
             [:first (m/app Float/parseFloat ?content)]
             (m/$ [:second-nested {:q ?q}])]])
          {:y-val ?y
           :q-val ?q
           :parsed-content ?content})

(m/find example
        {:tag :message
         :attrs {:type "X"}
         :content (m/seqable {:tag :detail
                              :attrs {:field ?y}
                              :content (m/seqable
                                        {:tag :first
                                         :content (m/seqable
                                                   (m/app Float/parseFloat ?content))}
                                        (m/$ {:tag :second-nested
                                              :attrs {:q ?q}}))})}
        {:y-val ?y
         :q-val ?q
         :parsed-content ?content})