Clojurians Log v2

@pez i think i have something that might be worth trying out now -- have you used tree sitter at all? if not, i can write up some notes on how to use things. if you're already familiar with tree sitter stuff, you can probably just replace the grammar.js file in one or the other of the tree-sitter-clojure repositories with this: https://gist.github.com/sogaiu/a6c7a6dca4259480ca960c4f8a594019 (i used tree-sitter-cli 0.16.4)

No, never used tree sitter. And I'm not sure where I would plug it in, even if I think I have an idea about how it could fit in.

the tree-sitter cli provides a parse command which can measure execution time -- so one can at least get some sense by running it on individual clojure files.

My processing of core.clj takes 150 ms, btw. Measured with checking the clock before and after. I don't know how much of that is the parsing. There is quite a lot going on in that proceess.

ah thanks for the numbers. borkdude and i did some measurements with clj-kondo's parsing -- we both got about what you got.

ofc we are all using different machines though 🙂

if you're interested, i'll write up some instructions. i've also made a few sample vscode extensions that use the grammar.

I have a pretty speedy machine, even if it is a laptop.

i've been testing on a notebook as well.

I'd like some instructions, yes. Not sure when I will have time to act on it, but I'd like to do some experimentation to try figure out a bit more about what kind of work would be involved.

ok, i'll work on putting a repository together with instructions then.

If you want to do the measurement yourself, here's where I put my time stamps: https://github.com/BetterThanTomorrow/calva/blob/dev/src/cursor-doc/model.ts#L388

At entry of that function and at exit.

ty! i hope to take a look soon -- gotta sleep first though 🙂

Crazy TZ diffs! 😃

he he -- we can leverage the differences to get more work done sooner 😉

vscode Clojure has someone at the watch 24/7.

🙂

I'll try to time the rainbow paren parser as well. Even if that one is in no way a full parser.

I imagine tree sitter would be the thing returning my tokens here: https://github.com/BetterThanTomorrow/calva/blob/dev/src/cursor-doc/model.ts#L41

you can essentially get back a tree of nodes (https://github.com/tree-sitter/tree-sitter/blob/master/lib/binding_web/tree-sitter-web.d.ts#L50)

one way to use the results is to use tree sitter's tree traversal api. another way to use the tree is to construct a query to it and then get back matching info.

i also fed the root node to:

(defn node-seq
  [node]
  (tree-seq
    #(< 0 (.-childCount ^js %)) ; branch?
    #(.-children ^js %) ; children
    node)) ; root

it's a bit ugly, but here are two ways of doing the same thing:

(defn doc-syms
  [^js doc ^js tok]
  (if (= "clojure" (.-languageId doc))
    (if-let [uri (.-uri doc)]
      (if-let [^js tree (get @trees uri)]
        (let [id-nodes (find-id-nodes (node-seq (.-rootNode tree)))]
          (->> id-nodes
            (map (fn [id-node]
                   (make-fn-sym-info (.-text id-node)
                     doc
                     (make-range id-node))))
            clj->js))
        #js [])
      #js [])
    #js [])) ; XXX: how to reduce this repetition

;; uses tree sitter query api
(defn doc-syms-2
  [^js doc ^js tok]
  (if (= "clojure" (.-languageId doc))
    (if-let [uri (.-uri doc)]
      (if-let [^js tree (get @trees uri)]
        (let [q (.query @clj-lang
                  (str "((list "
                       "   (symbol) @head "
                       "   (symbol) @name) "
                       " (match? @head \"^def[a-zA-Z]*\"))"))
              ms (.matches q (.-rootNode tree))]
          (->> ms
            (keep (fn [match]
                    (when match
                      (when-let [caps (go/get match "captures")]
                        (when (= 2 (count caps))
                          (when-let [name-node (go/get (nth caps 1) "node")]
                            (make-fn-sym-info (.-text name-node)
                              doc
                              (make-range name-node))))))))
            clj->js))
        #js [])
      #js [])
    #js [])) ; XXX: how to reduce this repetition

those basically implemented provideDocumentSymbols

I think it would be nice to build Calva's token cursor from a parser that more projects cared about.

But I'm pretty fond of the token cursor as such, because it is very natural and intuitive to work with.

i'll have to study it to appreciate those observations 🙂

I might whip up a symbol provider using the token cursor and we can compare, for the fun of it. 😃

sounds good to me -- hope that fits in your schedule 🙂

Haha, well, not that I really have a schedule, but yeah, lots of things to do, no doubt.

i'm off for the evening -- good luck with your endeavors 👋

So this is how symbols would be collected using Calva's LispTokenCursor:

function docSyms(document: vscode.TextDocument): vscode.SymbolInformation[] {
    const cursor: LispTokenCursor = docMirror.getDocument(document).getTokenCursor(0);
    let symbols: vscode.SymbolInformation[] = [];
    do {
        cursor.forwardWhitespace();
        cursor.downList();
        cursor.forwardWhitespace();
        const token: Token = cursor.getToken();
        if (token.type === 'id' && token.raw.startsWith('def')) {
            while(cursor.forwardSexp()) {
                cursor.forwardWhitespace();
                if (token.type === 'id') {
                    symbols.push(makeSymbol(document, cursor));
                    break;
                }
            }
        }
        cursor.forwardList();
        cursor.upList();
    } while (!cursor.atEnd());
    return symbols;
}

I would implement symbols using nrepl, probably, but anyway, if a static method would be called for, the token cursor is very neat, imo.