Clojurians Log v2

is there way to find all non-local bindings of a clojure form? For example:

(non-local-bindings
  '(fn ([{:keys [items ks]}]
        (let [foo :bar]
          (conj [foo] (not.locally/bound))))))
;; should return:
;; => #{clojure.core/fn clojure.core/conj not.locally/bound}

@denik my first hunch is to use (keys &env) and filter out everything in that

(what remains is not local)

@noisesmith since the form is quoted that wouldn’t work or would it?

it would require jumping through some hoops if it's possible, I don't know anything directly in clojure that answers your question - maybe some feature of tools.namspace?

Nope

sorry I was thinking of tools.analyzer and mis-remembered the name (answer might still be no)

Tools analyzer would be the thing, the general term for this is closed/free

Well I guess free, you are looking for the free variables in an expression

you could make something with tree-seq I bet, but you'd need to hard code every binding form to make it work

Which toops.analyzer will do, and a ton more

Nah, tree-seq won't work

@denik you could also use clj-kondo which will report non-locals as unresolved symbols

You need to analyze in the lexical context, which tree-seq will peel away

my thought was the "children" function could return an empty list for non-binding forms, and a list with locals filtered otherwise

but that might be flawed

In general it is not an easy analysis to do, if you can avoid it

It is really impossible before macroexpansion and after macroexpansion things like fn are gone, replaced with fn*, which as a special form is usually not considered free

$ cat /tmp/example.clj
(fn ([{:keys [items ks]}]
     (let [foo :bar]
       (conj [foo] (not.locally/bound)))))

$ clj-kondo --lint /tmp/example.clj
/tmp/example.clj:1:15: warning: unused binding items
/tmp/example.clj:1:21: warning: unused binding ks
/tmp/example.clj:3:21: warning: Unresolved namespace not.locally. Are you missing a require?

The clojure compiler sort of analyzes names to determine which of three states it is in: free (a free name that doesn't resolve to a var is an error) closed over (compiles to an instance field lookup) or local (gets a slot in the methods stackframe)

I’m already using sci 😛 this will a borkdude-heavy project 😄

I have some code from a sort of replacement for core.async's go macro I was writing that uses tools.analyzers postwalk-transforms feature to collect free variables https://gist.github.com/hiredman/5644dd40f2621b0a783a3231ea29ff1a#file-yield-clj-L655-L679

@borkdude is there a way to use clj-kondo on forms directly

@denik yes, let me create an example

also using sci, so if they share a ctx I could plug that in

@denik

(require '[clj-kondo.core :as clj-kondo])

(def example '(fn ([{:keys [items ks]}]
                   (let [foo :bar]
                     (conj [foo] (not.locally/bound))))))

(def findings
  (:findings (with-in-str
               (pr-str example)
               (clj-kondo/run! {:lint ["-"]}))))

(require '[clojure.pprint :as pp])

(pp/pprint findings)

[{:type :unused-binding,
  :filename "<stdin>",
  :message "unused binding items",
 ...

@denik Feel free to drop by in #sci if you have questions

thank you. looks like the symbols pat of a string. I’ll try to find lower-level fns that return the symbols

@denik we could add that to the data. I think that could be useful too.

what are you trying to accomplish with the symbol once you find it?

I’m storing function bodies in datalevin, example here: https://clojureverse.org/t/datalevin-powering-environment-and-runtime/7243

and want to resolve symbols from the db and replace them with something that is invocable

also wondering if ctx can be shared between the analyzer (kondo) and sci (evaluation)

@denik What you can do maybe is: use sci/parser + sci/parse-next which will get you the function s-expression. Then you can do some processing on that (postwalking, replacing) and then you can feed that to sci/eval-form

This is also how you can implement a REPL in sci: https://github.com/borkdude/sci#repl But you can do the processing step in between the parsing and evaluation.

that works for parsing into edn form but how would it solve the local-bindings problem?

(sci.impl.parser/parse-next
  sci-ctx
  (sci.impl.parser/reader (str '(fn ([{:keys [items ks]}]
                                     (let [foo :bar]
                                       (conj [foo] (not.locally/bound))))))))
=> (fn ([{:keys [items ks]}] (let [foo :bar] (conj [foo] (not.locally/bound)))))

It doesn't solve that part, but that's the place to potentially fix it. Why would you store s-expressions with unresolved vars/locals?

exactly, I wouldn’t! But since I store forms in a database (not namespaces) I need to find unresolved symbols in the db and replace them with their value and throw and error otherwise

is an unresolved symbol a local or a var in your problem? and is it namespaced? and what would you replace it with?

I'm trying to understand the problem, not entirely clear to me yet

I’ll do my best to explain! If I can use sci’s ctx it is a symbol (namespaced or not) that is not available in ctx’s :namespaces

And how do you know what to replace it with?

yes I have a function that inlines the form

but how do you know what to replace with what? can you give an example?

sure,

(snipf all-tweets
       []
       (->> (datoms :aev :com.twitter/tweet-text)
            (mapv (comp entity :e))))

; =>
{:var cells.lab.code-db/all-tweets,
 :id "1MCsHLjz56",
 :created-at #inst"2021-02-26T22:19:12.408-00:00",
 :updated-at #inst"2021-02-26T22:19:12.408-00:00",
 :form (clojure.core/fn [] (->> (datoms :aev :com.twitter/tweet-text) (mapv (comp entity :e)))),
 :db/id 38}

(snipf all-tweets-ui
       []
       (into [:div]
             (map (fn [{:com.twitter/keys [tweet-text tweet-author]}]
                    [:div (str tweet-text " by " tweet-author)]
                    ))
             (cells.lab.code-db/all-tweets)))

; =>
{:var cells.lab.code-db/all-tweets-ui,
 :id "1YiWEQrLlG",
 :created-at #inst"2021-02-26T22:19:28.533-00:00",
 :updated-at #inst"2021-02-26T22:19:28.533-00:00",
 :form (clojure.core/fn
         []
         (into
           [:div]
           (map (fn [#:com.twitter{:keys [tweet-text tweet-author]}] [:div (str tweet-text " by " tweet-author)]))
           ((clojure.core/fn [] (->> (datoms :aev :com.twitter/tweet-text) (mapv (comp entity :e))))))),

all-tweets-ui uses all-tweets which has been added to the db through the snipf macro. we can see that the form of all-tweets-ui inlined the form of all-tweets

so it's more or less a dependency problem?

dependency-resolution, yes. anything that is not provided in sci’s context will have to get inlined from the db. if it doesn’t exist, it should throw

I think you should solve this differently and store some information on which other vars the var depends and load those first

you can possibly store "require" expressions along with the fn expressions, to ensure the namespace gets loaded first

hmm, it’s less about loading and more about existence of a form for a given var in the database. it would be great if this could be figured out during analysis

in sci you could try to eval the form, catch the exception and try to load the other form with data from the exception, maybe

none of the forms should need to be evaluated to know whether some of their contained symbols need to be replaced

evaluating a fn expr is more or less the same as analyzing the fn body

am I wrong in thinking that the sci-ctx and a form-walker that ignores local bindings should be enough to do this?

sci does have a separate analysis step but this is not exposed. you will still get evaluations for macros that are expanded for example, so it isn't guaranteed to be side-effect free

hmm ok time to ponder this for a bit. thanks so much for being helpful!

> am I wrong in thinking that the sci-ctx and a form-walker that ignores local bindings should be enough to do this? this is more or less what happens during analysis

I looked at it earlier in sci and unfortunately the analyzer closes over function arities so that I cannot inspect them

what you can inspect is the parsed form

can I?

(sci.impl.analyzer/analyze sci-ctx
                             '(fn ([{:keys [items ks]}] items))
                             )
#:sci.impl{:fn-bodies [#:sci.impl{:body [#object[clojure.lang.AFunction$1 0xfbaf833 "clojure.lang.AFunction$1@fbaf833"]],
                                  :params [p__46761],
                                  :fixed-arity 1,
                                  :var-arg-name nil,
                                  :fn-name nil,
                                  :arglist [{:keys [items ks]}]}],
           :fn-name nil,
           :arglists [[{:keys [items ks]}]],
           :fn true,
           :fn-meta nil}

this is the hack that currently works:

(defn- local-syms [params]
  (set
    (sp/select
      (sp/walker symbol?)
      params)))

(defn- resolve-arity-tail [arity-tail]
  (let [params     (->> arity-tail :params :params (mapv second))
        local-syms (local-syms params)
        body       (-> arity-tail :body second)]
    (apply list params
           (clojure.walk/postwalk
             (fn [x]
               ;; FIXME need something that ignores all local bindings
               (if (symbol? x)
                 ; fixme should resolve from sci' ctx
                 (if (or (local-syms x) (resolve x))
                   x                                        ; provided
                   (if-let [{:keys [form]} (get-db-var (namespace-sym x))]
                     form
                     ;; FIXME
                     (do (println (str "Couldn't resolve " x ", assume locally bound"))
                         x)
                     #_(throw (ex-info (str "No :form found for " x) {}))))
                 x))
             body))))

(defmacro snipf [name & f-bodies]
  (let [[arity# tail-or-tails#] (s/conform ::clj-specs/fn-tail f-bodies)
        ari# (case arity#
               :arity-1 (resolve-arity-tail tail-or-tails#)
               :arity-n (map resolve-arity-tail tail-or-tails#))
        f#   (conj ari# `fn)]
    `(db/transact-entity!
       code-conn
       {:var  (~namespace-sym '~name)
        :form '~f#})))

but it would not catch vars that don’t exist in either the sci ctx or the db at def-time

with parsed, I meant what you get back from parse-next. the analyzer is an impl detail, the output from that isn't meant for third party consumption

So what is the use case of saving arbitrary snippets of code out of context in the db? Can you add the requirement that people can declare some kind of dependency so the function snippets become more standalone and easier to execute?

I'm afk now

it’s a little cumbersome but that could be done. I thought the namespaced symbol itself could be that requirement

this would work if it was possible to pass a resolve-symbol function https://clojurians.slack.com/archives/C06E3HYPR/p1614378661043400?thread_ts=1614373036.036400&cid=C06E3HYPR

https://github.com/borkdude/sci/blob/master/src/sci/impl/evaluator.cljc#L417