Clojurians Log v2

@slipset Hope I'm not messing up any pending commits you might have while I blast several changes into Eastwood repo today... Finding a few small things and fixing them (I hope).

@andy.fingerhut No problem! I’m just hugely grateful for all the help I’m getting from you.

I have some more stuff on the way wrt cleaning up the callback code.

I know I’ve asked this before, but say I have a project which contains two namespaces.

Would it be possible to do something like:

(let [f1 (future (lint-and-analyze ns1 linters)
        f2 (future (lint-and-analyze ns2 linters)]
    (results @f1 @f2))

My main question being, would the analyzer be tripping over itself because it’s doing analysis on two ns’s in parallell?

unforunately, because namespace loading in clojure is side-effectful , this is not possible to do in the general case

Imagine (ns foo) (def a 1) (ns bar) (def a (resolve 'foo/a))

which unfortunately is valid clojure

Hmm, but would stuff break, or would I just end up analyzing both ns’s in both futures?

I guess my end goal would be to analyze several ns’s at the same time. I wouldn’t mind getting the occasional penalty from analyzing the same ns in two threads, but I wouldn’t want the two threads to mess up each others analyzis.

If you see what I’m getting at?

analysis is not the problem, t.a in theory has no issues analysing two namespaces in parallel

the problem is that clojure makes no guarantees that all the namespace dependencies will be declaratively defined in ns

so side-effecty stuff in the namespace could depend on a particular order of loading namespaces

and given that namespace loading is not transactional in clojure, if we mess up and accidentally load in the wrong order, we might end up with 2 broken namespaces, which could lead to a snowball effect of further namespaces breaking down the line

would this comment in eastwood have something to do with that situation?

;; Create all namespaces to be analyzed.  This can help in some
       ;; (unusual) situations, such as when namespace A requires B,
       ;; so Eastwood analyzes B first, but eval'ing B assumes that
       ;; A's namespace has been created first because B contains
       ;; (alias 'x 'A)
       (doseq [n namespaces]
         (create-ns n))

no

Ok.

I'm not saying it's never possible to do this parallel loading, in fact it's likely the case that for > 80% of the clojure projects it could be for a certain subset of the ns graph

just that there's no clear way to know whether it's possible or not, doing analysis

other than trying and catching errors

So a :parallel true option would be nice 🙂

yep that's what I was going to suggest

possibly true vs a vector of top-level namespaces?

But out of curiousity. It seems like the order eastwood processes ns’s is somewhat arbitrary?

IIRC it's the order derived by tools.namespace's topological sort?

Hmm, doesn’t seem like there is any sorting?

https://github.com/jonase/eastwood/blob/master/src/eastwood/lint.clj#L417

hmm maybe it does.

Probably here:

https://github.com/jonase/eastwood/blob/master/src/eastwood/lint.clj#L320

yeah I think that's it

I guess it would be nice that the fact that namespaces has some sort of order was somewhat clearer in the code.

I believe the :namespaces option the user can provide does not really have an order, even though it is a vector. That is, they are sorted in dependency order regardless of the order the user provides them. IIRC

So I see that in nss-in-dirs we get the namespaces in a toplogical sorted order (from extracting :tools.namespace.tracker/load from the tracker map`)

But then we seem to bash the test-ns’s and source-ns’s together. How are we sure that the correct order is preserved?

It should do the topological sorting on the combination of source-ns's and test-ns's, I thought. Looking...

ok, I probably said something wrong earlier -- perhaps Eastwood does honor the order the user provides in the :namespaces vector. Within that vector, if it finds the keyword :source-paths it replaces it with a topologically sorted sequence of source namespaces, and independently also does that for :test-paths. If that description is correct, then you could often mess up Eastwood's linting by specifying :test-paths earlier than :source-paths.

It would still work in cases where loading the namespace twice in the same JVM, without unloading it in between, gave the same results, because even if you do not lint a namespace, the namespace can still be loaded by Clojure normally via eval'ing ns and require statements.

FYI, regarding magnitude of false positives in the current version of the new :implicit-dependencies linter, I have been updating lots of project versions in the crucible set of projects, and one, carmine, has 1003 such warnings with the latest Eastwood code. I suspect most of them are probably noise, but haven't looked at them in any detail.

@slipset You should regard this recent commit of mine with suspicion -- I have tested that it achieves the effect I had in mind of printing stack traces of exceptions thrown by linters (as older versions of Eastwood did, but 0.3.1 and maybe some other versions don't), but it may clash with your plans for the callback restructuring.

https://github.com/jonase/eastwood/commit/5bf325469ed98801ba4d52e1efb54fe62a5b2be3

I would understand if maybe you'd prefer the default behavior not to spew all that stuff to the output when Eastwood code throws an exception, but there ought to at least be a debug flag to enable it, if it is not the default.

I’ll take a look. I’ll push a branch later tonight CEST to show where I’m heading.

https://github.com/jonase/eastwood/tree/remove-callbacks