Clojurians Log v2

Is there a way to disable or configure the addition of ns forms to files? It currently tries to write (ns main.foo.bar) where main comes from src/main/foo/bar.clj , and sometimes it seems to write the ns form multiple times. I'm using

clojure-lsp --version
clojure-lsp release-20200819T134828

Yeah, I already faced that issue, clojure-lsp tries to add a ns when creating a new file but I think clojure-mode also add this making a duplicated addition

I'll try to address a PR to make that optional 🙂

Actually, for emacs it's clj-refactor that adds a namespace already: https://github.com/clojure-emacs/clj-refactor.el/blob/master/clj-refactor.el#L53

i think he was using neovim in this case

Yes, but this new setting may work too 🙂

@snoe :reviewplease: https://github.com/snoe/clojure-lsp/pull/175

ah missed that part 🙂

Thanks for the help! I for some reason had glanced straight over that option in the readme 😅

Hey I just updated to the new versions with clj-kondo and my lsp is just hanging my pc (fans just going to full speed) and the starting lsp server never finishes. Do I need to add any special config anywhere, looks like it's trying to lint all depencies of my project.

@rafaeldelboni yeah I've run into that with large classpaths, working with kondo to fix it but for now you might have to use an old release https://github.com/snoe/clojure-lsp/releases/tag/release-20200805T150718

Thanks I will rollback

@snoe what more do you need from clj-kondo to fix this?

I think if it's affecting others it'll be a kondo release, but even after the vec changes I managed to run into a stackoverflow yesterday. Still investigating from my end but I think it'll be a change like you said to use transducers or recur in the recursion

One other option is to not use any return values from the linted libraries. I think most things are written to some local atoms anyway

I'll make an issue for it

https://github.com/borkdude/clj-kondo/issues/972

I also want to look into parallel linting, that would maybe make it easier as well

One workaround in clojure-lsp might be to not lint the entire classpath at once, but maybe in groups of 10 libs or so in a doseq

ah good idea:bulb:

Yeah, we could try to do that, as we are not using the kondo analysis data, we could only lint the current file/text

WDYT @snoe?

You can also make linting the entire classpath optional and then do it in groups of 10

The cache gives us signature diagnostics

ah ok

(and I want to add the cross-ns unresolved symbol lint)

:thumbsup:

> (and I want to add the cross-ns unresolved symbol lint) Yeah, with that we can only use kondo for almost every analysis

Shouldn't be too hard to get in

Yes, I started a branch doing that, but is not that easy haha

https://github.com/snoe/clojure-lsp/pull/176

Feel free to help with that @snoe 😉

Oh :) I think it can be done entirely in impl/linters.clj in clj-kondo, as an optional linter. Already discussed with @snoe in #clj-kondo I think

Right now it only warns about unresolved symbols from the current namespace. It's only a small step to extend that to other namespaces probably

oh wow @ericdallo nice start

Thanks! Yes @borkdude, It would be really cool to left all analysis to kondo

It would be nice to have some repro of the out of memory or stackoverflow issue

E.g. post the classpath in the issue

@borkdude mind if i dm it to you? not quite comfortable sharing publicly

sure!

It might be one specific jar file. Since I can't reproduce it by concatenating one directory 10k times:

$ clj-kondo --lint $(bb -e "(println (str/join \":\" (repeat 10000 \"src\")))")
linting took 3670ms, errors: 0, warnings: 0

looking a bit I think running in a go block in lsp probably contributed to the stack size

Linted twice successfully with the released clj-kondo and the cp snoe gave me: > linting took 43652ms, errors: 2176, warnings: 3996

@snoe do you think it's really clj-kondo that is causing that slowness? My guess is that the jars crawler that are causing these issues

so the jar crawler only happens once and hashes your project file. kondo always runs in the released version of lsp. But yeah i've isolated it to kondo locally and like I said it's inconsistent when the overflow happens but I'm also manipulating how lsp calls it so I suspect we're just on a threesshold.

Hum, I see

@borkdude if I could get a release of kondo with https://github.com/borkdude/clj-kondo/pull/964 I could push an lsp release that at least is much more consistent for people and I'll just keep an eye on the new issue and try to recreate.

@snoe What about dependending on "2020.07.30-20200826.174236-11". It is a SNAPSHOT, but it won't change from under you

I'd rather revert the change really and fix it with a proper repro, so we know what we are doing

I haven't seen this issue with anyone using clj-kondo in its 1.5 year lifetime and it's used a lot

is that in maven? then that works for me

yup that works! thanks

yes, every SNAPSHOT has a "real" version that is based on the date that it was pushed

I did some analysis on the calls to process-file in clj-kondo.impl.core and the stack depth isn't any deeper than 1 really, so I don't see how vec helps there

it only recurses for jars and dirs, but it's only one level

However in analyzer.clj we do a deep recursion for nested forms, so if there's a very deep call, it may cause the issue. I'll try to repro that

No problem with 300 nested calls. And I've never seen an expression that big in any real code

@rafaeldelboni can you try the new release? https://github.com/snoe/clojure-lsp/releases/tag/release-20200827T175529

First run I get:

INFO  clojure-lsp.crawler: clj-kondo scanned project classpath in 69024ms
DEBUG clojure-lsp.main: :initialize 105825 ()

INFO  clojure-lsp.crawler: skipping classpath scan due to project hash match
DEBUG clojure-lsp.main: :initialize 9707 ()

@rafaeldelboni you can get the logs from /tmp/lsp.out

Do you have any stacktraces / dumps of when the error happened?

unfortunately no. just got the error message, i was throwing away the trace

can you try to reproduce with the change reverted?

On a big project... first run I get:

INFO  clojure-lsp.crawler: clj-kondo scanned project classpath in 46185ms
DEBUG clojure-lsp.main: :initialize 101477 ()

INFO  clojure-lsp.crawler: skipping classpath scan due to project hash match                                                                                                                  
DEBUG clojure-lsp.main: :initialize 14369 ()   

Meanwhile I'll work on a change where no return values are needed anymore. I'm only using it for a small thing which can be refactored

so we can just use run!

So before the lazy-fix there actually wasn't a stack trace:

(catch Throwable e
  (log/info "pre kaboom ")(.printStackTrace e)                        (log/error e "kaboom"))

INFO  clojure-lsp.crawler: pre kaboom
ERROR clojure-lsp.crawler: kaboom
java.lang.OutOfMemoryError: GC overhead limit exceeded

Hmm, GC overhead. In clj-kondo.core there is a doall around all of this mapcat stuff, so it would get realized anyway. Not sure how vec would have fixed that?

I think it's cause concat/mapcat are creating nested linked-lists that can are building more objects than you want

I think i linked to the blog post in the pr that explains it better than i can

ok

And whatever graal does must be better at cutting down on number of objects or something like that

hmm

my hope is that if we switch to kondo for analysis we can then work on moving lsp to graal too

I'm using the JVM for clj-kondo.lsp which is used in Visual Studio Code

but it can also be used with emacs probably

it was a great alternative for Windows when GraalVM didn't work there

and people don't have to install anything as long as they already have java

just download the VSCode plugin, done

ah, good to know

Really good idea indeed

Looking good now First Run:

INFO  clojure-lsp.crawler: clj-kondo scanned project classpath in 38472ms
DEBUG clojure-lsp.main: :initialize 86873 ()

INFO  clojure-lsp.crawler: skipping classpath scan due to project hash match
DEBUG clojure-lsp.main: :initialize 6413 ()

@snoe Version "2020.07.30-20200827.215744-12" should be even better. I now avoided all map(v) and mapcats when processing files, there's only run! over files now

Less garbage

This is the version that corresponds to master @ 157cc964a180432c27c447cb32d45bac7f391c0f

thanks!