Clojurians Log v2

Having the same issue 😕 @stefan.toubia were you able to fix it?

the code for the plugin is here: https://github.com/borkdude/clj-kondo.lsp maybe you can run using a local version with some debug statements in it. Without any repro I can't help

I wish I knew what causes it 🙂 It works fine for hours but then suddenly some java process starts eating all CPU. And it stays even after I quit VSCode. After restart everything is fine. I’m trying to edit the same files, but it works ok.

using jps you can view which pid is related to clj-kondo

17481 clj-kondo.lsp-standalone.jar

and you can kill it of course, but other than that, I don't know. VSCode should be killing the server itself

so it might as well be a bug in VSCode

the source code of the language server itself hasn't been changed in over 4 months

Here's a similar bug: https://github.com/forcedotcom/salesforcedx-vscode/issues/968

I’ve tried to quit vscode now and kondo process was killed as well You know, maybe it is killed every time, but when it is in “100% CPU” mode it takes more time, will wait next time :thumbsup:

hmmm, I’ve just realized that I ran test via lein doo not sure how they work, I didn’t do their setup, but I see that it copies a lot of files to target/cljsbuild/public/... folder during start phase. Could it be the reason?

And their content, well, is a copy. Can it confuse kondo? “Oh there are more than 1 definition”, somthing like that

clj-kondo only lints files you are editing

ok, will keep an eye on it thanks:+1:

which OS are you on?

could it be some virus scanner thing maybe

macOS 10.14.6 no AV software installed It happens in the evening. Maybe OS schedules something like indexing (not sure why it affects kondo/java though), maybe something happens just b/c it works for a day and smth like memory leakage is happening

What we can do is put a bunch of debug statements in the plugin, make a vsix package and you watch the clj-kondo output panel

it should already output that it starts (and stops, but it's likely you won't see that since it only happens on workspace change )

Let’s wait, I will avoid running tests on my machine. For some reason I have a feeling that it is somehow connected. Unfortunately I don’t remember the exact day when I ran them for the first time and when the issue started to appear.

I have to leave right now but I have not found anything useful yet

clearing the config didn’t help, eventually it just gets back to a bad state and I have to kill the process

I’m also running tests fairly often in editor, I wonder if there’s any relation

I haven’t seen anything helpful in the output yet

thanks for the link @borkdude that’s a good lead, I’ll read through and see if it helps

actually something else I noticed is the CPU for the clj-kondo extension is shooting through the roof as I type as well. That first spike was just typing a bunch of ((((((())))))), That last block I just wrote a tiny function and the CPU stayed high for 30s after I stopped

would definitely be helpful to get more verbose output on the language server, will check back in if I find out more

also, jps shows two clj-kondo processes. Only one of them is misbehaving

I do see CPU activity as I type, which is expected, but nothing abnormal. do you use one or multiple workspaces at the same time?

Can you also try disabling Calva and see if you can trigger the same problem?

@stefan.toubia @il.tmfv Added a bunch of debug statements which are visible in the clj-kondo output pane

Please install this VSIX

and keep me posted

I made the VSIX from this branch, fyi https://github.com/borkdude/clj-kondo.lsp/tree/debug

to build yourself:

script/build-server
cd vscode-extension && vsce package

oh very good point I forgot I had two workspaces open

thanks for the updates extension, I’ll test and see if I get any useful log info

No issues today 🙂 But I worked on another part of the app

I also have had no more issues :man-shrugging: but I’ve been doing the same thing

Ok so this issue has come up again, I’ve gotten to a state where I have high clj-kondo CPU usage and I’m getting this in the clj-kondo debug output, even without touching anything in vscode it seems caught in a loop where this is output every couple seconds

[Error - 1:12:24 PM] java.io.PrintWriter@258c7635
changed file, linting: file:///Users/stoubia/src/liftoff/clj/lib/user/src/liftoff/user.clj
found config dir at /Users/stoubia/src/liftoff/.clj-kondo
[Error - 1:12:32 PM] java.io.PrintWriter@56aa5236
changed file, linting: file:///Users/stoubia/src/liftoff/clj/lib/user/src/liftoff/user.clj
found config dir at /Users/stoubia/src/liftoff/.clj-kondo
[Error - 1:12:41 PM] java.io.PrintWriter@6c52e3cd

aha, this is interesting, thanks

can you force the error by visiting a certain file inside that project?

hmm, it seems like it wants to print an error but somehow calling str on PrintWriter is printing the class https://github.com/borkdude/clj-kondo.lsp/blob/4be61735ce538814fd9e66e6b0159d35d21ccada/server/src/clj_kondo/lsp_server/impl/server.clj#L67

Let me see if I can improve that bit

@stefan.toubia @il.tmfv Please install this updated version. It won't fix the problem, but it fixes error reporting, so then we will probably discover what's going on next time

The branch off which this is built is debug, here: https://github.com/borkdude/clj-kondo.lsp/tree/debug

The error might be this one: > (throw (Exception. (str cfg-dir " must be a directory"))) so is there a .clj-kondo file in your project which is not a directory maybe?

Ok thanks will give this a try

something else I noticed I think the error message snippet I gave was only part of the picture, it seemed related to the calva output.calva-repl file as it kept alternating between that file and another file I had open

maybe next time try disabling Calva to see if it's triggered by that extension? it has happened before

anyway, it seems we're getting closer

I’m running this build now. Seems I am working with things today that triggers the clj-kondo+calva combo to freak out.

thanks!

Could it be that the output window gets some funny data printed to it?

did you install .91?

.90 had funny output, that should be fixed with .91

Yes. .91. I was speculating about wether the contents of the Calva output could be what triggers the condition. But maybe not, since we have disabled all linters for it…

you disabled the rules, but clj-kondo is still linting it

can you post (maybe in a gist) the problem that clj-kondo prints to its output panel?

it should say somethig like [Error ..] blabla in case of an exception

This may be the cause of this CPU spike, if calva keeps printing to the REPL and triggers clj-kondo. But please post output so we know for sure.

We can hard code the Calva output file name in the clj-kondo plugin so it's just always ignored

but first proper diagnosis

You can maybe replicate this scenario by printing output to the REPL every second or so

and then see what happens

So far no errors and my computer is cool.

Do you see any output in the clj-kondo output panel. Like, is it trying to lint the REPL output?

Yeah, it prints when the files are changed.

changed file, linting: file:///Users/peter/Projects/app/.calva/output-window/output.calva-repl
found config dir at /Users/peter/Projects/app/.calva/output-window/.clj-kondo
publishing diagnostics

For project files it says the same thing except no file name after the message

found config dir at 

that's probably a nil:

(debug "found config dir at" dir)

I’ll notice immediately when I trigger the error, because I have my computer on my lap.

nice physical feedback and also a nice way to stay warm in these colder days

yeah!

so I think we can safely ignore output.calva-repl in the plugin right. this is just waste of clj-kondo cycles :)

> For project files it says the same thing except no file name after the message I think you might have no .clj-kondo dir for that project, right?

That’s correct.

Would it make sense to have a way to ignore dirs and/or files via the config?

so far that's not been needed, by default clj-kondo only lints .clj(s)(c) files

I'll just hardcode it for now, until clj-kondo gets support for it. This saves me doing another clj-kondo release for this one plugin only

Please install .92

diff: https://github.com/borkdude/clj-kondo.lsp/commit/878c704a229196cb97eb36e467bda051782489c7

How come it lints a .calva-repl file then?

I’ll wait installing .92 I think. I want the cpu to burn my lap up so that we can try figure out what causes the error. 😃

You have config {:linters ^:replace {}}: this will still lint the file but not report any errors

I think Calva sets the output window to Clojure mode and this will make it lint with clj-kondo probably

My question was re: > by default clj-kondo only lints .clj(s)(c) files

@pez so, this probably does it: https://github.com/borkdude/clj-kondo.lsp/blob/878c704a229196cb97eb36e467bda051782489c7/vscode-extension/src/extension.ts#L56

I haven’t got a hot cpu in my lap since I started wishing for it. But I have two stack traces in the clj-kondo output that look like so:

[Error - 10:15:53 AM] java.lang.OutOfMemoryError: GC overhead limit exceeded
	at clojure.lang.PersistentHashMap$BitmapIndexedNode.ensureEditable(PersistentHashMap.java:815)
	at clojure.lang.PersistentHashMap$BitmapIndexedNode.assoc(PersistentHashMap.java:894)
	at clojure.lang.PersistentHashMap$BitmapIndexedNode.assoc(PersistentHashMap.java:876)
	at clojure.lang.PersistentHashMap$TransientHashMap.doAssoc(PersistentHashMap.java:327)
	at clojure.lang.ATransientMap.assoc(ATransientMap.java:64)
	at clojure.lang.ATransientSet.conj(ATransientSet.java:27)
	at clojure.lang.ATransientSet.conj(ATransientSet.java:15)
	at clojure.core$conj_BANG_.invokeStatic(core.clj:3366)
	at clojure.core$conj_BANG_.invoke(core.clj:3358)
	at clojure.lang.ArrayChunk.reduce(ArrayChunk.java:58)
	at clojure.core$reduce1.invokeStatic(core.clj:942)
	at clojure.core$set.invokeStatic(core.clj:4113)
	at clj_kondo.impl.linters$lint_unused_namespaces_BANG_.invokeStatic(linters.clj:364)
	at clj_kondo.core$run_BANG_.invokeStatic(core.clj:119)
	at clj_kondo.lsp_server.impl.server$lint_BANG_$fn__11528.invoke(server.clj:126)
	at clj_kondo.lsp_server.impl.server$lint_BANG_.invokeStatic(server.clj:125)
	at clj_kondo.lsp_server.impl.server.LSPTextDocumentService.didChange(server.clj:152)
	at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.eclipse.lsp4j.jsonrpc.services.GenericEndpoint.lambda$null$0(GenericEndpoint.java:65)
	at org.eclipse.lsp4j.jsonrpc.services.GenericEndpoint$$Lambda$12/2115597658.apply(Unknown Source)
	at org.eclipse.lsp4j.jsonrpc.services.GenericEndpoint.notify(GenericEndpoint.java:152)
	at org.eclipse.lsp4j.jsonrpc.RemoteEndpoint.handleNotification(RemoteEndpoint.java:220)
	at org.eclipse.lsp4j.jsonrpc.RemoteEndpoint.consume(RemoteEndpoint.java:187)
	at org.eclipse.lsp4j.jsonrpc.json.StreamMessageProducer.handleMessage(StreamMessageProducer.java:194)
	at org.eclipse.lsp4j.jsonrpc.json.StreamMessageProducer.listen(StreamMessageProducer.java:94)
	at org.eclipse.lsp4j.jsonrpc.json.ConcurrentMessageProcessor.run(ConcurrentMessageProcessor.java:113)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

Both are printed after the output has reported that a file has changed, and none of the files have been the output/rep file.

interesting

Maybe a memory leak

I think the maps containing all the stuff needs to be cleaned afterwards. Clj-kondo normally runs as a short-lived binary, so it's never been a problem there

I'll see if I can fix that

Hmm, there aren't any global things, each lint starts with a fresh context

so I'm not sure why this happens

Is there one particular file that triggers this?

Can you repro it?

I’ll try

My first guess was that the content in these atoms would get large over time: https://github.com/borkdude/clj-kondo/blob/master/src/clj_kondo/core.clj#L99-L117 but they are re-created on each lint, so the old ones would get garbage collected, I think

can you also check the memory usage of the corresponding java process (find the PID with jps) when it happens?

Seems like the process went to bed after the last GC error:

[Error - 10:34:21 AM] Connection to server is erroring. Shutting down server.
[Error - 10:34:21 AM] Connection to server is erroring. Shutting down server.
[Error - 10:34:21 AM] Connection to server is erroring. Shutting down server.

@pez I have a suspicion where the resource leak might be. I use memoized functions to check things on a config, and this might hold on to something for too long. Can you check the memory usage of the clj-kondo java process over time for a few hours during work?

I think I have a repro!

$ clj -J-Xmx1g
Clojure 1.10.1
user=> (require '[clj-kondo.core :as clj-kondo])
nil
user=> (dotimes [i 10] (do (clj-kondo/run! {:lint ["src"]})))
nil
user=> (dotimes [i 10] (do (clj-kondo/run! {:lint ["src"]})))
nil
user=> (dotimes [i 10] (do (clj-kondo/run! {:lint ["src"]})))
nil
user=> (dotimes [i 10] (do (clj-kondo/run! {:lint ["src"]})))
nil
user=> (dotimes [i 10] (do (clj-kondo/run! {:lint ["src"]})))
user=> (dotimes [i 10] (do (clj-kondo/run! {:lint ["src"]})))
Execution error (OutOfMemoryError) at (REPL:1).
Java heap space

(from clj-kondo's own repo)

also high CPU

Found it, I think

yep :)

Fix underway

How did you go about figuring it out?

I already suspected it had to do with memoize since that's the only place where things are stored in a global atoms and then I saw with VisualVM that there were sci things in memory after a GC... which was weird. Then I discovered I was also using memoize in the hooks namespace and effectively holding on to all the state forever by memoizing over a ctx argument... https://github.com/borkdude/clj-kondo/issues/1036

This is great news 🙂 many thanks for looking into this!

Very good find. Guess it is a valuable stresstest to have it as a long running process that makes all that work on every single keystroke. 😃

yep :)

Great to hear 🎉