Clojurians Log v2

Why is there a #queue tagged literal in ClojureScript, but not in Clojure?

Hoping someone here has this handy: https://twitter.com/RobStuttaford/status/1409758371167735813

Wouldn't you need to tokenize beforehand so that each element is a pair of string,number? Something like

(->> ["item_10" "item_2"] (map #(re-matches #"(.+?)(\d+)" %)) (map (fn [[_ a b]] [a (Integer/parseInt b)])) sort)

If there isn't a regular pattern in each input element, then I'd immediately think that sorting would be ambiguous

Eg ["10_item" "item_2"]

Maybe this is helpful? https://medium.com/@wilkerlucio/natural-sorting-in-clojure-script-123749bf3ba

I'm looking for a function that gives me a snapshot of jvm stats (heap, threads, classloading, gc...) as clojure data. I'm aware of jvm metrics libs like micrometer and dropwizard, but I'm looking for something dependency free. not too hard to create myself, but ofc if it exists I'll use it 😉

is there anyway to get clojure.pprint/pprint to work with *print-meta*?

(binding [*print-meta* true]
  (clojure.pprint/pprint (with-meta {:foo 1} {:bar 2})))
;; => ^{:bar 2} {:foo 1}

I found this - https://clojure.atlassian.net/browse/CLJ-1445 and suspecting I’m on an older Clojure version perhaps since the above example doesn’t print the meta for me

Yep, Clojure 1.10.1

We have started to get a "java.io.IOException: Connection reset by peer" in our production environment (stacktrace in thread). The exception is coming from a call using the Java 11 HTTP client. We have not changed our Java version (running Azul 11.0.10-11.45.27) in some time. I have seen the exception come from a call to two different services. One that we are running internally (AWS API Gateway -> Datomic NLB -> Ion HTTP), and the other is Auth0's management API. This exception indicates that the remote server has dropped the connection. The part I cannot figure out is why it has "suddenly" started occurring. We have been continuously deploying new releases, and some of those releases may have changed the classpath deps, but I cannot see how any dep change would affect the Java 11 HTTP client library. I'm a bit at a loss on how to debug this. Does anyone have any insights into this particular exception or what could provoke something like this?

elided internal code calls

while you may not have changed anything, AWS may have

just another variable to consider

My thought as well. Likelihood slightly decreased by the two different services causing the issue. No idea if Auth0 runs on AWS & uses the same services.

My current thought on a fix is to just retry those exceptions since it seems that a subsequent call will go through. It feels so wrong since I have such a poor understanding of the root problem.

seems similar to things datomic team spent a whole lot of time looking at (was aws related), but I don't know the details

We've also seen an increase of this error with our AWS-based services (mostly Elastic Search but also a few third party services that are hosted on AWS).

I have found some references to AWS services on Auth0's blog, so I think they have at least some things running there. I could totally see this being related to an AWS level change. If that is so, tracking that down to a repro to present to them seems like a very time consuming task 😬

there are a lot of subtle things around nlbs... let me dig up some posts

We also see more random timeouts as well, so we are having to build in more retry logic etc.

https://medium.com/tenable-techblog/lessons-from-aws-nlb-timeouts-5028a8f65dda and https://medium.com/swlh/nlb-connection-resets-109720accfc6 might be the ones i'm thinking of

@seancorfield thanks for that info. Very helpful. How are you thinking to tackle it? Retry IOException w/ "Connection reset by peer" string?

when did medium add a paywall? 😞

Thanks @rutledgepaulv. Will read through. I'm definitely not an expert on the intimate details of NLBs.

@kenny Two-fold approach: yes, retry on certain IOExceptions and also add explicit timeouts and retries on some calls. Not all calls can safely be retried so there's no one-size-fits-all approach 😐 It's very annoying...

np. here's another i was remembering.. https://www.niels-ole.com/cloud/aws/linux/2020/10/18/nlb-resets.html

@rutledgepaulv Medium's had a $5/month paywall for a long time. I pay. Lots of great content on there.

If you have the info readily available @seancorfield, what other IOExceptions are you retrying?

(I think I pay about $40/month now for written content online?)

@kenny I don't have specifics I can share, sorry.

@seancorfield would you be able to share which HTTP client library you are using that you've been seeing these connection reset messages?

http-kit

(we have legacy code that still requires Java 8 even though most of our code is on Java 11)

@rutledgepaulv That first article was great! I am definitely seeing a good number of TCP_Target_Reset_Count in the CW metrics. However, they don't align as perfectly as I'd like with when I am seeing the connection reset exception thrown. I am now wondering if the IOException "Connection reset by peer" exception can be thrown in a situation that isn't the remove server sending a RST packet.

New to Clojure and have been using Atom + Chlorine, but have heard great things about VS Code + Calva. I was wondering what differences / pros / cons there are between the two? Should I switch?

Sean Corfield has written a post about it: https://corfield.org/blog/2020/11/09/vscode-clover/

Chlorine and Clover are essentially the same, using a Socket REPL, which I like. Calva is great: it provides a lot of useful functionality and you can use it with Clover. In the Calva settings, turn the nREPL UI off and then you get all of Calva's great features with Clover's Socket REPL instead of nREPL.

I switched because Atom just doesn't have the same level of support as VS Code: VS Code is updated more often, has much better/broader extension support, and is much more widely used. Since you're already using Chlorine, you should be able to switch to VS Code + Clover and retain the exact same editing experience -- you can have the same hot keys, the same config.cljs customization, and so on.

Then you can add Calva (and turn off the nREPL UI), and benefit from built-in LSP (static analysis), clj-kondo, auto-formatting, great paredit support, rainbow parens, etc. Without needing a whole bunch of extensions added -- just Clover and Calva.

@seancorfield Since you're here - do you work with jar files in any way? With IDEA+Cursive it's so easy, I sometimes don't get it how people don't look at the sources of libraries they use. It's as easy as looking at your own files, even if a library is in a jar with compiled classes. Last time I checked, there was no Java support for VSCode that would be advanced enough for that kind of navigation/search.

Non-Clojure JAR files you mean?

I can see the docs or source for any Clojure code, even in JAR files...

It sounds like you are exceeding the limit on the # of connections allowed in a connection pool

if one more IO client connects... it can bump an earlier client out of the pool

Java dependencies, in particular - IntelliJ has first-class support for downloading (and decompiling, if necessary) java deps

if this is for Solr/Elastic Search... there is a config to set the pool limit

something I miss in... anything else, really

Sorry, right - I meant Java jars in particular.

I have seen that error when concurrently calling either Solr/ElasticSearch pools or PostGres pools

It's the Java 11 HTTP client.

In over a decade of working with Clojure, I've never found a need to do that...

the client is saying... something closed it's connection

since it's happening with both AWS services... it sounds like a scaling issue...

are you scaling AWS up and down? burst traffic?

a retry is a good idea... no API calls are ever 100% especially when AWS scales up and down

Just like some people never seem to see how a REPL-DD could be useful. ;) Just as a single example - for me personally, I just find it to be so much faster, easier, and more convenient in general to understand how to work with some library as compared to reading docs. It's not always the case, but it usually is.

but if the error is happening frequently... I would examine the types of tcp connections... persistant versus non persistant

have to look at the % failure rate

Even to just better understand how Clojure itself works - I'd rather read Compile.java and know exactly how something works then rely on my ability to find the doc that's just right and not outdated or incomplete. And being able to navigate to all the .java files of Clojure right from its .clj files is a bliss.

I have the Clojure repo locally so it's easy enough to browse if I ever want to look at the Java code. ¯\(ツ)/¯

I just don't find I need to look at the source of libraries very much...

Fair. Alas, for me it's pretty much the only reason why my move from IntelliJ to VSCode is so slow. I've started using the latter but for projects that don't involve Clojure.

@natesire for us this is happening against third-party services that are hosted on AWS so we can't adjust their services 🙂

But, yes, retries is the most effective approach we've found 🙂 for service calls that can safely be retried.

ahh... AWS built in services... no controls over your Ec2 or NLB

I only installed the Java extensions to VS Code very recently. I'd resisted for quite a while. I was working on some tools.deps.alpha stuff and finally got sufficiently annoyed at being prompted about Java extensions that I gave in and let it install them 😐

does it happen frequently? or just .01%

I mostly run my own stuff on Ec2 and beanstalk... interested to know about it

For us it happens in short bursts "occasionally". We'll get a flurry of either connection reset and/or connection timed out errors from a particular service and then it's just fine. For a while we were actually suspecting some sort of network glitches in the data center we use but then we saw that, while that was causing some of this behavior, it seemed that more of it was caused by something at AWS since we were only seeing connection issues for AWS-hosted services in most of the interruption windows.

(funnily enough, we just saw a short burst of connection timeouts while this thread has been active: immediately before and after, everything looked just fine, latency was good, response times were good, then a few dozen connection timeouts for maybe a minute, and back to normal)

Now it just pesters me to "import Java projects" which I'm mostly ignoring 🙂

In the middle of one of these "bursts" of failures, I actually got an email from AWS on "Amazon EC2 Instance Retirement" on one of the instances behind the NLB, which confused me even more, haha. I don't think that's related anymore though.

sounds like AWS is using their burstable instances to run these services... I'd have to check if you can configure those...

I am guessing that's the root cause. I need to update my AWS certs....this is the exact stuff they ask on their tests

I can give you the ultimate solution...

use GraphQL... it has a que that handles this exact problem

the que can backup during AWS scale up events... and retry the call

Since you are new to Clojure, @emi, give Calva a spin. See https://calva.io/getting-started/ and feel very welcome to chat us up in #calva

Heh, if only the services we used had a GraphQL API... 🙂

yea... it might only work for some services... I will think on a full proof solution

it's really the failed calls that are unsafe to retry... that you need solved

I have an idea for you...

check what size your ElasticSearch instances are....you can try smaller instances that scale up/down more frequently. It could be that your instance size is too large... so when another instance is spun up... the spin up time takes too long.

hence the burst of failed connections

you can adjust many config options on services like ElasticSearch....to match the cpu, memory, shards with the needed instance sizes

you should be able to spin up an extra instance ahead of time... before the burst of failed connections

CloudWatch on AWS can help you tune the config

That’s fantastic, thank you for the response @seancorfield! Found your blog post super useful too. Gonna give VS Code a spin right now. Quick question: if I understand your setup correctly, you’re basically using Calva for formatting, linting, tooling, etc., and Clover for running a socket REPL instead of Calva’s default nREPL. Apologies if this is a super noob question, but what’s the benefit of using socket REPL vs nREPL?

I am still working on a way to format lein's output... so the java stack and errors sits at the bottom of my screen. I had to switch my monitors to vertical....to see the java errors. Anyone try using tail or reverse to pretty format the lein output?

I would really like to switch my monitors back to landscape (wide) and find way to focus the output on the java error... not 100s of lines

maybe I should write a grep script to filter all java errors out of the stack trace?

if you're at a repl clojure.repl/pst is quite handy and does some of this. not sure what your dev setup looks like though

yes, REPL helps... I mean at the command line

when I am auto testing some tests... or looking at the raw stack trace of leiningen

does almost everyone just rely on the REPL to eval their code? not stack traces?

can I ask which version of leiningen and Clojure you are using? some of the default behavior changed in Clojure 1.10.1

ok... I will check that now

Leiningen 2.9.1 on Java 11.0.11 OpenJDK 64-Bit Server VM

let me check on the latest version if it's different...

(clojure-version) will get you the clojure version

the difference is from your Clojure version

but it might depend exactly what command you're running as well (lein test, lein run, something else)

1.10.1

clojure version

and are you running lein test or something else?

yes

I am running tests with lein... inside a shell

and this is an exception thrown during the test?

yes

and normally... the useful java error will be 100 lines above the bottom of the shell

ok, so it's really clojure.test error reporting in the context of lein test (which iirc monkey patches part of clojure.test)

so I switched my monitors to vertical

yes @alexmiller

I'm just trying to be clear on what you're seeing. we've spent a lot of time trying to reduce places where you get thrown a stack trace out of clojure and instead format a useful error but in some cases the tooling is doing its own thing

let me write a quick demo... I will show you a photo of it

give me a few mins

that's ok, I've repro'ed it

in this case, it's clojure.test error reporting

now if I scroll up... I can see the useful error... like this...

Exception in thread "main" Syntax error compiling at (hello/core.clj:7:3).

but the java stack trace can be hundreds of lines... so I switched my monitor to vertical... I was going to try some ideas to format and filter out the most useful part... or maybe... reverse it

there is actually a dynamic var to set the max stack trace depth at clojure.test/*stack-trace-depth* but I'm not exactly sure how to bind that appropriately

bash tail or reverse has a ton of options... with grep

I guess you could use a fixture in your tests

ahhh. ok I will try that first...

lein has a thing for setting dynamic vars iirc.

I think the bash approach could become tedious... maybe not 99% effective either

https://github.com/technomancy/leiningen/blob/master/sample.project.clj#L289-L290

I'm not sure that dynvar exists yet to set there

does most engineers just use some IDE/Vim plugins to run tests?

(defn short-stack [f]
  (binding [clojure.test/*stack-trace-depth* 5]
    (f)))

(use-fixtures :once short-stack)

^^ that should work if you put it in each test file (which is tedious I admit)

ok... I will play with it and try to bind it into my tooling

I will follow up... I have always run my tests from the shell on any stack... so it's just a nice thing to have with the tooling

I think there are a wide range of approaches so hard to say "most" do any particular thing

but this is something I'm interested in improving the default case

you bet... I think it helps with the java error handling that everyone often complains about

it's not a show stopper... just makes it more friendly... tooling... when first learning Clojure

Hmm, with lein test I was unable to get a stacktrace. I only got the new .edn file stuff.

yep. the other option here would be to redefine the clojure.test/report :error method to set a default

@dominicm you can also just try "lein run"

scaffold a lein app... lein new app helloworld

lein run with clojure 1.10.1 should not print a stack trace anymore if an exception is thrown out of main

I purposely created an error in main... just to demo this

I got this:

/tmp/helloapptest 8s
❯ lein test                
Syntax error compiling at (helloapptest/core.clj:7:3).
Unable to resolve symbol: i in this context

Full report at:
/tmp/clojure-12954933124682389143.edn
Tests failed.

that's a compilation error on load

that doesn't even get to the test runner

(ns hello.core (:gen-class)) (defn short-stack [f] (binding [clojure.test/stack-trace-depth 5] (f))) (use-fixtures :once short-stack) (defn -main "I don't do a whole lot ... yet." [& args] (println "Hello, World!"i))

but if you have something like (throw (ex-info "I suck" {:a 1})) in that test that will work

"lein run"

@natesire Same thing for lein run:

/tmp/helloapptest
❯ lein run 
Syntax error compiling at (helloapptest/core.clj:7:3).
Unable to resolve symbol: i in this context

Full report at:
/tmp/clojure-14415791333264548638.edn

hmmm weird

these are all not what was reported above

you need a deftest that throws at runtime

then clojure.test reports the error with full stack in clojure.test/report :error

I have not upgraded anything in a few months...

@alexmiller I'm going off the screenshots above, which are a syntax error.

it sounds like it's already being addressed... awesome

wish I could help more

Actually, also the example above ^ which is exactly what I'm doing 😛

oh, I didn't look at the screenshots. yeah, that case should already be handled.

I just need to upgrade.... awesome

I really like Clojure... I think once we have a web toolchain... this language could skyrocket

@emi Since you are using Atom + Chlorine, I assumed you were already using a Socket REPL?

If you were using Chlorine with nREPL, you might just as well use Calva on its own, without Clover (and leave the nREPL UI enabled).

FYI, there are multiple web toolchains already

@seancorfield sorry - yes, was using Atom + Chlorine with socket REPL obviously, was just wondering whether there are any benefits to using nREPL?

BTW I’m playing with VS Code + Calva + Clover right now and it’s awesome

I prefer simpler tools, so I avoid nREPL/Orchard stuff, and I like that I can start up any process that includes Clojure and specify a JVM option and get a Socket REPL with no code needed.

There are definitely advocates of nREPL here who will extol the virtues of it and the tooling based on it, but I just don't want that much code between my editor and my program. Definitely a YMMV situation.

Thought that might be the case 🙂. As a beginner, I found a socket REPL much easier / simpler to grasp than an nREPL, which is what drew me to the Atom + Chlorine setup.

Thanks Sean, appreciate your time. Loving the Clojure community!

I have my .clojure/deps.edn file up on GitHub, along with my config for both Atom/Chlorine and VS Code/Clover/Calva, in case you want to take a look...