Clojurians Log v2
observability
https://clojurians.slack.com/archives/C1DV21X9P/p1591957214110300
@seancorfield Apart from being able to reload configuration without loosing log events (which is important enough in itself), were there other advantages of picking log4j2 over 1?
Do you have an example of what your deps.edn logging deps looks like, Sean? I’ve been setting up a new project and as plexus noted in his post, there are so many deps required to set everything up for logging its indeed confusing to know whats required. My current setup is based off stuart’s clojure.dev logging setup.
This is what I currently have, just two dependencies, although it's log4j 1: https://pastebin.com/sQbAHQne It logs into a file and the console.
Example of what's usually in my project.clj: https://gist.github.com/ataggart/ac208c289c5d01dacf3b4b341a1c37f0
@seancorfield What's the use case for logged-future? I didn't find any documentation, but I infer it's just wrapping future around a logging call.
@sveri Since we were switching away from Timbre, we figured we might as well use the most modern logging library (2 over 1).
;; use log4j 2.x:
org.apache.logging.log4j/log4j-api {:mvn/version "2.13.3"}
;; bridge into log4j:
org.apache.logging.log4j/log4j-1.2-api {:mvn/version "2.13.3"}
org.apache.logging.log4j/log4j-jcl {:mvn/version "2.13.3"}
org.apache.logging.log4j/log4j-jul {:mvn/version "2.13.3"}
org.apache.logging.log4j/log4j-slf4j-impl {:mvn/version "2.13.3"}@ataggart It means we can just replace future with logged-future everywhere and not have to worry about adding in code everywhere to get this:
(future
(try
,,, body ,,,
(catch Throwable t
(logger/error t))))Thanks a lot everyone for contributing to the conversation. I learned a few things, and I'll be updating the post!
Really nice summary! For simple applications it might be tempting to stick with simple solution as described in this older article here https://juxt.pro/blog/posts/logging.html I guess it makes sense to build on the Java logging ecosystem as soon as you have integrations with other Java building blocks and/or want to use existing tools 🙂
Seems they basically roll their own logging, which I would not recommend. One of the key things I need from a logging library is fine grained control over log levels per namespace. I may want WARN log level as default (so at least I see warnings/errors logged from 3rd party libs), INFO or DEBUG for everything that falls under my own code (thanks to hierarchies I can configure that with a single line), and then maybe switch specific namespaces to DEBUG or TRACE when I'm working on them.
I'd like to do a separate post about making good use of logging, I spent some time with a mentee talking about that recently, and it's really something that's underappreciated. People use logging calls like printlns, putting them in when they're debugging, then taking them out again. If you do it well though you have logging calls at different levels throughout different namespaces, and you can dial up or down the amount of detail depending on which part you're looking at.
Yes! Learning more about that would be awesome! It’s definitely something we do not talk about enough 🙂 It’s interesting to hear your intensive use of log levels. In the projects I worked, we worked mostly only with “tags” to filter for things. I can imagine that certain “debug” information shouldn’t be logged by default if the system deals with a lot of load 😄 Would also be very curious to see if people actually have a system to dynamically toggle what logs are active for a running system.. Logs of 3rd-parties is also an interesting topic where this becomes a lot more relevant. Something I haven’t had to deal with much so far..
Another interesting thing related to effective use of logging is to decide what should be a log and what a metric and what both. We often like to use logs as just plain data on the lower level and let the higher up components decide where to send this information, what to ignore and so on 🙂
These two are definitely converging, you also see that in the design of mu-log, or something Ghadi mentioned that he wrote a backend for SLF4J that simply outputs JSON which he then pipes into Prometheus (I think, might be missing on the details). Wading through text-based log files doesn't make a lot of sense any more, we have platforms to aggregate and filter this stuff
We've gone back and forth a lot at work over the years. We started with clojure.tools.logging with its default behavior (slf4j), tried a few different back ends, then switched to Timbre for a while (with slf4j and routed it into Timbre), then finally we switched to log4j2 which I think is enough of an improvement to make it worthwhile.
We like not having to use XML -- we use .properties files -- and we like being able to select a configuration file via a JVM property and via an environment variable so it fits well with different environments needing different logging configurations.
Also, from the log4j2 docs about auto-reloading configuration: "Unlike Logback, it will do so without losing log events while reconfiguration is taking place."
I should point out that when we switched from Timbre to log4j2, we also switched back to clojure.tools.logging and the final piece of the puzzle was the (recent) enhancement to c.t.l that added support for a JVM option to override its default factory selection -- -Dclojure.tools.logging.factory=clojure.tools.logging.impl/log4j2-factory
ah so you use the c.t.l API and log4j2 as the backend? that's really interesting and makes a lot of sense. I might add a note about that.
@plexus Yes, I think we should never have switched from c.t.l to Timbre, to be honest. And it would have been real nice if Timbre had simply provided a c.t.l factory implementation so we could have stayed with c.t.l's API instead of changing all the calls in our code 😐
I think the only thing we "kept" from Timbre was logged-future and we just added a version of that (based on c.t.l) in our "utility" ns.
Hi everyone, what are some good books on observability?
all of these https://landing.google.com/sre/books/
Thanks! Will check these out. Hopefully some of these are available as part of my Oreilly subscription. EDIT: Oh, these are free to read. Fantastic.
https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/0988262592 is also good if you want something that covers more of the human side of things.