Clojurians Log v2

Is the raw data available to the latest State of Clojure anywhere? I'd like to get a sense of how many people target Java versions after 11, but the survey only has the overlapping data, so I can't tell if the 10 percentish on 12, 13, and 14 are mostly distinct or mostly overlap. (Alternatively, is there a survey or data source of what Java versions Clojure projects typically use other than the State of Clojure?)

FYI to people who haven't seen this before, it's a useful survey: https://clojure.org/news/2020/02/20/state-of-clojure-2020

The raw data for all years is linked at the bottom of that article. 2020 is https://www.surveymonkey.com/results/SM-CDBF7CYT7/

The totals shown in the table beneath the graph should indicate whether it was multiple-response (overlapping) or single-response I guess?

Oh sorry. I was referring to the raw response data, which was available for some of the older surveys, like 2014, not the full SurveyMonkey report.

This isn't detailed enough?

Looking at the numbers, they add up to more that the total respondents so it must be a multiple-response question.

To be honest, it gives me enough of an idea in practice. I was just hoping to have a more definitive answer to "how many people use non-LTS versions"

Yeah, I was a bit surprised nearly 60% are still using 8.

RROR StatusLogger Unrecognized format specifier [d]
ERROR StatusLogger Unrecognized conversion specifier [d] starting at position 16 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [thread]
ERROR StatusLogger Unrecognized conversion specifier [thread] starting at position 25 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [level]
ERROR StatusLogger Unrecognized conversion specifier [level] starting at position 35 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [logger]
ERROR StatusLogger Unrecognized conversion specifier [logger] starting at position 47 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [msg]
ERROR StatusLogger Unrecognized conversion specifier [msg] starting at position 54 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [n]
ERROR StatusLogger Unrecognized conversion specifier [n] starting at position 56 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [d]
ERROR StatusLogger Unrecognized conversion specifier [d] starting at position 16 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [thread]
ERROR StatusLogger Unrecognized conversion specifier [thread] starting at position 25 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [level]
ERROR StatusLogger Unrecognized conversion specifier [level] starting at position 35 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [logger]
ERROR StatusLogger Unrecognized conversion specifier [logger] starting at position 47 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [msg]
ERROR StatusLogger Unrecognized conversion specifier [msg] starting at position 54 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [n]
ERROR StatusLogger Unrecognized conversion specifier [n] starting at position 56 in conversion pattern.

https://www.mail-archive.com/clojure@googlegroups.com/msg106499.html - I found someone else has this issue too

@seancorfield I guess you are the one who can know the answer. It touch topic around https://github.com/seancorfield/depstar

hmm I updated to new ver of depstar and I have different errors

1) option 1

:exec-args {             :aot true
                                 :main-class api.core}

Exception in thread "main" java.lang.IllegalArgumentException: No implementation of method: :get-logger of protocol: #'clojure.tools.logging.impl/LoggerFactory found for class: nil
	at clojure.core$_cache_protocol_fn.invokeStatic(core_deftype.clj:583)
	at clojure.core$_cache_protocol_fn.invoke(core_deftype.clj:575)
	at clojure.tools.logging.impl$fn__881$G__865__888.invoke(impl.clj:25)
	at api.core$_main.invokeStatic(core.clj:43)
	at api.core$_main.invoke(core.clj:42)
	at clojure.lang.AFn.applyToHelper(AFn.java:152)
	at clojure.lang.AFn.applyTo(AFn.java:144)
	at api.core.main(Unknown Source)

:main-opts ["-m" "hf.depstar.uberjar" "api.jar" "-C" "-m" "api.core"]

2020-12-17 13:17:06,472 main ERROR Unable to locate plugin type for JsonTemplateLayout
2020-12-17 13:17:06,473 main ERROR Unable to locate plugin type for JsonTemplateLayout
2020-12-17 13:17:06,497 main ERROR Unable to locate plugin for JsonTemplateLayout
2020-12-17 13:17:06,508 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.ConsoleAppender for element Console: java.lang.NullPointerException java.lang.NullPointerException
	at org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.findNamedNode(PluginElementVisitor.java:104)
	at org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.visit(PluginElementVisitor.java:88)
	at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.injectFields(PluginBuilder.java:185)
	at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:121)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
	at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:622)
	at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:695)
	at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:712)
	at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:267)
	at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:245)
	at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:47)
	at org.apache.logging.log4j.LogManager.getContext(LogManager.java:174)
	at clojure.tools.logging$eval136.invokeStatic(NO_SOURCE_FILE:0)
	at clojure.tools.logging$eval136.invoke(NO_SOURCE_FILE)
	at clojure.lang.Compiler.eval(Compiler.java:7177)
	at clojure.lang.Compiler.eval(Compiler.java:7132)
	at clojure.core$eval.invokeStatic(core.clj:3214)
	at clojure.core$eval.invoke(core.clj:3210)
	at clojure.tools.logging.impl$log4j2_factory.invokeStatic(impl.clj:183)
	at clojure.tools.logging.impl$log4j2_factory.invoke(impl.clj:178)
	at clojure.lang.Var.invoke(Var.java:380)
	at clojure.tools.logging$call_str.invokeStatic(logging.clj:316)

all the time work in REPL

Hello clojurians Anyone here using Clojure + gRPC ? Curious about Which libs do you use?

Is there a way to build a Clojure library that automatically runs some code when it is in the list of dependencies of an application?

Not that I’m aware of

Seems like that would be a security issue :)

Usually this is done by executing code at the top-level. This would be executed on require.

(I just fixed a couple of problems with some lib that did this when trying to use it with GraalVM :))

https://twitter.com/borkdude/status/1339592701952442378

Is that something like emacsclient for nrepl?

not really, it's just reply but then compiled to native

Would you guys say it’s a bad practice for a lib to run some code at the top level?

It depends.

What is your use case?

I am providing a logger lib on top of timbre that meant to be used only internally

The logger lib exposes a (configure settings) function

sounds like something someone should require and call some kind of init function

Before configure is called, the behaviour of the lib is undefined

The problem is that settings are usually read from a service

Being explicit about this like @dpsutton suggests is usually best

read from a service means network call?

The place where you would do this is in your apps -main method for example

Yes: service means network call And sometimes, an app needs to log before reading from a service>

and your logger is explicitly undefined behavior before this network call succeeds?

The current solution is that the app calls configure twice

you could make configure idempotent

or accumulative

Even if it’s accumulative, calling configure twice is cumbersome

@dpsutton the logger could work before the network call as it can accept an empty map as settings

Where can I find examples of tools.cli using the :in-order option? I’m having a hard time figuring out how to use it correctly.

Just a moment @ccidral. Let us complete the discussion

we shoudl use a thread. people don't have to wait to ask other questions

ok. let’s do it

Sorry @ccidral Keep going

Maybe you should use locking around your config... j/k ;)

@dpsutton @borkdude let’s continue here

i kinda want to recap as i don't quite have a good grasp. You need to initialize the logger with a network call. The logger's behavior before being initialized is undefined. You have seen instances of needing to log before initialization

is all of this true?

Correct. To be 100% precise: I need to initialize a logger with a map that is received from a netwok call

Is this in CLJS? Why would you do a network call to init a logger?

so what's your question? You could block awaiting the results of the network call. But fundamentally you need to handle failing network calls or slow calls. And does the app crash?

@borkdude It’ CLJ. The logger settings come from the network

if not, and you allow work to happen while its waiting, do you buffer all of these logs and then send them through once the network has (hopefully) succeeded? Do you have a fallback and log it to a file and then push that file through whatever configured logging settings eventually return?

I’m fine with blocking

then do that

What about logs that are sent before the request to the network is sent?

have (my-logger/init!) make the network call and block. i'd do this before anyone needs to log otherwise you get into weird territory

i wouldn't allow that

No!

but if you want to, then you need to figure out a way to store them and then log them in the future with the correct logging settings

but this could be a crazy hard problem. what do you do with those logs when the network request fails, or the program dies before initializing.

The way it works is

(init-logger! {})
(log "App is up")
(let [settings (read-from-network)]
    (init-logger! (:log-settings settings))
    (log "after settings are read"))
      

maybe not bring the app up before you have the required info to log

that's probably what I would do

It’s not a big issue that the logs that are sent between the two init-logger! are not using the same settings

then what's wrong with what you've posted?

Nothing

you could queue the pending logs until you have the right info, another option

It’s just that it’s cumbersome for the apps to have to call init-logger! twice

why not have the init logging do that instead?

I was thinking of calling (init-logger) in the library code

you call init and it starts immediately with an empty config and does a network call, updating the logging settings when that resolves

The logger doesn’t know about the network call. It’s an app related stuff

There are dozens of apps

that use the same logger lib

Seems like you just have to pass it into cli/parse-opts, so something like:

(cli/parse-opts args specs :in-order true)

i don't think i know how to help then

My question is: is it a bad practice to call (init-logger!) inside the logger lib. @borkdude you wrote earlier that https://clojurians.slack.com/archives/C03S1KBA2/p1608223187132500 On what side of the “it depends” my use case fall?

considering that it is your own library, I wouldn't worry about it

maybe document it in the README

You mean: it’s ok or not ok?

we're just people on the internet. but he's saying its ok

up to you

Why the fact that this is my own library matters?

because this would be crazy for something the general population used. but for an internal logging library its tied to your particular usecase

this is 100% your code top to bottom. so do what you need to solve your goals

You mean that I hold both sides: the lib and the usage of the lib

yes

?

Ok. Thank you guys

I'm in a meeting now, have to drop

Enjoy your meeting @borkdude

@p-himik Yes, I tried it but the missing piece is how to structure the specs so that they are interpreted correctly.

The docs at https://github.com/clojure/tools.cli are very vague about that.

I don't think it has anything to do with the specs. As per the docs: "`cli/parse-opts` accepts an :in-order option that directs it to stop processing arguments at the first unrecognized token." What exactly are you trying to achieve?

So e.g. if your specs are [["-p" "--port PORT"]] then cli/parse-opts will parse only -p and --port. Anything else will become arguments.

@p-himik I’m trying to write a CLI tool that accepts commands and subcommands as command line arguments, like eg: the git CLI

If my understanding is correct, then you have to create a global spec that does not include your subcommands and use it with :in-order true. Then, you create a spec for each subcommand and feed it to cli/parse-opts along with the arguments received from the cli/parse-opts called with the global spec.

Oh I see, that makes sense.

Livestream happening now: https://www.youtube.com/watch?v=KES-lKTq-3M

what’s the most efficient way to take a vector of maps like this

[{:a 1 :b "foo"} {:a 1 :b "bar"} {:a 2 :b "baz"}]

[{:a 1 :items [{:b "foo"} {:b "bar"}]} {:a 2 :items [{:b "baz"}]}]

I would imagine it’s a combination of group-by and map but trying to avoid multiple traversals if possible

You could probably rig up a single-traversal version with reduce

it depends on if things are sorted or not

but it is some kind of join

a self join

Thanks @p-himik I get it now.

I think this is possible via Java static initializers The trick that java classes are present regardless whether you :import them. So you could invoke clojure from a java class static initializer. If such class is in the classpath, mission accomplished I doubt it's a pleasant "API" though :)

use the specter library?

I ended with

2020-12-17 22:45:53,526 main ERROR Unable to locate plugin type for JsonTemplateLayout
2020-12-17 22:45:53,527 main ERROR Unable to locate plugin type for JsonTemplateLayout
2020-12-17 22:45:53,550 main ERROR Unable to locate plugin for JsonTemplateLayout
2020-12-17 22:45:53,560 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.ConsoleAppender for element Console: java.lang.NullPointerException java.lang.NullPointerException
	at org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisito

But I see this in verbose mode /Users/kwladyka/.m2/repository/org/apache/logging/log4j/log4j-layout-template-json/2.14.0/log4j-layout-template-json-2.14.0.jar so it should be there

and this Found META-INF/org/apache/logging/log4j/core/config/plugins/Log4j2Plugins.dat in multiple dependencies, selecting the largest maybe this is the reason

@kwladyka That message is expected, when multiple Log4j2Plugins.dat files are provided in dependencies. Selecting the largest is usually the correct strategy but perhaps you have an unusual library on your classpath that is pulling in a larger version that does not include the default plugins cache. If your project is on GitHub, I can take a look. If not, I can't debug it so I can't help you.

https://github.com/arctype-co/log4j2-plugins-cache Do you know how to use this in deps.edn ?

This is a design problem with log4j2 that people have complained about for years and the maintainers have agreed it's a design problem and said they would change how it works in 3.0 but we have no idea when that will be release.

That's a Leiningen plugin. It cannot be used with deps.edn.

yes… just thinking if not log4j2 and choose something else. I am trying whole week to make a simple JSON log output in custom format…

for google cloud jsonPayload logging

When I write depstar 2.0, I will be able to have external dependencies and I can address the log4j2 problem using similar code. But for now depstar cannot have external dependencies.

Is any way to choose Log4j2Plugins.dat source?

Nope.

ok, this already help with decision 🙂

thank you

As I said, if your project is on GitHub, I can take a look at it and see what might improve the log4j2 file selection.

it is not

As you can see, logging in Java is a giant mess.

yes, I really hate logging in Java.

Java is the worst part of Clojure 😉

I assume you'll be switching to a different logging library now?

I guess logback as blind choice 🙂

I will see the difference with log4j2. I have never do so deep logging things.

hmm maybe I can also try to copy Log4j2Plugins.dat to jar as an extra command somehow form the right place if this will even work. I use only 1 plugin at once, so in this specific case it should.

oh logback is not supported by clojure.tools.logging, I have bad luck today

Hello. Is there a way to include java source paths using clojure cli? Something like :java-source-paths for lein? It tried adding the path to :paths in deps.edn, but that didn't seem to work.

no, and furthermore clojure cli doesn't do java compilation for you - there are extension tasks that can do it though

Oh I see, thanks for the info

Could you elaborate on "extension tasks?" I'm unfamiliar

they are deps that add special tasks https://github.com/clojure/tools.deps.alpha/wiki/Tools

the only one I notice right away today that compiles java is mevyn, and it does that by delegating to mvn

I had hope to add :exclude ["META-INF/org/apache/logging/log4j/core/config/plugins/Log4j2Plugins.dat"] and add this file into resources but it doesn’t work with the same exception