Clojurians Log v2

> the db function should be wrapped in a defstate also @lucian303 this would not be a good solution

calls to db are just functions

defstate should be only used for stateful things (I/O, thread pools, etc.)

take a look at this sample web app: https://github.com/tolitius/stater/tree/master/neo

@tolitius in my actual code i'm replacing a def w/ a defstate. it's stateful because it depends on the db as well as an external file. so (def ^:dynamic *nwords* (load-dict)) becomes (defstate ^:dynamic *nwords* :start (load-dict)) where load-dict makes the actual call into the db namespace and also loads the external file. does that seem like a proper use case? i tried to pare all that down for the example on github so it's not apparent there.

is it possible with mount to change a state in a function ? for instance I have a logger that logs to the console and later during startup I want to add logging to a message broker if a connection becomes available

I just saw in the hubble repo that there is an on-change function I will look into that

@lucian303 yes, depending on the use case this is correct. for example if your application has configuration you are reading in, depending on what it is in dict, it may or may not be loaded as a part of a :start function of a config state. for example, in case you use: https://github.com/tolitius/cprop, it could be a part of:

(defstate config :start (-> (load-config) (merge (load-dict))))

@yenda how / where do you check > if a connection becomes available ?

@tolitius it eventually becomes available in the config at some point during the startup of the application

> I want to add logging to a message broker and stop logging to console?

not necessarily does it matter ?

depending on the logging lib you use, this could be simply done with something like logback appenders. For example here is from the SocketAppender:

Consequently, if the remote server is reachable, then log events will eventually arrive there. Otherwise, if the remote server is down or unreachable, the logging events will simply be dropped. If and when the server comes back up, then event transmission will be resumed transparently. This transparent reconnection is performed by a connector thread which periodically attempts to connect to the server.

i.e. I am not sure why you would need a state for a logger, as this is usually baked into the logger lib

I use my own logging library

ah.. ok, that's interesting, why not any of libs that exist in JVM: log4j2, logback, etc. ?

because I want to log structured logs and I made macros that capture variables in scope among other things such as project name and version, thread and hostname...

right, so this is a function, not a state

but what do you usually use to log?

i.e. clojure.tools.logging / timbre / ?

my macros

macros is evaluated at compile time: i.e. there is no way to pass it a connection / open file

do you have a dependency on clojure.tool.logging?

in boot.biuld, project.clj?

no my macros then expand to a call to my log function

ok, and what does that function do, does it use something like clojure.tool.logging?

no it writes to stdout and publish the message if the broker connection is available

interesting, how do you control the log level: i.e. info vs. trace vs. error, etc.. ?

or you don't need to?

I am currently experimenting by doing a rewrite with mount

yea, sure, I'd just like to understand the problem better

before log would call an append multimethod dispatching to the list of appenders

I see, so you have a list of appenders and those are states because they need access to external resources?

yes

actually this is not really my problem the logger was one expemple, in general I would like to have a mean to redefine states as the program startsup because I have lots of chicken and egg problems

then I would have that "broker appender" as a state that would always log, but would drop or buffer log messages until the connection is available rather then mutate the logger state

why is it preferable ?

I thought this was similar to my use-case https://github.com/tolitius/stater/blob/master/hubble/src/clj/hubble/watch.clj

is it bad to mutate state in mount ? instead I could just keep the way things were with the multimethod and have the list of available appenders to dispatch to in the config state

and only this one will mutate and the logger will not be a state at all

it's not "bad", but it is usually better solved on the "function" level. the hubble watcher watches changes from consul: i.e. there is an envoy: https://github.com/tolitius/envoy watcher that triggers the change. In your case, if I understood it correctly, it seems like you'd like to simply delay logging messages to a socket until the connection is there, which could simply be done with message buffer, in which case you won't lose any messages

since you have a list of appenders that you always want to keep around (until the app is stopped) I would not solve it by changing the state, but rather by creating apenders once at the app start

well actually in some cases I would not want all appenders it can be configured

right, but you would read that config at the start time, right?

that is the chicken egg problem I was talking about earlier the config can be either a file or fetched from a configuration service that is accessed via rabbitmq

right, so you are reading a config and create you appender state(s) based on it: appender(s) state depends on a config state:

(defstate config :start (load-config))

(defstate appenders :start (do-with config))

to get the config it reads the config string which can be a file url or rabbitmq connection, if it is a rabbitmq connection it fetches the config from there else it looks for a rabbitmq config in the file

I think I see where you are coming from. hubble example does a similar thing: i.e. it has a config in a file + the one in consul

so it merges the one in consul on top of the one in a file

i.e. this way once the config is loaded:

(defstate appenders :start (do-with config))

and appenders will be started after the config since it depends on it

yes but the trick is that the logging should start first with a console appender and update the list of appenders once new ones are added

the problem is that it needs to be really dynamic to accomodate with different use cases

I have a working version that doesn't use any library: it makes things available as soon as possible using delays and bootstrap namespaces to avoid circular dependencies

I see, so you want to be able to plug appenders at runtime

not necessarily delaying them on start

yes I want to make things more dynamic and also add mount as the system management library

basically the project is a library for config and logging and messaging for microservices

so a microservice code would define a few defstate and mount/start in the main

and be able to mount/stop during dev

right now you start with (config/load) which loads the config and starts the broker-connection if one is needed but you can't restart cleanly in the repl during development

than it would probably be better to have an appenders state which would be a vector behind an atom: i.e. you can start / stop it, and also swap! conj it whenever you need to add to it. the problem with simply restarting that sate would be other components that are "currently" accessing it at runtime

> you can't restart cleanly in the repl during development why?

for instance tif you close the rabbitmq connection you also close open channels

restarting it won't make the rabbitmq appender work again

> you also close open channels what do you mean?

it's just a rabbitmq thing to connect to rabbitmq you need a tcp connection and then a channel that is some lightweight not thread-safe connection on top of it

if rabbitmq connection is a state, you don't have to restart it if you don't want to:

(mount/stop-except #'app/rabbit-conn)
(mount/start)

sorry I wasn't clear I was talking about the current working solution that doesn't use mount

I am trying to make a better version that is less complex and has a clearer and cleaner bootstraping and can be started/stopped during dev time

ah, ok. yes, the restart in REPL should not be a problem if you use mount. if you need to restart both a connection and channels, you rabbit state would be {:conn conn :channels channels}, then your stop function would stop both

or (mount/stop-except #'app/rabbit-conn) in cases you don't need to stop / start it

or ^{:on-reload :noop} ( https://github.com/tolitius/mount#on-reload ) in case you don't need it to get restarted as the namespace recompiles

but usually you would just do (mount/start) / (mount/start) and it would successfully restart this connection along with the channels

there is no problem on that part before the heavy refactoring I tested it with mount and the old bootstraping model and it was perfect

I usually keep something like this around for dev: https://github.com/tolitius/stater/blob/master/smsio/dev/dev.clj

currently the problem could be put that way: states are defined in multiple steps - appenders: 1. starts with [:console] 2. update appenders when available in config - config: 1. starts reading from param 2. gets file to read or rabbitmq connection 3. fetch config from rabbit when connection is available - rabbit-connection 1. starts when available in config

I think I need to play a bit more with mount first to find a better design

sure

so far the way I see it is something like:

(defstate config :start (-> (load-config)
                            (merge (read-rabbit-config))))

(defstate rabbit-connection :start (r/connect (config :rabbit))
                            :stop (r/diconnect rabbit-connection))

+ a possible logger / appenders state that would be a vector behind an atom

@yenda: just curious, why do you need to implement your own logging solution vs. using clojure.tools.logging / timbre / slj4j, etc..

Convenience and structured logs, it captures variables from the scope as well as informations that other loggers don't

but since this is done by your macro, can it just wrap an existing logging solution?

to add "variables from the scope as well as informations that other loggers don't"

It wouldn't really add any benefit

for one you would not have to deal with appenders

As far as I remember you still need to configure the appenders in timbre

And you can't do it at runtime

in both, log4j and logback you can

I'll think about it. Thanks for all the help !

sure, good luck. let me know where you end up

@tolitius for now I didn't work on the error handling part but I went again with the bootstrap namespaces method to avoid the circular dependencies

and it work quite nicely it starts/stops and everything including logging seems to work nicely