Clojurians Log v2

What's the best way of debugging why an application takes so long to start? I'm using Clojure component along with a jetty HTTP server, a HikariCP instance and a few RabbiMQ listeners:

$ lein start
2021-02-04 05:50:48.848 INFO  [org.eclipse.jetty.util.log] - Logging initialized @17043ms to org.eclipse.jetty.util.log.Slf4jLog

:aliases {"start" ["with-profile" "dev" "trampoline" "run"]}

FWIW I have found significant improvements in startup time when I got rid of lein altogether. It's been quite some time ago and I didn't try to dig deeper.

Are you using deps.edn? How does it compare to lein in your opinion?

The Clojure CLI and deps.edn is focused on building a classpath and running some code. That's it. Leiningen is a full-on build tool that does "everything".

I find it much simpler and much more pleasant to work with in my use cases. The only hiccup for me was working with Java sources that are shipped within the same project. But there are ways around that.

That said, there are lots of community-provided tools for the CLI that provide all the same "features" as Leiningen but a la carte instead of bundled.

We run a 100K line codebase with the CLI. A monorepo with three dozen subprojects and we build over a dozen production services from it.

(we switched away from Leiningen to Boot about five/size years ago, then switched to the CLI/`deps.edn` back in 2018 I think)

As for your original Q @tanel.kriik building an uberjar with AOT compilation will really help startup time.

We had several processes that took 10-20 seconds to start up from source but only take up to about five seconds to start as AOT'd uberjars.

I see. Thanks for the suggestions!

> What's the best way of debugging why an application takes so long to start? How much time are we talking about? deps.edn can shave a few seconds (1 fewer JVM + cached classpath calculation) but that might not make a difference. My app at work takes ~4m to boot from source, it basically boils down to how many clojure namespaces are there. The clj compiler isn't particularly fast

@vemv See the OP -- it showed about 17 seconds to get to initializing the logging.

@vemv So 17 seconds is quite fast compared to yours. 😅

@ Sean whoops, didn't squint hard enough 😇 Yeah in this case deps.edn can make a difference. To be sure, you can ps aux | grep java and copy the java command that Lein generates (`lein run` is a JVM program which ultimately simply builds a java invocation to be executed in a new process). Then you can run that java invocation from scatch, measuring your app with zero Lein overhead If you perceive a difference, sure go ahead with deps! (remove trampoline for extra accuracy)

@tanel.kriik If it's a fairly straightforward process, it's probably just going to be clojure -M:dev -m your.main.namespace where :dev is a deps.edn alias that is equivalent to whatever special stuff you have in your dev profile in project.clj and your.main.namespace would be, well, your main namespace.

How do I use a java dependency that’s not on maven central repository? I want to use https://www.w3.org/Style/CSS/SAC/ in my Clojure project.

are you sure it's not on maven? https://mvnrepository.com/artifact/org.w3c.css/sac

to answer your original question, are you using deps.edn? if so, you can use a local jar https://clojure.org/guides/deps_and_cli#local_jar

I just found it on maven after searching.

That was dumb lol

But thank you.

Ok, so now I can use it in my deps.edn yeah?

:deps {org.clojure/clojure {:mvn/version "1.10.2"}
        clj-css/clj-css     {:mvn/version "0.1.0-SNAPSHOT"}
        garden/garden       {:mvn/version "1.3.10"}
        org.w3c.css/sac     {:mvn/version "1.3"}}

yea, adding the maven dep should be the most straightforward

Ok, I’m probably missing something dumb here. What’s going on?

if it's a java class, then you need an import, rather than require

ah ok let me try that

I think I got it. I used a dot when I called one of the methods, then I got a different error saying it got the wrong type. I think this is closer to what I’m looking to do.

Thanks for your help!

You might want to uberjar/aot the app if it is for prod, startup will then be faster too

Ok, I really don’t understand how to interop with Java libs at all. I’m trying to read http://cssparser.sourceforge.net/gettingStarted.html and get the methods working in clojure. net.sourceforge.cssparser/cssparser {:mvn/version "0.9.29"} Are there any resources on this topic you can recommend?

There isn't any way of knowing what a namespace currently requires just by using clojure.core, is there ? (ie. without resorting to tools.namespace)

@adam678 You can use clj-kondo for this which determines this through static analysis. See https://github.com/borkdude/clj-kondo/blob/master/analysis/README.md

note that t.n.parse does not have any side-effect (unlike (refresh) and friends)

Hello all - anytime I try to run lein repl now anywhere, even if there is no project.clj, it throws the following error:

$ lein repl
OpenJDK 64-Bit Server VM warning: Options -Xverify:none and -noverify were deprecated in JDK 13 and will likely be removed in a future release.
Retrieving fipp/fipp/0.6.23/fipp-0.6.23.pom from clojars
Retrieving fipp/fipp/0.6.23/fipp-0.6.23.jar from clojars
clojure.lang.Compiler$CompilerException: Syntax error compiling at (fipp/ednize.clj:1:1).
#:clojure.error{:phase :compile-syntax-check, :line 1, :column 1, :source "fipp/ednize.clj"}
 at clojure.lang.Compiler.load (Compiler.java:7648)
    clojure.lang.RT.loadResourceScript (RT.java:381)

@bendy This might come from something in your ~/.lein

Good call but needs to be dynamic here

@borkdude legend, that solved it! No idea what was even in there 😂

Almost! Unless I am doing it wrong, it won't show namespaces which aren't aliased nor referred. But it might be good enough, who knows... thanks!

root cause is https://github.com/brandonbloom/fipp/issues/72

Does mutating transient vector not change the root reference? E.g.

(let [t (transient [])] (dotimes [i 20000] (conj! t i)) (persistent! t))

@kumarshantanu it's allowed to, but doesn't promise to

your code will break if you rely on it

> Note in particular that transients are not designed to be bashed in-place. You must capture and use the return value in the next call. https://clojure.org/reference/transients

"mutating a transient vector" - a conj! is allowed to mutate, doesn't promise to mutate

Because for transient maps the root ref updates, so I was a bit surprised at transient vectors.

another thing to remember is that reading code doesn't particularly help you. This is explicitly not in the contract so reading code has to be phrased as "the implementation right now does..."

The conj! impl of transient vectors https://github.com/clojure/clojure/blob/140ed11e905de46331de705e955c50c0ef79095b/src/jvm/clojure/lang/PersistentVector.java#L579-L608, but as others have pointed out, the docs instruct us to capture the return.

The supported contract is always to use the return value of any transient operation, just as you would for a persistent operation. It might be identical? to what you gave it, it might not. If you use the original reference for further operations, you are breaking the contract, and might never get any error or warning that you are doing so.

and the confusion arises because in many cases, it works as you would hope, even if you break the contract

There is a pending ticket to make the doc string clearer about this

I wonder why that is, however. Is the reasoning to keep code close to the functional equivalent (as the docs describe), or to have flexibility to return a different thing in the future (much like array-maps are switched out to hash maps)...

yeah i was disappointed that (find-doc "bash") didn't return any warning. glad there's a ticket

I would guess the second reason.

transients are sort of a fusion between a mutable datastructure and an immutable one. internally they build the same tree structure as the immutable datastructure, they just mutate the structure when they can, otherwise they return a new tree. so for example if each node in the tree can hold 32 elements, when you add elements 0 to 31 you just mutate in place, and then once you add the 32 it has to split the tree

If Rich didn't want that flexibility, then why would the current implementation sometimes return a non-identical transient collection?

I have a very difficult time believing the answer "that was an accident of the implementation"

> why would the current implementation sometimes return a non-identical transient collection? I didn't know this is the case.

but it is really two questions right? why does that behavior pattern exist, and why do transients expose it instead of hiding it

That is the reason that some people bring up the question of "hey, this code using transients isn't working" -- i.e. they wrote code that ignored the return value, and sometimes it works (when the implementation returns an identical object), but in other cases, it fails (when the transient operation returns a non-identical object)

@hiredman Agreed that an implementation of transients could guarantee to return an identical object always, if the implementer wanted to.

> why do transients expose it instead of hiding it The docs being what they are, isn't it fair to say that transients do hide this?

it depends on what you think is being hidden

"the fact that the output may be identical?"

they expose the fact that sometimes you get a new structure instead of mutating the existing one

they could hide that if the whole thing was wrapped in a mutable reference

ahh, I'm coming at it from the other side.

The true answer to "why is it this way?" is truly only known by the implementer, if you want all the details and tradeoffs involved in the decision

While it would be possible to design transients that guarantee returning an identical object always, I believe it would involve an extra level of indirection near the root, in cases where the implementer wants to switch data structures, e..g from array-map to hash-map. That would have some performance cost. Maybe one that some implementers would find acceptable, and others would not find acceptable.

La mort de l'auteur for programming

What's a good lightweight library for making/unpacking tar files?

apache io probably has that

@wombawomba clj-commons/fs also comes with this lib (the apache one): https://github.com/clj-commons/fs/blob/master/src/me/raynes/fs/compression.clj but note that tar is now available on all major OSes, including Windows, so shelling out isn't even that strange to avoid this dep

cool, thanks — will take a look 🙂

I need a get-in that works not only for sequence of keys but also for functions, I can probably spend some time writing it, but perhaps someone already has done that before. So I basically I want something like this:

(get-in* {:foo [{:bar 1}]}
         [:foo first :bar]) ; => 1
(get-in* {:foo [{:bar 1}
                {:bar 42}]}
         [:foo second :bar]) ; => 42

Please don't suggest to use Meander or Specter.

its not fully general but you can use 0 here

it doesn't work for functions but in this tangible example vectors are associative with respect to index. so (get-in coll [:foo 0 :bar]) will achieve your result

ah, right... that does it. thanks Dan!