Clojurians Log v2

I'm thinking about a feature for clj-kondo where library authors can publish config specific to their library and clj-kondo can detect this somehow. So their artifacts should have some predictable and unique location for this config. Has something like this been done for other tools, so I could look at some prior work?

I've been thinking about exactly this lately for clj, but have not settled on an answer

one answer in the new clj world is to put it in :aliases of course

in general, we're trying to avoid "magic name" though

yup but deps.edn itself isn't often part of a published artifact right

right - have been thinking about some way to allow tools to be self-describing

I guess it doesn't have to be unique. If every library would publish their exported config under /clj-kondo, I guess you could just do .getResources. But maybe that doesn't work for dirs.

user=> (enumeration-seq (.getResources (.getContextClassLoader (Thread/currentThread)) "README.md"))
(#object[java.net.URL 0x50acf55d "jar:file:/Users/borkdude/.m2/repository/org/clojure/google-closure-library-third-party/0.0-20170809-b9c14c6b/google-closure-library-third-party-0.0-20170809-b9c14c6b.jar!/README.md"] #object[java.net.URL 0x3cae7b8b "jar:file:/Users/borkdude/.m2/repository/org/clojure/google-closure-library/0.0-20170809-b9c14c6b/google-closure-library-0.0-20170809-b9c14c6b.jar!/README.md"])

I guess it does work:

user=> (enumeration-seq (.getResources (.getContextClassLoader (Thread/currentThread)) "clojure"))
(#object[java.net.URL 0x27a7ef08 "jar:file:/Users/borkdude/.m2/repository/org/clojure/data.json/0.2.6/data.json-0.2.6.jar!/clojure"] #object[java.net.URL 0x280e8a1a "jar:file:/Users/borkdude/.m2/repository/org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar!/clojure"] #object[java.net.URL 0x11e33bac "jar:file:/Users/borkdude/.m2/repository/org/clojure/core.async/0.6.532/core.async-0.6.532.jar!/clojure"] ...)

There is still nothing like $HOME or ~ in tools.deps, am I right?

correct

ok

The reason I'm asking is, I'm considering sort of this for clj-kondo's config resources:

:config-paths ["../base/clj-kondo", "~/dev/clj-kondo", "~/.m2/repository/rum/rum/0.12.3/rum-0.12.3.jar!/clj-kondo"]

I think the users would invoke clojure -Spath for their project and clj-kondo would then scan this classpath for clj-kondo top level dirs, spitting them out to some file which it uses for future reference

but then they'd still would add their $HOME based paths manually, which is probably fine

not going to interpret ~

I think you should still consider alias data more, since it has existing sources and merging

and that data could point to files

Oh, the :config-paths will be part of .clj-kondo/config.edn, not of deps.edn. I was just thinking how clojure (or other tools) could help, since some of the config could come from libraries

Do ns-aliases work like :alias/foo?

no, they are aliases like namespace aliases

it's a map of alias to namespace, and you can use the alias in place of the namespace in your function name

Ohh, it's not for keys in the arguments. It's for the function name.

I see now "symbol" in the reference, my mistake 🙂

right. in your deps.edn {:aliases {:mine {:ns-aliases {foo my.other.thing}}}} and then clj -X:mine foo/bar to invoke my.other.thing/bar

and may be used for other things in the future

Curiosity getting the better of me, why aren't aliases referenced like #ref or [:alias :mine] or something that would distinguish it as a value for reference? That would allow for :exec-args to reference aliases, or the CLI.

sorry, not sure I'm getting the use case

:exec-args can reference aliases, but it's up to your program to do that resolution (via the injected basis)

I have thought about some kind of automatic resolution and using a tagged literal for that might be a good idea, but not ready to commit to that yet

@alexmiller I'm really surprised at the removal of -R and -O since those have been around since day one, as well as the removal of several -S options. I can imagine that breaking some people's scripts (it definitely breaks a couple of ours at work). And the impending removal of the ability of -A to run :main-opts is also going to break a bunch of our workflows at work.

interested in that feedback if others share it

some of those could be made to work again for some period of time if widely desired

We have a helper script that you pass a list of aliases into and it runs clojure -A:those:aliases in multiple directories -- now that script needs to know whether/when to use -M instead of -A based solely on whether the alias include :main-opts... which... I mean how would it know?

length of "period of time" tbd as well

can you take a look at frequency of that?

What do you mean "frequency of that"?

are 95% :main-opts or 50%?

or 5%

Pretty much any library/project out there built on deps.edn that includes instructions for running tests is going to suggest using -A:test or -A:test:runner and that would be broken if -A can't run :main-opts

See https://github.com/cognitect-labs/test-runner#usage for example -- and almost any project using that test runner is going to assume that usage.

yes, that's why it still works

but suggests to use -M instead

and only at some later time tbd will stop working

this is step 1 of a multi step plan towards a different place

but, also I'm trying to gauge the level of turmoil this will introduce :)

I think changing -A will introduce a lot of turmoil 🙂

I await more feedback :)

@alexmiller I just checked, comparing behavior between stable (1.10.1.561) and prerelease (1.10.1.672) and there's no compatible way to specify arbitrary combinations of aliases that may or may not include :main-opts in a way that avoids the warnings from -A encountering :main-opts but still includes all the non-`:main-opts` deps. In other words, you can't provide a single command-line that works across stable/prerelease CLI versions the same way, without warnings, as far as I can tell? In stable, -M only brings in :main-opts so you either need -A or repeat the aliases with both -R and -M; in prerelease, if you use -A you get the warning and you can't use -R. Is that a correct reading of the expected behavior?

What will be new the new equivalent for clojure -R:foo -C:foo -Spath? So getting the classpath with alias foo deps and paths included?

It seems like -M works as a -A replacement in prerelease but that can't be used on stable (because -M doesn't bring in general alias content in stable)?

@borkdude I don't think that's possible in the prerelease version? Since your only options are -A (which will still run :main-opts), -X, or -M...

This is pretty much what I'm asking about. I don't want my code to be dependent on aliases necessarily, then I have to wrap everything... It just seems like there's pitfalls.

Well, I think you can say clojure -Spath -X:foo but there's no command that is compatible across multiple clojure versions?

right, but what if you need that? I know I'm using that now sometimes

(you can't say clojure -X:foo -Spath since it tries to interpret -Spath as a function to be executed)

I use -R/C also when I have an alias with main opts, and dependencies and I only want the deps

https://github.com/borkdude/clj-kondo/blob/7364c6a5dc36904c3c5e8edb3d95e0b9c33e8f0c/script/diff#L5

if you want just parts of an alias, you should break that into multiple aliases and combine the parts you want

so make an alias for the deps, and an alias for main-opts, and use either one or both

from what I can tell, the number of people that use this very targeted usage of -R/-C is probably mostly the people in this conversation :)

maybe we represent thousands of silent users struggling with this behind their corporate proxies ;)

j/k

@alexmiller Am I correct that -M in stable does not include other stuff from the alias(es) but in prerelease it does include them?

yes

I was hoping there was a single command that could be used across multiple CLI versions (without warnings), but that seems not possible.

I don't think so. I want you to move to the new thing. :)

so in summary: decomplect your aliases and move on?

Every project generated by clj-new so far is going to run into this (because of the :test:runner combination).

because -A:test:runner uses :main-opts and produces the warning?

Yup.

In particular, :runner adds the Cognitect runner and its :main-opts.

well the solution is to tell them to use -M:test:runner which is where I want people to end up

But -M:test:runner will not work on stable CLI today.

or even better to coax tools into providing -X friendly entry points

there may need to be a period of dual instructions

another option would be to leave -A alone, but basically deprecate it entirely and only doc other stuff going forward

Removing -R/`-C` completely means there is a non-zero amount of tooling that cannot work at all across multiple CLI versions.

I think that would be the sanest: don't remove existing things unless you absolutely have to, undocument them and then wait for a year or so

there is a limit - leaving existing things complicates the impl. some of that's worth it, some isn't.

I think very very few people use -R and -C

but a lot of people are using -A

yes

and it continues to work

with the suggestion to do something else :)

printing a warning / hint is a good thing too imo

it does

you said above Sean "cannot work across multiple versions" but -A does

At my job they are just starting to adopt deps.edn. I was away for 1.5 week and my colleague starting porting other projects to deps.edn, after I did only our front-end. He's even using -X. I told him that's only release candidate material ;)

he's a man of the future

He also used your boot deps edn thing @seancorfield, since we still rely on boot for most of the stuff, but slowly migrating away

I'm hoping it will soon be in the stable version

so I hope the breaking changes won't affect their initial impression

but if warnings are printed, I think it's fine

@alexmiller So what is the recommended replacement for clojure -R:foo -C:foo -Spath ?

Sean: he already told: break foo up into multiple aliases and use -A

(I think: https://clojurians.slack.com/archives/C6QH853H8/p1599242896182500)

So this is accepted as a breaking change? That will force "lots" of users of the CLI to modify their deps.edn files?

clj -Spath -M:foo -P

Because -P suppresses :main-opts?

it suppresses execution

that's also slightly confusing to me, always has been, like -A:foo -Spath.... is it going to execute main or not... I always have to try it since I can't remember

it may be that -Spath should imply -P, I haven't thought about that much

But there's no compatible equivalent that works across both stable and prerelease?

I will make a note to look at that next week

like I said, hadn't thought about it

What’s the reasoning for -Spath to continue to exist but -Stree to move to -X:deps tree?

Because -Spath is handled in the script and doesn't need to run a program.

Thank you

that's not entirely true though? -Spath uses the tools jar which is a JVM program

if you have a cache, then it's indeed script only, I get the difference now

Only if it isn't cached -- it's related to just the first "part" of the CLI: build classpath etc.

yeah, what Sean said

it seemed silly to pass a path to a Clojure program so it could print it :)

we originally had it in the list of things to change but just didn't make sense (-Sdescribe is similar)

I think my biggest concern here is that several things that work in stable will simply break in the next version.

I view clojure as a tool for creating classpaths, and incidentally also executing them ;)

yes, there are breaking changes

Which means that if you use CLI tooling across a team and across various servers/CI/etc, you have to have a coordinated upgrade of the scripts, JARs, and all the changes to your code/tooling.

I've spent a week deciding which migration strategy to take on each of these settings. I tried to make what I judged to be the common things continue to work

none of these is a final decision, that's why it's out here as a prerelease for feedback

the only thing I've heard that is actually breaking you now is the -Spom I think?

well let me flip it and say, please give me a list of what is breaking you

It breaks our "outdated dependency" script, which relies on -Stree and now there's no compatible way to run that script across multiple versions of the CLI.

making -Stree or -Spom continue to work is an easy thing to do

(by internally rewriting as -X)

We don't use -R/`-C` but that will be a breakage for others with no compatible way to do it across multiple versions of the CLI.

I don't believe there are more than a handful of people doing that

(but I will be watching to see if I'm wrong)

False if old, true if new, just dropping this here :)

$ clojure -Sdescribe | bb '(-> *input* :version (str/split #"\.") (->> (mapv #(Integer. %))) (compare [1 10 1 672]) nat-int?)'
false

what am I reading?

CLI version sniffing 🙂

oh, ok

@seancorfield do you use -Stree with other args like -A/R/C?

We use it with -A

Currently, we use -A everywhere in our scripts that drive CLI stuff, because we have :main-opts in "separate" aliases -- but those :main-opts are mostly accompanied by :extra-deps to bring in deps for those mains.

The latter means we can't switch to anything that portably runs across multiple CLI versions that doesn't also produce warnings that might interfere with any output parsing we might do.

What about a CLJ_VERSION variable that invokes the right version of your CLI tool?

if the warnings were all stderr, would you not care?

(right now they are not all stderr, but they could be)

Well, at this point I'm now considering putting clojure and the tools JAR into our repo under version control so we can guarantee a given version everywhere, even if a developer or server environment has a different version of the tooling...

That's never been a problem before because there have not been any outright breaking changes.

I'm willing to consider something for deps.clj which allows compatibility with both versions

based on some env var

(we used to keep the lein and boot scripts under version control to avoid these sorts of problems)

I guess that also works

fyi, there are two jars now :)

-X uses a bootstrap program that does not have any other dependencies and that is in its own jar as of this release

Yeah, a couple of our servers are old enough that I can no longer run the CLI install shell script on them so any upgrades I do on those servers is manual and that was a bit painful because there's more than just a single script to update.

lein/`boot` both used wrappers that auto-downloaded the JARs if they were missing.

But making that work reliably across every environment, including older envs, is tough -- and not good for controlled corp envs with firewall restrictions etc.

@seancorfield I'm doing the same with deps.clj (downloading the jar that is). It also has support for proxies.

I'm considering adding download instructions when it fails for some reason. I'm not sure if I'm allowed to bundle those jars, so I'm doing it that way mostly for that reason

I'm taking off - have a good weekend all. more feedback is fine, just won't see it today :)

Have a good weekend!

The lack of "portable" CLI commands/instructions across multiple (unknown) versions of the tooling is definitely my biggest concern about this...

Is the issue that the alphabet is too short to use different letters? What about longer option names?

I don't think that solves anything @borkdude -- -M is the natural naming to choose here. It's just the transition between versions that is going to be so painful.

I guess especially on CI where you can't interactively react on a warning, yes

Checking in the CLI script is a good idea, probably we should also adopt it.

Since clj-new generates new projects for people and those include instructions on how to run tests etc, I'll have to "dual instruction" all of those...

Old model: -A:test:runner New model: -M:test:runner New command doesn't work on old CLI (because -M doesn't pull in :extra-deps) and old command produces a warning on new CLI and users -- especially beginners -- will complain that clj-new is "doing something wrong"...

clojure -A:test:runner -Spath works on both stable and prerelease because the -Spath option prevents execution -- but the prerelease produces a warning about switching to -M, but if you do what the warning says: clojure -M:test:runner -Spath treats -Spath as an argument for clojure.main

$ clojure -Sforce -A:test:runner -Spath
WARNING: Use of :main-opts with -A is deprecated. Use -M instead.
/Users/sean/.m2/repository/org/apache...

$ clojure -Sforce -M:test:runner -Spath
Unknown option: "-S"
Unknown option: "-p"
Unknown option: "-a"
Unknown option: "-t"
Unknown option: "-h"

USAGE:

clj -m cognitect.test-runner <options>

(don't get me wrong: I think the changes overall are good and the direction is good -- it's just the breakage and potential confusion that I think is going to be very problematic)

My 2 cents, fwiw, is that breakage is totally acceptable for an alpha release. Since so many folks are relying on this alpha release component, what would be nice, where possible, is clear guidance. For example I just tried the following with v1.10.1.672:

> clojure -Spom
Option changed, use: clj -X:deps mvn-pom

@lee I agree that tools.deps.alpha itself is clearly alpha and subject to change, but I think a lot of people would read https://clojure.org/guides/getting_started and not consider the CLI itself to be alpha and therefore not expect it to "randomly break stuff".

And if two different beginners have installed the CLI at different times (one before this change and one after this change), their experience following instructions out there on the web are going to be confusingly different.

This will be compounded if they try to follow tutorials out there about the CLI and create new projects (with clj-new), they'll also have confusingly different experiences...

I sympathize that the path from where we are today (at least on the stable CLI) to where we want to be "tomorrow" (when the prerelease becomes stable.next) is complicated by retaining backward compatibility here...

Unrelated, is it/will it be considered to specify extra-deps via an alias map? Between that & paths, it would abate most complaints about copy/paste between aliases / remembering which combination of aliases to compose together. Clasically it's :test:run because :run-tests requires copying the paths/deps from :test and people aren't so keen on that. Not something I'm looking to drive, but curious about how that would impact future endeavours of mine.

I think leaving -A alone -- retaining exactly its stable behavior, without even printing a warning message -- and just removing it from the documentation might be the best path. And perhaps at some future date adding a printed warning about using an undocumented option. That will allow a lot of tutorials etc to continue working -- without any potentially confusing warnings -- and let new/updated tutorials start talking the new options (but with caveats about CLI versions, to cater for folks who have older installations).

@seancorfield and @borkdude I too empathize with folks who will be surprised by any cli breakages. And agree that it is likely not clear to the average Clojurian that they have been using an alpha component.

@dominicm I already have :test:runner so I can use :test in various contexts without needing Cognitect's test runner, but today -A:test -M:runner doesn't work with that setup (since -M today does not bring in :extra-deps). So either have to split :runner in two and use something like -A:test:runner -M:run-tests to avoid warnings and be compatible across both versions or provide dual instructions that are version specific (`-A:test:runner` today, -M:test:runner "tomorrow").

Just another vote for what @seancorfield is saying. I use -C for mixing in a classpath into a -A invocation. I see how this can be changed for the next stable version, but having a window of deprecation would really be nice.