Clojurians Log v2

Reading over the prerelease Deps/CLI reference again, and reviewing the discussions above, it seems the -A option is being promoted as the way to do everything-except-execution whereas before it used to include execution. Given that it's being referred to as the "REPL option" and, for the most part, -R used to be somewhat similar to -A without :main-opts, I wonder if keeping -R but making it also pull in the classpath stuff (but not :main-opts) would be the less disruptive path?

Then -A could quietly be undocumented, eventually deprecated, and finally removed.

This would be a minor expansion to what -R currently does (it would respect :extra-paths which only -A/`-C` did before) and it would still ignore :main-opts. I suspect -C is the least used option and almost all -A uses could be replaced by either -M (in the new world) or -R (in either world) with no changes in behavior.

And -R more closely relates to REPL. So we'd have -R/REPL, -X/execute, and -M/main.

Hmm, I guess -R would also have to subsume -O for that to work fully.

Going back to the :test:runner example, it seems like -A:test:runner -M:runner works identically on both stable and prerelease and produces no warning on prerelease (presumably because -M is present?).

Ugh! So... I guess instructions could change now to recommend -A... -M... and that would work now and tomorrow even though it's a bit redundant.

@seancorfield I like your suggestion of extending R to include C, I've always found it weird that these two were separated. Also the -A -M trick is something I tried yesterday, but somehow didn't work for me, since I got file not found -M, but now that I'm trying it again, it seems to work.

Yeah, having integrated CLI/t.d.a into our repo and started to update all our script to match this "new world", I'm very torn between "extending -R and undocumenting -A" and just "documenting -A... -M..." as the preferred approach.

I would much prefer a solution that works for today and tomorrow (which is the -A... -M... approach, especially since that suppresses the warning on -A).

Either way, our build script that previously just turned build aliases subproject options into cd subproject; CLJ_CONFIG=../versions clojure -A:aliases options needs to be way smarter about handling the options part...

Yeah, I guess clojure could override certain deprecated options when the more preferred ones are used, which provides a migration path

Well, it already does.

Yes, but also for other deprecated options?

If you use -A:test:runner -M:runner you don't get the warning, even when :runner includes both :extra-deps, :extra-paths, and :main-opts.

Which deprecated options?

-R, etc

-R, -T, and -O aren't deprecated, they're simply removed.

-A's use of :main-opts is deprecated.

I've never used -T or -O, so R is what I'm concerned about. So if you use -R ... -A ... in the new world, does that do the same as in the old world?

-R cannot be used in the new world. It does not exist.

but is it ignored?

as in, no error

It spits out an error and halts the script.

that's what I meant

(! 2172)-> clojure -R:test -A:test
-R is no longer supported, use -A for repl, -M for main, or -X for exec

Sat Sep 05 00:13:57
(sean)-(jobs:0)-(/Developer/workspace/wsmain/build)
(! 2173)-> echo $?
1

but I guess -A would also override -R in the old world, so in the old case it also doesn't work as a polyfill for both

Right, the problem is that -A in the old world also executes :main-opts

But at least changing -A:... to -A... -M... works identically and doesn't issue a warning in either world.

that's already a win

But anything that uses -R today is plain ol' broken tomorrow.

There's no compatible way to run anything that uses -R today in a way that works regardless of clojure version I think.

I have a fixed version on CI, so I don't think it will break for me: https://github.com/borkdude/clj-kondo/blob/70b3e54130978692c61e53859b126ffbe8a492f0/.circleci/config.yml#L24

So you pin to an old version?

I never thought about it really hard, just copied the install instructions and that seems to contain a version number

Fine for CI for you, but not fine for any library code or tools that have to live in a version where members could have any version of CLI installed...

I do have one custom installer script for mac CI as the linux script didn't work there: https://github.com/borkdude/clj-kondo/blob/master/.circleci/script/install-clojure

But in that arena people see warnings and can react to them

which can still be confusing, no argument there

I have a solution for clj-new now so I'm somewhat mollified 🙂 but it doesn't help anything that uses -R and has to live in a multi-version world.

I guess clojure could print a link to a more elaborate blog post about the situation

I don't consider that a solution. "We broke your shit. Go read this to fix it!"

Seriously, that is not the right approach, esp. given Rich's whole pulpit-bashing about no breaking changes without renaming things.

I agree that it's far from ideal, but assuming that it's going to be like it is now, it provides people a place to read up if they are curious about why it changed.

Considering how many people learn Clojure by using lein and still trip over ~/.lein/profiles.clj stuff, I don't think we should just throw our hands up and accept this and just say "Well, read these docs about why we broke stuff". I really don't.

You don't have to convince me that avoiding breakage would be preferred. I think I'm going to take a break from this channel for the rest of the day :)

I can change all the -A uses in clj-new's docs and in the generated projects to be -A... -M... and at least I then say, "Hey, the latest version does recommend using both options so you're future-proofed" but it's still a fairly sucky experience for users.

Enjoy your weekend @borkdude 🙂

Proposal: * Recommend authors change -A:stuff to -A:stuff -M:stuff for compatibility with both versions where they want :main-opts executed. * Keep -R and expand it to include :extra-paths and :jvm-opts so it mnemonically matches REPL: -R/REPL, -X/eXecute, -M/Main. * Undocument -A for now, officially deprecate it later (and add that warning about using -M instead), and eventually drop it altogether.

This provides a migration path for the most common use cases that is minimally disruptive (most uses of -R today won't care about paths or JVM options I suspect). It improves naming for the three desired use cases: REPL, eXecution, and Main opts. It breaks the least amount of code out there.

(question for Alex when he gets back: why are there two identical copies of exec.jar in the CLI distribution -- one at the top level, one in libexec?)

Keep in mind that most Clojure users are on Mac and install via brew and that brew aggressively updates you to latest version (our stable) even if you're upgrading something else. So within some period of time (couple months maybe), many people are using newest version.

Also keep in mind that we are in work on a 1.10.2 Clojure cycle and we tie brew versions to Clojure versions, so that might be a natural point to either coax an upgrade or drop some kind of compatibility

If you don't introduce the warning for -A -- per bullet 3 -- then, yes, bullet 1 isn't really necessary.

I think there aren't two in the distribution, the install script copies the root one to libexec which is the structure brew expects, although I think maybe I didn't sync the clojure script up to that and it's using the root one. I'll check on that

the warning is what tells people how to move to the new world - without that, won't people be stuck on -A forever?

Bullet 3 suggests introducing the warning some time later, after everyone has had a chance to update.

People don't like warnings from their day-to-day development tools. Beginners in particular will complain to library authors about "incorrect instructions" if you introduce that warning before anyone has had a chance to update instructions/tutorials.

I'm ok with that :) I'm not sure I'm in agreement with you on the scope of this as an issue

you're right that they don't like warnings, so they'll change what they do, which is what we want

library authors can either do nothing and situation is same or they can use dual instructions for a time or they can doc new instructions and say clj > X.Y.Z (which also gets people on new version)

or they can start using -X style invocation (which also drives to new version)

> I'm not sure I'm in agreement with you on the scope I'm pretty sure we're just not in agreement on that 🙂

there is a middle ground - no warning for -A on first stable release

then warning

then (much later?) removal

Having been on the receiving end of complaints about incorrect instructions (or version-specific instructions that weren't absolutely clear on how to determine the version and how to "solve" the problem, on all platforms), I'm perhaps more sensitive to this whole issue than you... 🙂

Yes, bullet 3 is that middle ground.

I'm on the receiving end of those complaints too

That only covers 55.36% of the survey responders, so this doesnt seem to be the strongest of arguments. Of course I am happy to help encourage upgrading to newer versions in any way I can, once I've understood the changes myself.

I wonder what percentage of the 35%-ish that are on Linux are using brew vs manually installing via the .sh file?

There are legion of package managers for Linux that brew competes against, maybe a question for the next survey? I find it easier to use the Linux script install.

I use brew on Ubuntu (on WSL) because it's easy to keep clojure up-to-date that way but that's recent for me -- I used to use the manual install script, but I tended to only upgrade when I remembered to do it, or I tried to do something that my older, installed version didn't support 😐

I have updated the depstar README to show usage with the prerelease -X option: https://github.com/seancorfield/depstar/blob/develop/README.md#clojure--x-usage

It works quite nicely that way...

I expect I'll update clj-new next, to support -X usage...

FYI, confirmed that script uses the root one:

-classpath "$cp:$install_dir/exec.jar"

> Keep in mind that most Clojure users are on Mac and install via brew Have you considered that those Mac users will probably also use the linux scripts on CI? Do you get any stats from that usage? It may be less of an issue since the install script is versioned, but since people tend to develop CI scripts locally (I often do), it might become confusing.

There might also be less Windows users using this because for some people it's harder to get going than lein. I got this report yesterday and it's not the first time I hear these kinds of issues. > Clojure until 1.6 used to work on Windows from https://chocolatey.org/packages/clojure too, without problems. Clojure Tools broke everything and it seems that OSX ( Homebew ) is the only "first class citizen" now. https://github.com/babashka/babashka.process/issues/15#issuecomment-687035007 Don't shoot the messenger, I don't really have a stake in Windows support because I know how to work around it :)

Yeah, you have to be pretty persistent to use Clojure CLI/`deps.edn` on Windows, and willing to work with very "alpha" tooling. Which a lot of Windows users are not willing to do: they often have different expectations of the user experience than devs on macOS/Linux.

One personal data point: I found it difficult to install clojure on Windows CI because the powershell stuff you had to enable and whatnot. That's why I also went with lein to run tests there: https://github.com/borkdude/clj-kondo/blob/master/appveyor.yml

I do use clojure on the other two major OS CIs

I'll be using Windows for development a lot more going forward (Microsoft had a great deal on Surface Laptop 3 so I just ordered an i7 with 16GB RAM and 256GB SSD). But I'll be leaning on WSL2 very heavily, and using (Linux) brew for installation.

I have an odd case -- at least from my pure Clojure background (didn't start in Java.) I have a Java dependency that doesn't publish to maven. I can run ./gradlew build on it to get a library .jar. What's the best way to include this into deps.edn dependencies?

Can I publish just a .jar to maven under my own repo? I could try updating the library to properly publish to Maven & PR'ing it, but I'm just unfamiliar with the gradle ecosystem. The library author says he'll start publishing to maven soon...

You can use a local .jar file directly in deps.edn via a :local/root dependency. We use that for a couple of things that have to build locally because they aren't published anywhere.

oh man, I immediately get what you mean, but how did I utterly fail to find this in a doc somewhere? I spent 20 minutes searching around

anyway thanks Sean, you're a big help in many channels 🙂

It's mentioned here https://clojure.org/guides/deps_and_cli#_using_local_libraries

And here in more detail https://clojure.org/reference/deps_and_cli#_dependencies

It doesn't "scale" so well if you're part of a team or want to use it in a CI pipeline, but it's fine for local development and testing.

Ahh, I'll try just reading through the whole deps guide. I kept searching for "clojure deps include .jar library" and similar terms, never thought of "local library"

As far as "scaling" this approach, I was just going to git submodule the dependency library. I could see how many layers of builds would take a long time / cause odd failures. Thankfully still at "fun personal project" size for now

For the up-coming release of Clojure CLI tools: I have been using clojure -R:cognitect-rebl -A:nrebl to run REBL with nREPL. The -R:congnitect-rebl load in dependencies from an alias that also has a :main-opts (which it not the main namespace to run) and -A:nrebl which has a :main-opts I do want to run. If I create a separate alias for cognitect-repl without the :main-opts and use clojure -M:rebl-deps:nrebl it works. Or I could just add all the rebl dependencies into the :nrebl alias. Does that seem the right approach?

@jr0cket I suspect if you used -A:cognitect-rebl:nrebl it would run the :main-opts from :nrebl since they overwrite each other -- last one wins.

You could also use -A:cognitect-rebl:nrebl -M:nrebl which would work on both today's stable version and the current prerelease version without the -A warning about using -M to run the :main-opts (since the "winning" :main-opts here is using -M).

I think the guidance from Alex has fairly consistently been to separate out :main-opts from other stuff if you need to mix'n'match main opts from several aliases. But the expansion of both -M and -X to respect :extra-deps and other stuff in the latest prerelease feels like the pressure is in the opposite direction -- toward bundling more things into a single alias for convenience @jr0cket Not sure how closely you've been following the discussions generally in this channel?

(but of course, at this point, there's still some potential for changes in the prerelease that the -A/`-M` usage recommendations may change)

I happened to watch Spec’able before reading Alex’s post. Seems pretty ironic.

Spec'able?

deps.edn didn't seem to pick up on the .jar (silent failure?) as I couldn't load any expected classes. I realized there's no pom.xml, probably why. At a minimum, I believe I need to tweak the library to use https://docs.gradle.org/current/userguide/maven_plugin.html

sorry, sepc’ulation

Ah yes, https://www.youtube.com/watch?v=oyLBGkS5ICk

(transcript: https://github.com/matthiasn/talk-transcripts/blob/master/Hickey_Rich/Spec_ulation.md )

TL;DR: don't break stuff, only add stuff and fix bugs 🙂

@quest You shouldn't need a pom.xml for this usage -- but if that Java library has any dependencies, you need some way to tell clojure about them (if the JAR contains a pom.xml the t.d.a machinery would take care of that, else you could just add those deps directly to your deps.edn file).

@seancorfield Hmm, I could uptake them into my deps.edn. Do you know of a way to determine if the jar was loaded or not? I've just been trying to run an :import on a class that I expect to exist, but wondering it there's a better way to poke around

Use the -Spath option on the clojure CLI to see what the classpath contains.

Or even inside your REPL: (System/getProperty "java.class.path")

user=> (run! println (clojure.string/split (System/getProperty "java.class.path") #":"))
/home/seanc/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar
/home/seanc/.m2/repository/org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar
/home/seanc/.m2/repository/org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar
src
nil
user=>

^ I figured there was something from the REPL, but -Spath is a good tip. Appreciate the knowledge-share, you had a hand in me starting adoption of deps.edn a year ago too and I doubt I would've tried it otherwise :thumbsup:

Cool. We've never regretted switching to CLI/`deps.edn` in 2018 at work.

(and we're using the bleeding edge prerelease version right now -- as of yesterday)

I've read the whole conversation in here and reading through the docs Alex wrote. Still digesting what it all means. I'm currently testing all the aliases I have to understand the changes

I'm working on updating clj-new to work with -X, much as I've updated depstar. I'll probably add :new-x and :depstar-x aliases to my .clojure/deps.edn file so folks can use those tools via -X as examples. It'll be a while before that can become the default 🙂

at least once you decide you are out of alpha

Feedback on the "WARNING: When invoking clojure.main, use -M" message (for @alexmiller) -- this usage produces that warning: clojure -m my.entry.point -- is that intentional? Do you really want folks to write clojure -M -m my.entry.point instead?

Yes

@jr0cket would love to hear your feedback too

Oh I definitely think that’s a factor

Is this to discourage that usage in favor of -X? It seems to be another maybe unnecessary breaking thing that's been there for a long time

It’s not to discourage it as much as that we are trying to open up the repl arg space for other things

And that means making clojure.main usage explicit

Interesting choice. A lot of tutorials out there talk about running simple Clojure projects with just clojure -m my.entry.point

I think beginners will (quite rightly) ask "Why do we have to specify two m options?"...

“Lot” = ? 2 or 3?

And if these tutorials transition into talking about -X...

We want clj to do more. In some cases that means restructuring these options. It’s either that, or clj2 and clojure2 which do not seem better to me

Sure, if all the books and tutorials that exist today are updated to use -X instead of -m, it's not going to be a problem 🙂

Thanks for balancing all these things in a spreadsheet so lots of concerns are managed. On a Saturday no less. Looking forward to the previews

(I just want to be clear that I think all the new functionality is great -- all of my concerns are around how existing (stable) functionality is deprecated and/or removed over time)

And, ultimately, I'm not going to be the one complaining about the changes since I'm already on the latest prerelease version on every machine and I'm aggressively adopting the new functionality!

(I am somewhat concerned about users of my CLI/`deps.edn`-based projects, of course 🙂 )

Coming in the next version of clj-new: https://github.com/seancorfield/clj-new#clojure--x-usage

There was a time when the clojure CLI wasn't driven by tools.deps.alpha right? Wasn't it non-alpha back then? Isn't only tools.deps alpha but the CLI / shell script now also got alpha?

(`depstar` added support for -X in its most recent 1.1.104 release, last week)

I could be misremembering things, but I think there was a clojure package already years ago, before tools.deps.alpha? Not sure what it did back then, but -m was maybe already a part of it back then

I know there was an unofficial clojure script floating around which behaved differently than the official one, once it appeared.

maybe it just did java -jar clojure.jar $@

Yeah, I think that was pretty much it. My recollection of the official CLI was that it appeared in tandem with t.d.a during the 1.9 cycle once Spec was split out of the core clojure.jar since you could no longer just run a REPL with the JAR file alone so it was supporting machinery for that... but I'm having a hard time finding any of the original posts about it (maybe I should scan back through Inside Clojure since I bet Alex blogged about it).

I've used Linux for many years, the brew stuff is extremely rare.

ah right, thanks

Hmm, the pre-1.10 Inside Clojure posts are fairly sparse...

This is the first mention of t.d.a on the Clojure mailing list: https://groups.google.com/g/clojure/c/FpMqtNu0TCE/m/wc2n1tPHBwAJ

(July 2017)

Yup, I'm just glad it exists for clojure. Getting stuff into the more standard installer repos on Linux seems very laborious? What relatively painless options exist for Linux package managers that the Clojure core team could leverage @dominicm?

Ha! 😂

If only you knew the rabbit hole you'd just asked about

Every distribution has its own rules and timelines. Consider Ubuntu LTS. They don't take changes. Canonical pay people to backport security patches. Every now and then you get a system upgrade (like mac) and that's when you get changes. New features, bug fixes, etc.

There are, of course cutting edge distributions like arch and void. They'll update whenever the package maintainer feels like it. Usually regularly for important changes.

Some distributions care about reproducibility. They have other constraints I don't understand.

I am a void clojure package contributor, and my life is pretty easy. The installer script was "ported" to the void build system in ~10 lines and it's just worked since then. Maybe I'll need to make changes with the new system.

The deps/cli reference page only talks about the library being in alpha. The guide doesn’t mention it at all, and besides “But, that is not to say just leave your thing 0.0.967. At a certain point, you are going to have users, and whether you change it to 1.0 or not, they are going to be depending on your stuff.”

Been playing with the new features, and it's nice that you can have the following

:new {:extra-deps {seancorfield/clj-new {:mvn/version "RELEASE"}}
        :exec-fn clj-new/create
        :exec-args {:template lib} ; default
        :main-opts ["-m" "clj-new.create"]}

That's probably what I'll do in my dot-clojure repo by way of education for folks and as a way to encourage folks to transition (or at least see what the options available are).