Clojurians Log v2

Alex, would it at all be in line with the direction of the clojure tools to allow -Spath '{:replace-deps {my/lib {,,,}}'. Or perhaps a switch like -Srepro that would turn off any local deps.edn sources?

So that running tools from the command-line that don't need the project's classpath could become a little less verbose?

See https://github.com/tonsky/uberdeps/issues/33 for example

@imre what were you doing when you wanted that feature?

uberdeps, for example, as linked above

Ah, I'd put an alias into -Sdeps for that, and activate it that way.

Yeah, I understand that's the current solution 🙂 Just asking whether it could be expected that an alias won't be needed sometime in the future

you can clj -Sdeps '{:aliases {:foo {:replace-deps {my/lib {,,,}} :main-opts ["-m" "my.main"]}}' -M:foo

I think there are definitely more things to do on tools and ad hoc tool usage like this is something that could be included

Thanks both of you. -M:foo is what we've been using since we realized it doesn't work as we expected without an alias 🙂 It is grand for the moment, although it won't work if there is an invalid deps.edn in the current folder. I look forward to future developments wrt tools and ad hoc usage improvements.

@borkdude (and others): I'm going to add a question on ask.clojure about having additional deps.edn files read in via clj/t.d.a. so I'd like some input from folks on what problem(s) this would solve for them -- in a thread please.

The first scenario that comes to my mind is a monorepo but Alex has asked for the problem statement to be couched in more general terms.

is this related to the CLJ_CONFIG + mvn-pom problem?

So I'd state the problem in these terms: 1. a development team works together on multiple projects that each have their deps.edn file, and 2. those developers have their own preferred tooling configured in their own ~/.clojure/deps.edn file, and 3. the team as a whole wants a level of consistency in terms of certain library or tooling versions across multiple projects, by providing some standardized aliases and some "pinned" versions (e.g., :override-deps). Currently, 1. and 2. are supported by the CLI/t.d.a. but there's nowhere to hang the multi-project standardization.

@borkdude no, not really. that problem is about how to convey cp modifications to -X programs

Currently, folks who want 3. are using CLJ_CONFIG to provide a team-standard deps.edn file but losing 1. in the process.

The cp/deps modification to -X programs is a separate (but also a real) issue that would benefit from some sort of standardized approach that could be used by tooling in general (not just mvn-pom). So let's keep that discussion separate.

@borkdude Does my three-point problem statement match up with your CLJ_CONFIG use case, or is yours a separate scenario?

A separate issue that needs to go on ask.clojure to get votes and feedback is that it is fairly common for tooling to be want to read the deps.edn environment as if it were under a set of aliases, i.e., for a tool to want to see the world as if a specific set of aliases were provided, that are different to the aliases needed to run the tool. Discussion in a thread...

This touches on how -X.. mvn-pom works, as well as other tools on https://github.com/clojure/tools.deps.alpha/wiki/Tools such as depot: the environment needed to run the tool is not the same as the environment the tool wants for analysis.

depstar "works around" this problem by relying on its own runtime environment being the same as what goes into the JAR file but just excluding itself. This is only tenable because it is a very small tool that has zero external dependencies. You can override this and tell depstar to use a different classpath, but that approach doesn't work for tools that actually want to look at the versions of dependencies, such as mvn-pom, depot, etc.

I do have this problem staked out in a table btw, just trying to decide on the right answer

1) yes, 2) yes. 3) we use :default-deps for this with nil versions (as documented).

not sure yet whether it should be a feature, a set of conventions, a set of conventions + code to use those conventions, etc.

Ah, OK. So no need for an "ask" on this one?

the datomic ion dev tools is another example

would be fine to have one, just by virtue of working on it for the datomic tools it's well on our radar already :)

Right now it works well enough for me. Any tools I need for development I can also just put in our work repo's deps.edn. But it would be nice if I should not have to do that.

In clj-kondo I have a concept called config-paths which is a list of dirs that possibly have a configuration for clj-kondo to be merged in. It's similar to CLJ_CONFIG, but CLJ_CONFIG only supports one and not a list of dirs.

Basically a classpath for config.

And your home dir's clj-kondo config is implicitly part of your config paths, unless you opt out of that

OK, I'll add a question for it then...

> depstar "works around" this problem by relying on its own runtime environment being the same as what goes into the JAR file but just excluding itself. I also have a couple of tools that just receive a --classpath arg. But it depends on the tool. For mvn pom that might not be so convenient, and getting a list of aliases to resolve versions with might be better.

I put a few constraints to get multiple unified deps configs to work well, i.e. subrepo aliases must be namespaced, since that makes them simple to merge and call from another directory

A prerelease version of clojure 1.10.1.739 is now available: • When resolving pom models for local deps, use System properties so jdk profile activation works • In tools.deps, updated specs to cover alias changes

let me know if you see anything weird

replied there 😄

@vlaaad Can you elaborate on how that would help with invoking mvn-pom?

Is there an example of what that first bullet means that you can link me to (or easily describe)?

not sure that's what was asked.. did I read it wrong?

The question is how can tools like mvn-pom be made aware of aliases that should be considered in adjunction to the local deps.edn

ah, well... tools like mvn-pom should accept basis as argument instead of a list of aliases. Converting aliases to basis should be the outer-most place in such tools.

I'm guessing that it means that when specifying a jvm-specific dependency, that now works. @vlaaad :)

@vlaaad how do you pass a "basis" from the command line?

you don't, that's why you need to use t.d.a as a library

huh, me?

That's not a useful answer to this question.

I mean, yes, of course the tool has to use t.d.a but the point of this is to standardize an API so that all tooling can work from a set of aliases without having to reinvent the wheel and rely on knowing so much about how t.d.a works under the hood.

@vlaaad didn't you ask for this in relation to jfx deps?

And this also builds on the other question I posted on ask.clojure which allows for additional deps.edn files to be included in the process.

well, cljfx uses activation profiles, but it always worked ¯\(ツ)/¯

We don't want every single tool out there to have to write that boilerplate code.

Oh. So now I'm extra confused about what's new!

here is hoping for tools.build 😄

That still wouldn't address either of these two issues.

(and we also wouldn't want to tie arbitrary tooling to tools.build usage either)

Updated. Will let you know if I run into any issues (but I doubt either of those changes will affect our workflow).

I have a feeling it is possible to build a variation of tools-deps that accepts multiple deps.edn files and configuration with aliases as args using tools.deps... as a library, that is then used with -X

These two issues both need to be solved inside t.d.a really.

Since -X mvn-pom needs a standardized way to do this, the solution will essentially be right there inside t.d.a -- once Alex et al have figured out the best way to solve it. I created the ask as a public placeholder for something that is already on Alex's plate, in a table of pros/cons for various solutions.

@alidcastano Interesting, but it looks/feels clunky as a workflow and is still just papering over the underlying problem -- which should really be solved inside t.d.a itself.

(monorepo is only a special case of the ask that I posted)

I think my suggestion answers your question 😕 There is a problem with mvn-pom , but any tool built on deps.edn can already build a basis given a set of aliases — which is what you asked

See https://ask.clojure.org/index.php/9847/how-do-i-resolve-maven-dependencies-for-a-local-jar for an example failure case

Wouldn’t be an issue with maven deps, only with local

I’d be interested in hearing what you feel is clunky about it. Unifying all subrepo deps and starting a clojure program with that unified config seems less clunky (imo) than overriding the cross-user project config and having to put all user dev related aliases into git history. i would have waited for this to be addressed in tools.deps but from slack searches, this issue has been present for over 2 years and only suggested solution in convos has been overriding the CLJ_CONFIG value, or using Lein. So I figured some experimentation in user space would be useful.

"having to put all user dev related aliases into git history" -- huh? No, no one is suggesting that.

I think you're right: it solves a problem users are experiencing now. Eventually this library might become obsolete, but in the meanwhile, I think some people may benefit from it.

Unless you mean "common tooling for the monorepo" -- in which case, yes, that sort of stuff should be under git.

@alidcastano Maybe I'm reading your project wrong, but it looks like you would need to run that preprocessing step to create the actual deps.edn file after any changes?

Your suggestion is actually the problem that I want solved. It isn't a solution 🙂

It’s not just aliases, there are other relevant params like -Srepro

We don't want every tool to have to write the same code over and over.

@seancorfield I think what he's doing is spitting out a deps.edn at CLI invocation time and then your script invokes clojure using that deps.edn: https://github.com/rejoice-cljc/interdep/blob/c248b47d9e4a6f45982cac41f2ab3990d6f5e310/example/scripts/run.clj#L60

Yes, incorporating how -Srepro affects the merging of deps.edn would be important too. I don't use that flag right now because of course we have to rely on CLJ_CONFIG to "insert" that extra deps.edn file, which is why I didn't think to include it in the question 😐

so it writes on the fly, while you're invoking your commands

so the used deps.edn is ephemeral

which is a robust way of working around the current issues, since reading a single deps.edn is well supported

I understand that this is not the way things should end, but it's a nice temporary hacky workaround :)

> “having to put all user dev related aliases into git history” -- huh? No, no one is suggesting that. @seancorfield I imagined that by overriding CLJ_CONFIG and not having a user cross project config, that devs would need to put their own custom configurations in git history. but perhaps you work around that, too, not sure.

@alidcastano I see your example script runs clojure in a .main subdir. Doesn't this cause issues with relative file paths in an app?

right now this is a simple library that provides utilities for processing nested deps.edn configurations, along with constraints that standardize the experience, document it, and make it more robust. it’s up to the user how they want to start their program, since there’s many different ways someone may want to do that. the way the example in the repo does it, is it calls the two helper utilities (`multi-repo/process-deps` and multi-alias/with-profiles) and those return a map with the values of the main deps config and the activated aliases. then it writes that deps config to a subdir and uses babaskha to start the clojure process from there https://github.com/rejoice-cljc/interdep/blob/c248b47d9e4a6f45982cac41f2ab3990d6f5e310/example/scripts/run.clj#L57-L72

I think it's better to start clojure from the directory you are in. So maybe it should support reading files that are not named deps.edn so it can spit out to deps.edn locally (because clojure doesn't currently have the option to point at another local deps2.edn file)

Note that https://github.com/borkdude/deps.clj (also works with babashka btw) supports -Sdeps-file. I'll end the hackathon discussion here. Going to bed now :).

> I see your example script runs `clojure` in a `.main` subdir. Doesn’t this cause issues with relative file paths in an app? @borkdude multi-repo/process-deps accepts :out-dir option that makes any paths relative to the desired directory but reading deps file from root directory is a good idea. I’ll see if that can be done with tools.deps; thanks for feedback, rest well!

@alidcastano > I imagined that by overriding CLJ_CONFIG and not having a user cross project config, that devs would need to put their own custom configurations in git history. but perhaps you work around that, too, not sure. Yeah, it's definitely a compromise. We have an agreed subset of tooling that's in the "master" deps.edn file, that satisfies everyone's needs for working on the monorepo. Luckily, we're all fans of fairly minimal tooling so we work with either plain REPLs or Socket REPLs. Mostly, the shared dev aliases are around testing and packaging (which of course we want standardized across the team, across all subprojects) and we have a :dev alias that points at a subproject in the monorepo that knows how to start various REPLs/tooling, based on whatever happens to be on your classpath (i.e., based on the set of aliases you used).

I think if we decided to switch to some sort of rewriting tool at startup, we'd want to keep it pretty simple: just slurping the "master" deps.edn file, reading it as EDN, sucking out a couple of the aliases, and just assoc'ing those into the local project deps.edn and writing it back out. FWIW, we already have a tool that merges all the subproject deps.edn files and writes to an everything pseudo-subproject that we can use as a convenient place to start a REPL if we want the entire codebase available (which is normally how I work). We manually build it, as needed, and we keep it under git. It still assumes the "master" deps.edn file is in play with its :defaults alias to specify :override-deps for everything.