tools-deps

Discuss tools.deps.alpha, tools.build, and the clj/clojure command-line scripts! See also #depstar #clj-new
salam 2020-07-23T03:06:30.476700Z

a better approach would be to create an alias in the project deps.edn: https://github.com/seancorfield/dot-clojure/blob/develop/deps.edn#L66 i'm a cursive user myself and this is how i start a socket repl server in the terminal ( clojure -O:socket ) and connect cursive (a socket repl client) to it by creating a remote socket repl run configuration in intellij idea.

tengstrand 2020-07-23T05:48:28.482400Z

I can set up my deps.edn at the root so that it includes a "module" (that has its own deps-edn) like this:

:aliases  {:dev {:extra-deps {env/dev {:local/root "environments/dev"}}}
           :test {:main-opts ["-m" "polylith.clj.core.cli.poly" "test"]}}
Then I can run: clj -A:test:dev ...and it will run my tests (that is what the 'test' alias does for me). But the deps.edn under environments/dev contains its own aliases with some :extra-paths that I want to include also. Is that possible? I imagine that I need to do something like this (which doesn't work): clj -A:test:dev:dev/local-alias ...where local-alias is defined as an alias in environments/dev/deps.edn:
:aliases {:local-alias {:extra-paths ["../../components/help/src"]}

tengstrand 2020-07-24T06:04:53.017700Z

Okay, but here I suggest to use existing aliases that lives in environments/dev/deps.edn, that we "import" from deps.edn at the root. Then we don't "mess up things from the outside" just selecting aliases within that "imported" deps.edn. If I "cd environents/dev" first, then I have access to all those aliases which have the purpose to add :extra-paths which I need. But here I have several "sub modules" (not just the one in this example) and wants to do the same from the root deps.edn without having to manually merge all those deps.edn files into one huge deps.edn at the root.

seancorfield 2020-07-24T06:06:49.017900Z

As I said @tengstrand "That would be kind of dangerous".

seancorfield 2020-07-24T06:08:28.018100Z

:local/root isn't special -- it's just another way to identify a dependency. You're asking for dependencies that include deps.edn to have their aliases combined with the current project. That's not a practical way to deal with reproducibility in project builds.

tengstrand 2020-07-24T06:11:44.018300Z

Then I guess I need to solve it with code generation or create my own REPL or similar.

seancorfield 2020-07-24T06:15:43.018500Z

What we do at work, with our 100k+ line monorepo containing 30+ subprojects, is that we have a special subproject (called versions) that contains the "master" deps.edn that has all the aliases needed by all projects, including :defaults which brings in all the :override-deps to pin versions etc, and then all the other subprojects really just have :deps and an alias for :test-deps to add anything needed for testing that subproject (`versions/deps.edn` has a :test alias for all the common testing infrastructure). Then we use CLJ_CONFIG=../versions clojure -...aliases in each subproject.

seancorfield 2020-07-24T06:16:36.018700Z

CLJ_CONFIG lets you treat a specific folder as if it contains your user-level deps.edn (and so any actual ~/.clojure/deps.edn file is ignored -- which is good: the monorepo then control all dependencies and aliases!)

tengstrand 2020-07-29T15:24:31.311Z

Thanks for the information, can be very useful!

p-himik 2020-07-23T08:54:01.482800Z

There's no way to prevent clj from reading deps.edn in the current directory, is there?

borkdude 2020-07-23T09:56:20.483300Z

@p-himik don't think so.

p-himik 2020-07-23T09:59:15.484600Z

Thanks. That's seems a bit inconvenient because if I want to run some tool with -Sdeps (like Capsule in my particular case) within my project, I cannot really do it without mixing two different classpaths.

plexus 2020-07-23T10:02:15.485300Z

it's fairly annoying, I've had scripts that temporarily move deps.edn out of the way because of this...

borkdude 2020-07-23T10:04:54.486100Z

deps.clj, an unofficial re-implementation of the clj script does have an option to choose a different deps.edn file than the one in the current directory

vlaaad 2020-07-23T10:05:23.486700Z

I think setting :paths in aliases to replace source path from deps.edn is as close as it gets to what you want

vlaaad 2020-07-23T10:06:13.487Z

and :deps

borkdude 2020-07-23T10:06:38.487600Z

it's also possible to remove deps from the classpath in an alias: https://github.com/borkdude/babashka/issues/152

vlaaad 2020-07-23T10:06:39.487700Z

see https://clojure.org/reference/deps_and_cli#_aliases (“-A” section)

p-himik 2020-07-23T10:07:33.488500Z

But then I'd have to restructure my whole deps.edn just to make some thirdparty tool happy. That's not good.

vlaaad 2020-07-23T10:08:19.489200Z

not sure I’m following, can you put that third-party tool as a user-level alias?

borkdude 2020-07-23T10:08:44.489900Z

you can also manually construct the classpath (using a script) and pass it to -Scp

vlaaad 2020-07-23T10:08:53.490200Z

that way you don’t touch your project’s deps.edn at all

p-himik 2020-07-23T10:09:40.490300Z

The tool itself - yes. The thing is, it must be run in the presence of deps.edn. But it would be best to not let clj (and I run the tool using clj) also interpret deps.edn.

p-himik 2020-07-23T10:10:15.490500Z

The tool itself is used via -Sdeps, so it's not in the deps.edn.

p-himik 2020-07-23T10:14:15.490900Z

Oh, it's now possible to use :deps in aliases. There's hope!

borkdude 2020-07-23T10:15:47.491100Z

That overrides the base deps?

p-himik 2020-07-23T10:15:59.491400Z

Yep, seems like it.

p-himik 2020-07-23T10:16:22.491900Z

So basically I should be able to mimic an empty deps.edn. Will experiment later today.

borkdude 2020-07-23T10:16:54.492200Z

note that you will still have clojure and spec on the cp by default

p-himik 2020-07-23T10:17:52.492800Z

Hmm, right. But that should've really interfere with tools that are already meant to be run with clj.

borkdude 2020-07-23T10:27:31.493500Z

I also use deps.edn for babashka projects, the extra deps don't hurt, but they can make searching the classpath a tad slower (like a few ms maybe)

p-himik 2020-07-23T10:29:55.493600Z

In my case, they don't anything really bad as well, but I really want to get rid of those slf4j warnings.

alexmiller 2020-07-23T12:27:56.494700Z

yes, using :deps / :paths in an alias is what we call "tooling" and will replace the project :deps/:paths (for the purpose of using a tool that shouldn't use the project classpath)

alexmiller 2020-07-23T12:28:43.495500Z

until yesterday, you could only use those via -A, but as of the latest dev release yesterday, tooling -T is now a thing

p-himik 2020-07-23T12:38:02.495700Z

Awesome, thanks!

alexmiller 2020-07-23T12:40:41.495900Z

more to come on that and other changes in latest, just been kind of busy with ... other things :)

👍 1
Tamizhvendan S 2020-07-23T13:06:54.497100Z

Hi, I am getting FileAlreadyExistsException while trying to build an uberjar Here is my deps.edn file

{:paths ["src" "resources"]
 :deps {io.appium/java-client {:mvn/version "7.3.0"}
        org.clojure/data.xml {:mvn/version "0.0.8"}
        metosin/reitit-ring       {:mvn/version "0.5.1"}
        metosin/reitit-middleware {:mvn/version "0.5.1"}
        metosin/reitit-spec       {:mvn/version "0.5.1"}
        mount                     {:mvn/version "0.1.16"}
        ring/ring-jetty-adapter   {:mvn/version "1.7.1"}
        org.slf4j/slf4j-simple    {:mvn/version "1.7.30"}}
 :aliases {:dev     {:extra-paths ["dev"]
                     :extra-deps  {org.clojure/tools.namespace {:mvn/version "1.0.0"}}}
           :uberjar {:extra-deps {seancorfield/depstar {:mvn/version "1.0.94"}}
                     :main-opts  ["-m" "hf.depstar.uberjar"]}}}

Tamizhvendan S 2020-07-23T13:07:49.497600Z

Here is the command that I am running clj -A:uberjar my-app.jar

Tamizhvendan S 2020-07-23T13:08:17.497900Z

Building uber jar: my-app.jar
Execution error (FileAlreadyExistsException) at com.sun.nio.zipfs.ZipFileSystem/createDirectory (ZipFileSystem.java:422).
META-INF/versions/9/

Tamizhvendan S 2020-07-23T13:09:19.498300Z

Am I missing anything obvious here?

cap10morgan 2020-07-23T17:19:37.002500Z

What are folks using to store / set their project's version number so it makes it into the generated pom.xml (in an automated way; think CD pipeline, not hand-editing the pom.xml)?

seancorfield 2020-07-23T17:31:23.003200Z

I'm treating the pom.xml as the "system of record" for the version number and deriving it elsewhere from the POM file.

seancorfield 2020-07-23T17:33:16.005300Z

(and that's the workflow that I use for all my deps.edn-based projects that get deployed to clojars -- note that Contrib libs such as core.cache, core.memoize, tools.cli that I maintain already follow this workflow with "Maven Release" process in CI editing the pom.xml to change the version number explicitly based on the version you enter in that Jenkins form)

cap10morgan 2020-07-23T19:53:22.005500Z

@seancorfield OK, thanks. What tools do you use to read / bump it in the POM file? Curious if you have anything resembling the lein release workflow going there.

alexmiller 2020-07-23T20:00:20.006Z

there are maven plugins for that although it's all pretty tedious

alexmiller 2020-07-23T20:01:05.006800Z

the mvn release / deploy infrastructure does a lot for you but you have to buy pretty far into it all the way for it to be worth doing

alexmiller 2020-07-23T20:01:20.007200Z

(but it's a lot less fragile than the lein release stuff)

cap10morgan 2020-07-23T20:02:53.008Z

Thanks Alex. lein release can indeed be fragile. 🙂

alexmiller 2020-07-23T20:12:44.008800Z

the idea to put the version inside a file was probably bad from the beginning

seancorfield 2020-07-23T20:13:50.010100Z

@cap10morgan For my clojars-deployed stuff, I just update it as part of my general release process where I'm updating the README and the CHANGELOG and I update both the <version> and the <tag> fields in the pom.xml file all at the same time.

seancorfield 2020-07-23T20:14:57.011300Z

(so I grep for <old-version>|<new-version> so I can see all the places that refer to the current release and what will be the next one, and then I can visually verify that all references get changed)

seancorfield 2020-07-23T20:15:52.012300Z

In most of my libs, that includes several places in the README and usually in the documentation files as well that refer to installation/getting started and when new features were introduced.

alexmiller 2020-07-23T20:19:06.012600Z

I've automated that step on some libs :)

seancorfield 2020-07-23T20:21:07.013100Z

I still have artisanal documentation 🙂

slimslenderslacks 2020-07-23T22:35:28.013500Z

hey @cap10morgan, we're treating git tags as the source of truth for versions. When we deploy, the version in the generated pom.xml is made to be in sync with "git describe". Since the pom.xml is only ever needed for the deploy action, the pom.xml doesn't actually have to be checked in.

seancorfield 2020-07-23T22:42:34.013900Z

Note that for http://clojars.org and also for http://cljdoc.org there's an expectation that pom.xml contains a lot more than just the dependencies and the version, so having it as a "loose" file outside git would be... ill-advised if the generation process doesn't include all that information (which lein does put there, but clojure -Spom does not, so I'm talking about deps.edn projects -- note that clj -A:new would generate initial pom.xml files with all the appropriate fields for http://clojars.org and http://cljdoc.org)

slimslenderslacks 2020-07-23T22:51:34.014300Z

Ya, our process is still using lein for the pom.xml generation. That's super interesting @seancorfield - I was probably going to hit this in the near term as I had been planning on migrating to use clojure -Spom. Thanks!

seancorfield 2020-07-23T22:53:34.014500Z

clojure -Spom is fine for updating <dependencies> in an existing pom.xml file but it only generates a very bare-bones version if no file exists. That's why I added a full-featured pom.xml to the templates that clj-new uses for app, lib, and template.

slimslenderslacks 2020-07-23T22:57:44.014700Z

When git describe is the source of truth for version maybe we'd compose clojure -Spom to update the dependencies, with a git-describe->pom-version , and keep a clj-new generated pom.xml checked in.

slimslenderslacks 2020-07-23T23:01:47.014900Z

It sounds like a checked in pom.xml might currently be the best place to version the extra metadata that clj-new adds.

seancorfield 2020-07-23T23:05:10.015200Z

clj-new creates a baseline pom.xml. It never updates it after that.

👍 1
seancorfield 2020-07-23T23:06:52.015600Z

clojure -Spom updates an existing pom.xml with the latest dependencies (from deps.edn) so as long as you always do it prior to a build/deploy cycle, I guess those changes don't really matter.

seancorfield 2020-07-23T23:07:23.015800Z

Your git describe update also seems to be "do it before build/deploy" but don't really care about the changes.

seancorfield 2020-07-23T23:08:06.016Z

However, a lot of people would be very confused by a pom.xml file in a repo that didn't match the dependencies in deps.edn and/or didn't seem to reflect the current released version 🙂

seancorfield 2020-07-23T23:08:35.016200Z

So whether you check it in or not after those changes probably depends on whether it's going into a public repo I suppose 🙂

slimslenderslacks 2020-07-23T23:12:24.016500Z

Ya, I agree. It could be confusing to see a pom.xml with a weird version. You'd almost want the checked in copy to have a version of :project-uses-git-tags , and maybe have an empty dependencies, so it appears to be there just to define the extra maven stuff that you can't derive from elsewhere.

slimslenderslacks 2020-07-23T23:13:54.016700Z

I don't mind having a file with the correct version, I was more just trying to avoid the commits that go along with those workflows.

slimslenderslacks 2020-07-23T23:14:32.016900Z

Our unit tests and commit messages are the signals the produce Tags, and Tags are the signals that create new publishes to maven repos.

slimslenderslacks 2020-07-23T23:15:21.017100Z

(from a CI/CD perspective)

seancorfield 2020-07-23T23:23:43.017400Z

Yup. I think internal CI/CD processes are under different constraints and, unless you are publishing every successful build to (public) Maven Central / Clojars and also publishing every version to a (public) GitHub repo or similar, for internal releases you can definitely just keep a baseline pom.xml under version control and update that as needed for each build to your repo (and throw it away afterward).