tools-deps

Discuss tools.deps.alpha, tools.build, and the clj/clojure command-line scripts! See also #depstar #clj-new
Oliver George 2020-07-28T03:12:09.048500Z

I'm curious how RELEASE resolves in this maven dep...

{:extra-deps {clj-kondo {:mvn/version "RELEASE"}}
   :main-opts  ["-m" "clj-kondo.main"]}
Perhaps it's a normal maven feature. Hard to google for "release"!

alexmiller 2020-07-28T03:26:56.049200Z

RELEASE (and LATEST) are special "virtual" version numbers that tell Maven to look for the newest released version (non snapshot)

alexmiller 2020-07-28T03:27:30.049900Z

they are officially unsupported by clj (but do kind of work with some big caveats)

alexmiller 2020-07-28T03:28:08.050700Z

big caveat #1 - they break caching and you will never see a new release after the one that it first resolves to unless you -Sforce (or otherwise cause a cache recompute)

alexmiller 2020-07-28T03:29:50.052600Z

at some point, we may actually prevent caching at all if you use these "versions"

alexmiller 2020-07-28T03:30:42.053100Z

subtle caveat #2 - the answer you get is only as good as the metadata index in the repository. for maven or clojars, no worries. if you're using a bespoke s3 repo, then you probably aren't updating the right metadata files and you usually won't see the newest version

alexmiller 2020-07-28T03:32:04.054400Z

generally, I think it's harmless to do this in -Sdeps to grab "latest". it's mildly bad to do it for tools in your deps.edn aliases (like the example here). it's very bad to do it for your project deps.

2020-07-28T05:21:53.054800Z

Does there happen to be an equivalent for git/sha coordinates? That could be quite handy…

alexmiller 2020-07-28T07:09:04.080500Z

well, that's all of the work. not much point to caching otherwise

alexmiller 2020-07-28T07:11:21.080700Z

expanding and downloading a full deps tree can easily mean dozens of jars and many megabytes of network traffic (if your m2 and/or git cache is cold)

alexmiller 2020-07-28T07:16:04.080900Z

even if you have all the jars, expanding the tree is more than you'd want to wait, example in tools.deps.alpha repo (~80 jars in the full tree), all jars in cache so no downloads:

alexmiller 2020-07-28T07:16:22.081100Z

amac:tools.deps.alpha alex$ time clj -Sforce -e nil

real	0m3.028s
user	0m12.567s
sys	0m0.756s
amac:tools.deps.alpha alex$ time clj -e nil

real	0m0.745s
user	0m1.647s
sys	0m0.133s

alexmiller 2020-07-28T07:17:39.081300Z

(one caveat here is that the clj uber jar is not currently aot compiled - that's coming soon and will improve that first time, which includes a lot of loading)

alexmiller 2020-07-28T07:18:22.082100Z

but it's still not time anyone wants to wait

alexmiller 2020-07-28T07:20:51.083300Z

with a cold m2 cache, time is about 7.5 s (aot may knock a couple seconds off that)

2020-07-28T15:00:22.113800Z

I can see why that approach will be faster, though it breaks down (cache becomes stale) in the presence of dependencies that are expressed using a “floating” version reference (RELEASE, LATEST, SNAPSHOT, etc.). Those were deprecated in Maven3, and perhaps tools.deps should follow suit and emit a warning when it encounters such a thing?

alexmiller 2020-07-28T15:00:46.114Z

perhaps

1👍
seancorfield 2020-07-28T05:24:33.055400Z

@pmonks My dot-clojure deps.edn shows how you can resolve the head of the default branch -- in the comments.

seancorfield 2020-07-28T05:26:00.055900Z

And then (load-master 'clojure/tools.trace)

seancorfield 2020-07-28T05:26:36.056600Z

(following the http://github.com URL pattern, not the org.clojure/tools.trace Maven artifact pattern)

2020-07-28T05:41:29.057600Z

Potentially dumb question, but is it expected that code inside an EDN file would be executed? Or does that rely on a coincidence of how the Clojure EDN reader functions?

seancorfield 2020-07-28T05:53:23.057900Z

@pmonks Can you give a specific example?

seancorfield 2020-07-28T05:54:01.058800Z

I mean, EDN can contain tagged literals and you can provide reader functions that process those... but that's not exactly "code inside an EDN file".

alexmiller 2020-07-28T05:58:57.059500Z

edn files are data, not code, and are not evaluated

alexmiller 2020-07-28T06:00:33.061100Z

They are read but that will only construct tagged literals, and clj does not provide a way to install readers for tagged literals

alexmiller 2020-07-28T06:01:35.061700Z

Intentionally not, for the same reasons re caching

2020-07-28T06:03:04.062800Z

@seancorfield your example includes code (`(require '[clojure.tools.gitlib :as gitlabs]) …`) - how is that code evaluated?

2020-07-28T06:04:59.062900Z

I guess I don’t understand the caching issue. If you resolve “RELEASE” to a concrete version, and that version happens to exist in the cache already, wouldn’t tools.deps use it?

2020-07-28T06:05:28.063700Z

(and conversely, if it doesn’t, wouldn’t that version be added to the cache, and thereby eligible for future “cache hits”)

alexmiller 2020-07-28T06:06:15.065100Z

The caching is based on time stamps of deps,edn files. If the file never changes (set to RELEASE), the class path is never recomputed, even if newer versions come out

seancorfield 2020-07-28T06:06:40.065700Z

Those are comments showing what you would execute in your REPL...

alexmiller 2020-07-28T06:06:57.066400Z

Caching of classpath (not lib version)

seancorfield 2020-07-28T06:07:03.066700Z

The files contains lots of comments explaining how to use the aliases, with links to repos.

2020-07-28T06:08:24.066900Z

Ah ok. I assume there’s a good reason for making the deps.edn file the cache key, rather than just recomputing dependencies the classpath every time and having those in the cache?

alexmiller 2020-07-28T06:09:17.068500Z

Recomputing deps starts a second JVM so it’s at least a second, usually more depending on io

2020-07-28T06:09:30.069Z

Oh ok. So I would not put those in a deps.edn then.

seancorfield 2020-07-28T06:09:52.069900Z

Huh? Those are comments and they are in a deps.edn file.

alexmiller 2020-07-28T06:09:53.070Z

This is why clj starts faster than lein usually

alexmiller 2020-07-28T06:10:15.071100Z

It also goes to build repeatability

2020-07-28T06:10:16.071400Z

Yes they are, and as a result I would expect to uncomment them and have them Just Work:tm: .

seancorfield 2020-07-28T06:10:23.071800Z

I mean, that repo literally is my ~/.clojure/deps.edn file.

2020-07-28T06:10:36.072100Z

But if I’m following you, you’re saying that’s not correct - that those lines need to be copypasta’ed into a REPL.

seancorfield 2020-07-28T06:11:03.072600Z

Forgive me but I'm a bit shocked you would think code in EDN files would be evaluated...

2020-07-28T06:11:14.073Z

I didn’t - that’s why I asked the question.

alexmiller 2020-07-28T06:11:29.073800Z

Or “clj repeatability”

2020-07-28T06:11:48.074300Z

I was quite surprised to see commented out code in an EDN file.

2020-07-28T06:12:26.074500Z

(and by “quite”, I mean “very”…)

2020-07-28T06:13:37.075900Z

Regardless, @alexmiller separately explained that by design you can’t do an equivalent of RELEASE for git/sha coords in a deps.edn file.

seancorfield 2020-07-28T06:13:40.076Z

https://github.com/practicalli/clojure-deps-edn/blob/master/deps.edn contains comments with lots of shell commands, and even some elisp -- you wouldn't expect any of that to be executed just by reading the EDN file, would you?

2020-07-28T06:14:34.076400Z

Why would you assume I’ve read some random deps.edn file? Sadly I don’t have that much disposable time…

seancorfield 2020-07-28T06:17:03.076600Z

Well, you read my file and came to a ... strange conclusion ... so I offered another example deps.edn and wondered what conclusion you would come to from that...

2020-07-28T06:17:04.076800Z

Ok. I’ll assume there’s a good reason for needing a second JVM to do that (not requesting an explanation, btw, unless you’d like to provide one), and understood full well the performance implications of starting a JVM.

2020-07-28T06:18:17.077Z

If you re-read my original question (https://clojurians.slack.com/archives/C6QH853H8/p1595913713054800), you might see why I was assuming a deps.edn-only solution, and therefore didn’t understand your non-`deps.edn` answer.

2020-07-28T06:18:58.077300Z

In general, answering questions people didn’t ask is a wonderful source of confusion. 😉

seancorfield 2020-07-28T06:20:36.077500Z

With the Clojure CLI/`deps.edn` the "equivalent" to Maven/Leiningen/etc is often "some code that you run" -- which seemed like an acceptable answer to your question.

seancorfield 2020-07-28T06:21:31.077700Z

That's why I was so puzzled by your follow-up question, as it seemed unrelated to my answer (I was thinking "No, of course reading EDN won't execute code!").

seancorfield 2020-07-28T06:22:37.077900Z

I guess I should add "Here's some code you can run in your REPL" to my deps.edn file where there are code examples 🙂

1☝️
2020-07-28T06:23:04.078200Z

That would have avoided any confusion, no doubt. 😉

seancorfield 2020-07-28T06:24:20.078400Z

It is sometimes so hard to predict your audience's state of mind 😆 (when writing documentation, I should add).

1☝️
2020-07-28T06:27:40.078800Z

Professional technical writing is such an under-appreciated skill, I’ve found.

alexmiller 2020-07-28T06:41:24.079Z

it's running a Clojure program with a different classpath the program you're executing

alexmiller 2020-07-28T06:43:23.079300Z

specifically, it's running tools.deps to compute the classpath based on your deps.edn (talking to Maven, maybe S3, maybe Git, etc). in the end it spits out the results in cache. Your actual program is then started with the classpath you intended. Once we've cached that work, we can spit the first part and just run the program.

2020-07-28T06:52:28.079500Z

So the advantage of caching based on the deps.edn file, rather than the classpath it contains, is all the I/O to talk to Maven, S3, git, etc.?

alexmiller 2020-07-28T07:09:04.080500Z

well, that's all of the work. not much point to caching otherwise

alexmiller 2020-07-28T07:11:21.080700Z

expanding and downloading a full deps tree can easily mean dozens of jars and many megabytes of network traffic (if your m2 and/or git cache is cold)

alexmiller 2020-07-28T07:16:04.080900Z

even if you have all the jars, expanding the tree is more than you'd want to wait, example in tools.deps.alpha repo (~80 jars in the full tree), all jars in cache so no downloads:

alexmiller 2020-07-28T07:16:22.081100Z

amac:tools.deps.alpha alex$ time clj -Sforce -e nil

real	0m3.028s
user	0m12.567s
sys	0m0.756s
amac:tools.deps.alpha alex$ time clj -e nil

real	0m0.745s
user	0m1.647s
sys	0m0.133s

alexmiller 2020-07-28T07:17:39.081300Z

(one caveat here is that the clj uber jar is not currently aot compiled - that's coming soon and will improve that first time, which includes a lot of loading)

alexmiller 2020-07-28T07:18:22.082100Z

but it's still not time anyone wants to wait

2020-07-28T07:20:31.083200Z

Hey guys. If i have a setup as described in https://clojure.org/guides/deps_and_cli#_using_local_libraries. is there a way to fire up a repl in the hello project which reloads the time-lib deps if they are changed?

alexmiller 2020-07-28T07:20:51.083300Z

with a cold m2 cache, time is about 7.5 s (aot may knock a couple seconds off that)

alexmiller 2020-07-28T07:22:14.084100Z

no, that's not a built-in feature of Clojure (or Java). there are ways to do some variants of this but it's not common

alexmiller 2020-07-28T07:22:28.084300Z

well, do you mean local dir or local jar?

alexmiller 2020-07-28T07:23:02.085Z

if local dir, you can just use Clojure's normal :reload or :reload-all mechanism built into require

2020-07-28T07:23:24.085300Z

i meant local dir

alexmiller 2020-07-28T07:23:37.085700Z

yeah, so that's just normal reloading

2020-07-28T07:23:45.085900Z

ok. cool. thx

2020-07-28T13:06:18.086600Z

any idea why running brew upgrade clojure/tools/clojure installs 1.10.1.561 not 1.10.1.590 ?

alexmiller 2020-07-28T13:28:48.086800Z

561 is the newest stable release

alexmiller 2020-07-28T13:28:58.087Z

590 is a dev release

1👍
2020-07-28T15:00:22.113800Z

I can see why that approach will be faster, though it breaks down (cache becomes stale) in the presence of dependencies that are expressed using a “floating” version reference (RELEASE, LATEST, SNAPSHOT, etc.). Those were deprecated in Maven3, and perhaps tools.deps should follow suit and emit a warning when it encounters such a thing?

alexmiller 2020-07-28T15:00:46.114Z

perhaps

1👍
seancorfield 2020-07-28T16:10:48.115Z

Following up from my comment in the announce thread:

{:paths [:clj-paths :resource-paths]
 :aliases
 {:clj-paths ["src/clj" "src/cljc"]
  :resource-paths ["resources"]}}
Is that correct, or should :clj-paths and :resource-paths be underneath an alias?

seancorfield 2020-07-28T16:11:24.115800Z

If that is correct, I don't see the value since they are fixed labels and values -- am I missing something @alexmiller?

alexmiller 2020-07-28T16:12:28.116100Z

oh, that is right - they are aliases

alexmiller 2020-07-28T16:12:40.116300Z

an alias is just a name for some data

alexmiller 2020-07-28T16:13:37.117200Z

this was the original meaning here but we have kind of gotten off talking about things that way by tying it into clj's usage of alias data to affect classpath construction

seancorfield 2020-07-28T16:14:11.117700Z

OK, so how is that useful since they are just fixed values?

alexmiller 2020-07-28T16:14:20.118100Z

at the moment, it's not very useful

seancorfield 2020-07-28T16:14:33.118800Z

Also, you have this paragraph twice "Replace project environment ("tool")" -- one with plenty of detail, one with very little.

alexmiller 2020-07-28T16:14:36.118900Z

but if you had say, a build tool that wanted to also use a subset of paths it might be useful

alexmiller 2020-07-28T16:14:42.119100Z

;)

alexmiller 2020-07-28T16:16:01.119500Z

that's actually intentional - the first part is kind of an overview, and the second is more how the tool works with more detail

alexmiller 2020-07-28T16:16:35.119700Z

doing a lot of heavy editing on this page lately but will likely change more over time

seancorfield 2020-07-28T16:17:13.119900Z

OK, I think it's just really confusing at the moment with two identical headers...

seancorfield 2020-07-28T16:17:37.120500Z

The Maven install example is missing an alias tho' (even if the paths one isn't), yes?

{:aliases
 {:fn clojure.tools.deps.alpha.install/install
  ;; :args map could be provided but can pass on command line instead
  }}

alexmiller 2020-07-28T16:17:38.120600Z

I'm ok with that for the moment :)

alexmiller 2020-07-28T16:17:55.120800Z

the toc at the top side bar may make the structure more evident

alexmiller 2020-07-28T16:18:25.121500Z

I've gone through about 5 different structures for this page recently, and I don't think I'm done

1😆
seancorfield 2020-07-28T16:18:28.121800Z

That's meant to be executed with -X:some-alias (most people will pick -X:install I suspect)

alexmiller 2020-07-28T16:18:44.122100Z

ah, yes that is

alexmiller 2020-07-28T16:18:53.122400Z

I actually saw that late last night and forgot to fix

seancorfield 2020-07-28T16:20:39.123700Z

I'm curious as to why that ability surfaced as a built-in option, given that JAR files can already be used as :local/root deps?

alexmiller 2020-07-28T16:21:24.124Z

it's not a built-in option, it's a built-in program

alexmiller 2020-07-28T16:22:01.124700Z

didn't seem big enough to make into its own thing, depended heavily on all the code in tools.deps, wanted it for dev-local, etc

seancorfield 2020-07-28T16:22:16.124900Z

Interesting...

alexmiller 2020-07-28T16:22:23.125100Z

serves as a good example :)

alexmiller 2020-07-28T16:22:59.125500Z

the plan is to shift some of the magic switches in clj over to just be programs you can -X

alexmiller 2020-07-28T16:23:33.125900Z

some of those might become built-in to the root deps.edn, not sure yet

seancorfield 2020-07-28T16:23:40.126Z

On another wordsmithing issue: now that page refers to "tool" as something that might want to override :deps and/or :paths, this section name might be confusing (a different meaning of tools) https://clojure.org/reference/deps_and_cli#_clojure_tools_usage

alexmiller 2020-07-28T16:24:05.126200Z

yeah, I've been fighting that, haven't decided what to do about it yet

alexmiller 2020-07-28T16:24:32.126400Z

not enough words :)

alexmiller 2020-07-28T16:24:40.126600Z

it's like naming things is hard

seancorfield 2020-07-28T16:28:10.126800Z

Tools are just so useful 🙂

seancorfield 2020-07-28T16:30:02.127300Z

The install line you gave in the announcement doesn't seem to work

Error: No available formula with the name "clojure/tools/clojure@1.10.590" 
==> Searching for a previously deleted formula (in the last month)...
Error: No previously deleted formula found.
==> Searching for similarly named formulae...
Error: No similarly named formulae found.

seancorfield 2020-07-28T16:30:23.127500Z

That's from brew install clojure/tools/clojure@1.10.590

seancorfield 2020-07-28T16:30:31.127800Z

(after a brew uninstall clojure)

seancorfield 2020-07-28T16:33:16.128500Z

Should be 1.10.1.590

alexmiller 2020-07-28T16:33:29.128700Z

Ah, shoot

alexmiller 2020-07-28T16:34:41.128900Z

Edited

1
2020-07-28T16:54:37.130100Z

wow need to get my head around these new tools.deps features; they might make some stuff I’ve been planning to do completely redundant :thinking_face:

2020-07-28T16:58:11.131200Z

how is the -T alias different to having an alias with a :deps key?

alexmiller 2020-07-28T16:58:43.132100Z

It’s not

seancorfield 2020-07-28T16:58:57.132800Z

I need to figure out how to get the clojure script updated on a couple of our older servers since this would help eliminate some of the extra script stuff we've written (but I think there's a blocker for installing clojure on a couple of our old servers).

alexmiller 2020-07-28T16:59:13.133500Z

It is exactly what was already there via -A, just has its own switch now

2020-07-28T17:00:38.133900Z

ok so it just makes the intent clearer on the command line?

alexmiller 2020-07-28T17:01:03.134200Z

Yep

1👍
alexmiller 2020-07-28T17:01:51.135300Z

A is meant to be an “all” kind of thing but this didn’t have its own name, so now it does

2020-07-28T17:02:48.136600Z

so what would -T do with :extra-deps instead of :deps?

alexmiller 2020-07-28T17:03:02.137100Z

(Which also helped further clarify some things in the code which has gone through some substantial changes)

alexmiller 2020-07-28T17:03:12.137500Z

It would be ignored

1👍
2020-07-28T17:04:08.138800Z

wow I had no idea how many deps had unqualified names. I thought it was just a handful, but it’s way more than I expected

1☝️
alexmiller 2020-07-28T17:04:20.139400Z

Each of the sub processes takes what it expects and ignores the rest, like all good Clojure programs :)

1👍
seancorfield 2020-07-28T17:06:34.140400Z

Should this check include tools_aliases now?

-h|--help|"-?")
      if [[ ${#main_aliases[@]} -gt 0 ]] || [[ ${#all_aliases[@]} -gt 0 ]]; then
        break
      else
        help=true
        shift
      fi
      ;;

alexmiller 2020-07-28T17:08:10.140600Z

No?

seancorfield 2020-07-28T17:09:55.141900Z

Maybe I'm just missing the actual flow of the script then, nm.

alexmiller 2020-07-28T17:10:30.143400Z

Yeah this has to do with falsely interpreting a -h appended to main aliases as a help opt

seancorfield 2020-07-28T17:10:30.143500Z

(I was just grepping for (tool|all).alias through the new script out of curiosity)

alexmiller 2020-07-28T17:11:52.144Z

Would be curious if you try the win install :)

seancorfield 2020-07-28T17:13:01.144800Z

@alexmiller re: -h -- that means that clojure -A:test:runner -h and clojure -T:test:runner -h behave differently.

seancorfield 2020-07-28T17:14:10.145800Z

(! 1537)-> clojure -T:test:runner -h|more
Version: 1.10.1.590

...
(! 1538)-> clojure -A:test:runner -h|more
Unknown option: "-h"
Wasn't sure whether that was intentional or not since you said -T and -A were "the same"?

seancorfield 2020-07-28T17:15:38.146300Z

(the first case displays help -- -h is not passed to the main program; the second case passes -h to the main program in the test runner here)

2020-07-28T17:24:26.146900Z

Just tried doing -X:myfn in one of my projects and I get the error:

2020-07-28T17:24:28.147100Z

clojure -X:myfn
Execution error (FileNotFoundException) at clojure.main/main (main.java:40).
Could not locate clojure/tools/deps/alpha/exec__init.class, clojure/tools/deps/alpha/exec.clj or clojure/tools/deps/alpha/exec.cljc on classpath.

alexmiller 2020-07-28T17:25:45.147600Z

are you on clj 1.10.1.590?

2020-07-28T17:26:16.147800Z

I thought I was

2020-07-28T17:26:34.148Z

yep

2020-07-28T17:26:44.148300Z

$ clojure --help
Version: 1.10.1.590

...

alexmiller 2020-07-28T17:27:39.148400Z

this is kind of a corner case - the condition is guarding against having -A/-M supplying the first part of the main args and then -h on the command line where it is not actually the main args to pass to clojure.main. I'm not worried about it.

seancorfield 2020-07-28T17:30:50.150Z

I get that too, also on 1.10.1.590:

(! 974)-> clj -Sdescribe
{:version "1.10.1.590"
 :config-files ["/usr/local/Cellar/clojure@1.10.1.590/1.10.1.590/deps.edn" "/Users/sean/.clojure/deps.edn" "deps.edn" ]
 :config-user "/Users/sean/.clojure/deps.edn"
 :config-project "deps.edn"
 :install-dir "/usr/local/Cellar/clojure@1.10.1.590/1.10.1.590"
...
(! 975)-> clj -X:example
Execution error (FileNotFoundException) at clojure.main/main (main.java:40).
Could not locate clojure/tools/deps/alpha/exec__init.class, clojure/tools/deps/alpha/exec.clj or clojure/tools/deps/alpha/exec.cljc on classpath.

Full report at:
/var/folders/p1/30gnjddx6p193frh670pl8nh0000gn/T/clojure-13599808261801984368.edn
(! 976)-> cat deps.edn 
{:aliases
 {:example
  {:fn clojure.core/println
   :args {:n 1}}}}
(! 977)-> 

alexmiller 2020-07-28T17:30:52.150100Z

if you clj -Sdescribe and then jar tf <the-install-dir/clojure-tools-1.10.1.590.jar | grep exec.clj

alexmiller 2020-07-28T17:30:58.150300Z

do you see it?

alexmiller 2020-07-28T17:32:32.150700Z

what installer are you using?

2020-07-28T17:32:51.151100Z

the jar is further down than that in /usr/local/Cellar/clojure@1.10.1.590/1.10.1.590/libexec/clojure-tools-1.10.1.590.jar

alexmiller 2020-07-28T17:33:09.151500Z

sorry, bad multi copy/paste on my part, that's right

2020-07-28T17:33:16.151700Z

exec.clj is in there yes

alexmiller 2020-07-28T17:33:28.152Z

this is mac? linux?

2020-07-28T17:33:34.152200Z

$ jar tf /usr/local/Cellar/clojure@1.10.1.590/1.10.1.590/libexec/clojure-tools-1.10.1.590.jar  | grep exec.clj
clojure/tools/deps/alpha/exec.clj

2020-07-28T17:33:47.152400Z

mac

seancorfield 2020-07-28T17:33:58.152700Z

(yup, same here)

2020-07-28T17:33:59.152800Z

brew was the installer

alexmiller 2020-07-28T17:34:25.153100Z

clj -Sdeps '{:aliases {:myfn {:fn clojure.core/pr}}}' -X:myfn is a standalone example

2020-07-28T17:34:46.153400Z

edn is broken

seancorfield 2020-07-28T17:34:55.153600Z

Fails for me

alexmiller 2020-07-28T17:35:07.153800Z

edn is broken means?

2020-07-28T17:35:17.154200Z

ah wait sorry was in a project with a broken edn file

seancorfield 2020-07-28T17:35:23.154400Z

(! 982)-> clj -Sdeps '{:aliases {:myfn {:fn clojure.core/pr}}}' -X:myfn
Execution error (FileNotFoundException) at clojure.main/main (main.java:40).
Could not locate clojure/tools/deps/alpha/exec__init.class, clojure/tools/deps/alpha/exec.clj or clojure/tools/deps/alpha/exec.cljc on classpath.

Full report at:
/var/folders/p1/30gnjddx6p193frh670pl8nh0000gn/T/clojure-12219293780956477884.edn

alexmiller 2020-07-28T17:35:36.154600Z

ah, I was able to repro

2020-07-28T17:35:41.154900Z

$ clj -Sdeps '{:aliases {:myfn {:fn clojure.core/pr}}}' -X:myfn
Execution error (FileNotFoundException) at clojure.main/main (main.java:40).
Could not locate clojure/tools/deps/alpha/exec__init.class, clojure/tools/deps/alpha/exec.clj or clojure/tools/deps/alpha/exec.cljc on classpath.

Full report at:
/var/folders/ln/536xqskd3_g80n5pdbsjkrjw0000gn/T/clojure-8308868078358895829.edn

2020-07-28T17:35:43.155100Z

too

alexmiller 2020-07-28T17:35:54.155300Z

it matters whether you're in a directory with a deps.edn

alexmiller 2020-07-28T17:36:17.155800Z

well, that's definitely a bug :)

alexmiller 2020-07-28T17:36:20.156Z

will fix

alexmiller 2020-07-28T17:36:24.156200Z

thx

seancorfield 2020-07-28T17:36:38.156500Z

Crashes for me whether there's a deps.edn file or not.

2020-07-28T17:36:44.156800Z

yeah same

alexmiller 2020-07-28T17:36:57.157400Z

hm, maybe it's a cached classpath for me or something

2020-07-28T17:37:20.157700Z

there are 3 hard things in computer science…

1🙉1🙈1🙊
seancorfield 2020-07-28T17:37:52.158400Z

I tried it via an alias in a deps.edn file (shown above), then tried via the command line with just {} in deps.edn, and then tried in a directory with no deps.edn -- all of them crash the same way. Even tried -Sforce and it still failed.

2020-07-28T17:38:35.159200Z

I tried two of those; just tried with -Sforce too — and same

alexmiller 2020-07-28T17:38:46.159400Z

well, that's weird. :) I've been using this for a while and others have tested it as well, somehow never saw an error

alexmiller 2020-07-28T17:39:13.160Z

I'm sure it will make sense when I understand why it's doing it ... this is why it's a dev release!

seancorfield 2020-07-28T17:40:24.161Z

Fair enough. Just wanted to point out that it is a difference between -A and -T 🙂

alexmiller 2020-07-28T17:40:45.161800Z

-X has to go at the end, in case that affects anything you tried

2020-07-28T17:41:16.162300Z

nope, I had -X as my only alias

2020-07-28T17:42:38.163700Z

Well I’d be happy to try and help narrow debug it but I’m afraid I need to call it an evening 😞 I’ll check back here tomorrow. Thanks for your help, and also thanks for the feature… it looks like it’ll be really useful.

alexmiller 2020-07-28T17:43:56.164100Z

ah, this makes sense to me now, and why it worked for me

alexmiller 2020-07-28T17:44:29.164600Z

and why things others tried worked. such a combination of coincidences... anyhow will fix

seancorfield 2020-07-28T17:45:47.164900Z

Now I'm curious, what's the bug?

alexmiller 2020-07-28T17:46:23.165200Z

it's not putting that class on the classpath, obviously :)

alexmiller 2020-07-28T17:46:32.165400Z

I was testing in the tools.deps.alpha repository itself

alexmiller 2020-07-28T17:46:38.165600Z

where it is

alexmiller 2020-07-28T17:47:01.166100Z

and others were testing things that used the install program, which requires you to pull in tools.deps.alpha

alexmiller 2020-07-28T17:47:15.166300Z

so those all worked

alexmiller 2020-07-28T17:47:27.166600Z

I may actually have flubbed a merge somewhere

alexmiller 2020-07-28T17:57:25.166700Z

there's no difference in how :deps and :paths work, which is what I meant

alexmiller 2020-07-28T19:00:31.167400Z

@seancorfield @rickmoynihan ok, new clj 1.10.1.596 is available, should fix this

1👍
seancorfield 2020-07-28T19:05:37.167800Z

Looks fixed! Thank you!

alexmiller 2020-07-28T19:08:47.168300Z

if you try it on windows (or linux), would be happy to hear there too

seancorfield 2020-07-28T19:09:49.169500Z

@vlaaad If you want to get strings from the command line in your -X-executed function, you can always use *command-line-arg* (although you'd have to discard leading arguments passed to the CLI itself I guess)

seancorfield 2020-07-28T19:11:37.170200Z

(! 1010)-> clj -A:test -X:example :test value
("-X:example" ":test" "value") ({:test value})
Looks like you'd only have to skip over the first arg?

vlaaad 2020-07-28T19:12:50.170800Z

I don't want to make every arg a string 🙂

seancorfield 2020-07-28T19:14:21.171600Z

How would you want the CLI to distinguish between "I want this as a string" vs "I want this as an EDN value"?

seancorfield 2020-07-28T19:17:09.174100Z

It seems like you'd have to try to read as EDN and if you got an error or a symbol then use the whole argument as a string, else whatever the EDN reader produced? Which is open to all sorts of surprises I would expect...?

vlaaad 2020-07-28T19:20:12.176Z

I guess it's impossible to have it both simple and easy 😞

vlaaad 2020-07-28T19:24:34.179500Z

I don't want to deal with escaping because I use both powershell and bash that have different escaping rules. With those, whenever I have a need to escape I wish I just used a file. This might be a bit too limiting for convenience, but I'd be fine with this: everything that can be read as number/boolean/keyword/nil -> number/boolean/keyword/nil everything else, including things that can be read as collections -> string

vlaaad 2020-07-28T19:27:09.179900Z

it doesn't work with repeated args though

vlaaad 2020-07-28T19:32:55.181900Z

I've been thinking a bit more about simple vs easy, and currently I think I'm overreacting, I think escaping might be an okay price to pay for the limitations different approaches provide

alexmiller 2020-07-28T19:33:37.182900Z

when you have static data, the best thing to do is put it in deps.edn and reference via alias

seancorfield 2020-07-28T19:34:30.184Z

> including things that can be read as collections -> string I think that's a bad idea since it's common to want to pass a map of stuff when calling Clojure function (in general) so that choice would force EDN parsing on everyone

alexmiller 2020-07-28T19:35:02.184700Z

with -X you are always passing a map - the command line is overrides to the data in deps.edn

alexmiller 2020-07-28T19:35:18.185200Z

and the overrides support nested paths

seancorfield 2020-07-28T19:35:20.185300Z

Right, I meant as one of the arguments (inside that map)

alexmiller 2020-07-28T19:35:40.185900Z

nested paths gets you out of another set of cases

seancorfield 2020-07-28T19:35:45.186Z

i.e., to produce a nested map

vlaaad 2020-07-28T19:36:13.187200Z

like -X:deploy :envs '#{"prod" "qa"}'

seancorfield 2020-07-28T19:36:26.187700Z

Yeah, having :test 1234 and :test [1234 5678] produce a number and a string respectively would be very strange.

vlaaad 2020-07-28T19:36:38.188100Z

I agree 🙂

alexmiller 2020-07-28T19:36:44.188300Z

it turns out vectors are actually ok bash if you use the comma

vlaaad 2020-07-28T19:36:55.188700Z

not ok pwsh

alexmiller 2020-07-28T19:37:04.189100Z

:test [1234,5678] is actually ok

seancorfield 2020-07-28T19:38:42.190600Z

I sympathize with the additional pain on PS (I too use it from time to time with the Clojure CLI) but since the vast majority of Clojure CLI users are on macOS/Linux I'd rather not make all their lives more painful, just to make PS less painful -- and it's already a bit painful to use for the Clojure CLI...

alexmiller 2020-07-28T19:40:55.191300Z

maybe this is a topic for #clj-on-windows but what's the story on the new Windows terminal?

vlaaad 2020-07-28T19:40:56.191400Z

hmm, I might be tripping, [1,2,3] worked fine in pwsh

seancorfield 2020-07-28T19:45:20.192200Z

@alexmiller The new Windows terminal is just a fancy tabbed shell around existing PS, cmd, and Linux terminals.

alexmiller 2020-07-28T19:45:52.192500Z

so nothing new

nate 2020-07-28T19:47:05.193800Z

out of curiosity, why are unqualified lib names now deprecated? was there a deeper issue that needed addressing or is it just because qualified names are unambiguous and more correct?

nate 2020-07-28T19:48:11.194200Z

ah

seancorfield 2020-07-28T19:48:15.194400Z

and clj -X:example [:test,:value] 1234 works on macOS so, yay, portability! 🙂

nate 2020-07-28T19:48:23.194600Z

I saw it in the blog post that he posted https://insideclojure.org/2020/07/28/clj-exec/

nate 2020-07-28T19:48:47.194900Z

which mentioned the deprecation again, but I was curious if there was a deeper reason for it

alexmiller 2020-07-28T19:52:15.195300Z

not sure what I can add beyond what's in the blog

alexmiller 2020-07-28T19:52:58.196Z

unqualified names are bad so we're going to stop encouraging them in this area

nate 2020-07-28T19:53:50.196300Z

that makes sense, thank you

vlaaad 2020-07-28T20:17:23.196800Z

PS C:\Users\Vlaaad\Projects\tdeps> cat .\deps.edn
{:aliases {:prn {:fn clojure.core/prn}}}
PS C:\Users\Vlaaad\Projects\tdeps> clj -Sdescribe
{:version "1.10.1.596"
 ..."}
PS C:\Users\Vlaaad\Projects\tdeps> clj -X:prn
Execution error (FileNotFoundException) at clojure.main/main (main.java:40).
Could not locate clojure/tools/deps/alpha/exec__init.class, clojure/tools/deps/alpha/exec.clj or clojure/tools/deps/alpha/exec.cljc on classpath.

alexmiller 2020-07-28T20:21:00.197300Z

hrm, let me check on that, shouldn't be using that class anymore

vlaaad 2020-07-28T20:21:53.197600Z

-Spath is

C:\Users\Vlaaad\.m2\repository\org\clojure\clojure\1.10.1\clojure-1.10.1.jar;C:\Users\Vlaaad\.m2\repository\org\clojure\spec.alpha\0.2.176\spec.alpha-0.2.176.jar;C:\Users\Vlaaad\.m2\repository\org\clojure\core.specs.alpha\0.2.44\core.specs.alpha-0.2.44.jar;src
`

alexmiller 2020-07-28T20:22:49.198600Z

path is fine, it's the class, I missed one change there

alexmiller 2020-07-28T20:25:05.198800Z

windows 1.10.1.600 has the fix

dominicm 2020-07-28T20:25:31.198900Z

To encourage use of groupid

dominicm 2020-07-28T20:26:21.199100Z

And discourage hiccup/hiccup in favor of com.weavejester/hiccup. But in a general sense, not the global sense.

dominicm 2020-07-28T20:26:43.199700Z

Does -X compose?

borkdude 2020-07-28T20:28:24.199900Z

What do you mean by that?

alexmiller 2020-07-28T20:28:32.200100Z

depends what you want to compose with :)

alexmiller 2020-07-28T20:28:36.200300Z

with other -X, no

alexmiller 2020-07-28T20:28:43.200600Z

with all the classpath stuff, yes

alexmiller 2020-07-28T20:29:04.201Z

but if you want to run multiple things, write a program that does so and run it with -X :)

sveri 2020-07-28T20:29:20.201100Z

Finally 🙂 That was one thing that always made me wonder why one would have unqualified library names when clojure is so heavily rooted in java.

dominicm 2020-07-28T20:31:05.203Z

I don't want to write lots of programs! I want to change my programs on a whim at the cli. Start a socket server, start a whatever, run the program in the local project that does setup, run my custom dev function.

dominicm 2020-07-28T20:31:50.204200Z

(We've discussed before though, not really trying to convince, but I hoped that exec would provide this)

dominicm 2020-07-28T20:32:49.205900Z

Right now I'm (ab)using the fact you can supply an init and a main to load a dev.clj in my home directory, which will run things based on the directory the jvm started in...

alexmiller 2020-07-28T20:34:18.206300Z

there's this thing called a repl that lets you do like ... anything

1😆
dominicm 2020-07-28T20:35:09.207100Z

Yeah, but I have to like type all these weird ( characters.

1
vlaaad 2020-07-28T20:35:39.208100Z

PS C:\Users\Vlaaad\Projects\tdeps> clj -Sdescribe
{:version "1.10.1.600" ...}
PS C:\Users\Vlaaad\Projects\tdeps> clj -X:prn
Execution error (ExceptionInfo) at clj-exec/check-first (clj_exec.clj:32).
Invalid first arg to exec: prn

vlaaad 2020-07-28T20:36:33.210400Z

could it be PS escaping away the :? :thinking_face: other args like -A:alias work fine thogh

alexmiller 2020-07-29T18:42:02.336200Z

hrm

vlaaad 2020-07-29T18:48:52.340100Z

hacked it to work with this:

$sym, $params = $params
$ExecAlias += "$arg$sym", $params
now it doesn't work with "-X:prn" though

vlaaad 2020-07-29T18:51:05.340500Z

seems to be PS parsing issue:

> echo -X:prn
-X:
prn

vlaaad 2020-07-29T18:51:47.340700Z

> echo -A:prn
-A:
prn

vlaaad 2020-07-29T18:52:28.340900Z

hmm, that works:

s> clj -A: beep:boop:bap
WARNING: Specified aliases are undeclared: [:beep :boop :bap]
Clojure 1.10.1
user=>

alexmiller 2020-07-29T18:54:10.341100Z

is there a difference in powershell parsing a single thing (into leading char / remaining char) or many things?

vlaaad 2020-07-29T18:55:16.341400Z

you mean -XXX:yyy vs -X:y ?

alexmiller 2020-07-29T18:58:49.341600Z

no, I mean are we getting different behavior on clj -X:foo vs clj -X:foo :a 1 ?

vlaaad 2020-07-29T19:01:34.341800Z

no:

PS C:\Users\Vlaaad\Projects\tdeps> clj -X:prn :a 1
{:a 1}
PS C:\Users\Vlaaad\Projects\tdeps> clj -X:prn
nil

vlaaad 2020-07-29T19:02:47.342Z

but this wont work: clj "-X:prn", because I made the script to expect args -X: and prn separately

vlaaad 2020-07-29T19:06:24.342200Z

so what we really need to do is accept both -X:alias and -X: , alias for powershell to make it predictable

vlaaad 2020-07-29T19:06:37.342400Z

I'll try to write some powershell..

vlaaad 2020-07-29T19:11:58.342600Z

if ($arg -eq "-X:") {
        $sym, $params = $params
        $ExecAlias += "$arg$sym", $params
      } else {
        $ExecAlias += $arg, $params
      }

vlaaad 2020-07-29T19:12:16.342800Z

PS C:\Users\Vlaaad\Projects\tdeps> clj "-X:prn" :a 1
{:a 1}
PS C:\Users\Vlaaad\Projects\tdeps> clj "-X:prn"
nil
PS C:\Users\Vlaaad\Projects\tdeps> clj -X:prn
nil
PS C:\Users\Vlaaad\Projects\tdeps> clj -X:prn :a 1
{:a 1}

alexmiller 2020-07-29T19:24:29.349200Z

the code for other aliases in tools.deps is actually tolerant of the split here, maybe that's necessary for the -X stuff too. in other words, we've been masking this problem

dominicm 2020-07-28T20:37:28.212100Z

In earnest, there's a little bit of a desire to hide implementation behind flags. Telling someone to run clojure -A:nrepl:piggieback is easier than explaining that they need to run a set of functions once waiting for the repl to connect, and then press buttons on their editor, in an exact sequence to make it work. Right now I use jvm options.

alexmiller 2020-07-28T20:47:03.212200Z

will take a look, not sure

alexmiller 2020-07-28T20:50:33.212400Z

maybe I'm missing the first arg I match in the pwsh for this, but seems like we'd get nothing here, not part of the arg

alexmiller 2020-07-28T20:53:09.212600Z

how hard is it for you to just modify the ClojureTools.psm1 ?

alexmiller 2020-07-28T20:54:08.212800Z

I think line 84 just needs to be $ExecAlias += $arg, $params

alexmiller 2020-07-28T20:54:28.213Z

it's losing the matching $arg right now

seancorfield 2020-07-28T20:56:13.213600Z

Dang, we use a lot of single-segment lib names at work! 😞

seancorfield 2020-07-28T20:57:39.214200Z

(updating 27 deps.edn files in our monorepo!)

alexmiller 2020-07-28T20:58:09.214400Z

I have https://www.youtube.com/watch?v=jsW9MlYu31g running through my head ...

alexmiller 2020-07-28T21:00:27.215600Z

feedback on the warning message? I tried to make it useful

seancorfield 2020-07-28T21:00:49.216200Z

Am I right in thinking the new "alias data" feature is going to allow people to add all sorts of configuration stuff into deps.edn as data under an alias?

alexmiller 2020-07-28T21:01:08.216700Z

if they like

seancorfield 2020-07-28T21:01:17.216900Z

Folks have been asking for that for quite a while 🙂

alexmiller 2020-07-28T21:01:26.217300Z

like most things, taste is helpful

seancorfield 2020-07-28T21:01:36.217700Z

(and have been told in the past, "don't do that" 🙂 )

alexmiller 2020-07-28T21:01:44.218Z

it might be a good place to put like, build configuration

seancorfield 2020-07-28T21:02:34.218700Z

Hmm, and because aliases don't compose, it's "just" fixed data so I guess that's a nice place to draw the line.

seancorfield 2020-07-28T21:03:21.219300Z

I suspect shadow-cljs and other tools might leverage it for build config to avoid having a separate file for that...

seancorfield 2020-07-28T21:03:38.219400Z

It's perfect.

seancorfield 2020-07-28T21:03:42.219700Z

Annoying, but perfect.

seancorfield 2020-07-28T21:04:01.220200Z

The only slightly weird thing was that I seemed to get all of the warnings twice?

dominicm 2020-07-28T21:07:16.222700Z

Aliases can compose, but it's down to their interpreter to decide how.

alexmiller 2020-07-28T21:08:39.224300Z

btw, I don't know if add-libs works with latest (haven't tried). I am still injecting the lib map atm which is what add-libs works starts from, but not sure if it also uses anything else in tools.deps that's changed. the branch I have has the stuff to switch to basis but also other stuff too so some interpolation will need to be done, not sure when I'll get to that

seancorfield 2020-07-28T21:09:44.225100Z

NP. I just updated my dot-clojure deps.edn to rename the alias from :deps to :add-lib and add notes that it should be considered experimental and likely to break or go away 🙂

seancorfield 2020-07-28T21:10:14.225600Z

(I had forgotten that the installed deps.edn contains a :deps alias when I originally added it)

seancorfield 2020-07-28T21:11:09.226700Z

I also just cleaned up the :rebl* aliases to reflect the latest REBL having its own deps.edn files for java8 and openjfx15ea (plus some reorganization and updating several versions).

alexmiller 2020-07-28T21:11:40.226900Z

cool

alexmiller 2020-07-28T21:12:33.227Z

that dimly rings a bell. I wrote this like a month ago. I think because it's encountered during canonicalization and also during expansion or something?

1👍
borkdude 2020-07-28T21:13:57.228100Z

My work is now finally using deps.edn/tools.deps/clojure CLI (how do you even call this stuff?). I moved out our front-end build to it (using figwheel.main / cljs.main) last week while we still do the uberjar and REPL stuff with boot. Also now we can finally run Java 11 everywhere, since we were stuck, the boot frontend stuff didn't work with it (at least not for us).

ghadi 2020-07-28T21:15:06.228500Z

that's exciting @borkdude

alexmiller 2020-07-28T21:15:37.228800Z

nice!

borkdude 2020-07-28T21:17:23.230200Z

Also we use this dev script to start 3 processes simultaneously: clojure -A:cljs/dev, clojure -A:less/dev and ./boot dev, while having only one terminal tab open: https://gist.github.com/borkdude/8f5dff7c2330ca520403eb44c9013a83, just as a convenience

dominicm 2020-07-28T21:20:47.231500Z

3 jvms! I bet that doesn't integrate well with cider jack in. Might I suggest starting 3 threads instead?

borkdude 2020-07-28T21:21:36.232400Z

I've always avoided cider-jack-in, I always use cider-connect. Even more so now I'm developing on a remote machine but editing files in my local emacs using tramp.

borkdude 2020-07-28T21:25:40.233500Z

Threads may work but it's not a huge deal right now. We need at least two JVMs given we're using two different build/classpath/dependency tools right now.

seancorfield 2020-07-28T21:26:14.234Z

"I've always avoided cider" -- ftfy 🙂

1👏
seancorfield 2020-07-28T21:27:46.235500Z

As part of this monorepo deps cleanup I'm doing (to prevent all those DEPRECATED warnings!) I'm also removing the last few references to nREPL in our setup since neither I nor my team mate use that now.

dominicm 2020-07-28T21:28:04.235800Z

seems unfair to people who use jack in. ¯\(ツ)

borkdude 2020-07-28T21:28:25.236200Z

Well, I asked my teammates and this was all OK for them, so I guess we're good 🙂

2020-07-28T22:36:59.244500Z

I’m curious why the new tools.deps adds a tool to do local maven installs. Is this something people frequently need? I’ve probably only ever needed to install free form jars into my local maven repo a few times in the past ten years. I mean sure if I’ve built a jar myself I will frequently want to install it, but it seems strange to supply a method to install jars, but not build them. I’m just curious what problem it’s solving for people.

2020-07-28T22:39:06.245900Z

Am I missing something in my workflow; or is it just that this happens to have been baked before other build tasks in tools.build?

2020-07-28T22:39:58.246Z

works now! Thanks 🙂

kenny 2020-07-28T22:56:37.247100Z

Perhaps related to Datomic's dev-local needing to be installed into your local Maven repo to be used (it's not distributable).

seancorfield 2020-07-28T23:03:38.247500Z

^ That is what Alex said when asked earlier @rickmoynihan

2020-07-28T23:06:41.247800Z

ahh ok makes sense in that use case. Thanks @seancorfield

alexmiller 2020-07-28T23:10:25.248300Z

thx

2020-07-28T23:10:35.248700Z

@alexmiller Robot Chicken has some pretty darned funny riffs on Star Wars scenes, including the one you linked above: https://www.youtube.com/watch?v=WpE_xMRiCLE

1😂