Clojurians Log v2

yeah that’s precisely my point… Personally I feel like deps-deploy and depstar should probably be integrated into the same project. And use each others :exec-args keys etc.. Though I guess that depends on whether there are any plans for the cli tools to ever support task composition. (I know it’s been raised here a few times). Here’s hoping tools.build lets you compose tasks intelligently 🤞 The project I’m working on wants to build and deploy multiple builds (essentially themed versions of the same thing), and to do that right now I need to fire up two JVMs for each build — and we might need 10 builds at some point… So that’ll be a lot of time wasted in CI starting/stopping JVMs. Not the end of the world; but might add a minute to the CI time; when it could be more like 10 seconds.

Silly question, why not write a short clj file which calls them with the right args?

Assuming a functional api

Just found this https://github.com/applied-science/deps-library

> Silly question, why not write a short clj file which calls them with the right args? Well this is my point. I can definitely do that, and likely will. But I’m only solving it for myself in one project. If you build a jar you almost always want to deploy it. depstar and deps-deploy would ideally be aligned and integrated around common keys etc. It looks like deps-library is essentially what that might look like, though with a traditional cli interface not an :exec-fn one.

The iteration etc is fairly unique to my project; but the integration of deps-deploy and something like depstar is common enough that it can/should be combined in my mind. If we had that it would make the job of my iteration/script even smaller (but also I think be widely useful)

Well it turns out this is a bit of a problem to doing it as a clj script: https://github.com/seancorfield/depstar/blob/5b1f9f02b34d07f88b42cbf2c2109d04b829cd8b/src/hf/depstar/uberjar.clj#L525

@seancorfield: would you take a PR to refactor the bulk of this function into a new function that does everything except call System/exit?

https://github.com/seancorfield/depstar/blob/5b1f9f02b34d07f88b42cbf2c2109d04b829cd8b/src/hf/depstar/uberjar.clj#L525

oh actually looks like it’s not system-exit that is the problem… but shutdown-agents

Just to be clear, I’d be happy to move deps-deploy in (almost) any direction that might help solve problems.

well the system/exit is also bad as it kills my repl when I make a mistake in the config

That’s pretty much my thoughts on what should happen too… I feel we need some community consensus on what that direction might be though.

It calls System/exit in multiple places.

yeah so I’ve seen

I often try to separate the shutdown-agents/System.exit type stuff into a wrapper

yeah me too

that’s what I’m suggesting

Yeah, I would normally but I haven't quite gotten there yet.

There's also the call to help-and-die in there. I'll give it some thought this weekend.

shall I make an issue for it?

Sure. It'll be a good reminder for me.

another option is to parameterize the "on-exit" function

I usually write a -main and main. Main only returns the exit code (or a map with exit code) and -main does System/exit + shutdown-agents

I then use main in my tests

yeah that’s the same pattern I do… though this isn’t main but an :exec-fn not that it makes much difference.

it lets you essentially test command line opts etc

yes, this is how I test all my CLIs using the exact same input as from the command line

Whilst I’m hear, as you may have seen I’ve been tweaking deps-deploy to support the new -X stuff. If you’re going to use deps-deploy however; you essentially need to use depstar (or similar), so it’d be good if these tools could be aligned a bit to respect the same keyword vocabulary. How do people feel about that? Similarly in the deps-deploy changes I made I thought it would be nice to use :mvn/version as it means the same as in tools.deps. However I also wanted :mvn/artifact-id and :mvn/group-id etc. Would those terms in principle be the same as tools.deps / tools.build etc might use? Is it legitimate for me to coin keys in the mvn namespace?

to use in what context?

like in calling with -X or something?

yeah precisely, my patches to deps-deploy allow you to override the :mvn/version via :exec-args and the cli, but also :mvn/artifact-id etc…

probably won't line up to tools.build but I don't know if you should care

well I don’t know what tools.build does… so hard to know if I should care or not 😀

I guess I have a slightly deeper question here though… Which is essentially how deep does RDF’s influence over best practice around keywords in clojure go? Rich has mentioned RDF publicly multiple times over the years. And clojure’s namespaced keywords etc are essentially equivalent to predicates etc. The promoted strategy to handling them has been “if they have the same meaning” it’s fine to reuse them (open world, open for extension etc…). However the linked-data world also has a convention of only coining new terms in namespaces you own. However clojure best practice glosses over that here. Does tools.deps own :mvn, is it legitimate for me to coin keywords in that namespace?

Also is the concept of keyword vocabularies perhaps a useful one?

What is "RDF"?

https://www.w3.org/TR/rdf11-primer/

TLDR; A web-scale datomic-like data model described as a set of W3C standards and practices for how people can build databases to support a global graph of knowledge

I do not consider tools.deps to "own" :mvn

qualifiers and their usage are contextual. spec argues for sufficient qualification to create a universe of unambiguous attribute definitions

in more limited contexts (inside your project or inside another appropriately scoped data context), less qualification may be sufficient

deps.edn has a competing design constraint of wanting to be succinct as this is a format people are frequently writing

(Also acknowledged by Rich to be a big influence on the design of clojure and datomic too)

we ended up leaning more on the latter constraint in that balance

Thanks for that @alexmiller, that’s essentially my perception too. What then do you think of collaborating around common shared definitions like :mvn/version etc and agreeing in principle on other ones to avoid future conflict in areas such as this?

for the purposes of invoking tools, I don't think there's any need to collaborate

and I don't see it as conflict

well it’s not a conflict if I chose :mvn/group-id and you choose :mvn/groupid, but it is irritating if we mean the same thing and our terms flow through the same channels. It would be a potential conflict if we both independently chose :mvn/repository but I defined it as a map, and you a string; and they happened to flow through the same context.

I don't see them flow through the same channels or exist in the same context, which is why I say this is not a conflict

really, I'd say neither of these tools "own" :mvn. use a qualifier you own if you want to own it.

otherwise, don't complain when it conflicts :)

depstar's arguments to -main are pretty fixed at this point, and I've just named the :exec-args after the long-form command-line arguments I believe. Changing them would break backward compatibility so I don't want to do that. Re: deps-deploy -- I think it's pretty important that pom.xml be the source of truth for the group/artifact/version -- at least insofar as the version that is included in the JAR and uploaded to Clojars with the artifact -- so I don't think it's a good idea to override the version in deps-deploy and end up with the internal POM not matching the external POM.

Well that’s certainly pragmatic. I’ll take that as permission to follow your example and use :mvn/group-id here. I mean it seems sensible to follow similar conventions for human reasons.

And then there is me who thinks that pom.xml it's just a temporary file that should be deleted as soon as deployment is done

At this point, I would prefer to defer all of this to tools.build but I don't know whether deployment is in scope for that (we have local mvn-install already in t.d.a but no remote deployment).

I don't want to have to keep maintaining a community tool if there is an official tool from the core Clojure team that does the same job.

@seancorfield: yeah that’s fair… I’d much rather overriding the :mvn/version etc happened in depstar. I said this the other day in fact.

I think deployment is one of many things that could be invoked by tools.build

whether that's something provided as part of tools.build or community is a separate question. I've looked at it in depth, have a lot of opinions, have not built it.

@vlaaad Well, the reality is that there's a lot of stuff in pom.xml that other tooling out there relies on, far beyond the minimum that clojure -Spom produces, so you pretty much have to maintain that information somewhere under version control.

@rickmoynihan Yeah, there are parts of pom.xml that it would be nice for depstar to be able to override but if you override <version>, you pretty much have to override the SCM <tag> to match, because of tooling that uses that.

the code under -Spom has been refactored to make it more amenable to having more options/features and I think longer term that's going to be something we push to primarily being done as part of a build, with probably ever-increasing scope to refine as part of a build

Yeah… For my needs I think treating pom.xml as a template, and moving my override feature from deps-deploy into depstar would make most sense.

yeah @dominicm mentioned this the other day… that’s trivial to do too though.

there are all sorts of special cases you might want to handle in making a pom for deployment

A guiding principle for depstar is no additional dependencies so the more functionality it accretes, the more I end up rewriting other tooling into the depstar codebase.

many require some kind of data manipulation you'll want to do during a build

Of course, I just think that pom.xml is a horrible storage for this information, since there is a lot of complexity with regards to updating it

just fyi on tools.build - it exists, it has a lot of functionality but we are trying to finalize a lot of the details around how tasks are defined, stitched together, invoked, etc. b/c that kind of affects everything, we're delaying the public release of it but we'll get there.

Oh totally agree! XML is a horrible source format -- but we're "not supposed to use" deps.edn for this stuff, which is where I would otherwise expect it to be... although with :exec-args etc, that's getting more reasonable and "sanctioned" as appropriate by the core team.

I have no opposition to using that name :)

And I'm still going to bleat on and on about some sort of built-in solution for either the monorepo case or where folks want to combine/insert an extra "master" deps.edn file into the t.d.a mix that everything should then use 🙂

Why no additional dependencies? A stable xml lib would be handy, and tools are usually separate from an apps :deps anyway.

(ideally, in t.d.a so that tooling could rely on it easily)

even tools.deps has dependencies!

and clojure 🙂

Some sort of "include" or "reference" functionality in deps.edn would work for that -- but I think you've said the core team are opposed to going down that path @alexmiller?

Look at the way depstar works and you'll see: it packages up what it finds on the classpath you used to run it, simply excluding itself.

my initial lean would be towards an additional deps.edn source but haven't talked about it enough

That ensures you get exactly what you expect in the JAR based on the aliases you provide, and mean that you don't have to tell depstar itself how to read/combine aliases/`deps.edn` etc.

It also allows you to set an explicit :classpath though right?

Yes, that's an alternative option but I don't want it to be the default.

Lots of people seem to struggle with the other JAR-makers out there and end up at depstar because it is simple and does the obvious, intuitive thing.

sounds more like easy to me… simple would be having the :classpath supplied explicitly.

That would satisfy me too. I'd like to get away from my current CLJ_CONFIG hack, and I'd like individual developers to be able to keep using their own ~/.clojure/deps.edn file for dev/test stuff without us needing to repro a common core set into our "master" deps.edn file 🙂

The original version of depstar that I inherited had pretty much no options and just created a JAR from the classpath. It was about as simple as it could be. Adding AOT and main-class and other stuff has made it less simple as a tool, but still easy 🙂

FWIW I think this would be useful too.

yeah I understand, and there’s nothing wrong with easy. It just sounds like that’s a requirement for the uberjar side of depstar; but the library side is I think a far simpler problem. Perhaps it should be split into two separate libraries, depstar.jar and depstar.uber?

It could be refactored to keep those two things separated. Lots of things could be done with depstar but, right now, I'm waiting to see where tools.build ends up.

Like I say, I don't want to keep maintaining a community tool when there's an official core Clojure team tool that does the "same" job.

If it seems like tools.build won't handle the cases that depstar can handle, then I'll be happy to invest a lot more time into it.

that’s fair enough 🙂 but I need solutions now, not in 2+ years 🙂 Being able to build and deploy a library into a private s3 bucket is a common requirement that’s far more awkward than it should be right now.

@alexmiller: Which problem does tools.build solve more of? Orchestrating arbitrary tasks together, or providing ways to build jars, deploy stuff etc? My feeling is that it’s more like make than a way to build jars? Is that right? Or is it both? I appreciate you don’t want to go into too much detail; but it’d be useful to know if depstar etc might be replaced wholesale by it; or whether something like depstar would just integrate with it?

Ok sorry folks this is going to have continue another time… Many thanks to everyone involved in the discussion. I hope you all have fab weekends. 👋 👋

https://github.com/seancorfield/depstar/issues/45

I would love to see the built-ins become de facto defaults and make them great with community help

I'm very curious to see how/if caching is handled. Notoriously difficult task.

caching of?

I would guess caching of task results

E.g. make

if you mean like compiled classes, that's one thing we're still thinking through

Fwiw, I think this would be a valuable article on http://clojure.org. Explaining the purposes of namespacing and when you can skip it, when to use #db/id instead of #com.datomic/id Or #datomic/id.