Clojurians Log v2

Would it make sense to add a npm resolver?

Right now, it seems a ClojureScript lib can't depend on another ClojureScript lib that depends on a JS lib unless someone repackages the JS lib into Maven. But since tools.deps has custom resolvers, if it could resolve from npm, it seems it would allow this scenario to work

When you say "Maven", does that include the cljsjs infrastructure or whatever it is?

Ya it does

cljsjs just repackages NPM packages into Clojars

As it does so, it might also go ahead and perform some minification on it, and it'll add the required extern files.

So cljsjs is convenient when you're too lazy to minify or write extern files yourself. But right now, it is also mandatory just to be able to resolve deps and deps conflicts. At least this is my understanding.

I guess the challenge of a NPM resolver would be transitive NPM deps. Tools.deps would need to understand the npm package.json (the equivalent of pom.xml) and all that

I would imagine the #clojurescript channel would be the best place to petition for such a thing? How does David Nolen feel about cljs dependencies from the NPM/JS world? I saw the recent changes to make it easier to publish cljs stuff to NPM (as I understood the news)...

Hum, ya not sure. From my understanding, the new bundle target isn't really about publishing to NPM. Basically, when you want to deploy a ClojureScript app, you need to do a sort of uberjar, but in JS world it's called a bundle. The ClojureScript compiler has the capability to bundle your ClojureScript code and all its dependencies together. Since all users will need to download your bundle when viewing your site, bundle size is a sensitive topic. ClojureScript code and ClojureScript dependencies as well as JS dependencies that are designed with Google Closure support can be bundled by the Google Closure compiler in a super optimized way (in terms of having a very small size). It renamed all vars to super short names, removed all whitespace, and does dead code elimination at the function and variable level. That said, JS dependencies not designed for Google Closure can't go through it. So ClojureScript compiler lets you specify those as "foreign-libs". In which case, it will still bundle them up, but it won't optimize them at all. For a project that depends on a lot of such library, that isn't very good, because it still results in a large bundle size. There exist other bundlers for JS, which can take those type of libraries and bundle them with a reduced size. But until now, if you tried to used them to bundle the already bundled output of the ClojureScript compiler, it would fail. The new bundle target for the ClojureScript compiler fixes this. It makes sure to output a bundle that can be further bundled by JS bundlers. Nothing about it really has anything to do with package management. At least this is my current understanding.

FWIW I don't think it make sense to move this into deps.edn. I doubt Alex wants to maintain and interact with the insanity that is npm. We already have the mechanism to declare npm dependencies and installing them in CLJS directly, just a bit basic maybe. can always add more options to that when needed.

Is that through shadow-cljs? Or without?

CLJS has --install-deps via cljs.main and shadow-cljs just does it automatically on startup

Hum, so there's something I don't understand then. Install-deps is unaware of versions. And it does no transitive deps handling. Or I haven't figured out how

So for example, if lib A wants to declare some npm-deps, and app X wants to use lib A, app X won't automatically pull down lib A's deps

in CLJS automatically no, you have to run --install-deps in app X

in shadow-cljs it is automatic

Well, I'll definitly be trying shadow-cljs eventually. But for now, --install-deps, even run manually doesn't pull transitive dependencies for me

both look for deps.cljs files in the libraries and use :npm-deps {"some-lib" "some-version"} to install

Oh, what is a deps.cljs file 😛

well do this libs you are testing contain the proper deps.cljs? many that still just depend on cljsjs don't

I am writing the lib. First time I hear about a deps.cljs file

eg. https://github.com/reagent-project/reagent/blob/master/src/deps.cljs

Ok, official doc seems here: https://clojurescript.org/reference/packaging-foreign-deps

hmm that doesn't even mention npm-deps

maybe there are no docs for this? I always thought there were

There doesn't seem to be. And I felt like I had gone over most of the docs looking for something like this. Maybe it just needs some better doc

So in shadow-cljs you just leverage the compiler install-deps feature under the hood? Or you make use of npm directly? Or you implemented a commonJS repository resolver of your own?

no I don't use the CLJS install-deps ... mostly because it didn't exist when I wrote the feature for shadow-cljs

I don't handle any package installing whatsoever, it just runs npm install the-package@the-version nothing more.

(same as CLJS does nowadays for --install-deps)

Ok, and you grab the-package and the-version from the deps.cljs file on the resource path?

Just trying to figure out how to best package my lib. So if I make use of a npm deps, I just declare it in the deps.cljs along with foreign-libs and externs. And now anyone would be able to just add a dependency to my lib and unless they have a newer version of the npm deps themselves, everything would work as is and they'd get my lib and my transitive npm deps (assuming shadow or needing to run install-deps manually otherwise)

generally it is either/or. either you depends on :npm-deps OR you use foreign-libs. don't mix them.

And where am I supposed to put the deps.cljs ? That weird doc mentions inside a src folder? But I assume just anywhere on a resource path of my deps.edn? Or should it be project root?

the file needs to be in the published .jar file at the root

so generally any source-path will do

Ok, and if I'm temporarily using a local dep? Same logic applies?

for CLJS don't know. for shadow-cljs yes, it just loads any deps.cljs files on the classpath

👍 alright, will try it out. Thanks.

This seems to be the spec in case there is interest: http://wiki.commonjs.org/wiki/Packages/Registry

the idea of using npm to find transitive cljs deps has come up here several times and it's interesting but needs better thinking starting from the goals. tools.deps really has several parts and it seems like the relevant part here is using the tools.deps dependency expander to trace the transitive set of dependencies, starting from a cljs project. tools.deps is designed to be extended for this by implementing various multimethods in the ctda.extensions ns. I've looked at the npm repo and it has everything you would need to do so and would be pretty simple to plug in in that way.

however, it is then at odds with the rest of tools.deps (and clj) in that the other parts are about making a jvm classpath and I don't think that's what you need or want for cljs.

so yes, you could use tools.deps to get a transitive dependency expansion from a heterogenous set of sources including maven, local, git, and npm. I think you want to do that from some cljs tool that then knew what to do with all that stuff to appropriately run the cljs compiler (or whatever).

if that big picture made sense, I'd be happy to lay out what needs to be done to plug in an npm procurer into the tda extensions. That part is not hard, but it's not useful unless this in service of some greater vision.

I think npm is useful outside of cljs. It would make a good alternative to webjars.

useful to do what

I use webjars to serve assets (js, css) for libraries like codemirror

those are different things though, one is providing dependencies to the cljs compiler, the other is providing runtime resources

it seems like tools.deps is mostly about runtime resources (building a classpath), it just happens that those things almost always overflap significantly in clojure, and people have built tools on top of tools.deps that reinterpret it as build time dependencies (building uberjars out of the stuff it puts on the classpath) because of that overlap

there is, from my very limited perspective on cljs, a lot less overlap between compile time and runtime dependencies there because of the split between the compiler and runtime, and the way js is usually delivered

thinking about it, it also strikes me as very unlikely that packages right from npm are suitable to be served up ala webjars, my guess is you need some kind of intervening build step, so really both cases are inputs to some build/packaging process

To me, this is the same. We want to provide the ClojureScript compiler (which is a Clojure application) the necessary resources that it needs. Because it relied on npm currently, it makes an assumption to not look for them on the classpath, but instead look for them in a node_modules folder. But I feel if it could leverage tools.deps, it's not unrealistic for it to look for them on the classpath instead. With target :bundle it could create a node_modules folder of symlinks to the deps pulled down by tools.deps and thus allow JS bundlers to "find" the tools.deps managed JS dependencies. Or maybe, tools.deps could just pull the npm deps into that folder directly. Not sure if it's a core part of tools.deps to put all the deps in some central place? Or if it would be fine for it to put them in different places?

Also, I'm not super familiar with JS bundlers, do they expect to bundle node_modules stuff? Or could the compiler just copy over all the necessary JS on the output folder and the JS bundler can just bundle it up.

they are not dependencies of the clojurescript compiler, they are dependencies of the code it is compiling

Right, I see them as resources needed by the ClojureScript compiler. I was using dependencies loosely here.

Woa, webjars already mirrors all of npm and Bower into Jars?

That means I'm theory, we can already use tools.deps.to.depend on those.

sticking things on the class path because we have a tool for building class paths does not seem like a considered approach

The only missing piece is from the cljs compiler. But you could easily write a tools.deps alias that extracted the content from the jar and put them in a node_modulrs folder and then created the deps.cljs file. So you could run that, and then run the compilation

Have you found webjars is up-to-date with npm?

And mirrors absolutely everything?

it sounds like the problem you want to solve is transitive js dependencies

Well, personally, I find an explicit list of paths to resources is more considered then an implicit one. But NPM already made that choice. Though here it seems we could make a different one. The only difference is where to look for the resources, do you just scan the classpath, or do you also look for a specific relative folder. That said, it seems @alexmiller was saying that it would be possible that tools.deps doesn't do classpath generation for those, but just feeds the resolution to something else. So that could be an option as well. So maybe that something else can generate the node_modules folder instead. That said, I think going with the classpath is best. Because cljs already uses the classpath for ClojureScript deps, and for cljsjs deps. So reconciling those two seem like a nice outcome.

I'm trying to solve JS dependency resolution. So pulling down all required direct and transitive JS dependencies, detecting conflicts between them and letting the user resolve those. That's the first part. Feel tools.deps could do this. Right now the ClojureScript compiler itself is slowly building this functionality inside itself (though it delegates to npm mostly). The second issue is setting up the environment for running the ClojureScript compiler where it can find your dependencies. It currently relies on a mixture of the classpath + looking for an implicit node_modules folder. I'm not as adamant here, but I still feel if we moved the nodes deps to the classpath as well, it seems to simplify things. Also, it gives us a single place for ordering in terms of which lib to grab first if the JS class exists in more than one place.

But probably we should discuss these two things separately.

It's an automated process to request more

It‘a not - it’s up to the procurer to do that, and then tell tools.deps where it is