Clojurians Log v2

Would it make sense for tools.deps to be extended to have a CommonJS Package Registry resolver?

It would be a little tricky, since it would need to support package.json parsing to manage transitive deps as it does now for pom.xml with Maven registries

But if it did, one could depend on NPM deps freely. And one could have a ClojureScript lib with NPM dependencies. And tools.deps would be able to do a full dependency closure on all of it and resolve conflicts, pull down the defined versions, etc.

Npm works differently to tdeps expectations because of hierarchical dependencies. Also, clojurescript doesn't search the classpath for node dependencies.

You can already depend on npm dependencies. I think it's deps.cljs on the classpath

https://github.com/vouch-opensource/krell/blob/master/src/deps.cljs I can't find docs, but krell uses this feature

OK so I the behavior I see when using the :bundle target is that cljsjs libraries are not connected to the required namespace symbol. When using reagent for instance, in the expression (:require [react]) the react symbol does not get bound to the cljsjs/react library when the react npm package has not been installed to node_modules.

what I see is an npm_deps.js with no entries, and in the compiled code I see reagent.impl.component.node$module$react = require('react');

Just verifying that this is the expected behavior

Folks probably need to understand the ins and outs of all this more as we move forward

@bhauman not expected, file an issue in JIRA this seems pretty minimal

got it

basically if it's not in node_modules it should be resolved in the usual way

cool that’s what I thought the behavior should be

though you should check one thing

whether cljsjs.react has a deps.cljs with :npm-deps in it

@juhoteperi ^

reagent does of course

right so need to think about that then

cljsjs/react does not though

right sorry I mean reagent of course

oh makes sense

cljsjs/react wouldn't do that

yeah open to ideas here, not sure what makes sense

the first thing that comes to mind is :npm-deps supporting excludes

though my feeling is that this feels a bit like overkill

if you're going to use :bundle mixing it with CLJSJS seems silly anyway

especially if the lib you are using declared an npm dep

yes it does but dependencies are trees

but Reagent has done something funny

which is to declare two different deps

the same dep two very different ways

I thought that was the pattern to support both

not the way Reagent has done

Reagent has created a conflict

this is completely different from a user managing stuff

which is why I'm suggesting :excludes to remove an upstream conflict

So install-deps is inly intended to work for libraries that declare npm-deps by default

Huh, what's the problem?

the issue is that Reagent I believe both declared a dep on Maven dep and a NPM dep for the same library

in this case React

though ...

I guess we could only believe node_modules

and ignore :npm-deps

deps.cljs is only used if install-deps is enabled?

@juhoteperi uh no

no no

@juhoteperi we pinged you to find out something we answered already

@dnolen that makes sense to me

hrm

I audited the code a little bit

I'm not saying you aren't observing what you're observing

but I don't see how it wouldn't pick up cljsjs/react , it appears we do use node_modules as the source of truth

yeah it is strange for sure

please make something tiny and I can poke around - it's might be something else

I’ll put it in the jira

thanks!

its just the simplest example without installing react via npm

i’m actually using the the webpack guide example

just skipping the npm add react react-dom step

but there are many strange things about what you're talking about - no errors

which seems impossible

it's either in node_modules or the classpath or you're going to get an error

I’m not excluding cljsjs

but that's what I'm talking about

so it must have found the classpath one

it does find the classpath one and includes it

but doesn’t bind it to the namespace symbol

BTW sorry for punctuating your days like this

no what I mean is there are two incompatible things

A) classpath library is found

B) node.js require emitted even though A) means it cannot be indexed under :node-module-index

FWIW in shadow-cljs I completely dropped support for foreign-libs because I didn't want to deal with "mixing" dependencies. one library depending on cljsjs while the other going to npm directly. you might want to do the same for :bundle builds?

there's no need as far as I can tell as of yet

but there are foreign-libs that are just javascript includes for the local build

just a hassle for existing users who are transitioning too

yeah that’s the problem that someone came to me with this morning

in anycase what you're observing seems like an impossible situation - from looking over the code

you somehow have a node lib and a foreign dep

OK it’ll be in the JIRA

I just wrote a stub library that maps the most common cljsjs package pack to a regular require and exposing the global

eg https://github.com/thheller/shadow-cljsjs/blob/master/src/main/cljsjs/react.cljs

Could it be a transitive dependency?

Eg webpack depends on react or something

he never ran --install-deps so I doubt it

making :bundle work w/ everything that came before isn't that hard and is in the spirit of not breaking stuff that used to work

not going to change anything here

Must have installed webpack though, regardless of using cljs.main to do so?

let me detail the steps so there is little confusion

@bhauman you verified react isn't at the top level yes?

it does not matter who installed it by the way

yes

k

https://clojure.atlassian.net/browse/CLJS-3258

I double checked it to make sure

thanks!!! will take a look

Hey, I’m the one chatting to @bhauman. Wasn’t really expecting this to go here =)…

And here is another question: is this code broken or are we supporting it? https://github.com/JulianBirch/cljs-ajax/blob/master/src/ajax/xml_http_request.cljs#L29

basically its using cljs.core/*target* to detect if its in node.js

unfortunately under :bundle its “nodejs”

not really interested in making a new target

actually

people should detect the platform by some other means

its a problem with the way its written

I was wrong

the require is emitted

either way so it doesn’t work in advanced

right that code just seems problematic

and we can override the *target* in closure defines

if we someone needs to

unfortunately a very popular library

I debated whether or not to actually emit a new target

but in the end that just means another thing for people to hook onto that they probably shouldn't

yeah I’m going to advise they use goog.global.require to not trigger the bundler, and use a different way to detect nodejs

that won't work. require is not a global.

@thheller oh cool

bad idea then

@thheller how do you solve this problem, you must have come across it

cljs-ajax specifically

its not a problem since I don't extract js/require calls in the code for normal builds

so for node builds it takes the correct path and just requires at runtime

for browser builds it also does the correct thing and just uses XHR

and that’s the trick

do you special case other things besides XHR?

no special case at all here. it is the logic in their code that does this.

the problem only appears if you try to extract js/require calls when trying to feed them to the other bundler

I only do this for react-native builds but not browser/etc

OK I’m trying to understand here the

CLJS is emitting require("xhr...")

and we pass that to the bundler

and the bundler does the wrong thing

I’m not understanding the word extract in this context

well we pass all the code to the bundler so that it resolves the requires

oh and I’m talking about advanced builds here

not normal builds

advanced or simple builds

oh right .. well I don't pass anything to a bundler so the "rogue" require call does nothing

its only for react-native builds that I actually extract js/require calls from the code since that will go through metro

but all other builds won't

well that’s food for thought

and this case is particularly interesting because I don’t see how to make this code work at all now, if its getting passed to a bundler

a macro won’t work

because target is “nodejs”

to be honest I still think the library should just fix this

yeah its not a great pattern but I've seen other libs doing this

the question is how?

because it's hard coded to a dynamic require

but how do they fix it?

well its not a problem in shadow-cljs so doesn't require fixing. no clue about :bundle.

but conditional requires are not fun to work with at all so I'd like there to be a better solution. just not sure how.

I don't have any interesting ideas that I'd be willing to implement or see implemented at the moment

the issue is way beyond this particular lib

any library that leaks js/require has this problem

they can’t use :npm-deps

they want a library thats cross-platform but it doesn’t look possible

doing this in JS wouldn't work either?

it's not specific to ClojureScript

it does work in JS and is quite common. webpack eliminates some basic conditionals

eg. https://unpkg.com/react@16.13.1/index.js

oh freak

"webpack eliminates some basic conditionals" != "it does work"

you mean some cases work no?

but that honestly looks like the answer

yes, it works for some cases

not our problem

do what JS does

problem is that it likely won't understand the cljs.core/*target* conditional. I have only seen this work for actual const locals and process.env.NODE_ENV

but that's what I mean

oh OK then that’s doable

kinda

you would emit that process.env.NODE_ENV

the important thing here is this

:bundle has to mean stuff that Webpack could consume sensibly if used

so for a library to adopt :bundle it has to adopt these bundler conventions

about platforms

inventing stuff is not that appealing IMO

NODE_ENV is only about development and production though

you can always ignore the xmlhttprequest require via webpack config when building for the browser

oh well that’s interesting as well

resolve: {"xmlhttprequest":false} or whatever the config for that was

resolve alias

got it

yeah I have something similar in shadow-cljs https://shadow-cljs.github.io/docs/UsersGuide.html#js-resolve .. for some npm packages there is just no other way without manual intervention

good stuff

That means that libraries couldn't change their underlying library without breakage. But that's js too

so yeah day8.re-frame/http-fx doesn’t work for advanced because

of this

but if the underlying lib fixes the issue then it goes away right?

to be honest I don't see any simple way to fix this

you could legitimately sprinkle your code with js/require knowing you will use a bundler

btw, above I wasn't shooting down ideas to fix it - just stating I don't have any good ones

which is why I was suggesting adopting bundler conventions

because I'm afraid we would just interfere with established conventions

Can anyone here shed some light on an aspect of the compiler analysis state cache? In particular, in the following JS console inspect snippet, • why isn't the macros field at the "top" level (same level as extrerns? • how can you have multiple null keys in the map and what does that entail?? • what does edit mean?

6: {ns: null, name: "cljs.core.async", str: "cljs.core.async", _hash: -159169011, _meta: null, …}
7:
meta: null
cnt: 16
root:
edit: {}
bitmap: 221921953
arr: Array(24)
0: null
1: {edit: {…}, bitmap: 34603008, arr: Array(8), cljs$lang$protocol_mask$partition1$: 131072, cljs$lang$protocol_mask$partition0$: 0}
2: {ns: null, name: "externs", fqn: "externs", _hash: 221720677, cljs$lang$protocol_mask$partition0$: 2153775105, …}
3: {meta: null, cnt: 3, arr: Array(6), __hash: null, cljs$lang$protocol_mask$partition0$: 16647951, …}
4: null
5:
edit: {}
bitmap: 3211264
arr: Array(8)
0: {ns: null, name: "use-macros", fqn: "use-macros", _hash: -905638393, cljs$lang$protocol_mask$partition0$: 2153775105, …}
1: {meta: null, cnt: 2, arr: Array(4), __hash: null, cljs$lang$protocol_mask$partition0$: 16647951, …}
2: {ns: null, name: "excludes", fqn: "excludes", _hash: -1791725945, cljs$lang$protocol_mask$partition0$: 2153775105, …}
3: {meta: null, hash_map: {…}, __hash: null, cljs$lang$protocol_mask$partition0$: 15077647, cljs$lang$protocol_mask$partition1$: 139268}
4: {ns: null, name: "macros", fqn: "macros", _hash: 811339431, cljs$lang$protocol_mask$partition0$: 2153775105, …}
5:
meta: null
cnt: 3
arr: Array(6)
0: {ns: null, name: "go", str: "go", _hash: 1493584872, _meta: null, …}
1: {meta: null, cnt: 8, arr: Array(16), __hash: null, cljs$lang$protocol_mask$partition0$: 16647951, …}
2: {ns: null, name: "alt!", str: "alt!", _hash: 1759993452, _meta: null, …}
3: {meta: null, cnt: 8, arr: Array(16), __hash: null, cljs$lang$protocol_mask$partition0$: 16647951, …}
4: {ns: null, name: "go-loop", str: "go-loop", _hash: 692273294, _meta: null, …}
5: {meta: null, cnt: 8, arr: Array(16), __hash: null, cljs$lang$protocol_mask$partition0$: 16647951, …}

An odd (to me) thing is if you look at this in cljs-repl there are no null keys and :macros is at the top level of the map

you are looking at the internal structure of the map implementation. thats not very interesting. use cljs-devtools or just prn the result to get a useful representation

As I mentioned, the 'useful' representation has the :macros field at the top level of the map. Here's a snippet:

((-> state deref :cljs.analyzer/namespaces) 'cljs.core.async)
{:rename-macros {},
 :renames {},
 :externs {Error {}, Array {}, Object {}},
 :use-macros {go cljs.core.async, go-loop cljs.core.async},
 :excludes
 #{reduce take map transduce into partition merge partition-by},
 :macros
 {go
  {:arglists ([& body]),
   :doc
   "Asynchronously executes the body, returning immediately to the\n  calling thread. Additionally, any visible calls to <!, >! and alt!/alts!\n  channel operations within the body will block (if necessary) by\n  'parking' the calling thread rather than tying up an OS thread (or\n  the only JS thread when in ClojureScript). Upon completion of the\n  operation, the body will be resumed.\n\n  Returns a channel which will receive the result of the body when\n  completed",
   :line 4,
   :column 1,
   :file "cljs/core/async.clj",
   :name cljs.core.async/go,
   :ns cljs.core.async,
   :macro true},
  alt!
  {:arglists ([& clauses]),
   :doc
   "Makes a single choice between one of several channel operations,\n  as if by alts!, returning the value of the result expr corresponding\n  to the operation completed. Must be called inside a (go ...) block.\n\n  Each clause takes the form of:\n\n  channel-op[s] result-expr\n\n  where channel-ops is one of:\n\n  take-port - a single port to take\n  [take-port | [put-port put-val] ...] - a vector of ports as per alts!\n  :default | :priority - an option for alts!\n\n  and result-expr is either a list beginning with a vector, whereupon that\n  vector will be treated as a binding for the [val port] return of the\n  operation, else any other expression.\n\n  (alt!\n    [c t] ([val ch] (foo ch val))\n    x ([v] v)\n    [[out val]] :wrote\n    :default 42)\n\n  Each option may appear at most once. The choice and parking\n  characteristics are those of alts!.",
   :line 63,
   :column 1,
   :file "cljs/core/async.clj",
   :name cljs.core.async/alt!,
   :ns cljs.core.async,
   :macro true},
  go-loop
  {:arglists ([bindings & body]),
   :doc "Like (go (loop ...))",
   :line 95,
   :column 1,
   :file "cljs/core/async.clj",
   :name cljs.core.async/go-loop,
   :ns cljs.core.async,
   :macro true}},
 :name cljs.core.async,

this is self hosted and core.async isn't self-host compatible

I am using Andare which is supposed to be self-host compatible. I did discuss a bit with mfikes, but it seemed the issue was more related to the compiler

I find it strange that the internal structure differs in the simple respect that the array map impl has (multiple?) null keys

don't look at the JS representations of the CLJS data structures. they are just going to confuse you. :P

they are null in clojure too .. you are just not looking at the internal java representation ...

eg. {ns: null, name: "cljs.core.async", str: "cljs.core.async", _hash: -159169011, _meta: null, …} ... this is the cljs.core.async symbol but its really hard to tell that from the output

Yeah, understand that - but it does not have null keys

watch some talks or read some blog posts about how the CLJ(S) data structures work. it'll make more sense then.

Putting that aside then (representation aspects), it would then seem that the compiler state does have the macros as in the cljs.core.async name space. But any attempt to reference them gives an analyzer error that the do not exist

for self hosted the macros will live in cljs.core.async$macros but I'm honestly not entirely sure how self-hosted macros work

but I do know that they live in a secondary namespace

Yeah, that stuff is there as well - and the macros are listed in there too

I don't know what you actual question is though. the JS "mess" you posted doesn't really tell you anything about the data you actually have. just pprint it or something to get a useful view of the data

trying to deciper the JS objs is just not useful

??? I posted the actual 'useful view' of the compiler state above

ah thought you got that from somewhere else

No, that is the real live data

@jsa-aerial it sounds like you’re trying to debug some issue with andare. Like @thheller is saying, the three questions you posted above about the JS is a red herring, it probably has nothing to do with your issue. can you explain the issue you’re having with andare?

As mentioned at 4:51 msg, despite the macros being in the compiler state for cljs.core.async, the analyzer claims they do not exist

Maybe this is just a fool's errand and unless you know some deep (and apparently opaque) magic, it just isn't possible to get this to work

where or how does it claim it? maybe your use is just incorrect?

but self-host macros are definitely magic ... 🙂

In a namespace if you issue (require '[cljs.core.async :refer-macros [go]]) for example, it just errors with illegal refer that macro cljs.core.aync/go does not exist.

what about (require '[cljs.core.async :refer [go]]) or (require-macros '[cljs.core.async :refer [go]])?

also I'm assuming you are talking about a REPL right? otherwise requires should be in the ns

You can (require-macros '[cljs.core.async.macros :refer [go]]) w/o analyzer error, but any attempt to use it (for example with <!) just errors with <! not in a go block.

where does cljs.core.async.macros suddenly come from?

That's the separate namespace you mentioned - cljs.core.async require-macros that

no I didn't. I said cljs.core.async$macros. which is different. thats the "secondary" namespace self-hosted will be using for macros

cljs.core.async.macros is another actual ns, which will also have the secondary cljs.core.async.macros$macros

again ... macros in self-hosted are weird 😛

> also I'm assuming you are talking about a REPL right? otherwise requires should be in the ns either repl or ns - either way behaves the same

Well, there is no cljs.core.async$macros in any of the code

then there is your problem

well it shouldn't be in your code .. but it should be in the analyzer state

OK, that seems to be how Andare is written. It uses this cljs.core.async.macros namespace

It's definitely not in the analyzer state. Does that get generated or is it how the code is expected to be written?

that you'll have to ask someone that actually knows how self-hosted works 😛

does (require '[cljs.core.async :refer [go]]) not work?

It does not work

It just says it does not exist

if the analyzer state has no cljs.core.async$macros then it does in fact not exist

Well, I guess this just goes back to the 'deep / opaque' magic bit. Andare does not have any such namespace (that I can find) but the claim is that it does work

it is the namespace the is created when the .clj files are compiled as macros

Oh, so it is generated

it doesn't exist on disk yes ... it is the result of compiling cljs/core/async.clj as a macro namespace

Well, there you are - maybe I just need to grab and cache that at compile time

probably, no clue how self-host handles that.

Let me try that and see what happens

Dependencies are hierarchical in tools.deps as well. Its a big tree

And ya, I would assume the resolver would make symlinks into a local node_modules folder, or to be honest, it be nice if the compiler did look them up in the classpath, and when :bundle target is chosen it would go and create the node_modules symlinks needed for the js bundler to run over afterwards.

I don't see much issue here. NPM seems pretty standard, it's got packages with versions, which can depend on other packages and optionally their versions if provided a package.lock

At any rate, the compiler is just trying to re-implemented what looks to me like a worse version of tools.deps bit for cljs specifically. So I feel that effort would make sense to be consolidated into tools.deps

Wouldn't that kind of conflict be something that would be solved if we reconcile it all in tools.deps? Tools.deps could find conflicts between a cljsjs and a node_modules or a git deps, etc. And just resolve which one to pull down

> maybe I just need to grab and cache that at compile time Nope that doesn't work either. In fact the compiler state at build time claims there is no cljs.core.async$macros namespace. I think I am going to give up and approach originating problem in a (less general, less simple, and less clean) fashion - but which should work, as I will be using core.async at the app (non self hosted) level (which definitely does work). 😒 this was one of the more total wastes of time I've had the misery to have engaged in.