Clojurians Log v2

hmm odd since it is always compiling the same amount of files. maybe some external tool (eg. your editor) touching all the files all the time?

but yeah most commonly this was caused by having :output-dir into something that is also a :source-paths entry

With the node-library target, is there a way to include all dependencies from node_modules in the target JS file? (asking because that would be useful in the context of cloud function setups like AWS Lambda)

you can post-process the :output-to with https://github.com/vercel/ncc or just include the proper node_modules for aws

I wonder why everyone keeps asking this? Is this not the normal thing you'd do with AWS? https://docs.aws.amazon.com/lambda/latest/dg/nodejs-package.html

I've never done anything with AWS so I really don't know what the best practices for this are but it seems straightforward?

Yeah, you’re right, I think just packaging with node_modules also works perfectly fine

Actually ncc does seem nice. One problem I’m seeing with it is that the source maps probably wouldn’t be accurate anymore(?) did anyone run into that?

Thanks for the tip - I'll take a look again next week

Whats the performance impact of having lots of builds? If the cache is shared between them it shouldn’t be too bad right? (Is it?)

@martinklepsch For AWS lambda the best possible approach is to pack dependencies and ship them as a layer

Ah, that’s interesting! Can you explain a bit more how this would manifest? Like if I have a lot of node_modules I’d put those in a layer and then the deploy payload would get smaller I guess? Does it have any impact on performance? Right now with ncc + advanced + zip I get pretty tiny lambdas (25kb) and the deployment story of a single JS file just seems more straightforward.

So I’m thinking layers yes, but probably only once your individual lambdas get too big?

@karol.wojcik what do you use for deployment and resource automation? I just played around with https://arc.codes/ a bit and I’m really liking how little assumptions it makes around build tools and what not

Did some exploration of using https://arc.codes to deploy CLJS to AWS Lambda today: https://github.com/martinklepsch/shadow-cloud-fns/tree/master/trying-arc — it all works pretty well. The only thing I’m still a little unsure about is the ncc source maps situation https://clojurians.slack.com/archives/C6N245JGG/p1620472416489100

Layers are super useful when you have a lot of dependencies since you pack your dependencies once. For this I strongly recommend trying yarn. I started using it 2 years ago and now I now that I will never go back npm. Regarding performance it should not vary much where dependencies are stored. I'm already doing the same for babashka runtime and I'm happy with the results. If your lambdas are 25kb it's probably not worth trying layers though. For deployment and resource automation I'm using AWS SAM. I have already used Serverless Framework and I find AWS SAM the simplest and the best solution. I have never tried Architect , but for me it's just a sugar around AWS SAM and cloudformation. For me having all resources of the stack in template.yml is super simple. https://github.com/FieryCod/holy-lambda/blob/master/examples/bb/native/sqs.example/template.yml Btw I'm planning to add support for Node.js in holy-lambda, so if you would like to help me out it would be lovely.

cache is not shared between builds

fwiw shadow-cljs also has some basic support for this directly. it does however have some issues so not all packages work. https://github.com/thheller/shadow-cljs/issues/290#issuecomment-459818765

don't know about source maps and ncc

in general I'd likely recommend creating a dedicated folder that you'll send to AWS

create a package.json in that folder and run npm install --production

that'll not install devDependencies and the resulting node_modules should be much smaller

the just :output-dir "that-folder/lib.js" and just zip and deploy that-folder

I’m trying to set up a React Native project using Fulcro 3. I get a strange exception when I run shadow-cljs watch :app. About piggieback. My deps.edn looks like so:

{:paths   ["src/main"]
 :deps    {org.clojure/clojure    {:mvn/version "1.10.1"}
           com.fulcrologic/fulcro {:mvn/version "3.4.21"}}

 :aliases {:dev {:extra-paths ["src/dev"]
                 :extra-deps  {org.clojure/clojurescript   {:mvn/version "1.10.742"}
                               thheller/shadow-cljs        {:mvn/version "2.12.5"}
                               binaryage/devtools          {:mvn/version "0.9.10"}}}}}

{:deps {:aliases [:dev]}
 :builds
 {:app
  {:target :react-native
   :init-fn <http://example.app/init|example.app/init>
   :output-dir "app"
   :devtools {:autoload true
              :preloads [shadow.expo.keep-awake]}}}}

~/Projects/rn-fulcro-shadow(main|✚3…) % npx shadow-cljs watch :app
shadow-cljs - config: /Users/pez/Projects/rn-fulcro-shadow/shadow-cljs.edn
shadow-cljs - starting via "clojure"
WARNING: When invoking clojure.main, use -M
[2021-05-08 18:43:55.123 - WARNING] TCP Port 9630 in use.
[2021-05-08 18:43:55.363 - WARNING] :shadow.cljs.devtools.server/nrepl-ex
Note: The following stack trace applies to the reader or compiler, your code was not executed.
CompilerException Unexpected error macroexpanding if-ns at (cider/piggieback.clj:22:1). #:clojure.error{:phase :macroexpansion, :line 22, :column 1, :source "cider/piggieback.clj", :symbol if-ns}
        clojure.lang.Compiler.macroexpand1 (Compiler.java:7019)
        clojure.lang.Compiler.macroexpand (Compiler.java:7075)
        clojure.lang.Compiler.eval (Compiler.java:7161)
        clojure.lang.Compiler.load (Compiler.java:7636)
        clojure.lang.RT.loadResourceScript (RT.java:381)
        clojure.lang.RT.loadResourceScript (RT.java:372)
        clojure.lang.RT.load (RT.java:459)
        clojure.lang.RT.load (RT.java:424)
        clojure.core/load/fn--6839 (core.clj:6126)
        clojure.core/load (core.clj:6125)
        clojure.core/load (core.clj:6109)
        clojure.core/load-one (core.clj:5908)
Caused by:
ExceptionInInitializerError 
        java.lang.Class.forName0 (Class.java:-2)
        java.lang.Class.forName (Class.java:468)
        clojure.lang.RT.classForName (RT.java:2211)
        clojure.lang.RT.classForName (RT.java:2220)
        clojure.lang.RT.loadClassForName (RT.java:2239)
        clojure.lang.RT.load (RT.java:449)
        clojure.lang.RT.load (RT.java:424)
        clojure.core/load/fn--6839 (core.clj:6126)
        clojure.core/load (core.clj:6125)
        clojure.core/load (core.clj:6109)
        clojure.core/load-one (core.clj:5908)
        clojure.core/load-one (core.clj:5903)
Caused by:
ClassNotFoundException com.google.javascript.jscomp.AnonymousFunctionNamingPolicy
        jdk.internal.loader.BuiltinClassLoader.loadClass (BuiltinClassLoader.java:606)
        jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass (ClassLoaders.java:168)
        java.lang.ClassLoader.loadClass (ClassLoader.java:522)
        java.lang.Class.forName0 (Class.java:-2)
        java.lang.Class.forName (Class.java:468)
        clojure.lang.RT.classForName (RT.java:2211)
        clojure.lang.RT.classForNameNonLoading (RT.java:2224)
        cljs.closure/loading--5569--auto----3990 (closure.clj:9)
        cljs.closure__init.load (:9)
        cljs.closure__init.<clinit> (:-1)
        java.lang.Class.forName0 (Class.java:-2)
        java.lang.Class.forName (Class.java:468)

@pez wrong CLJS version. 2.12.x requires 1.10.844

To add to thheller's answer, according to documentation, you simply don't need to specify any version of ClojureScript - shadow-cljs already depends on the needed one. That is, unless you need a very specific version of ClojureScript. https://shadow-cljs.github.io/docs/UsersGuide.html#failed-to-load

Thanks! I'll send a PR on the fulcro book about it.

Thanks. Will try it out when I'm back at the keyboard. But, I thought no piggieback was involved?

So I guess that means initial build times grow with number of builds. What do you think would the impact be if you had, say 20 builds, for incremental builds? I realize this is a bit outside of the usual but just curious what your potential concerns would be with such setup

shadow-cljs does not use it but it is added for compatibility with stuff that expects it (eg. cider-nrepl)

I don't have a clue what you are building so impossible to comment

shadow-cljs is very flexible so if the default targets don't do what you need it would be relatively easy to add one that fits better

wow, this looks very cool @karol.wojcik, I’ve just been thinking that babashka tasks seems like the perfect foundation for this type of thing

I've prepared a minimal repo and README capturing the difficulties I encountered getting a shadow-cljs plus tools.deps project working well with cider after transitioning over from years of clojure-cli plus figwheel-main https://github.com/codeasone/starter-cider-tools-deps-shadow - it was certainly more painful than I'd like it to have been.

that seems to be much more complicated than it needs to be

yeah, I would have been delighted if I could just run watch in a terminal, and establish a sibling cljs REPL connection, but I can't make that work

switching between and eval-ing forms in clj cljc and cljs code and having it target the correct REPL is essential for my workflow

why not? I mean cider does have a remote connect open, but everything you described seemed to be based on jack-in?

but you can also just run in embedded mode but you sort of flipped it to embed everything in shadow-cljs instead

I end up with two "sessions" in cider and the way they are managed is not smart enough

adding a custom shadow.user ns is definitely not the way to go here

the only thing that works - and it works well - is to have cider-jack-in-clj&cljs control everything - it's also really simple to bring everything up with one key binding

I guess what I’m asking is how shadow will perform with one build vs. 20 identical ones with different target files.

how am I supposed to answer that when I don't have a clue what you are building? shadow-cljs doesn't care, you can have a thousand builds, it does not matter to shadow. your computer will probable scream and blow up since memory and cpu consumption will go through the roof though

I do not know what you are doing. do you intend to build all the builds in parallel? sequentially? watch all? just release all?

what is the point of having 20 identical builds with different target files? why is it not one build and you copy the output file 20 times?

you are only giving me very vague information so I really cannot give you any reasonable answer 😛

I made a typo in one of the heading https://github.com/codeasone/starter-cider-tools-deps-shadow#jacking-in-to-clj-repl-and-adding-a-sibling-jack-in-for-shadow-cljs I actually tried establishing a M-x cider-connect-sibling-cljs which would have been ideal, but as I say didn't work out due to the way REPL sessions are managed.

I do not have a clue what the commands are called in emacs. the way I work is I run my backend manually outside my editor, typically just lein repl. then I start shadow-cljs server separately.

then I connect to the .nrepl-port for my CLJ REPL and .shadow-cljs/nrepl.port for the CLJS REPLs

two separate processes, completely separate connections, nothing shared

I would expect emacs to be capable of doing the same but I don't have a clue

you can instead run shadow-cljs in embedded mode so it runs in your backend process

you sort of flipped it and run your backend process in the shadow-cljs process. not something I recommend but you can do it.

A moment of inspiration struck me, I just wanted something that worked and fit my muscle memory of how to bring up my full-stack, I honestly don't know the intent behind shadow.user ns, and whether my usage is misuse, but I defer to your advice of course.

1. Unhandled clojure.lang.ExceptionInfo
   shadow-cljs has not been started yet!  In embedded mode you need to call
   (shadow.cljs.devtools.server/start!) to start it.  If you have a shadow-cljs
   server or watch running then you are not connected to that process.

did you ever try that? I mean running the shadow-cljs server in embedded mode in your backend process?

I assume that is how you ran figwheel previously?

> Picking up on shadow-cljs has not been started yet! I try running shadow-cljs watch app in a separate terminal (which worked fine):

running shadow-cljs watch will start a new JVM process, instead you are supposed to run (shadow.cljs.devtools.server/start!) in whatever JVM you were connected to when you got that error

> But I just started it!

exactly the point, you started it separately but still talk to the old JVM

Re: figwheel I just invoke vanilla cider-jack-in-clj&cljs and it just works, clj REPL spins up - I (start)the backend server and figwheel is piggybacked in to the same cider session, all orchestrated by Emacs

what is (start)? I'm guessing thats a user.clj function you have in your codebase that starts figwheel in embedded mode

wanted to get to same single-step bring up under Emacs, that's the point of the repo

(start) is just something to bring up the backend - just an example of something you might type into the REPL to start the HTTP server for instance

Unfortunately he cljdoc build is broken btw, if you run the following in your repo you can see an issue with cljdoc.edn

curl -fsSL <https://raw.githubusercontent.com/cljdoc/cljdoc/master/script/verify-cljdoc-edn> | bash -s doc/cljdoc.edn

as per the demo-rig code, which I tried to keep as minimal as possible, but reflect the various moving parts I need to content with day-to-day in my production code.

but this is not minimal at all

This can also be used as a CI step and will fail if there is an issue

okay, I wanted some clj some cljccode and some cljs code so I could ensure I could eval forms and the appropriate REPL would be targetted

I'm also learning to use shadow-cljs and understand it's capabilities in the process

sure I get that

to me it sounds like all your were missing is instead of calling just (start) you also call (shadow.cljs.devtools.server/start!)

https://github.com/FieryCod/holy-lambda/pull/39

or have the (start) do that for you wherever that was coming from. I assume you did something like that with figwheel

I'll try that out soon, thanks

point is when you run cider jack-in that starts a JVM for you with a nrepl server. if you then run shadow-cljs watch separately you'll get a second JVM instance with its own nrepl server.

Going to hit the sack now though, thanks for the feedback

Ok fair enough 😄 Maybe I’ll try to write up more specific information tomorrow 🙂

how you deal with that in emacs I don't know. if emacs doesn't support two separate remote connections then you likely want to run shadow in embedded mode https://shadow-cljs.github.io/docs/UsersGuide.html#embedded

> so if you would like to help me out it would be lovely. I’ve played with shadow-cljs quite a bit this weekend to see how it worked with different packaging strategies/deployment tools (like Architect) — if that sounds interesting maybe we can have a chat?