Clojurians Log v2

I am trying to understand the following type

user> (type in-ns)
clojure.lang.RT$1
user> 

that's the naming scheme javac uses for anonymous classes

it would be https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/RT.java#L240-L247

which is assigned to IN_NAMESPACE here https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/RT.java#L319

so RT$1 is the name assigned to the first anonymous class inside the RT class

Hello. Does anyone here know about some workaround for this issue with clojure compiler https://clojure.atlassian.net/browse/CLJ-1852? It seems to be a cause of this issue in Fulcro I reported a few days ago: https://github.com/wilkerlucio/pathom/issues/187. Perhaps @alexmiller might have some workaround idea?

Thanks for the link. I couldn't find this relationship between IN_NAMESPACE and IN_NS_VAR. Now, it makes sense. 🙂

The long name is the combination of nesting and existing names, so using shorter intermediate names and/or nesting less are the two main workarounds

That's what I thought as well. But if I'm looking correctly to that specific problem in compiling of pathom, it doesn't seem to use that-much of a nesting, does it? https://github.com/wilkerlucio/pathom/blob/v2.3.1/src/com/wsscode/pathom/core.cljc#L564

Maybe if one would macro-expand all that, it's actually nested a lot. But at first sight I don't see any obvious "low-hanging fruit" of easily reducing the nesting in that code.

isn't the code in question a bit problematic: nested go blocks

I suspect that could be one of the reasons why it explodes

@brdloush hard agree with mpenet - the big culprits for massive method size and deep unexpected nesting after macroexpand are go, core.match, and for / doseq - if you nest those things enough you will run into issues

usually it's a question of decomposing into functions to remove that nesting of "verbose" macros

the "low hanging fruit" in your example is to replace the nested go block with a call to a function that makes a go block

re-reading - it's probably actually to move the function definition out of the deep nesting where the current literal is (and factor any captured locals into a hash map arg)

I didn't dig deeper but for sure some refactoring might help

I see this thread, https://clojureverse.org/t/what-about-goto-in-clojure/3943/19, is it possible to create a goto macro in Clojure? Make a label somewhere and use (`goto label)` to continue execution at the label?

in theory, but it would all have to be in one method body for a literal vm goto

otherwise you end up needing an interpreter, which clojure intentionally is not

@erwinrooijakkers an option that I don't see in that thread: letfn plus trampoline allows a structured rewrite of any code block that uses goto

replace each segment of code between labels with a named fn inside letfn, and replace every fall-through to the next label with returning the function replacing that label

similarly, replace each goto of some label with returning the function replacing that label

it has the same semantics (but is a little slower thanks to function call overhead)

There's also an issue in JIRA about loop + recur-to I believe

;; x = 1
;; a: inc x
;; b: if (x > 12) goto :c
;; goto :a
;; c: return x*x
(let [x (volatile! 1)]
  (letfn [(a []
            (vswap! x inc)
            b)
          (b []
            (if (> @x 12)
              c
              a))
          (c []
            (* @x @x))]
    (trampoline a)))

it's a tedious way to write code, but if going through Knuth, like the example, the translation is very direct

(returns 169)

(nb. the gotcha with trampoline is you have to wrap a function in a container in order to return it)

a version of trampoline that works on (function, value) tuples instead of just function might be a fun exercise

@noisesmith @erwinrooijakkers Might be relevant https://archive.clojure.org/design-wiki/display/design/Named%2Bloops%2Bwith%2Brecur-to.html

you could use a continuation monad to get an extremely literal translation of goto containing code https://gist.github.com/hiredman/235655ccde8689499576db882095959f is of what using a continuation monad can look like, it creates labels and uses goto to jump to them.

with continuations a label is "capture the continuation here" and goto is "replacing the continuation here with this captured one"

@borkdude interesting that your link doesn't mention trampoline - I always thought that letfn (which it does mention) is pretty much pointless outside trampoline

(and trampoline addresses the non-tail-recurring complaint in that post)

Thanks a lot for your ideas, guys. I'll pass this information to the mentioned pathom issue. And if I find some spare time, I'll try to look into the issue myself just to learn what can be done with it.

https://gist.github.com/779827c2f1055c5ab080a32bff31656b is the little continuation monad library that example code uses, it is kind of customized around the needs of that particular library, no docs on it of course

it'd be nice if the compiler itself didn't have that issue though 😉

it's intrinsic to the compilation model - fn inside fn is a nested class

since go rewrites things into fns, your nesting can get very deep

fixing this compilation issue (without losing performance) would be one of those "clojure 2.0" things, and I suspect clojure 2 would never happen

I don't think it needs to be of that level of significance. it's on the list to think about for 1.11 at least

oh wow - I'm glad if that's the case

letfn is useful even with a single recursive function - to avoid having to type its name twice.

I'm not saying it's likely :)

in a way that (fn foo [x] ...) isn't?

(let [foo (fn foo [x] ...)] ...)

is the double name thing

Awesome! I don't want to push my luck, but speaking of list of things to think about.. I wonder whether this is still a possibility? 😇🤞 > Rich and I have been kicking around some interesting ideas on a class compilation cache that would be integrated with core and that research is in scope for Clojure 1.10 but no guarantees on anything. Potentially it would combine the benefits of publishing source-only but the performance of AOT. It is inherently silly to recompile the same lib source file over and over when it’s not changing. https://clojureverse.org/t/deploying-aot-compiled-libraries/2545/10

I disagree with the letfn hate, letfn is great, I prefer it to a regular let binding of functions

you can get 90% of the benefits of that using the approach in https://clojure.org/guides/dev_startup_time right now

(and probably fewer of the downsides)

a gist for all seasons https://gist.github.com/hiredman/1179073

I do too, but only when there are no non-fn bindings involved, to avoid an extra level of nesting.

here's my letfn: https://github.com/borkdude/sci/blob/master/src/sci/impl/namespaces.cljc#L284-L293

sure, creating a cyclic graph is trivial when you mutate stuff

Are people following the https://clojure.org/guides/dev_startup_time dev model at scale? It worked well for me in a small-medium project (where it shaved some 10s startup time) but in a larger app it would immediately cause issues. Which would not seem surprising when "AOT in dev" is kind of a frowned-upon thing (and a root cause diagnostic that I've seen given in this very channel many times) So I'm a bit conflicted as to what to believe

I think that guide undersells the tricky bits of making aot in dev work because it is written by someone with considerable clojure and clojure tooling expertise

I have a compile.clj script that, when loaded, will compile several top-level namespaces in our codebase to follow that "AOT in dev" model, but I don't update the compiled classes very often. I haven't noticed any unusual behavior with it but I'm probably not pushing it hard -- it does improve initial load time in dev noticeably but, OTOH, I only restart my REPL maybe once a week so I don't suffer that initial load time very often anyway...

thanks! do a lot of 3rd-party libs get compiled transitively?

Yes 😞

that's kind of good news as it means that many libs in the wild can survive AOT in dev? :) I think I had problems with two, although I'm not experienced in AOT to actually tell if they were wrong

it would be helpful to file questions on https://ask.clojure.org if you run into them

AOT compile is exactly the same compile as the not AOT compile, so generally all libs should "survive AOT" (because they are being compiled on load)

there can be issues with load ordering, particularly around protocols being reloaded after instances have been loaded

yes @seancorfield, about repl management, I can say I improved a lot my workflow thanks to your advices. I mostly start my repl for deps.edn update right now ! it has been a learning curve but the final experience is much more "sure", I feel like I am more mastering the state of my code, even during deep refactoring

if there are issues that people are running into doing this kind of aot dev env, I would love to be hearing about them so we can assess if there are things to fix/improve

in my experience, using pre-compiled libs is problematic (because that nails down use of a specific clojure version and to some degree fixes compilation order in a way that may be different than use), but using aot in dev is not inherently frowned upon - it is exactly the same work being done to compile at load time, just saved to disk

@vemv FWIW, I just cleaned out my local classes folder and kicked off the compile script and that produces just over 13K .class files.

(:foo (sorted-set 1 2 3)) => ClassCastException (can't cast Long to Keyword)

Hey all, old question I'm sure but I had trouble finding an answer: is there a way to intern a new dynamic var? Like (intern (ns-name *ns*) (with-meta 'blorp {:dynamic true})), except it works

Cool lisp (not clj) article on lisp shells https://ambrevar.xyz/lisp-repl-shell/index.html

(some #{:foo} (sorted-set 1 2 3))
=> nil
(some #{1} (sorted-set 1 2 3))
=> 1

mm no, by generic I mean something that works on ANY object without throwing exceptions

this requires my generic arg to be a collection

You could just filter for coll? first

I guess what's trickier is also avoiding a linear search

I also want this stuff to be fast, e.g. O(1) on maps, not O(N)

sorted-map cannot be O(1). :)

ehhh

(defn safe-contains? [maybe-coll val]
  (try
    (contains? maybe-coll val)
    (catch Throwable _ false)))

I mean reasonobly fast

yeah, probably catching is fine...

hey there, i’m running into an opaque macroexpansion error on clojure 1.10.2 that i did not have on 1.10.1, and i was wondering if any folks had some ideas (details in thread):

here’s a repro repo https://github.com/robhanlon22/clojure-1.10.2-compiler-error-repro

i have a project that depends on specter and methodical and a newer version of riddley than is specified in either of those projects

i’ve tried adding an exclusion for riddley to both methodical and specter’s dependency vectors, but this error persists

the error is specifically this:

{:type clojure.lang.Compiler$CompilerException
   :message Unexpected error macroexpanding com.rpl.specter/path at (com/rpl/specter.cljc:980:9).
   :data #:clojure.error{:phase :macroexpansion, :line 980, :column 9, :source com/rpl/specter.cljc, :symbol com.rpl.specter/path}
   :at [clojure.lang.Compiler macroexpand1 Compiler.java 7023]}
  {:type java.lang.NoClassDefFoundError
   :message clojure/core$seq_QMARK___5406
   :at [riddley.walk$macroexpand$fn__214 invoke walk.clj 14]}
  {:type java.lang.ClassNotFoundException
   :message clojure.core$seq_QMARK___5406
   :at [java.net.URLClassLoader findClass URLClassLoader.java 382]}

thanks folks!

note that lowering the clojure version to 1.10.1 causes the error to go away

thanks in advance!

Just in case you don't get an answer here - that statement about "working with 1.10.1, not working with 1.10.2" might make it worthy of a question at https://ask.clojure.org/

thanks! i’ll post in there in a bit.

there's not that much different in 1.10.2 so that is surprising to me

it’s surprising to me too! i just checked, and it actually even manifests on 1.10.2-alpha1

it looks like methodical is a compiled uberjar and includes riddley, potemkin, etc inside it

that's always bad

it's basically a hidden dep conflict in that you end up with two versions of those transitive deps on the classpath, one pre-compiled with a different version of the compiler

ah! no wonder the exclusion had no effect

nice catch 🙂

same bug is filed multiple times on the methodical repo

like https://github.com/camsaul/methodical/issues/40

I commented there

thanks so much mr. miller!

https://github.com/camsaul/methodical/issues/33

^^ closer to yours

If excluding potemkin and ridley has no effect, it sounds like some other dependency is including their source or more likely aoted classfiles

Which is often bad news

(oops. missed already done I guess)

Would it be reasonable to detect and issue a warning when you're trying to exclude something that was uberjar'd?

it is tricky

impossible in the general case, maybe sort of workable in someway?

clojure has kind of two languages of dependencies

the language of clojure itself is expressed as clojure namespaces, references to classes by name, etc

but overlayed on top of that is the language of artifact dependencies (maven mostly)

the names in those two languages have no mechanical linkage

and exclusions are expressed in terms of maven names, and compiled clojure classfiles exist in terms of clojure names

And it's not possible to robustly get the URL from which a particular ns was loaded, right?

something that would give you similar kind of warning, and might be easier to implement, is a utility to look at your dependencies and warn you if any of them contain both classfiles and .clj files

you could maybe download the transitive dependency tree of your exclusions, build a list of namespaces that seem to be defined there, then look through your dependencies and issue a warning if they are defined there

Mmm, right, I see. Thanks!

I believe this utility to examine your classpath exists

it certainly exists for java, and I seem to recall someone made a clojure friendly one too

but unfortunately I don't remember a link or context for it

I think I recently discussed a similar thing with @noisesmith. The way I solved a similar problem was just to do:

(io/resource "clojure/spec/alpha__init.class")

user=> (binding [clojure.core/*loading-verbosely* true] (require '[clojure.spec.alpha] :reload-all))
(clojure.core/load "/clojure/spec/alpha")

(clojure.spec.alpha is just a random example here, that wasn't the lib in question I had a conflict with then)

that kind of analysis is handy (an oracle that tells you which namespaces are defined in which jars), I started writing it the other day (for what feels like the 10th time) because I had the idea that some of our projects at work have unused dependencies, and getting rid of them would cut down on build artifact size (big dumb tree shaking)

But maybe clojure itself could emit this information during loading verbosely

for our purposes, while it is true that issues like this usually crop up because of aot dependencies, the aot bit really doesn't matter

you want it to be an error/warning when a clojure namespace froms a jar file you don't want it to come from

@hiredman there is also tools like this: https://github.com/SevereOverfl0w/vizns (based on static analysis, not on runtime data). This probably won't find the above problem though, since it relies on source for analysis, not AOT-ed stuff.

https://github.com/hiredman/clojure-dependency-grapher 12 years old

but yeah, vizns looks like what I was describing, not just namespace dependencies, but including maven artifact dependencies

so, on a different tack—it appears that methodical is including potemkin (and its dependencies) because of potemkin’s definterface+

it internally defines a java interface

is there a way to work around needing to AOT compile that so the interface is present?

or would that require a restructuring

That interface would be defined when you load those namespaces, so why is AOT needed?

ah, i see what’s happening

the interface namespace is never :required, it’s only :imported

it seems even somewhat likely that the lack of the :require caused them to aot, so they could import without requiring, the aot caused them issues where some code would use the aot'ed interfaces and some would use the newly created interfaces once the required code was loaded, which caused them to bring in potemkin to definterface+ which seems to have its only feature avoiding re-defining the interface

thank you for the help and great conversation!