Clojurians Log v2

A library that I'm publishing on clojars requires some AOT compilation due to :gen-class. I'd ideally like to allow users to have this library as a git dependency too. My question is this: does the Clojure compiler produce bytecode that's dependent on the JVM version it's running on? e.g. if I compile on openjdk11, will I be able to rely on the classfiles generated being able to run on openjdk8?

In https://clojure.org/guides/repl/enhancing_your_repl_workflow#writing-repl-friendly-programs why is print-number-and-wait re definable here:

(future
  (run!
    (fn [i] (print-number-and-wait i))
    (range)))

(future
  (run!
    print-number-and-wait
    (range)))

@suskeyhose More to the point, if you compile code against a specific version of Clojure, people using other versions might not be able to use your library.

Because a (fn [..] ..) compiles in a way that will still deref the name to get the current function value, but when passing a function by name the compiler derefs it to get the underlying function value.

That's a good point, but an unfortunate requirement. In this case, the actual generated code is exceedingly simple, just a call to = and a couple of keyword lookups on the state of the generated class, which extends java.lang.Error

(it’s a bit more complex than that but that’s the gist of it)

Oh and the class implements an interface for a protocol

So I'd imagine it's unlikely to see version conflicts.

Not a good idea

Aot compilation works best at the end of the dependency chain (you might aot your final program)

If you start inserting aot in the middle (aoting chunks of libraries) things can go very wrong

Yeah, I've been over this before. The unfortunate reality is that with the current facilities Clojure provides I cannot extend Throwable in a way that I can catch the resulting class without :gen-class

The hazards you have to navigate are largely related to aot compilation being transitive, and order dependent

So aot compiling code will aot compile everything it depends on

There is exactly one namespace that needs aot compilation, and it only needs to create a named subclass of Throwable.

@suskeyhose stay tuned for some new features in the hopefully imminent future

Thanks alex! I'm looking forward to it!

Which will, if you package it in a jar, result in something like an uberjar

Which depending on can lead to all kinds of dependency conflicts

Sure, except in this case there are no transitive dependencies of this namespace.

Well, clojure.core is a dependency I bet?

So compiling your single namespace will cause a bunch of compiled .class files to appear I suspect.

depstar allows me to elide all of the clojure.core class files

and that has worked fine when depending on it from clojars.

Eliding class files can be done, but it will also break things sometimes

clojure.core is already compiled so will not produce any class files there

Fair enough. I don’t like to encourage AOT for libraries (and I think you get a warning, yes?) but some folks insist…

Yeah, I would love to avoid aot if I could

Because the next time whatever elided clojure code is compiled it may not have the same class name

So your aot compiled code may refer to a class that doesn't exist any more

Yes, it will not produce any clojure.core class files, but I had to elide some clojure.core.specs.alpha class files

Yes, I would love to avoid aot completely, and if you have a way I can extend Throwable and catch the resulting class without aot, I will gladly switch.

ideally, relying on ex-info is better

Unfortunately that's not an option in this case because this is for control-flow constructs and I can't rely on third party code to re-throw the ex-info

For reference, this is for https://github.com/IGJoshua/farolero

@alexmiller If you compile a very simple namespace, even though you don’t get classes from clojure.core, you do get classes from core.specs.alpha in your classes folder.

Yup, I had to elide those

that is true

Since @alexmiller is around: what about the OP’s Q which we haven’t answered yet… “if I compile on openjdk11, will I be able to rely on the classfiles generated being able to run on openjdk8?”

should be, they are java 8 bytecode

whether or not you make calls to jdk apis added after java 8 requires more discipline

Cool. I had a feeling I’d seen that as an answer to another question here recently.

Awesome, that helps out a lot.

Yeah, I'm not calling any JDK api, all I'm doing is extending java.lang.Error with a named class that has a protcol implementation that checks if the argument is equal to an item in the state.

there is a lot of fud around aot. it's not conceptually any different than all java libs that "aot compile". there are some issues for sure - we may take on making those issues less of a problem.

Although now that I think about it, I could potentially reduce how much is AOTed by using extend-protocol in the other namespace.

rather than directly extending the interface

“fixing” the transitive compilation stuff would probably be the #1 thing folks seem to want (although there are tooling workarounds) — but I understand that there are complexities behind that.

Making it easy to compile some thing(s) at the point of use, i.e., via a command that runs code, would also probably satisfy what most folks actually need 🙂

Yeah, that or a way to extend classes while generating a named class is all I need

in order to completely remove aot from the equation

all of those things are up for consideration

On mobile, but would it be possible to put your error class as a stand alone maven project and just depend on that from your clojure project?

Then no aot and can use git deps and anything else you like

Hmm. Except for the ns form, I think I could actually entirely remove the dependency on clojure.core

That's actually a good question @dpsutton

I'll consider that. It depends on what gets generated once I pare down what's actually in the aot-ed part.

I think I can get everything that's there down to special forms.

I think two jobs ago we did something like this and just compiled it once and stuck it on the class path.

Checked in and everything if I remember correctly

Alright, well after getting it down to just special forms and an ns clause it's still a few too many class files. I think I might just make a java artifact that I can put out on clojars.

Thanks for the suggestion @dpsutton!

@suskeyhose one thing that i think might be worth investigating is just using https://bytebuddy.net/#/

making your class and loading it into the right classloader

I think that adds more overhead than just making an additional jar with the single class file in it, and it certainly increases the runtime overhead for users of my library.

Clojure has a censored version of asm

Vendored

If you are willing you can do all kinds of stuff with it https://git.sr.ht/~hiredman/reagents/tree/master/item/src/com/manigfeald/reagents/struct.clj

(oh, I forgot that is such a heinous arrow)

I’ve just adopted using component(https://github.com/stuartsierra/component) in my project, and I’d like to know how others think about the experience. I haven’t checked, but there seem to be quite a few alternatives like integrant and mount. It might be because I’m inexperienced, but restructuring a non-component project (which was made of many global singleton defs and defns) into a component project wasn’t easy..

I like that using these libs seem to direct code organisation towards a more decoupled way.

I've worked with all 3 and currently prefer integrant then component, lastly mount

> restructuring [...] wasn't reasy Seems like it's one of the reasons behind https://github.com/juxt/clip since it can work with regular functions. Personally, I use Integrant. Wasn't that hard given that I had to just add a bunch of multimethods only for the lifecycle parts that I was interested in.

What I like in integrant is that system structure and component config are in the same source data structure

At least in my experience

I prefer stuartsierra/component for the simplicity of the model. In the ends it's a topological sort and a function to map over your graph. I used the generic mapping in the past to implement "system wide" functions, other than start and stop.

Is anyone aware of a utility lib that has a hybrid let/`letfn` macro? I often want to use letfn but end up just going with a let as I also need let-like bindings in the same scope and don't like the idea of adding yet another level of indentation. I'm looking for something where you can mix let and letfn style bindings in the same binding vector

To add to what imre said - it can be separated with e.g. Aero. That's exactly what juxt/yada uses, and sometimes it makes sense - in my case, because I'm using shadow-cljs and I want to pull some of the config from the shadow-cljs.edn file.

I use juxt clip, across a variety of applications. Works really well.

Similar concepts as others (there is a small comparison here: https://github.com/juxt/clip#comparison-with-other-libraries)

I've used every library mentioned so far apart from Component, and I also prefer Clip. The only downside is that it is not (yet) possible to start/stop a partial system.

What I don't like about Mount and Integrant is that both conflate namespaces with components. I also found Clip pretty easy to grok, compared to e.g. Integrant. YMMV, of course.

I hardly use letfn, I just can't remember the syntax. Imo letfn is useful for mutually recursive fns, but in other cases, I would just use let

my code is usually littered with letfn , and I’ve also often wished to mix let with letfn but not often enough to write the macro.

@imre what would be the clojure ideomatic way to distinguish the two types of bindings? For example, I very well might want to bind a variable to a vector. would it be the case that a list of 3 or more components whose 2nd element is a vector is necessarily an attempt to bind a function? and the macro could simply insert the (fn …) around it? or would you want something more exotic?

the elements of a letfn binding vector are list s

destructuring works off vectors and maps

so if the next thing you see in the binding vector is a list, that would be your letfn thing

It's a pretty simple and convenient syntax actually if you only need to let fns

(letfn [(foo [] 1)] (foo))

yeah, once I look it up it makes sense, but I just can't remember it

We’ve recently found redelay and have been really happy with how simply it deals with state. https://github.com/aroemers/redelay

(let+ [bar           4
       (foo [x y z] [x y z])
       {:keys [baz]} {:baz 2}
       [_ _ qux]     [0 0 3 4]]
  (foo bar baz qux))

I prefer it a lot to using let when it's possible as it's a lot more concise

@tlonist.sang I just did the same but with integrant. It’s a lot of up front work but when you get into the “reloaded workflow”, it makes things so much smoother.

Took me some time to "get" component. I think my problem was Component seemed too ad-hoc and using them was often left up to the implementation. Looking at https://github.com/juxt/aero#using-aero-with-components I landed at a usage mode I liked: Very generic components. Implement interfaces on them, including IFn. Take "constructor" functions as arguments. Constructor functions take the entire component as argument and destructure the keys they require out of it. Because of how component works and how the aero config got merged in, the component's this contains all of its dependencies and options. And finally, don't pass components as arguments to functions which aren't constructors. It creates code which looks a bit weird, but very easy and pleasant to work with.

Is it possible to test macro expansion? I have a branch in my macro that throws an error when given incorrect code. I’d like to write a test to demonstrate that. Is that possible?

yes with macroexpand, macroexpand-1 or clojure.walk/macroexpand-all

I’ve only ever used that at the repl. They don’t return strings, they return the actual code?

correct

macroexpand takes a form and returns a form

Cool, I’ll give that a try

Thanks for all the comments. When using component, I felt its way of coding makes the whole structure more nested than I originally thought. For instance to track down where the datasource is made -whether it be that of jdbc or from hikari congig - I needed to go to the definition of record database first, and then to its realiing function, and finally to where it actually gets injected. In the meantime it was quite verbose, with its structure code and business code mingled together.

But I believe the directive of component is a truly needed one. Will look more into it with other suggestions, thanks!

yeah, it can make things feel much more interwoven/complex, but it also reveals how interconnected things already were through globals.

Great thread! @flowthing I have a question specifically for you. > The only downside is that it is not (yet) possible to start/stop a partial system. Usually the way we can test apps using these libraries is not starting real HTTP web server and replace it with a handler functions and so on. If you can’t start/stop a partial system, how do you test them? Do you start whole dependencies and then mock all HTTP calls or such? Please share how you test a Clip app, very interested.

@imre Let me take a stab at it - untested

😄

(defmacro let% [clauses]
  (loop [clauses clauses
         final-bindings []]
    (if (empty? clauses)
      `(let ~final-bindings)
      (if (list? (first clauses))
        (recur (rest clauses) 
               (conj final-bindings 
                     [(first (first clauses))
                      (cons 'fn (rest (first clauses)))]))
        (if (>= (count clauses) 2)
          (recur (rest (rest clauses))
                 (-> final-bindings
                     (conj (first clauses))
                     (conj (second clauses))))
          (throw (RuntimeException. "Not enough clauses in let")))))))
        

give this a shot

it won't let you do mutally recursive stuff like letfn but its somethign

nice one 🙂

I'll give this a go

Coming from ClojureScript I was suprised that defrecord behaves differently in both:

(defrecord Service [send-query])

(defn make-send-query []
  (fn [foo] foo))

(let [s (->Service (make-send-query))]
  (.send-query s "foo"))
; ^ Throws in Clojure (no matching method), but works in Cljs

you should not be using interop syntax to invoke

record fields should be accessed like maps

I see thx :thumbsup:

in both clj and cljs

(defmacro let% [clauses & body]
  (loop [clauses clauses
         final-bindings []]
    (if (empty? clauses)
      `(let ~final-bindings ~@body)
      (let [[fc & rcs] clauses]
        (if (list? fc)
          (recur rcs
                 (-> final-bindings
                     (conj (first fc))
                     (conj (cons `fn fc))))
          (if (>= (count clauses) 2)
            (recur (rest rcs)
                   (-> final-bindings
                       (conj fc)
                       (conj (second clauses))))
            (throw (RuntimeException. "Not enough clauses in let%"))))))))

did a bit of fixing & refactor on it

(let% [bar           1
       (foo [x y z] [x y z])
       {:keys [baz]} {:baz 2}
       [_ _ qux]     [0 0 3 4]]

      (foo bar baz qux))

;; => [1 2 3]

so this would be a proper use for Protocols I guess

((:send-query s) "foo") is I think what you're doing?

you can certainly do the equivalent with protocols too

(protocols are mostly just implemented as maps of types to functions)

yeah that works in both

thanks!

The lower level syntax for accessing these fields is (.-send-query s "foo") which works in both, but it's more preferable to access those via either keywords or protocol fns (in the case of deftype)

That (.send-query s "foo") works in CLJS might be just accidental since the . is quite overloaded in CLJS

mutual recursion could be tough 🙂

does anyone know how I can contact Sam Ritchie ?

@jimka.issy #sicmutils

Through generative testing I found this behavior which I find surprising. In the context of maps and sets, [] and (list) are equivalent:

{[] :vec, (list) :list}  ;; Throw, duplicate keys

(hash-map [] :vec (list) :list)  ;; Super weird => {[] :list}

;; And similarly with sets

In CLJS (at least, planck) {[] :vec, (list) :list} doesn't throw though, it's something only the JVM Clojure seems to do

They have the same hash for some reason and are equal in CLJ.

But Lumo (CLJS) returns {'() :list} , which is inconsistent with Clojure JVM

sequential collections compare equal. equal things have the same hash.

user=> (= [] (list))
true

Ah, right, makes sense.

hash-map docstring explains why different than literal (... "as if by repeated uses of assoc")

in clj -A:cljs -M -m cljs.main -r {[] :vec, (list) :list} returns {[] :list}

$ bb -e '{[] :vec, (list) :list}'
{[] :list}

Right, so this is not specific to empty colls then, it is indeed about hashing. It has never bothered me with = , but after years of Clojuring, I kind of freaked out seeing that behavior with keys :p

it's just a subtle difference in the behavior when the key is already present and a new value is assoced.

I rarely mock anything. The one application where I've had the opportunity to use Clip I just start the entire system and test against that. Might not be a viable option with larger applications, of course (although you might be able to split it into several systems, not sure).

https://gist.github.com/imrekoszo/5e5c007659264729fad26481206c2691

@emccue I saved that for later 🙂 https://gist.github.com/imrekoszo/5e5c007659264729fad26481206c2691

Yeah, I try to stick to [] (vectors as opposed to lists/seqs) nowadays for everything which is data but definitely not code in the “final” output; eliminates this kind of “worry”

Unless I specifically want to keep something as a, say, list, so I can keep adding to the front; That’s rare.

@alexmiller I watched your video on Clojure Spec from about 1 year ago. Just wanted to say thank you for all the hard work you have done leading the community. It has kept me interested in using Clojure. Spec is an amazing tool. And you picked the perfect images for your talk. Of the bird.

thank you

I was just seconding what hipster said

I saw the video too