Clojurians Log v2

We have a few "rogue" JAR files at work that aren't available in a repo (and we've had to build ourself), and we just stick them in a lib folder accessible to our code and point :local/root at them.

(I just checked and we're down to just one such JAR these days but Cognitect's REBL was also handled this way for quite a while: just download it somewhere and point a :local/root dependency at it)

@noisesmith Ah yes, CI/CD must be considered

@seancorfield This project is based on lein for the forseeable future.

right, @seancorfield’s solution is to check the jar into the repo it seems

(yeah, or build it on the fly from source and keep that source under git)

For dev-only stuff, you don't need to check it in, if it's just in your local deps.edn file tho'...

Yes, it will certainly be Java under source control, but I want non-Clojure devs to be able to build it easily, and Clojure devs to consume it easily.

I think for something like that, it makes sense to treat it as its own artifact in a maven repo

For the particular "rogue" JAR I mentioned above, it's never going to change so it was a one-off add+commit. Ugly but...

I'd treat a shared clojure library the same way

Aye, I agree that I would expect a Java lib that needs to be built from source to have the artifacts published to a repo for the company.

(since it's going to keep changing)

We ran Apache Archiva for a similar reason for a couple of years (but it's pretty flaky and didn't seem well-maintained). If we needed to do that again, we'd probably use S3 buckets since that's a fairly widely support repository provider now for Clojure.

Maybe an S3 bucket would work

this one is a little clunky, but it works https://github.com/s3-wagon-private/s3-wagon-private

there are alternatives I haven't tried

🙋 Roll call! Who else is here?

Thanks!

@ps hello

hey

(reminder that there are thousands of people here so keep things focused on Clojure)

why is (or (not nil) (nil x)) an error? I have an optional parameter in my function, a predicate that may either be a lambda or nil, so I call it like this: (or (not f) (f x)) . But it gives me an error when f is nil. Doesn't or short-circuit the evaluation?

Because nil is not a function, you probably want nil?

or does short-circuit the evaluation

user> (def f nil)
;; => #'user/f
user> (or (not f) (f 1))
;; => true

Don’t you want something like (when f (f x) ? Or you can (when (fn? f) (f x))

but the forms like this (or (not nil) (nil 1)) will fail because first of all this form needs to be compiled and Compile exception will be thrown because of the form (nil 1) In case of macro it is easy to catch this case

Just spotted LazilyPersistentVector in the clojure implementation, and I’m guessing this is part of the logic that keeps collections smaller than 32 elements as essentially linear arrays, but promotes them when they’re larger than that?

I have a question to reducers. I have a line that uses r/map . So far I had r/foldcat in my code, but while that ran everything in parallel as expected, I didn't get the result I wanted out of it. I see in the reducers reference it says "To produce an output collection, use https://clojure.github.io/clojure/clojure.core-api.html#clojure.core/into". When I wrap the call to r/map in an into will it still be run in parallel?

Nope, just r/fold or r/foldcat (which is an r/fold in disguise) enable parallelism for foldable collections (beyond a certain size)

Wait, if you mean on top of r/foldcat then yes

(require '[clojure.core.reducers :as r])
(def input (r/map inc (into [] (range 1000))))
(into [] (r/foldcat input))
> [1 2 3...

out of curiosity, how do Clojure’s short-lived functions behave with the JVM JIT? assuming I have a reduce in a hot function that uses a short-lived reduction function, I can only assume the JVM “forgets” the previous allocations (and invocations) of these short lived functions, right ?

Yeah that’s possible because (nil 1) makes no sense.

No, not on top of r/foldcat, because since the input to it is a vector of maps from r/map, it doesn't process them correctly (and seems to just merge the maps)

What I did now is write a reducef and wrap it in a (r/fold reducef (r/map ...)).

(defn- reducef
  ([] [])
  ([a b]
   (if (and (vector? a) (vector? b))
     (into a b)
     (conj a b))))

Worth remembering that r/foldcat is not an equivalent mapcat or concat of sort. The “cat” part in r/foldcat is mainly an implementation detail. But it seems to work as advertised in your case?

(require '[clojure.core.reducers :as r])
(def large-map (into {} (map vector (range 1000) (range 1000))))
(def vector-maps (into [] (repeat 1000 large-map)))
(def input (r/map #(assoc % :new :key) vector-maps))
(:new (first (r/foldcat input)))
;; :key
(count (into [] (r/foldcat input)))
;; 1000

What is "short lived"?

Unless you're using eval, all functions are compiled to a class and have their locals available.

Anonymous functions are compiled once, not each time the function around them is called.

And in general local functions defined within another function are compiled once, whether they are anonymous or have a name, so the "anonymous" part of my previous statement is irrelevant.

Let's say I have a macro that I want to conditionally define a few functions depending on its input — how can I break out the 'function-building' part into functions? For instance, let's say I have something like:

(defmacro foo [x]
  `(if (even? ~x)
     (defn ~(symbol (str "foo-" x) [] (inc x)))
     (defn ~(symbol (str "bar-" x)) [] (dec x))))

(defn make-foo [x] `(defn ~(symbol (str "foo-" x) [] (inc x))))
(defn make-bar [x] `(defn ~(symbol (str "bar-" x) [] (inc x))))
(defmacro foo [x] `(if (even? ~x) (make-foo ~x) (make-bar ~x)))

Are there any editors (or IDE plugins) that somehow highlight the S-expressions which return values from a function? Or maybe there's an easy heuristic that everybody uses? As a newbie-lisper, I find it really difficult to mentally parse a large nested structure of let/loop/if/do/implicit-do to find all the places where expressions' values are not discarded and they form function's return values.

actually maybe this is a bad example

What I'm actually doing is:

(defmacro foo [x]
  `(let [x# (some-fn x)]
     (if (even? x#)
       (make-foo x#)
       (make-bar x#))))

and what ends up happening here is that my defns end up getting returned to the code calling the macro without being evaluated

def  prefix explains that the function define a `var` in the namespace. And, `def*`  calls must be top level calls. Any other usage of them is not right.

I'd do ~(make-foo x#) to solve this but that doesn't work because x# isn't available unquoted

I recommend you to consider about your data-flow again.

The JVM JIT should see local Clojure functions as just another class with methods to execute. No different from top level functions.

@scknkkrer if that's the case, you should probably stop using defn 😉

Sorry to bother you, it’s a common knowledge, I just wanted to inform you.

I do not know of any editor/IDE help related to this, but I haven't looked for such a thing, either.

No worries

I would suggest that another technique that would help with this, and perhaps have other benefits as well, is to make some effort at breaking sub-expressions within a large block of code into separate functions.

I'm aware that non-top-level defs are typically discouraged, but there are places where it's appropriate to make use of them (like in the definition of defn)

just wanted to inform you back 🙂

In general, smaller expressions / functions make it easier to understand a lot of things about them, including what you asked about, and can also make it easier to develop and test the smaller functions, sometimes separately.

The most deeply nested code at any point is usually the final step I guess. But this isn't something I generally find myself searching for in my programs either.

@andy.fingerhut I agree. I was just making some quick-and-dirty java-interop megafunction, and wondered if there's already such a "tail detection" implemented somewhere :)

for example, something like this:

(defn my-transform [prefix input]
   (reduce (fn [xs x]
              (conj xs (str prefix x)))
           {} input))

(my-transform "prefix" ["a" "b" "c"])

especially since the behavior of this function depends upon an external input

but as i understand now, this is actually compiled once?

Compiled once, yep.

It is compiled once, and if you look at the JVM byte code, and/or the Java source code decompiled from that, you will see that the class created for the (fn [xs x] ...) function has a constructor that takes prefix as a parameter. So the local function is an instance of a class that is constructed once on each call to my-transform, but the class remains from one call of my-transform to the next. The only thing that is allocated fresh on each call to my-transform is an instance of that class.

Of the IDEs I know of, I don't have deep knowledge of them, but #cider has probably been around the longest, and #cursive has a full time commercial developer behind it, so those are the two IDEs that I'd guess have the most features, and therefore the most likely to have what you are asking for, if any of them do.

and if I recall correctly, the only thing in those instances are things like a local variable for each value in the inner function's environment, like prefix in your example, and the constructor does not do anything except assign values to those fields and return, so they should be as light weight as any constructor can be.

this is a great answer, thanks a lot! this was exactly what i was wondering, so the good news is that all this integrates really well into the JIT of the JVM 🙂

I’ve written some pretty gnarly clojure, and I’ve never had the problem of not being able to easily see the return form.

Curious if you have some sort of example. do blocks should be rare enough that they stand out.

Oh, you know what helps: indentation markers.

together with properly formatted code

(This is Cursive.) You can see the indentation markers on the left. Also, rainbow parens help to easily spot, e.g. where the finally block should go.

I am writing a generative test case in kaocha, using fdef and a local function, but it seems like I get a null pointer exception when calling spec/exercise-fn. The function exercise-fn is calling is declared with letfn. Could that be the reason for my problems?

`

@potetm yeah, using these already. They help, but really don't give me feeling that I have everything covered. For example, upon careful examination, I see three tail positions in my code: 1. (recur) 2. :error 3. (swap!) But the function spans up a couple of screens more 🙂

(test/deftest ^:integration-generative-concepts-paging generative-test-concepts-paging (test/testing "test main/concepts paging generatively" (let [_ (db/create-version-0) concepts (assert-concepts (rand-int 4)) ;; time consuming operation _ (versions/create-new-version 1)] (letfn [(test-concept-paging [query pagination] ;; Return true if the test passed. (let [params (generate-page-params (count concepts) query pagination)] true))] (let [res (spec/exercise-fn `test-concept-paging 10)] (doall (map #(test/is (= true true)) res)))))))

user=> (use 'kaocha.repl) user=> (run 'jobtech-taxonomy-api.test.generative-test/generative-test-concepts-paging)

Uncaught exception, not in assertion. Exception: java.lang.NullPointerException: null at clojure.core$apply.invokeStatic (core.clj:665) ... clojure.spec.alpha$exercise_fn$iter__2569__2573$fn__2574.invoke (alpha.clj:1881) ... jobtech_taxonomy_api.test.generative_test$fn__68371.invokeStatic (generative_test.clj:145) jobtech_taxonomy_api.test.generative_test/fn (generative_test.clj:135) jobtech_taxonomy_api.test.test_utils$fixture.invokeStatic (test_utils.clj:48)

generative_test.clj:145 is the line with spec/exercise-fn

I have written generative tests before, but never using a local function. The reason I have the local function is because I want to share some data across the test. This data takes a long time to generate, and I do not want to regenerate it for each time the tested function is called.

Here's a link to the code if anyone feels inclined to have a look. https://gitlab.com/team-batfish/backend/jobtech-taxonomy-api/-/merge_requests/143

Aaand I was wrong, these are not tail positions, because they turned out to be inside a doseq 😞

Since I have seen a similar question only a couple of weeks ago, and did most of this investigation at that time, I decided to put the results into a short article that you might find interesting: https://github.com/jafingerhut/jafingerhut.github.com/blob/master/notes/clojure-inner-functions.md

you're calling functions from a macro that just return lists of symbols. for make-foo and make-bar to be macro expanded into the defn for evaluation, they need to be macros not functions ... does that make sense?

but, if you're always passing in literal values like that, you could take the syntax quote out and just return the code from foo?

So, some things

This sort of side-effect heavy algorithm is never going to feel good in clojure.

There is an upside to that: It strongly encourages you to isolate side effects (which most developers are poor at).

The downside is: Any algorithm that really needs to be side-effect heavy is just going to feel bad.

IME those cases are relatively rare. It’s usually the case that I need to separate logic from side effects.

But those cases do exist. And they’re just not fun.

not quite sure if this is the right channel to ask this question, but I was wondering if anyone has ever done side-by-side pair programming with two REPL clients connected to the same REPL server?

yes ... I found it very confusing when someone else changed a function underneath me ... it was better to have one typing at a time

if you do things like load-file or cider-load-buffer then you can clobber changes that have been loaded in from someone else's machine without thinking about it ... it's a very easy thing to do

@afoltzm I've heard some people do this via LiveShare in VSCode

Ask in #calva for details.

@l0st3d yeah I figured it could get really weird really quick if you're not careful, but it also seemed like an interesting technique in the distributed-by-default setting we're all in now, which is why I wanted to ask about people's experiences

I think if you're doing pair programming in the sense that you're communicating about what changes each other are making and you're working toward a common goal, I could see it working, but if multiple people are redefining things in a REPL without communicating it, then it feels like essentially the whole "shared mutable state" problem, where the threads are people instead of program threads

For exercise-fn to run, your function needs at least :arg specs -- and you aren't writing a spec for test-concept-paging

user=> (letfn [(foo [n])] (s/exercise-fn `foo))
Execution error at user/eval24110 (REPL:1).
No :args spec found, can't generate

How would one set any of the clojure.core/*config-flag* type vars from Java?

Where is your fdef?

I'm writing a plugin in a Java environment that hands me a pretty restricted classloader and I think I need to set *use-context-classloader* to false before I invoke any clojure code.

var: #'clojure.core/assert-args is not public. - hmm this seemed like such a useful macro to use in my own code

I just copied it over to my own 'core' lib

The fdef is

(spec/fdef test-concept-paging :args (spec/cat :query ::concept-paging-query :pagination ::paging) :ret (spec/and boolean? #(= true %)))

But to evaluate everything you also need a few more functions from, https://gitlab.com/team-batfish/backend/jobtech-taxonomy-api/-/merge_requests/143/diffs

That would be a global (top-level) function tho', right?

Yes

But the problem might be that the implementation is local?

And in letfn, that's a different function -- it's a local name that shadows any global.

Ok, now I see. The top-level fdef doesn't find the local one.

test-concept-paging is only defined locally.

So then I need to define the fdef at the same local level as test-concept-paging?

I'm pretty sure fdef only works on top-level functions.

Thanks for the help. I'll see tomorrow, how to do that. But is this the right way to test a function that has some parameters constant and the same, and others that you want to vary?

Unfortunately it seems that way. I did a quick test

clojure.spec is not a type system 🙂

(local functions are generally just considered an implementation detail)

In the code you shared, I would expect generate-page-params to have the fdef and be the function under test.

Yes, I’ve always found shared keyboard control creates mutex issues. One person “driving” (typing) at a time, taking turns, seems to work better for me.

Ok, thanks! I'll try to rearrange the functionality tomorrow.

You might see an example in bukkure

Does anybody know how to get javadoc to work in cider? I've used it for a decade, and never seen it do anything useful.

@russell.mull Maybe ask in #cider?

oh right on, didn't know that was a thing. Thanks.

clojure.java.javadoc is aliased to javadoc in the repl by default, and uses the web browser

https://www.reddit.com/r/Clojure/comments/k105ud/inner_functions_in_clojurejava/ shared on Reddit if you don’t mind :)

it’s great

(it can even try to find javadoc using google’s I’m feeling lucky URL, which I found amusing when I stumbled over that)

(although that is broken right now as google changed the url)

thanks! I don't see any instances of use-context-classloader in there from a GitHub search.