Clojurians Log v2

Is there a more idiomatic way to reduce over multiple collections?

(reduce (fn [acc [x y z]]
          (assoc acc x (str y z)))
  {}
  (map vector
    [:a :b :c]
    (range)
    [:d :e :f]))

(reduce (fn [acc f]
          (f acc))
        {}
        (sequence (map (fn [x y z]
                         #(assoc % x (str y z))))
                  [:a :b :c]
                  (range)
                  [:d :e :f]))

@delaguardo wow 🙂 Have never thought of having the “item” in reduce be a function, cool.

Luxury of fp)

That's a really neat and mind-bending solution 🙂 Although in a sense it's just swapping the intermediate collections for intermediate anonymous functions?

(Not that I've tried profiling to compare the two, performance isn't really a concern)

I guess (into {} (sequence (map #(vector %1 (str %2 %3))) [:a :b :c] (range) [:d :e :f])) also works, although I'm not sure about the performance characteristics. No need to destructure, at least. :man-shrugging::skin-tone-2:

yeah that works for this toy example, but in general there would be more logic going on in the reducing function than a plain assoc

Right, gotcha.

Hello everyone. I'm having a really confusing past two days. Clojure code that has been stable for months/years and have been running and building correctly (`lein`) is now giving me weird errors when I try to run and/or build. The errors started on dev laptops (Mac OSX) and on our CI servers (linux) at the same time. An example; one of my namespaces has (:require ... [clojure.core.async :as csp] ...) and then when I run I get java.lang.RuntimeException: No such var: csp/to-chan!. This code has been in production for nearly two years at this point, except now it's not loading/running properly any longer. Maybe it's also good to mention that we're using integrant and using integrant.core/load-namespaces to require the namespaces dynamically based on an integrant config file. Any help and/or pointers would be appreciated.

Something must've changed and you now have an older version of org.clojure/core.async on your classpath.

to-chan! has been added in 1.2.598.

Oh wait, it's for CLJS, nevermind.

1.2.593 for CLJ.

Our lein deps :tree is clean, with no dep conflicts. It shows I have only [org.clojure/core.async "1.3.610"]

Just to be absolutely sure - can you see what version is used in the output of (System/getProperty "java.class.path")?

Not in a separate REPL but right in the same process where that error happens.

buried in there: .m2/repository/org/clojure/core.async/1.3.610/core.async-1.3.610.jar

One other potential reason is that something is either an uberjar with core.async in it or it ships its own core.async along with its sources.

I'm not sure what the best way of checking that would be, but you can at least try running:

(require '[clojure.core.async])
(:file (meta #'clojure.core.async/tap))

so maybe I should expand a bit here: This to-chan! is a symptom of a bigger issue we are having. Previously (earlier today) I had no such var on symbols in our own namespaces. I then modified our own namespaces with (println ...)'s (to introduce side-effects). Those println's changed the behaviour of the bugs to such an extent that we're now getting this no such var on a namespace that I cannot modify...

Oh. No clue what's going on then. Maybe something is modifying existing namespaces in runtime?

I got "clojure/core/async.clj" on that suggestion of yours

Have you added new dependencies recently or bumped any versions?

We keep our dependencies as up-to-date as possible (running lein ancient as part of CI) However, our dependencies are still the same as they were early last week when we did have builds working. Something in the environments may have changed (eg brew upgrade), but I don't know how to determine that.

Hmm… Check what version of java you’re running against; make sure “brew upgrade” etc hasn’t installed some esoteric JVM that is not supported (I’ve tried a few intentionally recently and they didn’t work); are you able to build at least once anywhere locally or on a new machine?

As a last resort, try running your project in a Docker container with one of the pre-built Clojure Docker setups on Docker Hub; at least it will give you pretty much a 100% guarantee of a stable environment.

I will try this docker suggestion: lein version gives me this: Leiningen 2.9.5 on Java 15.0.2 OpenJDK 64-Bit Server VM

Does a call to (require ...) perform any work in a background thread? IE, can it return before it completes loading all the namespaces?

Java 15 OpenJDK should work…

no threaded work but it is not thread-safe which can lead to code trying to use vars that have not been loaded yet

@thheller Is that assuming that you’re calling (require …) from the middle of a namespace somewhere? Or if you’re using top-level (:require …) it can still happen?

@pieterbreed perhaps reading through this can be of some help (if your problem is related) https://ask.clojure.org/index.php/9893/require-is-not-thread-safe

I don't really understand what this means... The thread that runs my main does a whole bunch of dynamic requires, after that, it starts the system which then starts a number of threads... Am I to believe that require may return without those namespaces being ready?

That’s probably the root cause of your issue, if I have to bet

https://clojuredocs.org/clojure.core/require#example-5fda2a16e4b0b1e3652d7419

;; Note that require is known _not_ to be thread safe in Clojure 1.10.x and
;; earlier, so avoid calling it concurrently from multiple threads.

;; See <https://ask.clojure.org/index.php/9893/require-is-not-thread-safe>
;; for some thoughts on approaches to using require from multiple threads
;; safely, which today boils down to "use locks to make all calls to require
;; guaranteed to execute one at a time".

those require's are only invoked serially from a single thread, that waits for them to complete... then it starts doing business logic... is this not safe?

Perhaps not … I am not 100% sure

But I’d try replacing those dynamic requires with:

(#'clojure.core/serialized-require 'tick.alpha.api)

All it does is it just wraps require in (locking …)

(locking clojure.lang.RT/REQUIRE_LOCK
  (apply require args))

> We keep our dependencies as up-to-date as possible (running lein ancient as part of CI) Just wanted to comment on this specifically - I don't think it's a good practice to do that automatically. Detect and notify automatically - fine. But not update. You can never know what the actual changelog will have between x.y.z and x.y.(inc z).

I am guessing that even though the require is invoked serially from one thread, later when your business logic starts, if that’s on different threads (and I bet it probably is, thread pools and all of that), that’s where you might run into issues;

I agree with @p-himik

@pieterbreed or perhaps you can use the public requiring-resolve Let me know how it goes, curious if this fixes it.

Thank you @raspasov, I'll try this and see how it goes. Thanks for the engagement :thumbsup:

Also, if you have Java code calling Clojure, that can be another possible cause… Attempting to do multithreaded programming is one of the most humbling things I’ve done. It will often prove me wrong even after multiple iterations and when I’m internally convinced that I finally “got it” 🙂 And that’s in Clojure, with all the help from immutability. Can only imagine doing it in C++/Java or something like that.

it is useful to understand how clojure loads code. it processes one form at a time sequentially. so it usually processed a ns form first, then a defn then the next etc

so if you are in a threaded scenario with two require running you can end up there on ns is trying to use a defn from another ns but failing since the original thread hasn't processed that yet

but if you run all require sequentially from that one thread that can't happen

What’s truly weird is this: we’ve been getting “no such var” type errors in namespaces which are ours. We’ve gone and added (println ::loading) to those same namespaces, right below the ns form, and then magically the errors move to a different “place”. Finally, we got it to this point where we are getting “no such var” on core-async. Why would adding println’s affect this?

These points are valid; however for us, it’s been easier to run into compatibility problems sooner rather than later. Also, staying up to date incrementally is easier than trying to catch up in bulk from versions that are old, like when a bugfix is only in a version that’s recent.

well sounds like you are racing and adding a println is going to make the one thread a tiny bit slower since it has to compile and execute the print

thus given the other thread a little bit more time to make further progress. definitely use a lock if you are using require dynamically

not sure why, but I’ve started getting email from the Clojure Jira

I think I submitted a ticket or two maybe ~7 years ago

sorry, should be fixed (wrong change to a notification scheme)

Does anyone use clojure.core.logic.unifier? Interested to know how you leverage it

I'd try hard to use either tools.namespace or Clojure's vanilla require system. Alternatives can be less polished/understood I'd also try to take threading out of the equation - all requires should happen in the main thread

It's not the first time you are experiencing issues btw :) https://github.com/adambard/failjure/issues/24 Personally I'd use that as evidence that a simple/standard system would be the best best

Hi, yes, you are right. I do think these issues are all related. I am only using require and only from the main thread before any business logic starts, so I’m uncertain how to improve things.

As you may have noticed integrant.core/load-namespaces performs some requires, so I'd try to investigate in which conditions you are invoking load-namespaces (no threads, retries, try/catch, etc...?)

Did you see https://github.com/weavejester/integrant/issues/52 as well?

Hi everyone, I'm struggling to create an uberjar using depstar while excluding a jar that is provided by the platform. Basically, something like <scope>provided</scope> in maven for deps.edn. Is there a way?

@sammerat What have you tried so far?

@seancorfield Thanks for the response Sean. I tried {:mvn/version "2.2.0" :scope "provided"} but it looks like it is still getting bundled

tools.deps.alpha doesn’t pay any attention to :scope as far as I know.

What library are you trying to exclude and why? An uberjar is intended to be a standalone runnable bundle.

@seancorfield I'm building an apache storm topology which depends on strom-client.jar which is also provided by the storm runtime

So you need the JAR on the classpath for compilation, but not included in the final (uber) JAR?

exactly