Clojurians Log v2

I'm not sure you even need to do this, can't you just bind *ns* with binding?

pmap will max out concurrency at max cpu + 2 or the chunk size if given a lazy-seq

So you can kind of control the concurrency level by controlling the chunk size like so:

(defn re-chunk [n xs]
  (lazy-seq
    (when-let [s (seq (take n xs))]
      (let [cb (chunk-buffer n)]
        (doseq [x s] (chunk-append cb x))
          (chunk-cons (chunk cb) (re-chunk n (drop n xs)))))))

(time (dorun (pmap (fn[e] (Thread/sleep 100) (inc e)) (re-chunk 100 (range 1000)))))

"Elapsed time: 1038.57 msecs"

So you can just call re-chunk on the collection before passing it to pmap, and give it the chunk size you want, that will also be the concurrency level.

That said, to the original question, I would use an Executor with some fixed thread count which is scaled either to the API I call, or the service I am using.

in-ns also creates the ns if needded

Did anyone encounter the following error on java11? Caused by: java.lang.Exception: Cyclic load dependency: [ /clojure/spec/alpha ]->/clojure/walk->[ /clojure/spec/alpha ]->/clojure/main->/clojure/core/server

IIRC I've seen people report this when they add Clojure sources to their classpath along with the jar. But maybe that's not it in this case.

Encountered it here: https://github.com/FieryCod/holy-lambda/issues/26

The error occurs when using clojure >= 1.10.0 only on java11

Hard to say exactly, but my guess is it is some kind of aot related issue. This is kind of hard to wade through (for example the source for the lein plugin doesn't seem to be the repo?) but we can discount the existence of an actual cyclic dependency in clojure. My guess is an aot issue, just because it often is, but I can't recall an aot issue presenting as a cyclic dependency. The repo looks like it is library code, but is also marked to aot compile everything when building an uberjar. Both of which are big red flags with library code. Library code should not be uberjared or aot compiled.

Thanks @hiredman! Will check whether removal of aot helps

@hiredman Removed aot compile for library. Allowed aot compilation for hello-lambda code. Still the same errors is there. Maybe Alex would know the answer? EDIT: When trying clojure:1.10.0 (removing require of clojure.walk) and compiling on java8, but still running on java11 the error is different:

Caused by: Syntax error macroexpanding clojure.core/defn at (clojure/spec/alpha.clj:78:1).
	at clojure.lang.Compiler.checkSpecs(Compiler.java:6972)
	at clojure.lang.Compiler.macroexpand1(Compiler.java:6988)
	at clojure.lang.Compiler.macroexpand(Compiler.java:7075)
	at clojure.lang.Compiler.eval(Compiler.java:7161)
	at clojure.lang.Compiler.load(Compiler.java:7636)
	at clojure.lang.RT.loadResourceScript(RT.java:381)
	at clojure.lang.RT.loadResourceScript(RT.java:372)
	at clojure.lang.RT.load(RT.java:459)
	at clojure.lang.RT.load(RT.java:424)
	at clojure.core$load$fn__6839.invoke(core.clj:6126)
	at clojure.core$load.invokeStatic(core.clj:6125)
	at clojure.core$load.doInvoke(core.clj:6109)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.core$load_one.invokeStatic(core.clj:5908)
	at clojure.core$load_one.invoke(core.clj:5903)
	at clojure.core$load_lib$fn__6780.invoke(core.clj:5948)
	at clojure.core$load_lib.invokeStatic(core.clj:5947)
	at clojure.core$load_lib.doInvoke(core.clj:5928)
	at clojure.lang.RestFn.applyTo(RestFn.java:142)
	at clojure.core$apply.invokeStatic(core.clj:667)
	at clojure.core$load_libs.invokeStatic(core.clj:5985)
	at clojure.core$load_libs.doInvoke(core.clj:5969)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.core$apply.invokeStatic(core.clj:667)
	at clojure.core$require.invokeStatic(core.clj:6007)
	at clojure.main$loading__6721__auto____8974.invoke(main.clj:11)
	at clojure.main__init.load(Unknown Source)
	at clojure.main__init.<clinit>(Unknown Source)
	... 58 more
Caused by: java.lang.Exception: #object[clojure.spec.alpha$and_spec_impl$reify__2387 0x69d45cca "clojure.spec.alpha$and_spec_impl$reify__2387@69d45cca"] is not a fn, expected predicate fn

Should I try brew cask install -ing both adoptopenjdk/openjdk/adoptopenjdk11 and adoptopenjdk/openjdk/adoptopenjdk8? Will it work? (while being able to switch versions at will) (don't want to try it right now, lest I break my dev env) ...I'm aware that jenv and sdkman exist. Hadn't had the need so far, although I'm willing to use them

should have rtm :) https://github.com/AdoptOpenJDK/homebrew-openjdk/tree/463eb266789e9e45b4a444923877861a4068c64d#switch-between-different-jdk-versions

I bumped into this with Figwheel. It culprit was cljc dir in the target dir, ie twice. Once in src/ and the second in target/.

I have a bunch of env vars set in my .profile so I can easily specify JAVA_HOME=$OPEN_JDK<n> at the start of any command to run whatever version I need for that command.

(I have 8, 11, 14, 15 installed right now)

lein clean often helps @karol.wojcik

Hi everyone. I've noticed that if you call realised? on a delay that is has already started evaluating, the call blocks until realised? is true rather than immediately returning false e.g. in the REPL

(def my-delay (delay (Thread/sleep 5000)))
(future (@my-delay))
(realized? my-delay)

Hi @seancorfield. Thank you for an advice! Unfortunately it does not help either. Moved with discussion to this thread https://clojurians.slack.com/archives/C1B1BB2Q3/p1609759205028100

Deref and realized https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Delay.java to https://docs.oracle.com/javase/tutorial/essential/concurrency/syncmeth.html methods.

> When one thread is executing a synchronized method for an object, all other threads that invoke synchronized methods for the same object block (suspend execution) until the first thread is done with the object.

I think it's at least partly to avoid Shrodinger stuff: a delay is either pending or ready -- you can't observe it "getting ready"

hmm ok. In my current design I've got a time-consuming task which several threads might be waiting for, so I put it in a delay, but I guess that's not what delay is meant for, since clojure code is supposed to avoid locking. I guess I need to re-think it.

You probably want to check .isDone on the future

Well, you can't use a value until it is available. What do those other threads do if the value is not available (yet)? If they simply spin and wait for it, why not have them just use a deref and block? If they have other work to do, use a future instead of a delay perhaps? realized? doesn't block on a future (it calls .isDone under the hood).

I'm trying to create a kind of cache of memoized results, where if a thread wants something that is already being processed, it just waits for the existing computation to finish. The cache is an atom so I can't swap! a future into it, because creating a future has a side-effect, so I create it as a delay then deref it once the swap is done. Now I'm trying to clear down stale results from the cache so obviously I want to ignore any that are still being computed.

i have a scripts/build.clj file that define a function deploy-build. how do i run that (from a terminal)?

maybe I can just put a promise in the cache and have a free-running future that signals the promise when it's done

Depends what's in it. See if this helps: https://clojure.org/guides/deps_and_cli https://clojure.org/reference/deps_and_cli

https://github.com/anisoptera/screeps-cljs/blob/master/scripts/build.clj

this the file

hm actually i need to run the whole .clj file

deploy-build is just something used during a build/watch

Then https://clojure.org/reference/deps_and_cli#_running_a_main_or_script

clj -M /path/to/myscript.clj

ok so i needed the -M option

now i get

clj -M ./scripts/build.clj
Syntax error (FileNotFoundException) compiling at (E:\workspace-clj\screeps-cljs\.\scripts\build.clj:1:1).
Could not locate cljs/build/api__init.class, cljs/build/api.clj or cljs/build/api.cljc on classpath.

i need to set something in the classpath? or something?

seems that it does not even pass the first require

@belun that file is part of a leiningen project with this project.clj: https://github.com/anisoptera/screeps-cljs/blob/master/project.clj

the only way you can run that is through leiningen

i have lein but still dont know how to run and trigger the whole thing

The README of that project suggests you run this script: https://github.com/anisoptera/screeps-cljs/blob/master/scripts/watch

damn i saw that in the readme but was not seeing the shell script

on windows

I guess on windows you would leave out rlwrap but the rest should more or less work, give or take a few slashes

clojure.lang.ExceptionInfo: failed compiling file:src\screeps\memory.cljs {:file #object[java.io.File 0x2bc0b8c8 "src\\screeps\\memory.cljs"]}

Caused by: clojure.lang.ExceptionInfo: No such namespace: cognitect.transit, could not locate cognitect/transit.cljs, cognitect/transit.cljc, or Closure namespace "cognitect.transit" {:tag :cljs/analysis-err
or}

now i am guessing i am missing some dependency

the project seems v old

yeah, that's weird. he mentions transit in the README, but it doesn't seem part of the project.clj

but you could add it

https://github.com/cognitect/transit-cljs

yea trying to add and run [com.cognitect/transit-clj "1.0.324"]

except i need the cljs one :

[com.cognitect/transit-cljs "0.8.264"]

yes

um

some warning on that transit script

but

export SCREEPS_USERNAME and SCREEPS_PASSWORD (optionally SCREEPS_BRANCH) to autodeploy code.

this is the error message from the build.clj

is great so far

I have a macro question that has me stumped. What's a good way to check if a passed in argument is a function? I thought fn? would work but it doesn't work in a macro as I expected since the macro receives a symbol or a cons with the fn definition. Simplified example of what I'm trying to accomplish:

(defn make-comparison-expr [field compare]
  (if (fn? compare)
    `(~compare ~field)
    `(= ~compare ~field)))

(if (or (and (symbol? compare) (fn? @#'compare))
            (and (seq? compare) (= 'fn* (first compare))))
      `(~compare ~field)
      `(= ~compare ~field))))

This kind of thing is usually a good indicator that you shouldn't be using a macro

When binding names to values (via let, def, function invocation, etc) the name is known statically (at compile time) while the value is dynamic (known at runtime). Macros are functions that run at compile time, so there is a fundamental mismatch if you are writing a macro that tries to examine the values names are bound to

There is kind of an exception to this for defs, because previous defs run, so their value is part of the compile time environment of the next top level form

But there are other ways to bind names, and your macro won't work correctly for them

Thanks for the details explanation. That's a good point about not knowing what's being referenced at macro eval time. I tried lifting this check to runtime, and it works but generates some pretty odd code. e.g.

(if (fn? 1)
  (1 1)
  (= 1 1))

You may also want to use ifn? instead too

fn? is only true for a surprisingly small subset of things that can be invoked as a function

For example if I recall, fn? Is false for multimethods

Ah did not know about multi methods. I originally chose fn? to avoid keywords returning true in this case. But maybe it makes more sense to go the other way around and use ifn? but give keywords special treatment.

or just assume you will always be passed a callable thing

and pass #(= % whatever) if you want the comparison case

(or (partial = whatever) )

macro sugar can often not be worth it. they feel pretty silly to me too if they just serve to prevent you from having to write (fn [] ) around a body

not saying that's your case here but that's a common occurrence in the wild

That would definitely simplify things a lot here. Probably too much of my common lisp thinking bleeding through when trying to write this 😄

Thanks for all the discussion and suggestions. I settled on having two versions of my macro. A plain version that handles the = case (which is 80% of the time), and a version with -by appended that only takes callable things. It really simplified the design and overall code.

are you sure you needed a macro? i didn't follow along too much but it seemed it was just performing a check or returning a function to do so?

I'm making a little query dsl. The main purpose being to keep the intent of the query simple and easy to follow. Mainly using the macro to make things easier for the readers of the code.

@hackeryarn You may find https://github.com/clojure-expectations/clojure-test interesting code to read over: it provides a testing DSL where you can say (expect 1 1) and it tests they are equal or (expect odd? 1) and it tests the value using the predicate.

(it macroexpands to regular clojure.test assertions)

Thanks @seancorfield that's exactly the type of syntax I was trying to accomplish!