Clojurians Log v2

I am trying to extend transit for arrays which (surprisingly) are not supported out of the box. E.g.:

(def array-write-handler (transit/write-handler "pods/array" (fn [o] (vec o))))
(def array-read-handler (transit/read-handler (fn [o] (into-array o))))

(defn transit-json-read [^String s]
  (with-open [bais (java.io.ByteArrayInputStream. (.getBytes s "UTF-8"))]
    (let [r (transit/reader bais :json
                            {:handlers {"pods/array" array-read-handler}})]
      (transit/read r))))

(def array-type (type (into-array [])))
(def long-array-type (type (into-array Long [])))

(defn transit-json-write [^String s]
  (with-open [baos (java.io.ByteArrayOutputStream. 4096)]
    (let [w (transit/writer baos
                            :json
                            {:handlers {long-array-type array-write-handler
                                        array-type array-write-handler}})]
      (transit/write w s)
      (str baos))))

I have trouble believing nobody has ever run into this before

I think I'll just use a postwalk instead...

anybody figured out workarounds for git deps and Apple M1?

@dnolen what's the problem needing working around?

git deps fail

a repro with any error messages would be helpful

I realize the error messages coming from that subsystem are not very useful, but e.g. is it just ssh: protocol, or https: too? is an ssh-agent active?

Error building classpath. /private/var/folders/q4/3d296lg10qd4nt3sxxl98fmw0000gn/T/jna--541193536/jna3022155084463275496.tmp: dlopen(/private/var/folders/q4/3d296lg10qd4nt3sxxl98fmw0000gn/T/jna--541193536/jna3022155084463275496.tmp, 1): no suitable image found.  Did find:
	/private/var/folders/q4/3d296lg10qd4nt3sxxl98fmw0000gn/T/jna--541193536/jna3022155084463275496.tmp: no matching architecture in universal wrapper
	/private/var/folders/q4/3d296lg10qd4nt3sxxl98fmw0000gn/T/jna--541193536/jna3022155084463275496.tmp: no matching architecture in universal wrapper
java.lang.UnsatisfiedLinkError: /private/var/folders/q4/3d296lg10qd4nt3sxxl98fmw0000gn/T/jna--541193536/jna3022155084463275496.tmp: dlopen(/private/var/folders/q4/3d296lg10qd4nt3sxxl98fmw0000gn/T/jna--541193536/jna3022155084463275496.tmp, 1): no suitable image found.  Did find:
	/private/var/folders/q4/3d296lg10qd4nt3sxxl98fmw0000gn/T/jna--541193536/jna3022155084463275496.tmp: no matching architecture in universal wrapper
	/private/var/folders/q4/3d296lg10qd4nt3sxxl98fmw0000gn/T/jna--541193536/jna3022155084463275496.tmp: no matching architecture in universal wrapper
	at java.base/jdk.internal.loader.NativeLibraries.load(Native Method)
	at java.base/jdk.internal.loader.NativeLibraries$NativeLibraryImpl.open(NativeLibraries.java:383)
	at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:227)
	at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:169)
	at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2383)
	at java.base/java.lang.Runtime.load0(Runtime.java:746)
	at java.base/java.lang.System.load(System.java:1857)
	at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:761)
	at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:736)
	at com.sun.jna.Native.<clinit>(Native.java:131)
	at com.jcraft.jsch.agentproxy.usocket.JNAUSocketFactory$CLibrary.<clinit>(JNAUSocketFactory.java:47)
	at com.jcraft.jsch.agentproxy.usocket.JNAUSocketFactory.open(JNAUSocketFactory.java:114)
	at com.jcraft.jsch.agentproxy.connector.SSHAgentConnector.open(SSHAgentConnector.java:93)
	at com.jcraft.jsch.agentproxy.connector.SSHAgentConnector.<init>(SSHAgentConnector.java:54)
	at com.jcraft.jsch.agentproxy.ConnectorFactory.createConnector(ConnectorFactory.java:104)
	at clojure.tools.gitlibs.impl$fn__1257.invokeStatic(impl.clj:31)
	at clojure.tools.gitlibs.impl$fn__1257.invoke(impl.clj:29)
	at clojure.lang.Delay.deref(Delay.java:42)
	at clojure.core$deref.invokeStatic(core.clj:2324)
	at clojure.core$deref.invoke(core.clj:2310)
	at clojure.tools.gitlibs.impl$call_with_auth.invokeStatic(impl.clj:51)
	at clojure.tools.gitlibs.impl$call_with_auth.invoke(impl.clj:43)
	at clojure.tools.gitlibs.impl$git_clone_bare.invokeStatic(impl.clj:73)
	at clojure.tools.gitlibs.impl$git_clone_bare.invoke(impl.clj:70)
	at clojure.tools.gitlibs.impl$ensure_git_dir.invokeStatic(impl.clj:112)
	at clojure.tools.gitlibs.impl$ensure_git_dir.invoke(impl.clj:102)
	at clojure.tools.gitlibs$resolve.invokeStatic(gitlibs.clj:33)
	at clojure.tools.gitlibs$resolve.invoke(gitlibs.clj:29)
	at clojure.tools.gitlibs$procure.invokeStatic(gitlibs.clj:47)
	at clojure.tools.gitlibs$procure.invoke(gitlibs.clj:41)
	at clojure.tools.deps.alpha.extensions.git$eval1323$fn__1325.invoke(git.clj:42)
	at clojure.lang.MultiFn.invoke(MultiFn.java:239)
	at clojure.tools.deps.alpha$expand_deps.invokeStatic(alpha.clj:422)
	at clojure.tools.deps.alpha$expand_deps.invoke(alpha.clj:390)
	at clojure.tools.deps.alpha$resolve_deps.invokeStatic(alpha.clj:495)
	at clojure.tools.deps.alpha$resolve_deps.invoke(alpha.clj:475)
	at clojure.tools.deps.alpha$calc_basis.invokeStatic(alpha.clj:655)
	at clojure.tools.deps.alpha$calc_basis.invoke(alpha.clj:629)
	at clojure.tools.deps.alpha.script.make_classpath2$run_core.invokeStatic(make_classpath2.clj:91)
	at clojure.tools.deps.alpha.script.make_classpath2$run_core.invoke(make_classpath2.clj:57)
	at clojure.tools.deps.alpha.script.make_classpath2$run.invokeStatic(make_classpath2.clj:119)
	at clojure.tools.deps.alpha.script.make_classpath2$run.invoke(make_classpath2.clj:113)
	at clojure.tools.deps.alpha.script.make_classpath2$_main.invokeStatic(make_classpath2.clj:169)
	at clojure.tools.deps.alpha.script.make_classpath2$_main.doInvoke(make_classpath2.clj:140)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.lang.Var.applyTo(Var.java:705)
	at clojure.core$apply.invokeStatic(core.clj:667)
	at clojure.main$main_opt.invokeStatic(main.clj:514)
	at clojure.main$main_opt.invoke(main.clj:510)
	at clojure.main$main.invokeStatic(main.clj:664)
	at clojure.main$main.doInvoke(main.clj:616)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.lang.Var.applyTo(Var.java:705)
	at clojure.main.main(main.java:40)

Yeah, that’s just not going to work

@dnolen Can you run this under Rosetta?

Hey all, can anyone help me make sense of this one:

(not (= ##NaN ##NaN))
=> true
(not= ##NaN ##NaN)
=> false

It’s basically part of the definition of NaN

http://github.com/jafingerhut/batman

“NaN is not equal to anything, including itself.”

Right, but that isn’t the issue.

Oh wait, not=

not= and (not (=))

Yeah, I’m catching up.

Here’s some more fun. I am working in a mixed clj/cljs env. Just discovered this:

(= (not (= ##NaN ##NaN)) (not= ##NaN ##NaN))
=> false

(= (not (= ##NaN ##NaN)) (not= ##NaN ##NaN))
=> true

Ooooh, cljs

but the cljs version is the one that behaves as expected

The answer for Clojure (not ClojureScript) is probably somewhere in this annotated output: https://github.com/jafingerhut/batman/blob/master/doc/macos-10.13.6-jdk-adoptopenjdk-11.0.3-clojure-1.10.1.txt

Yeah, I was looking at https://clojuredocs.org/clojure.core/not=, but that just the docs, not the src.

The source doesn’t really help:

(defn not=
  "Same as (not (= obj1 obj2))"
  {:tag Boolean
   :added "1.0"
   :static true}
  ([x] false)
  ([x y] (not (= x y)))
  ([x y & more]
   (not (apply = x y more))))

For 2 args, not= just calls (not (= x y))

Here be dragons. Avoid NaN whenever you can.

= is inlined w/o boxing

not= isn't

user=> (not (= ##NaN ##NaN))
true
user=> (not (= (identity ##NaN) (identity ##NaN)))
false
user=> (not= ##NaN ##NaN)
false

IIRC, (= ##NaN ##NaN) can sometimes return true because of the optimization to check for identical? first, and sometimes two ##NaN objects are identical.

If two ##NaN objects are not identical, then (= ##NaN ##NaN) returns false

indeed, when comparing ##NaN for equality as an object, it may return true

but as primitive doubles, it will not

Ok, but for 2 args, not= just calls (not (= x y)), so why doesn’t it behave the same?

because = has intrinsics in the compiler

There are some expressions here where the Clojure compiler inlines, vs. not, object vs. primitive double, etc. The transcript I linked above might cover all the possibilities, but it might not. Feel free to suggest additions if it is missing anything.

which avoid boxing the arguments if they are literal primitive values

but not= isn't special cased, so the primitive values always get boxed

if not= was a macro instead of a function (or if it had an inline annotation), it'd behave differently

Ah, ok, the boxing is invoked in the process of evaling the arguments, and by the time they get to =, they’re already boxed.

That’s interesting.

yes

There is almost no rule you can give about ##NaN behavior which is true in all cases, other than it will drive you nuts if you try to find general rules.

“The only way to win is not to play.” — WOPR.

##NaN is the only value where this can happen

and should just not be used in equality comparisons

Play with doubles/floats carefully, if you need them (read as much of https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html as you need for your purposes), but just stay away from NaN whenever you can.

A few more examples involving set elements and map keys: https://clojure.org/guides/equality#_floating_point_not_a_number

In my current use case, I definitely do not want ##NaN (or even doubles), but this does seem like a bug.

which part? 🙂

(not= not producing the same behavior as (not (=.

I haven't tried to examine what it would take to "fix" that, but I suspect it would mean slowing down operations on values that were not ##NaN

But it isn’t anything I’ll lose any sleep over. As we all say, the best solution is to prevent ##NaN in the first place.

what it would take to fix this would be slowing down every call to = to check for isNan before identity?

never gonna happen

I suppose the ##NaN literal could return a new instance every time and that would "fix it" for the ##NaN literal

Yeah, I wasn’t even suggesting it be fixed. Mainly just wondering why it was behaving as it was in the first place.

well there's also this

Double/NaN
##NaN
(not= Double/NaN Double/NaN)
true
(not= ##NaN ##NaN)
false

That is probably because of Object identity in one case, but two non-identical objects in the other

but there's still programmatic way to make it behave like this, it's just an edge case of how NaN work as a primitive (as defined by the float standard) and how it behaves as an object

##NaN eats babies for breakfast, shoots your dog, steals your Bible, and drives Darth Vader insane.

'tis evil, I tell you

For sure.

floats remain one of the more interesting inventions of computer science. They are incredibly clever, and incredibly … sharp and pointy sometimes

https://clojure.org/reference/atoms Could someone explain to me why, in this example, the memoized fib call returns in a fraction of the time of the not memoized fib call even on the first call to the memoized fib? I ran the example and verified that it does run way faster on the first call, but how does this work if the result for those args hasn't been saved yet? I understand that subsequent calls with the same args should return much faster since the results have been saved, just unsure about how that happens with the first call.

Furthermore, if I run something like (fib 444) unmemoized it ties up the repl, naturally, but if I memoize fib then call the same thing I get an actual result in milliseconds. :thinking_face:

And that result hasn't been saved before ^

Maybe I should be clear I'm using clojurescript targetting node, not sure if that matters though.

fib is recursive. (fib 5) calls (fib 4) and (fib 3). And if those are in cache, then (fib 5) will be fast. So now you have (fib 5) and (fib 4) in cache - exactly what's needed for (fib 6). And so on.

because fib calls itself. fib 35 calls fib 34 and fib 33. each of which call .... So "the first call" has lots of other first calls that are duplicated

Ohhhh 🙃

Thank you

I strongly suggest reading the 2nd edition of SICP. It has some great material that deals with all sorts of problems in the functional world and outside of it - fib is one of them.

Oh I see! 😄 Thank you.

it's the difference between 444 and 444! :)

Oh cool. I actually have this on my list.

I believe that most arithmetic operations are not associative using floating point arithmetic, at least for a large class of values, e.g. it is often the case that you get different results from ((a+b)+c) vs. (a+(b+c)). I believe Fortran and a few other programming language mandate that order of operations emitted by the compiler may not change from what is implied in the source code, so people can write floating-point numerical libraries and not have the compiler mess up what they intended.