Clojurians Log v2

well, aside from a lot of us in IRC trying to break it all the time for years, I agree 😄

it also does the nice thing and will ignore thing if it is prepended with _, like _thing to signal unused

Hi everybody

does somebody know how to convert this java line code into clojure code?

this.API = (WindowsNamedPipeLibrary) Native.loadLibrary("kernel32", WindowsNamedPipeLibrary.class, W32APIOptions.UNICODE_OPTIONS)

I've been trying this...

(defn get-API []
  ;this.API = (WindowsNamedPipeLibrary) Native.loadLibrary("kernel32", WindowsNamedPipeLibrary.class, W32APIOptions.UNICODE_OPTIONS)
  (let [interface-class (.getClass osquery-clj.interfaces.IWindowsNamedPipeLibrary)
        unicode-options (W32APIOptions/UNICODE_OPTIONS)
        API ^osquery-clj.interfaces.IWindowsNamedPipeLibrary (Native/loadLibrary "kernel32" interface-class unicode-options)]
    API))

You should print out the value of interface-class

what error are you getting?

Im getting that error Interface (Class) of library=kernel32 does not extend Library

For JNA, I've been using something like:

(def objlib (try
              (com.sun.jna.NativeLibrary/getInstance "CoreFoundation")
              (catch java.lang.UnsatisfiedLinkError e
                nil)))

(def api (try
              (com.sun.jna.NativeLibrary/getInstance "kernel32" (W32APIOptions/UNICODE_OPTIONS))
              (catch java.lang.UnsatisfiedLinkError e
                nil)))

UNICODE_OPTIONS is not a function so the parens look wrong around that.

that part works though

> (W32APIOptions/UNICODE_OPTIONS)
{"type-mapper" #object[com.sun.jna.win32.W32APITypeMapper 0x4a9f4493 "com.sun.jna.win32.W32APITypeMapper@4a9f4493"], "function-mapper" #object[com.sun.jna.win32.W32APIFunctionMapper 0x703d3680 "com.sun.jna.win32.W32APIFunctionMapper@703d3680"]}

it also works without the parens :thinking_face:

Source https://github.com/java-native-access/jna/blob/master/src/com/sun/jna/win32/W32APIOptions.java so...

that block is from my repl, so I can confirm it works with and without parens

unless my repl is doing something goofy

now it seems it is working

(defn get-API []
  ;this.API = (WindowsNamedPipeLibrary) Native.loadLibrary("kernel32", WindowsNamedPipeLibrary.class, W32APIOptions.UNICODE_OPTIONS)
  (let [interface-class osquery-clj.interfaces.IWindowsNamedPipeLibrary
        unicode-options W32APIOptions/UNICODE_OPTIONS
        API ^osquery-clj.interfaces.IWindowsNamedPipeLibrary (Native/loadLibrary "kernel32" interface-class unicode-options)]
    API))

And as for WindowsNamedPipeLibrary.class, in Clojure the class name evaluates to the class itself, so WindowsNamedPipeLibrary should be all you need?

yes you are right

I'm just kind of puzzled by a Java interface that seems to have data members...

with that change it seems its working

My Java is rustier than I thought? 🙂

thanks all

there's some odd things in JNA to make interop with c better

honestly, I'm not sure how all of it works

Apparently you can declare variables (data members) in an interface and they are treated as static final. TIL.

@pez hmm that sounds quite strange (esp the 200x)… what kind of code are you running and how many cores is the machine? What’s the (pipeline n …) number?

Is it pure functions or is there IO in the code?

I’ll make a repro.

I think this is similar enough to what I do in my program, where I have a boolean-array where some positions are set, and some are not. Most are not. Then I want the indexes of the set bits of every odd position from the array, as a sequence.

(let [n 1000000
      ba (boolean-array n)
      every-other (range 1 n 2)
      prob 0.15
      sample-size (long (* n prob))
      to-ch (a/chan 1)
      from-ch (a/to-chan every-other)]
  (doseq [i (take sample-size
                  (random-sample prob (range n)))]
    (aset ba i true))

  (println "filter")
  (time
   (count
    (filter #(aget ba %) every-other)))

  (println "transduce")
  (time
   (count
    (transduce
     (filter #(aget ba %))
     conj
     every-other)))

  (println "core.reducers/filter")
  (time
   (count
    (into [] (r/filter #(aget ba %) every-other))))

  (println "core.async/pipeline")
  (time
   (do
     (a/pipeline 4 to-ch (filter #(aget ba %)) from-ch)
     (count
      (a/<!! (a/into [] to-ch))))))

Output:

filter
"Elapsed time: 24.779003 msecs"
transduce
"Elapsed time: 23.854725 msecs"
core.reducers/filter
"Elapsed time: 23.759016 msecs"
core.async/pipeline
"Elapsed time: 3535.955765 msecs"
=> 74885

If I use criterium/quick-bench instead, the transduce and reducers wins are a bit more appearant:

filter
Evaluation count : 30 in 6 samples of 5 calls.
             Execution time mean : 23,654346 ms
    Execution time std-deviation : 301,776435 µs
   Execution time lower quantile : 23,352585 ms ( 2,5%)
   Execution time upper quantile : 24,050820 ms (97,5%)
                   Overhead used : 14,507923 ns
transduce
Evaluation count : 36 in 6 samples of 6 calls.
             Execution time mean : 20,129352 ms
    Execution time std-deviation : 595,084459 µs
   Execution time lower quantile : 19,646010 ms ( 2,5%)
   Execution time upper quantile : 21,079716 ms (97,5%)
                   Overhead used : 14,507923 ns
core.reducers/filter
Evaluation count : 36 in 6 samples of 6 calls.
             Execution time mean : 17,903643 ms
    Execution time std-deviation : 186,971423 µs
   Execution time lower quantile : 17,675913 ms ( 2,5%)
   Execution time upper quantile : 18,138291 ms (97,5%)
                   Overhead used : 14,507923 ns
core.async/pipeline
Evaluation count : 22230 in 6 samples of 3705 calls.
             Execution time mean : 27,089463 µs
    Execution time std-deviation : 136,898899 ns
   Execution time lower quantile : 26,919838 µs ( 2,5%)
   Execution time upper quantile : 27,291471 µs (97,5%)
                   Overhead used : 14,507923 ns

Interestingly (to me, at least 😄 ) for performs on par with transduce with this task:

(println "for")
    (quick-bench #_time
                 (count
                  (for [i every-other
                        :when (aget ba i)]
                    i)))

for
Evaluation count : 36 in 6 samples of 6 calls.
             Execution time mean : 19,456574 ms
    Execution time std-deviation : 100,364503 µs
   Execution time lower quantile : 19,370228 ms ( 2,5%)
   Execution time upper quantile : 19,618312 ms (97,5%)
                   Overhead used : 14,507923 ns

Hi everybody! I gotta hook up a call (sentry 🙂 ) to all of the exceptions (`Throwable`s) in my application. Is there a nice AOP way to do it by adding a hook somewhere once?

What kind of application are you developing? For example with a HTTP stack you can create a http middleware, which catches and re-throws the exceptions thrown by the handler(s) and reports them to sentry

You can try this:

(defn ^:private set-default-exception-handler
  [sentry-logger]
  (Thread/setDefaultUncaughtExceptionHandler
   (reify Thread$UncaughtExceptionHandler
     (uncaughtException [_ thread ex]
       (log/warn ex "Uncaught Exception!")
       (sentry-logger {:throwable ex})))))

You simply need to then provide a sentry-logger, which is a function that creates a sentry instance, configured as you see fit...

hmmm ☝️ that's one way to do it

Works for moi in my application 🙂

of course, I'd have to add the sentry hook to all of the caught exceptions - but that's mostly search and replace

yeah, I like this one. Thank you!

you're welcome. 🙂

btw, which sentry library are you using?

I think I'll go with the official one. Got any reccomendations?

Which one would that be?

I recently had some problems with the official one (`sentry-clj`?), it failed to report some clojure exceptions properly. raven-clj did not have this issue.

Which version of the sentry-clj are you using?

Out of curiosity - what about a plain loop?

@dharrigan sentry-clj

good question, I believe I tried the latest one a few weeks ago

I also saw one from exoscale - is it worth it? which one are you using?

I'm maintaining the semi-official one

(sorry, I didn't mean to hijack this thread with my own issues 😄 )

sentry-clj isn't official, in the sense that it's not sanctioned by Sentry. It's deemed a community-driven effort

@pez I see… Ok, I think the biggest gains from pipeline are to be had when the pipeline transducer is CPU intensive (think like parsing HTML into data, file compression, etc); here you have a pretty straightforward xf (filter #(aget ba %)) Also, I think 1,000,000 samples is not that much really, so (pipeline …) would be suffering from all the channel, etc overhead of passing the data around;

If you have issues with sentry-clj, raise an issue on github and I'll see what I can do 🙂

PRs are welcome 'tho 🙂

Also, a sidenote, (time …) is almost never a good benchmark strategy (but quick-bench is); I’ve seen cases where a simple (time …) benchmark would be “slow” but quick-bench would actually show a huge improvement since the JVM does its JIT magic and code really speeds up after a few iterations in some cases;

I think that’s a good idea @p-himik (loop []…)

That’s probably the fastest thing you can get in terms of raw single thread perf… pretty much Java speed;

I remember reading the code in Clojure compiler that would explicitly turn (MyClass/STATIC_FIELD) into a regular static field lookup. Can't quickly find it right now though. Perhaps it even worked that way for (.regularNotCallableField myObject)...

BOOM

(println "loop")
(quick-bench #_time
             (count
              (loop [res []
                     i 1]
                (if (<= i n)
                  (recur (if (aget ba i)
                           (conj res i)
                           res)
                         (+ i 2))
                  res))))

loop
Evaluation count : 84 in 6 samples of 14 calls.
             Execution time mean : 7,518441 ms
filter
Evaluation count : 30 in 6 samples of 5 calls.
             Execution time mean : 23,020098 ms
transduce
Evaluation count : 36 in 6 samples of 6 calls.
             Execution time mean : 19,090405 ms
core.reducers/filter
Evaluation count : 42 in 6 samples of 7 calls.
             Execution time mean : 16,328693 ms
for
Evaluation count : 36 in 6 samples of 6 calls.
             Execution time mean : 19,678977 ms

Yup, loop is the king 🙂

If you really care about perf. I highly recommend YourKit

I bet it will help you gain 50% in no time

I’ve used it, it’s like magic; the gains will come from a place you least expect… some reflection call that’s using 50% of your CPU time

@pez Now try making res a transient. :)

transient, huh? Doin’ it!

Try also unchecked-math 🙂

In my previous adventure with single-threaded high perf, I ended up writing a Java class. :D All my data consisted of integers and Clojure doesn't really like them.

Also, http://clojure-goes-fast.com (various ideas how to go fast)

I’ll be trying YourKit too. Though only out of curiosity really. I don’t have performance tasks often. This is a little toy challenge I have, mainly to learn more about Clojure. I profile it with tufte right now, which is pretty nice.

Seems like I should be able to parallelize the loop, no?

Absolutely, your problem is a textbook map(filter)/reduce problem.

I'm a bit surprised that transit doesn't offer support for serializing arrays out of the box:

user=> (write-transit [1 2 3])
"[1,2,3]"
user=> (write-transit (into-array [1 2 3]))
Execution error (NullPointerException) at com.cognitect.transit.impl.AbstractEmitter/marshalTop (AbstractEmitter.java:203).
null

I guess it focuses on Clojure data, and arrays are host-specific things:

(into-array [1 2 3]) => #object[“[Ljava.lang.Long;” 0x55cad8a0 “[Ljava.lang.Long;@55cad8a0"]

So are byte arrays, yet it supports them:

user=> (write-transit (.getBytes "foo"))
"[\"~#'\",\"~bZm9v\"]"

I see… I don’t have other guesses 🙂

Good Morning Clojurians, if I make a mistake and require something that I should not have, and get this:

user=> (require 'development)
Syntax error (FileNotFoundException) compiling at (app/model/session.clj:1:1).
Could not locate datascript/core__init.class, datascript/core.clj or datascript/core.cljc on classpath.

@timofey.sitnikov in the leiningen world you have pomegranate for this, in the deps.edn world there is something called add-lib

Yes. You can use a different classloader that allows adding things to the classpath in runtime. While it may sound complicated, there are already implementations out there. Probably the easiest way to start using such functionality is to look at Sean Corfield's dot-clojure repo: https://github.com/seancorfield/dot-clojure Namely, search for :add-libs in the README.

(I'm assuming you're using deps.edn)

I there a way to override a specific method implementation in a defrecord?

Hmm, maybe :extend-via-metadata helps somehow?

user=> (defprotocol Foo :extend-via-metadata true (foo [_]) (bar [_]))
Foo
user=> (defrecord FooRec [] Foo (foo [_] :foo) (bar [_] :bar))
user.FooRec
user=> (foo (->FooRec))
:foo
user=> (foo (with-meta (->FooRec) {`foo (fn [_] :overriden)}))
:foo
user=> (foo (with-meta (into {} (->FooRec)) {`foo (fn [_] :overriden)}))
:overriden

But not really I think

Do you mean a defrecord instance of the defrecord type btw?

The idea was to override the equality methods for a record, but I can roll my own with a minimal deftype

oh right

Things got less pleasant where I had to implement my own hasheq 😞

you should not do this

if you want custom equality/hashing semantics, you should deftype

dumb question about internals: is the compiler supposed to evaluate each top level form one-by-one when compiling? it has to learn about macros somehow as it goes

or is there some more advanced magic involved?

the compiler first reads (string -> clojure data)

then compiles, then evaluates

@vandr0iy you can make your own try catch

compiling an expr will look at the thing in function position. if that's a macro, it expands the macro (this happens until it's not a macro)

thats always on the table

a macro is just a function that takes a form (clojure data representing a function call) as input and returns an alternate form (clojure data) to replace it with

not recommending it but you can always make

(custom/try
   ...
   (catch RuntimeException e
     ...)
   (finally
     ...))

and expand it to have whatever calls to sentry you want

the unit of compilation/evaluation is a single top-level form (not a file)

that would be the AOP-iest way

yes, but defmacro is basically (do (defn x ...) (set-macro! (var x))) and that set-macro! call has to be evaluated for the compiler to know that x is a macro

ah right, that’s what I was wondering about

thanks @alexmiller 👍

obviously it is not called set-macro! in Clojure, used that name for brevity

That's what I ended up doing

Could someone please point me to the repo that tracks the Clojure CLI bash script?

https://github.com/clojure/brew-install/blob/1.10.2/src/main/resources/clojure/install/clojure

Thanks! Would've never thought to look in something that has "brew" in its name. :) And for some reason Google doesn't index the repo.

https://clojure.org/releases/tools is also a useful reference, listing all the releases and the changes in each.

I'm trying to use clojure.tools.namespace.repl/refresh and I've stumbled upon an interesting issue. When called for the first time, it seems to reload absolutely all Clojure files that it could find in the classpath. Regardless of whether they have been loaded before or not. The problem is that I have a few files that aren't loaded and require extra dependencies to be loaded. Clearly, I don't want refresh to load them at all. Is it possible to fix it on my end? Would it make sense to alter something in clojure.tools.namespace to support reloading only the files that have been loaded before?

transient shaves some more of the time, as hinted at 😃

loop
             Execution time mean : 7,704050 ms
loop-transient
             Execution time mean : 5,017702 ms
filter
             Execution time mean : 24,047486 ms
transduce
             Execution time mean : 19,687393 ms
core.reducers/filter
             Execution time mean : 17,303117 ms
for
             Execution time mean : 21,142251 ms

we’re also seeing this, iirc it’s a problem with tools namespace not seeding the timestamp it uses to track what needs to be compiled directly

Not sure. The docstring explicitly says that it will load all files upon the first call: > Scans source code directories for files which have changed (since the last time this function was run)

yeah, might well be a design decision but it’s annoying in our project where you need to wait forever the first time you call it

https://github.com/clojure/tools.namespace/blob/master/src/main/clojure/clojure/tools/namespace/track.cljc#L147

preseeding this timestamp might be worth a try

Oh, that might help, I'll try that!

please let me know if it does!

Yes, the repo name is bad and we have a ticket for that (but it requires updating a bunch of infrastructure so has not seemed worth it so far)

PSA: Clojars is having DNS issues - our provider (DNSimple) is having an outage, which is making it impossible to resolve http://clojars.org in some parts of the world. Hopefully they will get it fixed soon.

Note that the script linked above does have some variable replacement done on it before it gets published so you can’t just use it as is

OK, adding

(alter-var-root #'clojure.tools.namespace.repl/refresh-tracker
                assoc :clojure.tools.namespace.dir/time (System/currentTimeMillis))

Often when people ask this, their real need is something else which might have a better answer if you care to share

Thanks! Yeah, I was just curious about the history of commits that first introduced clojure.libfile and later removed it. Cursive seems to still set the clojure.libfile property.

this looks super cool. do you have any examples of usage?

Ah, yeah, that’s dead, replaced by the basis property. I was setting both so add-lib would keep working but that’s updated now

https://gist.github.com/hiredman/3443693c5994a8b0bb0a41f068107abd

awesome, thank you!

Got it working properly (or at least, as I want it to work) with this:

(alter-var-root #'clojure.tools.namespace.repl/remove-disabled
                (fn [orig-remove-disabled]
                  (fn [tracker]
                    (let [filter-loaded #(filter find-ns %)]
                      (-> tracker
                          orig-remove-disabled
                          (update :clojure.tools.namespace.track/unload filter-loaded)
                          (update :clojure.tools.namespace.track/load filter-loaded))))))

@alexmiller You're the latest contributor to tools.namespace - do you have any thoughts on the above? Would it make sense to add such a behavior to the lib, perhaps under a flag or via a function that changers some var, like disable-reload! already does?

please file a question on ask.clojure..

don't have time to look at it atm

I remember I found a tool similar to Ansible but written in Clojure, it’s not pallet, it was recently developed, I can’t find. any idea?

Spire? https://github.com/epiccastle/spire

https://ask.clojure.org/index.php/10277/preventing-clojure-namespace-refresh-loading-namespaces

Unchecked math doesn’t seem to make much of a difference for the particular problem.

I think that's because there's only a single math operation there, and its arguments' types are well known by the compiler. If you really want to pursue it further, I would try to get the bytecode for that code and see if there's something fishy going on. I've had some success with https://github.com/gtrak/no.disassemble/ and https://github.com/clojure-goes-fast/clj-java-decompiler before.

Yes thank you

Looks like you aren't invoking set-refresh-dirs beforehand? It's very recommendable to do so

It still doesn't solve item 1 in the post above. It's easy to forget to add a newly created dir there. It's impossible to have a file that you don't want to load in the same dir with a file that should be reloaded.

> It's easy to forget to add a newly created dir there. Can be solved in a number of ways; this is not a place to question t.n's API/design. > It will attempt to (re)load even the files that haven't been loaded before. This doesn't make sense, t.n's very job is loading code. Its first load will typically load all your project's code; if you aren't doing this you are following a custom/hybrid approach

And I don't think that it's an invalid approach. At the very least, mkvlr above seems to share it with me.

@vemv IMO there’s nothing to reload when you call refresh after you project boots without code changes

there's everything to reload because nothing can be assumed to have been required. If something else was requireing your code, making refresh perceivedly redundant, you should have a careful look at what/why is doing that. https://github.com/stuartsierra/reloaded is the canonical example of how to setup a project using t.n.

Still, it doesn't invalidate the desire to be able to adhere to, as you called it, a custom/hybrid approach.

I didn't suggest so

Unchecked doesn’t attract me so much. I would rather to figure out how to parallellize it. I can’t immediately see how:

(quick-bench #_time
                 (count
                  (loop [res []
                         i 1]
                    (if (<= i n)
                      (recur (if (aget ba i)
                               (conj res i)
                               res)
                             (+ i 2))
                      res))))

- Split ba into N chunks - For each chunk, run a thread that creates its own res - Combine the resulting collection of res vectors in a single vector, preserving the order

Just out of interest - why (+ i 2)? Does ba store something unrelated at even indices?

Yes, I am only interested in the odd indices. ba contains the results of an eratosthenes sieve, where I have skipped sieving even numbers, b/c we all know there’s only one even prime number. 😃

I was hoping there was some reducer or something that would do all those steps for me.

Oh, is that code just to find prime numbers up to n? If so, then even constructing the sieve could be made parallel. And I'm 95% certain there's already a Java library that does it. :)

Haha, I’m in this to learn about Clojure. 😃

That code is only to pick out the prime numbers I have found up to n.

Here’s the full thing, using loop and transient:

(defn pez-ba-loop-transient-sieve [^long n]
  (let [primes (boolean-array (inc n) true)
        sqrt-n (int (Math/ceil (Math/sqrt n)))]
    (if (< n 2)
      '()
      (loop [p 3]
        (if (< sqrt-n p)
          (loop [res (transient [])
                 i 3]
            (if (<= i n)
              (recur (if (aget primes i)
                       (conj! res i)
                       res)
                     (+ i 2))
              (concat [2] (persistent! res))))
          (do
            (when (aget primes p)
              (loop [i (* p p)]
                (when (<= i n)
                  (aset primes i false)
                  (recur (+ i p p)))))
            (recur  (+ p 2))))))))

I’m not tracking this thread so please make sure relevant info ends up on the ask question, thanks!

I haven’t ventured into how to speed up the sieving (beyond the obvious optimizations) b/c most of the time has been spent in picking out the indicies from the sieve.

We discourage cross-posting. People who want to see news and articles will subscribe to that channel. It's the same reason that articles/videos/etc are generally disallowed in #announcements -- it is for project/library releases only (and maintainers are discouraged from posting too often -- that's why we have #releases for more minor updates).

OK, so how do you start? Do you just execute a single command and everything gets re-loaded?

FYI, every new member is automatically subscribed to #news-and-articles -- so if that channel has fewer people it is because they've chosen to leave it.

Got it. Sorry 😬

I deleted it from #clojure as it seems you are getting feedback in #news-and-articles

Sometimes you just need to be patient before people get around to responding 🙂

Maybe I got greedy, it's not like nobody cared, I'll stick to announcements from now on. 😀

Thanks for the heads up @tcrawley — and glad it was sorted quickly.

Nothing gets reloaded. You require t.d.a (tools.deps.alpha) and call the right function to add a dependency in runtime. The linked repo mentions this near the description of :add-libs: "see the example load-master function in the comments in my deps.edn". That example explains pretty much everything.

Hello Clojure friends. It's your prodigal son who's spent some time wandering in statically typed functional land, especially OCaml as of late. Compared to Clojure, OCaml's experience for live editing is really subpar, and I want to improve it. There are 2 routes I'm considering: 1. Add better REPL support in my editor (similar to Calva/CIDER) 2. Don't do a REPL per se, but instead do something like https://quokkajs.com/. Whereas a REPL is stateful, Quokka just runs a file and prints results inline in the editor, discarding the context each time. What do you guys think? I'm guessing that experienced Clojurians probably think 1 is strictly better than 2. Could you explain why?

One thing that immediately comes to mind - CPU intensive work. It wouldn't be fun to wait for something to finish each time you run the file or to keep track of all the temporary files with intermediate results. Another thing is side-effects. You usually don't want to repeat them unless necessary.

Come to think of it, Quokka has some pretty crazy heuristics for finding exactly what code needs to be rerun to mitigate those issues @p-himik. Re implementing those for OCaml would not be fun, whereas with a REPL, the user decides what needs to be rerun.

I suspect a Quokka-style setup up is easier to get up and running since you don't have to maintain a connection so I don't know about strictly superior. Although if you're coming up with heuristics, it's probably not simpler anymore. Also, the reloading entire files avoids situations where, e.g., you're running (g (f val))) but the actual code running is (old-g (current-f val))) because you forgot to re-evalute your new definition of g.

@d4hines Perhaps - apart from having a glance at the website, I'm not that familiar with Quokka so I was talking about just running a file each time.

I think the REPL approach is ultimately better because like you said, the user can decide and you can include shortcuts to reload the entire file. But running everything from scratch has a certain simplicity. I don't think it's well-suited to Clojure due to the startup time, but it might work for ClojureScript or something like Lumo or Babashka.