Clojurians Log v2

Hello all. I created a #show-and-tell channel today. #announcements channel is lovely for libs but I want to be inspired by the systems and products people are building. What are you building with clojure today?  Show a screenshot and tell us about it. Few fun screenshots at #show-and-tell already.

I found the Figwheel docs and re-frame docs pretty good. But I'm only starting to learn cljs. I've been doing backend Clojure for a decade.

how can i use the threaded form but avoid particular parts of the thread based on the current threaded value, I'm aware of cond-> but that doesn't test on current value, eg:

(-> data 
    (fn1) 
    (fn2) 
    (thread-if #(= (:needs-doing %)) (needs-doing)) 
    (fn5)

cond->

Or lift the if into a fn above

hi please advice is there a smart trick to know if it is a last iteration inside the run! (the consumed sequence is lazy and it is important to keep it lazy)

there is no way to know if you're at the last element without looking ahead one element

is there a run! which accepts two elements? 🙂

I would just implement this using loop probably

makes sense!

thanks

almost. the nil? check would not be enough if the seq also can contain nils, so I would use:

(loop [xs (seq xs0)]
  (let [next-xs (next xs)
         last? (not next-xs)]
   (recur next-xs)

You could also do it with a transducer/eduction

(run! identity (eduction (fn [rf] (fn ([] (rf)) ([r] (prn 'done r) (rf r)) ([r i] (rf r i)))) (range 5)))

it is interesting idea with a transducer thanks!

You could make a function for this pretty easily.

(defn when-pred
  [val pred then]
  (if (pred val)
    (then val)
    val))

(defmacro if->
  [val sym test clause]
  `(when-pred ~val (fn [~sym] ~test) (fn [~sym] ~clause)))

(-> data
    fn1
    fn2
    (when-pred :needs-doing needs-doing)
    fn4)

(-> data
    fn1
    fn2
    (if-> val (= (:some-key val) :some-thing) (update val :some-key some-update))
    fn4)

Could you run! Over but-last and special case the last one?

seems like custom loop as simplier solution

thanks for the input

you could also pair each element with the next one and look for where the paired elements run out:

(let [data (range 5)]
    (run! (fn [[a b]] (if (= a ::last) (do (prn b) b) b)) (map vector (concat (drop 1 data) [::last]) data)))

thanks

I just got this. Who do I need to poke (if anyone)?: > WARNING: Illegal reflective access by clojure.lang.InjectedInvoker/0x0000000800232040 (file:/C:/Users/x/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar) to method sun.security.x509.X509CertImpl.getSubjectDN() > WARNING: Please consider reporting this to the maintainers of clojure.lang.InjectedInvoker/0x0000000800232040 > WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations > WARNING: All illegal access operations will be denied in a future release

What would be the best way to run command-line tools from Clojure? I do not just want to dispatch jobs, but also be notified when they are finished and learn the exit-code, std(out/err), and so on.

can you provide more context around the code that did that?

--illegal-access=debug @sweb add ^ to your JVM to get some better stacktrace

https://clojuredocs.org/clojure.java.shell/sh perhaps does what you need?

https://clojure.org/guides/faq#illegal_access

I like babashka/process https://github.com/babashka/process as the name implies, it's included in babashka

I am building a (Snake)make-like workflow management system where users provide the rules in text files to be parsed by my program to create a DAG for execution (phew!). The rules look like this (there would obviously be many more rules in a file):

(defrule samtools-sort
  "Sort the bams."
  {:wildcards [:sample :genome]
   :input     "bwa-map.bam"
   :output    ["bam/sorted.bam"]
   :shell     "samtools sort -T {{wildcards.sample}} -O bam {{input.0}} > {{output.0}}"}) 

:clojure (-> infile slurp process-data write-result-to-file)

Thanks. I'll look at both of them!

The actual code creating the issue is this: (->> cert (.getSubjectDN) (.getName) (drop 3) (apply str))] cert is an X509CertImpl pulled from an sslContext Solution is to use a different method but it asked to warn someone..so here I am.

So, I have fixed the issue in my code to not use a deprecated function.

unfortunately, because this is in code generated by the compiler on your code's behalf, it will always get reported as if it is Clojure ;)

Call a build tool from your program

Okies. Thanks

the issue is not actually one of deprecation, but rather calling into a function that you cannot access due to module visibility.

I guess. This would require me to learn lots more about tooling. Up to this day I've just cargo-culted and used luminus-recipes XD

Thanks, btw.

For now clojure.ava.shell.sh is enough. It is synchronous though. But I am sure I can just use Thread to make it non-synchronous.

And it's true that switching from getSubjectDN to getSubjectX500Principal still gives an error

this being one of the internal sun jdk methods, you're not supposed to use this class directly

can you post the full stacktrace from --illegal-access=debug?

I'd like to know if it's the Reflector at fault or something else

I'll have a look

yeah, would be curious to see the code

probably should be using java.security.cert.X509Certificate ?

ok here is what I got for you: > WARNING: Illegal reflective access by clojure.lang.InjectedInvoker/0x0000000800232040 (file:/C:/Users/sweb/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar) to method sun.security.x509.X509CertImpl.getSubjectX500Principal() > at clojure.lang.Reflector.canAccess(Reflector.java:49) > at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:156) > at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:438) > at url_check.core$wildcard_QMARK_.invokeStatic(core.clj:70) > at url_check.core$wildcard_QMARK_.invoke(core.clj:67) > at url_check.core$eval8394.invokeStatic(form-init16933106916591026780.clj:205) > at url_check.core$eval8394.invoke(form-init16933106916591026780.clj:205) > at clojure.lang.Compiler.eval(Compiler.java:7177) > at clojure.lang.Compiler.eval(Compiler.java:7132) > at clojure.core$eval.invokeStatic(core.clj:3214) > at clojure.core$eval.invoke(core.clj:3210) > at clojure.main$repl$read_eval_print__9086$fn__9089.invoke(main.clj:437) > at clojure.main$repl$read_eval_print__9086.invoke(main.clj:437) > at clojure.main$repl$fn__9095.invoke(main.clj:458) > at clojure.main$repl.invokeStatic(main.clj:458) > at clojure.main$repl.doInvoke(main.clj:368) > at clojure.lang.RestFn.invoke(RestFn.java:1523) > at nrepl.middleware.interruptible_eval$evaluate.invokeStatic(interruptible_eval.clj:79) > at nrepl.middleware.interruptible_eval$evaluate.invoke(interruptible_eval.clj:55) > at nrepl.middleware.interruptible_eval$interruptible_eval$fn__3452$fn__3456.invoke(interruptible_eval.clj:142) > at clojure.lang.AFn.run(AFn.java:22) > at nrepl.middleware.session$session_exec$main_loop__3553$fn__3557.invoke(session.clj:171) > at nrepl.middleware.session$session_exec$main_loop__3553.invoke(session.clj:170) > at clojure.lang.AFn.run(AFn.java:22) > at java.base/java.lang.Thread.run(Thread.java:834) > WARNING: Illegal reflective access by clojure.lang.Reflector (file:/C:/Users/sweb/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar) to method sun.security.x509.X509CertImpl.getSubjectX500Principal() > at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:167) > at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:438) > at url_check.core$wildcard_QMARK_.invokeStatic(core.clj:70) > at url_check.core$wildcard_QMARK_.invoke(core.clj:67) > at url_check.core$eval8394.invokeStatic(form-init16933106916591026780.clj:205) > at url_check.core$eval8394.invoke(form-init16933106916591026780.clj:205) > at clojure.lang.Compiler.eval(Compiler.java:7177) > at clojure.lang.Compiler.eval(Compiler.java:7132) > at clojure.core$eval.invokeStatic(core.clj:3214) > at clojure.core$eval.invoke(core.clj:3210) > at clojure.main$repl$read_eval_print__9086$fn__9089.invoke(main.clj:437) > at clojure.main$repl$read_eval_print__9086.invoke(main.clj:437) > at clojure.main$repl$fn__9095.invoke(main.clj:458) > at clojure.main$repl.invokeStatic(main.clj:458) > at clojure.main$repl.doInvoke(main.clj:368) > at clojure.lang.RestFn.invoke(RestFn.java:1523) > at nrepl.middleware.interruptible_eval$evaluate.invokeStatic(interruptible_eval.clj:79) > at nrepl.middleware.interruptible_eval$evaluate.invoke(interruptible_eval.clj:55) > at nrepl.middleware.interruptible_eval$interruptible_eval$fn__3452$fn__3456.invoke(interruptible_eval.clj:142) > at clojure.lang.AFn.run(AFn.java:22) > at nrepl.middleware.session$session_exec$main_loop__3553$fn__3557.invoke(session.clj:171) > at nrepl.middleware.session$session_exec$main_loop__3553.invoke(session.clj:170) > at clojure.lang.AFn.run(AFn.java:22) > at java.base/java.lang.Thread.run(Thread.java:834)

seems like https://github.com/davidkeen/url-check-clojure/blob/master/src/url_check/core.clj

line #s don't seem right there though

indeed I am doing something along the same lines. I hadn't seen this repo before though

or maybe that's your code? I'm just guessing from the ns

ha, no it's not although we have the same ns.

ok, well what's your code at line 70?

he seems to be checking for a well formed response from a server. I am checking cookie compliance, form usage, cert validity (where I am running into the interop issues) etc

(let [n (->> cert (.getSubjectX500Principal) (.getName) (drop 3) (apply str))]

any hints on how to debug a memory leak / performance leak in a clojure app on production ? The normal way to look on gc generation does not really help because clojure does not have classes.

It seems like this will be really easy with the clj command-line tool :thumbsup:

how are you starting the java process? if you want to download code from the internet and add it to your classpath, you may want to look at something like the add-lib branch of the clj tool (which is currently work in progress)

My Clojure program will be a jar 🙂

https://insideclojure.org/2018/05/04/add-lib/

Cool! Will keep my eyes open for updates on the tool 😎

yeah ... that's the bunny 😉

I've used this a little in the past too: https://github.com/pallet/alembic

Clojure does produce classes

for what it's worth: nREPL 0.7 got sideloader: #97: Added a sideloader, a network classloader that allows dependencies to be added even when the source/class files are not available on the server JVM's classpath (e.g. supplied by the client). https://github.com/nrepl/nrepl/releases (I haven't tried this feature myself)

I am following the https://clojure.org/guides/deps_and_cli#_using_a_main but I cannot find out how to run the example program from a different folder:

clj -X hello/run 

Perhaps with VisualVM: https://docs.oracle.com/javase/8/docs/technotes/guides/visualvm/heapdump.html

as per your faq, changing it to: > (let [n (->> ^java.security.cert.X509Certificate cert > (.getSubjectX500Principal) > (.getName) > (drop 3) > (apply str))] worked without error.

Cool to know about 🙂

bash-5.0# jcmd 1 GC.class_histogram -all > GC.class_histogram.csv bash-5.0# head GC.class_histogram.csv 1: num #instances #bytes class name (module) ------------------------------------------------------- 1: 96854702 3099350464 clojure.lang.LazySeq 2: 64655045 2068961440 clojure.lang.Cons 3: 5122755 394906984 [Ljava.lang.Object; (java.base@11.0.10) 4: 3382334 157481032 [B (java.base@11.0.10) 5: 3323357 79760568 java.lang.String (java.base@11.0.10) 6: 1509001 60360040 clojure.lang.PersistentVector 7: 2059526 49428624 clojure.lang.PersistentVector$Node

hmm, this seems to indicate holding onto the head of lazy sequences

yes, and because clojure lacks types it's hard to know where this is comming from

in Java, with types you would see the name of the class somewhere there

no, not with using clj -X

@eugen.stan did you try to force a GC before taking this data?

no

I see there is a -Jopt. Will read up on the possible flags I can send there.

if you take a thread dump can you see something in the list of things that the process is doing that's a common code path that looks like it might be holding onto the head of something large?

I don't have one now

thanks

it's a bit difficult since it's hard to reproduce the performance issue that triggers the issue and the app runs remotely inside docker

what kind of things is it connected to? if it's selecting large amounts of data from a db, and trying to serve that up over http it might be calling doall on something before trying to convert it to an http response?

Can you dump the heap on out of memory exception and have it written to a mounted docker volume to be inspected later? https://www.baeldung.com/java-heap-dump-capture#automatically

Yea exactly, I was gonna say may not be as hard as you think

thanks - the app does not throw memory exception - it starts lots of threads and it slows down to a halt

at the same time memory consumption is very high and these are related

but because there is no OOM it might not be a leak ?!

are you catching and logging/printing uncaught thread exceptions?

if there's a leak, you'll eventually get an OomException

which gc algorithm are you using?

have you tried the old strategy of doubling the heap size and seeing what happens over longer periods of time?

@eugen.stan > it starts lots of threads can you see what the app is doing in those threads?

@l0st3d default with java 11 - G1

@borkdude: I am working on that

@ivar.refsdal: not sure how to reply

I think @ivar.refsdal meant are you doing this: https://stuartsierra.com/2015/05/27/clojure-uncaught-exceptions ?

thanks @l0st3d, will check

thanks for the feedback so far, I'll see what I can do further

if you've got multiple docker processes to play with, you could always have one start up with the serial collector, which is a bit easier to follow what's getting cleaned up, because it just stops the world rather than trying to let all the threads progress ... (just as an idea of trying to work out what's going on 😉 )

Yes that's what I was suggesting

the performance issue is difficult to trigger, the app uses ~ 12 - 23 GB of ram and needs a cache, db and queue - these make things difficult to debug

that just lets you use any jvm option

class histo indicates a 96 million entry seq -- at least

you could take a memory dump and examine any instance of the LazySeq, check its .s field

I will do that, thanks

your step 1 class_histo was the absolute correct thing to do

Any advice on propagating errors when you have multiple long-running threads in a clojure service on the jvm? i.e. my-process returns [input-ch done-ch] and starts some background some work somehow -`ExecutorService`, async/thread, future, etc. The problem is all these wrappers to background work end up with uncaught exceptions not being handled (i.e. by top-level Thread$UncaughtExceptionHandler unless a specific uncaught exception handler is added to the thread, and this then needs to exit the process. It seems possible to return a future and deref futures from all these processes, but this doesn’t seem to be the way, as the deref order on the main thread then matters or you end up with the same problem :thinking_face: I’ve read this from a few years ago, but I’m not sure what the best approach is https://stuartsierra.com/2015/05/27/clojure-uncaught-exceptions

thank you, I found a very good presentation on the subject: https://www.youtube.com/watch?v=_gineh_HcoQ (shorter version available online someplace)

wow 3hrs 🙂

jcmd <pid> GC.heap_dump file_destination

thanks, I am putting some load on the system now (trying, I can't seem to be able 😄 ) .

https://github.com/clojure-goes-fast/clj-async-profiler. I found the flame graph quite useful in analysing both space and time issues.

this looks interesting, thanks rmcv

@eugen.stan if you can download the heapdump, you can offline analyze it with https://visualvm.github.io/download.html

I just re-tested it out and it was easy to tell what's in the LazySeq

thanks. Will try it out. I don't lknow how useful it will be for me now since I can't seem to reproduce the issue

I think it's affraid 🙂

you definitely have a thing hanging onto a LazySeq -- it's absolutely clear

seems I can't create a heap dump - size is 0. tried with both jcmd and jmap - files have size 0

Most (all?) of the ExecutorService implementations allow you to provide a ThreadFactory which would allow you to set an uncaught exception handler

The error handling strategy will depend on the use case. Generally, it's hard to recover from unexpected errors from arbitrary processes.

As far as using something like core.async, I would generally turn all "outputs" (including exceptions) into data. That can mean wrapping work in a try/catch

+1 on this, the ideal here is to start all your processes with core.async and then use alt to wait for the first one to fail and then exit on that.

Because the system is stalled or wrong pid? I did it like this some time ago:

$ docker exec my-service ps
PID   USER     TIME   COMMAND
    1 java       0:00 /dev/init -- java -Xmx8g -Dcom.sun.management.jmxremote.rmi.port=19090 -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=19090 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=true -Djava.rmi.server.hostname=localhost -jar /my-jar.jar
    8 java       0:26 java -Xmx8g -Dcom.sun.management.jmxremote.rmi.port=19090 -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=19090 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=true -Djava.rmi.server.hostname=localhost -jar /my-jar.jar
  256 java       0:00 ps

# Make a heap dump for PID 8
$ docker exec my-service jcmd 8 GC.heap_dump /tmp/docker.hprof

and then:

docker cp my-service:/tmp/docker.hprof .

it works now, I changed the path

There are some types in the transit spec like "point in time" that are listed as "extension" types

https://www.cognitect.com/blog/2015/9/10/extending-transit

but i'm unclear if making a type like "~y123" is doable

most of the examples make it seem like the one letter, string prepending tagged values are an internal thing

and custom types i support myself would have to be

["~#y" "123"]

I'm not clear what your question is

@emccue these types should be handled by registering transit handlers for those with a tag

for context, I am circling back to writing a transit impl for Elm

cool, I like transit and would like to see it be extended to other languages

One aspect is that it won't be idiomatic to have dynamic handlers registered

so its important to understand how these special cases work

like Symbol is listed as an extension type

but it affects how caching works

so that makes it an intrinsic thing

but ~i123 is a ground type...

and so on

transit has been extended to many languages

https://github.com/cognitect/transit-format/wiki/Community-Implementations

yeah one of the challenges is that all the community implementations seem to take the same "pass a map of handlers" approach so I don't have much prior art to look at

the simplest case would be something like

[["~#point", [3, 4]], ["~#point", [5, 6]]]

in most implementations you would do something like

(transit/read {"point" point-handler} transit-val)

and how to interpret a value tagged with "point" is encoded all the way down the tree

but in elm, we don't have much of a choice but to use an api similar to elm/json where you put explicit expectations at each point

Does anyone know if there's an existing macro/library/idiom for using something like let with channels for fan-in? For example:

(let* [thing1 (<!! (do-thing1-returning-channel))
       thing2 (<!! (do-thing2-returning-channel))]
  [thing1 thing2])

import Array
import Transit.Decode as TD

type alias Point =
    { x : Int
    , y : Int
    }

decoder : TD.Decoder (List Point)
decoder =
    TD.vector 
      (TD.map2 Point
         (TD.tagged "point" (TD.index 0 <http://TD.int|TD.int>))
         (TD.tagged "point" (TD.index 1 <http://TD.int|TD.int>)))
    |&gt; TD.map Array.toList

core.async/merge

(one way to do it)

Yeah, I can think of several good ways to get it done, just don't want to recreate someone's prior art needlessly

so how I would handle TD.tagged "y" is an open question

since a since character could be a string tag

or it could be a tag for a composite value

Manifold has its own version of let , IIRC.

ooh, that's very promising, thanks

regarding using deref to test future failure, you can use the second and third args to deref to time out and provide a placeholder value if the future isn't ready, then move to the next future

contrived example:

(cmd)user=> (def futs (map future-call (repeat 10 #(doto (rand-int 20000) (Thread/sleep)))))
#'user/futs
(cmd)user=> (do (doall (take-while #{:waiting} (map #(deref % 1000 :waiting) (cycle futs)))) (map #(deref % 1000 :waiting) futs))
(6165 :waiting 7143 1631 :waiting 1091 :waiting :waiting :waiting :waiting)

Hello all. I created a #show-and-tell channel today. (Resend for us time zones) #announcements channel is lovely for libs but I want to be inspired by the systems and products people are building. What are you building with clojure today?  Show a screenshot and tell us about it. Few fun screenshots at #show-and-tell already.

thanks! I will check them out

If I’m applying a filter on a set, what is the time complexity, where the size of the set n?

O(n)