Clojurians Log v2

Thanks. I see that now.

hey, some time ago I asked in this channel for a debugger that shows the intermediate steps of a function call in the buffer (like the clojure one) for clojurescript, and I got pointed to a tool (standalone) that works with both clj and cljs. But then I hit the slack message limit and I can't find it anymore. Do you recall the name?

hashp or spyscope

this are very interesting things, but the project I remember had a gui you connected to

https://vlaaad.github.io/reveal/ or https://github.com/jpmonettas/flow-storm-debugger

Hi all, I have the following (probably common) problem. I have a sequence (count: 100-300) of maps that need to be filtered by a series of predicates and then (on the two dozen or so ones that remain) embellished with a couple of new keys and values. The issue is that most of the predicates (and some of the embellishments) require REST or GraphQL calls, so are blocking. What would an experienced Clojure developer (of which I'm not) reach for first in this situation to make this run quickly and make the best use of cores/threads? The r/reduce r/fold things say they are for computationally intensive stuff, not i/o blocking. Maybe async/pipeline-blocking, I thought, but it doesn't seem to help much, unless I'm using it wrong.

You forgot to describe, what's the actual problem with blocking? (e.g. performance, sth else)

Sorry, yes, performance: I want to be quick.

Have edited.

is it acceptable for your use case to perform 100-300 requests in parallel? (if not, what's the max)

Yes, I have the threads. This is a command-line/batch tool. In Scala, I would probably do something like that: use futures and a big (100-200) threadpool.

given those requirements I'd simply use pmap in such a way that each item in the 100-300 sequence gets its own thread, with its own filter->embellishment steps happening in each thread pmap (and future, send-off) are perfectly fine for IO-bound workloads even if some other options may look fancier

But pmap doesn't allow you to specify the concurrency level, does it?

if you pmap a seq of 300 items you get a parallelism of 300 threads, which is desired in this case ...just make sure to (vec (pmap, to ensure such parallelism, since pmap is lazy

thanks @cursork I think it was flow-storm!

Thank you. I'll try that.

@vemv I don't think you assessment is correct. I just experimented, and I couldn't get more than 30-something threads. With hundreds of items and long sleep times. IIRC it's explained by the chunking. pmap derefs its futures by chunks which have a limited size.

@eamonn.sullivan FWIW I just found this in my notes: https://github.com/TheClimateCorporation/claypoole As per noisesmith: > there's a version of pmap in the claypoole library that's better [than Clojure's pmap] for compute tasks > the advantage of claypoole over just mapv future / deref is it lets you define a specific parallelism (if coll has enough elements in it, it will grind the jvm or even OS to a crawl as it loses all its resources to thread context switch overhead)

pmap uses future which uses a CachedThreadPool. It only grows if needed, which explains what you probably are seeing. If a given pmap step can be performed using a thread that was used-but-then-released from a previous pmap step, it will.

@vemv That's why I mentioned long sleep times. Try to use pmap with a huge collection and a blocking function, and see how much JVM threads you get. The amount will be between 30 and 40.

Pmap has some logic in it to not grow more than ncpu+2 tasks, it's enforced by the way it realizes the sequence and futures and not the thread pool that future spawns into. Look at the source

Parallelization capped near ncpu is intended primarily for cpu bound work and not blocking io, though you can certainly use it for either but you won't achieve the kinds of throughout you might be able to using larger number of threads if your work is primarily blocking

that's true @rutledgepaulv, I had only checked out future in the source but not that logic. future and send-off seem simpler then (the same CachedThreadPool will be used, but without a cpu-related limit) ...you must be sure that a reasonable amount of threads will be spawned though. 300 is OK, 10000 starts to be dangerous

Or just use a fixed executor directly and skip the clojure.core functions

If you’re actually concerned about performance, I would recommend reducing roundtrips by batching queries and then merging results in a post-processing step.

Yeah, I don't have control over these APIs (an internal one and Github's REST and GraphQL). I'm batching as much as I can (doing Github searches when filtering on topics, for example), but I have limited leeway on this side.

I'm writing a CLI querying tool, to help my teammates find which one of our hundreds of microservices and lambdas are using a particular dependency or runtime environment (e.g., version of node). This involves getting everything from an internal registry (which has things like whether it is lambda or an EC2, or what version of CentOS) and then poking Github to answer queries about particular dependencies, language or topics.

My initial attempt (single threaded) took as long as two minutes to get an answer. My second attempt, using pmap and async/pipeline-async, takes 15-45 seconds. I think I can make it faster, given that just about everything is blocking i/o.

(currently trying the fixed executor directly, but actually hitting Github API rate limits, so there's probably a ceiling on how much more I can squeeze out of this.)

is there a version of in-ns that functions like let? Something that temporarily overrides the namespace for a given expr like:

(with-ns (symbol "new-ns")
  (do (println "the current namespace is: *ns*)))

(defmacro with-ns [ns form]
  `(let [nsp# (.name *ns*)
         _# (in-ns ~ns)
         res# ~form]
     (in-ns nsp#)
     res#))

(with-ns 'clojure.core.reducers
  (prn (.name *ns*)))
;; => clojure.core.reducers

Since your original question was "what do people go to", I'd add core.async, since in my experience there's always a good solution you can come up with using those building blocks (pipeline for this one?). If you were looking for info tuning java threadpools, I will be very quiet 🙂