Clojurians Log v2

Hi guys, I have this project https://istd50043.github.io/project#implementation-requirements I'm trying to consider the possibility of doing this solo (so that I can do Clojure full stack) I'm thinking the "Production System" can be done with Clojurescript. "Analytics System" can be done with Clojure. But I also need to do automation scripts for AWS - not sure how possible is it to do this with babashka :o I know might be pretty intense to tackle alone. But am wondering how intense/possible is it :x (Also not sure if this is the best channel for this)

@zackteo If you're not already a member, I suspect the #aws and maybe #aws-lambda channels will be helpful to you. I don't know if there are channels specific to automating ops stuff on AWS.

In that case, I'll move this there

It's a big enough project that different parts of it will likely be best served by asking questions in several different channels but I think some of the core questions about AWS will need answering first and those are almost certainly "non-Clojure" questions. I think it's an interesting project and should be a good challenge.

I think it will be very good for me to attempt to tackle it myself. Especially with groups you end up missing out on learning the different parts. But the wonder is will it be too overwhelming a task that it would be better to take a step back

There are some Clojure-based projects for managing AWS ops stuff (Pallet comes to mind, there's a #dda-pallet but I don't know how active it is) but I'm not sure how active any of those ops projects are. I think people went back to more mainstream tech for that sort of thing?

@zackteo aws is a recurring topic in #babashka - welcome to join and thinking about it

@borkdude will do! 🙂

I ended up rolling my own mini library: https://github.com/ivarref/memoize-ttl

@seancorfield am gonna do the project in Clojure after all! :D I miraculously found 2 peers to join me to form a 3 man team :) so it isnt as ridiculously daunting anymore

What would be the best way to organize such a program in clojure ? I want to concurrently build a set of unique things existing across a cluster of "locations" (DBs in reality). Let's say for simplification that I have an expensive func (get-ids location) which might take a while to return. I have a list of locations, say a few thousands of them, and I want to share the load of calling get-ids for each of them, across, say, 16 workers, and finally get a unique set of values out of this. I have to limit the number of concurrent workers because in effect, each worker opens a connection.

Tried my way around with pipeline but ended up having bloated parts to rebuild the final list of results out of channels that I'm not sure they're done fetching.

https://github.com/TheClimateCorporation/claypoole

that looks great! thanks

just out of curiosity, how would you go about organizing such a thing with the core libs, as in, for an ad-hoc solution for a one time problem?

I would write a pipeline that fetches the location in a thread pool or asynchronously and plug the distinct transducer on the output channel.

I think I went more or less that way, wasn't sure how to ensure my output channel gave me all expected results though

(so I could dump my final list to a file and exit the program)

If you close your input channel after posting your locations, the pipeline-blocking function will close your destination channel.

So when your code detects that the destination channel closes, you can assume that you have the complete set of ids and can dump them in the file.

ah yes, good idea

And if you build your input-channel with to-chan on the collection of locations. you don't even have to bother closing this input channel, it will be done automatically.

would you assume that the chan is closed when you're taking nils out of it ?

yes

yep, feels weird but makes sense, thanks a lot

My first attempt would be something like this

(require '[com.climate.claypoole :as cp])
(let [adresses [address1 address2 address3]]
  (cp/upmap 10 (fn [address] (connect-address address)) addresses))

yep ok, using claypoole then, Tbf it looks simple enough to be pulled in, thanks

nil is not an allowed value in a chan

so if you take and get a nil, you know the chan is closed and drained

what's a good way to express "at least 2" elements in a sequence? I believe count will traverse an entire seq?

@kevin842 https://clojuredocs.org/clojure.core/bounded-count

you can use bounded-count

bounded-count came from one recent release, if you have older clojure, you can use simply (->> x (take 2) count)

oh, cool thanks :) tbh I wish there were a many? in clojure.core that did this; I find myself having to sprinkle this around in a few places for UI-related stuff since we humans like our grammatical numbers

it seems that nobody else needs it

Might also be worth looking at https://clojuredocs.org/clojure.pprint/cl-format

For simpler shell scripts you might consider using #joker and wrapping the AWS CLI tool rather than importing the AWS SDK into your clojure of choice.

example

actually, (count (range ..)) appears to be constant time in cljs, so it seems to always be ￱~￱4x faster to use count instead of bounded-count there -- not that it's meaningfully slow either way, at about 25 nanos for bounded-count according to simple-benchmark.

range implements ICounted so it will be constant time

not necessarily

only finite long ranges

ICounted
  (-count [rng]
    (Math/ceil (/ (- end start) step)))

(cljs)

infinite ranges obvs can't be ICounted

its nonsensical but it works

(count (range))
1.7976931348623157e+308

if "works" = wrong :)

yes. in cljs it seems they are never infinite. ([] (range 0 (.-MAX_VALUE js/Number) 1))

never delved into this before so fun to learn

I mean, technically even in Clojure counting is int-bounded due to Java collection apis right now

Would tell you guys what it returns in babashka, but still waiting for the answer

ah, the clojure version of (range) is (iterate inc 0) which obviously won't implement Counted whereas the "infinite" version of (range) in cljs does

also, the non-long version of range in Clojure does not implement ICounted

as there are many weird edge cases

yeah i remember you saying you spent a lot of work in there

thanks for that tedious work 🙂

user=> (counted? (range 0.0 10.0))
false
user=> (counted? (range 0 10))
true

user> (counted? '(1 2 3)) true user> (counted? `(1 2 3)) false

that I really don't understand.

the first is a persistent list, which is realized (and thus countable, like all persistent colls)

the second is a sequence

sequences are not (necessarily) realized so not necessarily countable

why it's a sequence is pretty obscure and probably should be considered an implementation detail

very interesting. think that's mentioned here https://clojure.org/reference/reader#syntax-quote

bb is JVM-based so same as clj

Hi, I'm trying to test a highly asynchronous test, and found out some bizarre behavior of clojure.test: sometimes, my assertions simply are not registered if they are on another thread. I was able to shring to the following code:

(t/deftest bar
  (t/testing "Why this does not work?"
    (.submit (java.util.concurrent.ForkJoinPool/commonPool)
             ^Runnable (fn []
                         (Thread/sleep 200)
                         (t/do-report {:type :fail, :expected 10, :actual 12})))
    (Thread/sleep 1000)))

The task submitted to the ForkJoinPool, when it calls do-report, prints the failure on my terminal (but not on nREPL / Socket REPL) but does not register on clojure.test that a failure occurs

fail how?

that pool submission code above does not convey dynamic bindings...

guess: do-report uses dynamic bindings?

@mauricio.szabo I think @ghadi is right -- clojure.test is full of dynamic bindings so that's likely to be your problem.

*testing-vars* and *testing-contexts* are the two used by do-report that jump out at me immediately.

Oh well... So I'll need to bring these asserts into the main thread, right?

It doesn't register that there was an assertion, nor register the failure. So run-tests thinks that it's an empty test and returns a success.

That is usually for the best, but you can also look at bound-fn

If you don't put the assertions on the main thread you also have to make sure your assertions on other threads have run to completion before the main thread exits, which can also be tricky

@hiredman yes, I was in fact working on a library to ease async tests in ClojureScript, and wanted feature-parity with Clojure. Seems that it'll not really be possible, unfortunately