Clojurians Log v2

If I have

(defn my-func []
  (async/<!! (async/timeout 100000)))

(async/go (my-func))

<!! is always blocking

I always wandered how that interacts with direct linking and aot compilation. Anything I should be wary of?

to you web stack gurus that I know lurk in here; I have a server giving me HTTP 500: Request timed out occassionally on long-lived requests. where could this be coming from? I have not implemented or configured any timeouts that I know of, and I would expect to see HTTP 408if it was timed out due to, well, too much time passing.

Your HTTP proxy (Apache or Nginx or similar) would be my guess.

it's jetty, but yes, it probably is. just can't figure out why or what I could possibly do about it

If you can confirm that it's your server and not some proxy, then set up a proper logging that will show where that error is coming from. With a proper setup, such questions are answered by the lots 100% of the time.

If it's a proxy that you control, same thing - look into its logging setup.

well, it's an azure app service, so no I don't have "direct" control of jetty, but there may still be logs somewhere

the timing of the drops also differs some over time (seems to be 5-6-7 minutes), so it doesn't look like a set timeout to me

I guess it's possible that Azure doesn't want long-lived requests and it's my job to design around that 🙂

If I want to follow the "Cryptographic Right Answers" in clojure, which libraries should I be looking at for e.g. symmetric encryption and hashing?

(assuming I know nothing about the Java ecosystem)

lvh / latacora are good places to look for advice. https://www.lvh.io/posts/cryptographic-right-answers/ https://www.lvh.io/posts/crypto-apis-and-jvm-byte-types/ . That said, the defacto standard in the java ecosystem is a set of libraries under the bouncy castle umbrella

buddy is used a lot for this

which is based on bountycastle

would try to avoid buddy if at all possible

Swiss army knife design of crypto libraries is a security antipattern

direct interop is the way to go?

you could look at lvh’s libsodium wrapper (caesium) or google’s Tink

Lvh's caesium (clojure lib) is likely worth looking at if you're doing a lot of crypto

If you’re just hashing, direct interop is fine. If you need encryption like AEAD then use something like Tink or libsodium

Basically for encryption you want to use something that doesn’t expose nonce management to the user

yes, that's what I'm looking for

Try Tink

at $previousJob we were using a version of .NET that didnt have AES-GCM implemented and had to cobble together some kind of AES-CBC + HMAC monster which never felt correct. I'll give Tink a look, thanks

(Of course you can do it all with interop, too)

@ghadi you said something about staying away from buddy. wanna go into more details? We use it in my current project and I’d like to avoid any issues

I was surprised to find out that split-with is literally implemented as [(take-while pred coll) (drop-while pred coll)], meaning that it does a redundant amount of work applying the predicate twice for every element that satisfies it. See how the following takes 800+ ms and prints 0 1 2 3 0 1 2 3

(time
  (mapv doall
    (let [xs   (range 10)
          pred #(do (println %) (Thread/sleep 100) (< % 3))]
      (split-with pred xs))))

@qythium I'm sure Alex welcomes an http://ask.clojure.org item for this which could lead to a JIRA ticket, which could lead to a patch, which could lead to a better Clojure future 🎉

If pred calculates hard, it make sence

Changing it would break stateful predicates (not that I know of anyone relying on the number of executions of a predicate for split-with)

Yep I'll post this on http://ask.clojure.org too, just wondering as it seems like such a glaring oversight

If your code depends on stateful predicates, relying on 2 times eval of them, possibly you are doing something wrong 🙂

I don't disagree. Just saying it's a possibility :)

@rutledgepaulv I was thinking about that too, but if anything I'd expect (apply concat (split-with pred xs)) == xs to always hold

and here's that expectation being violated:

(let [xs      (range 10)
      counter (atom 1)
      pred    (fn [x] (< x (swap! counter + 1/2)))]
  (split-with pred xs))
;; => [(0 1 2) (7 8 9)]

Oh, looks like Functor/Monad laws in the begining ))

But yes, Clojure has implicit side effects

But this is not a bad thing, just keep it in mind and don't step on this emergent area - like relying on order of function arguments evaluation etc.

There are more interesting things and leaky abstractions - when -> does not equal to as->

non-crypto expert users cut themselves when using crypto libraries with tons of choices

it was tricky trying to "couple" the two return results together while preserving laziness, but here's my attempt:

(defn split-with*
  "Like `split-with` but evaluates `pred` at most once on each element"
  [pred coll]
  (let [takes (take-while pred coll)
        drops (lazy-seq (drop (count takes) coll))]
    [takes drops]))

buddy is one such library, so is bouncy castle

https://buttondown.email/cryptography-dispatches/archive/cryptography-dispatches-nacl-is-not-a-high-level/

search for the rationale on Google's Tink and other high-level crypto APIs

Of course, at least not to count takes

Interesting! I’ll read through that. Thanks for the link

"misuse-resistance" is a general topic in crypto

Yeah that makes total sense

Do you have any examples of this? Just curious

Especially if pred allways true and takes have to be infinite and lazy )

unzipping into two sequences is not such an easy problem after all 🙂

but then again most predicates don't have a Thread/sleep call in them

They can query http requests 🙂

maybe that's not such a great idea

Yep, but we discussing about possibilities

I don't know, I use split-with on some pretty expensive predicates (eg. graph traversal to find common ancestor)

Of cousre, but for now in Russian video only https://youtu.be/0JAHb35QOYQ?t=781

Sounds reasonable - Heroku kills any request that’s longer than ~1 minute IIRC

In general, I would consider HTTP requests longer than a few seconds an antipattern, but if you can’t control the server there’s not much you can do about it I guess. The “proper” solution would be to perform the task asynchrounously (e.g. by putting it on a queue) and then poll the status of the task / set up a websocket or long-polling subscription that notifies the client when the task is complete

(defn sp-with [p coll]
  (->> coll
       (map p)
       (map vector coll)
       (split-with second)
       (mapv #(map first %))))

Hardweight predicate evaluates once. But beware stateful predicates and chunked implementations of lazy-seq functions.

I'm trying to create a Docker image for arm32v7 (RaspberryPi). Can anybody confirm that the official Docker images (https://hub.docker.com/_/clojure/) are not built for arm?

The docs on that page link to this for arm: https://hub.docker.com/r/arm64v8/clojure/

Yes, I'm sorry I wasn't clear enough. I really need images for arm32v7, as the Raspbian image I'm using is 32bit.

And unfortunately, the images for this platform are two years old (https://hub.docker.com/r/arm32v7/clojure/tags?page=1&amp;ordering=last_updated).

You could try updating/changing the underlying Dockerfiles, but from the official support docs it looks grim for out-of-the-box images, I agree

Yes, that sounds like the quickest route to go

IIRC azure app service has this timeout enforced. They say 230 seconds here: https://social.msdn.microsoft.com/Forums/en-US/05f254a6-9b34-4eb2-a5f7-2a82fb40135f/time-out-after-230-seconds?forum=windowsazurewebsitespreview

Something to keep in mind is those are "official" in the sense that http://docker.com calls it official, not in the sense that anyone from the clojure dev team has had anything to do with it

I say that because those images are not really images for running clojure programs, they are images for building clojure programs, which is why they include variants that include, basically, different clojure build tools

If you are looking for an image to run a clojure program by far the way to do that is by building an uberjar outside of docker, then use a jdk docker image to the uberjar

I'm looking at clojure.core.logic.fd and wishing for a version that can do... less logic programming and more constraint solving/optimization on rationals. I have a set of producers that can produce two things in any ratio, and a demand for each thing. I want to give a bag of constraints like:

(<= supply-1 demand-1)
(<= supply-2 demand-2)
(<= supply-3 demand-3)
(<= (+ supply-1 supply-2) producer-capacity-A)
(<= (+ supply-1 supply-3) producer-capacity-B)

does anything like that exist? I'm new to logic and constraint solving and don't know what to search for.

For solving/optimizing linear programming problems, I might look for something besides core.logic. There's some clojure wrappers of constraint solvers: • https://github.com/levand/prolin • https://github.com/tanders/clojure2minizinc/ It might also be worth finding a good linear constraint solver in java

Does anybody have any suggestion on how to call the function? Ideally I would call it 412 so I could just do (responses/412 "Error") but it’s not possible; I am not very convinced about r412 or generally speaking r{status-code}

precondition-failed?

yeah, http status codes all have textual descriptions

https://httpstatuses.com/

Hmm, good point, I could just use textual names

Also, as a precedent, all ring.util.response fns are named like that (https://ring-clojure.github.io/ring/ring.util.response.html).

I second looking into minizinc, which can be run from clojure via http://jacop.osolpro.com/.

I find I have this pattern too often, I'm curious how to make this cleaner. I have some data, I want to "compile" it, then i want to use the compiled version, but still want to access the data, eg:

(def myregex "x.*x")
(def pattern (Pattern. myregex))

(defn where-pattern-used []
   (...))

Would a let statement know to not keep compiling the regex?
Not looking for something specific to regex, this is generally question for newbie.

why do you think it keeps compiling the regex?

above, i don't think it would, but I find it awkward to use (def's) for so many of these.

ah

I see so that is an example of your work around, not an example of what you are trying to work around

defs of values are good

yeah, seems like "littering" of def's to get around a specific problem.

it isn't

seems like it could be localized better.

why should it be localized?

do you localize the number 5?

if something is a value, has no state, is freely shareable, then def'ing it is good

i suppose there's something like:

(defn compile [data] {:data data :compiled (compiled data) })
(def dc (compile "something"))

What I'm asking really is what are my options here, not "no, def is cool man"

re-pattern is a thing (vs Pattern.) btw

you may want to look at some kind of programatically built pipeline

e.g. you start with a map like {:a 1 :b 2} and have another map that says something like {:c {:depends [:a :b] :f +}} and some function f that given those two maps produces {:a 1 :b 2 :c 3}

https://github.com/plumatic/plumbing#graph-the-functional-swiss-army-knife is the first place I saw that pattern in clojure, but it is kind of old hat by now so there may be some more recent libraries for it, and it isn't too hard to just build it yourself

interesting @hiredman that might be generally useful for me. @alexmiller wasn't trying to solve regex, just general understanding... i thought maybe function inside of a let... but i you can't call those functions.

https://gist.github.com/hiredman/71b71e4e07e7666fe1b9def9a476c765 is an example of what it takes to write that kind of thing yourself (it has the additional wrinkle of doing everything via map instead of just calling functions)

But is "fred" called every fn call?

(def constant "stuff")
(defn x [data] (let [one-or-every-time (compile constant)] ... )

in the graph/pipeline stuff I mentioned, all the results are cached in the map

and if some key depends some other keys, those keys are computed first and cached in the map

https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems.pdf is a good, but much deeper dive in this kind of thing, in a slightly different context

these kind of systems where you have a sort of computational graph with dependencies and you want to build each node in the graph once and re-use it are like a build system like make

Gurobi is commercial, but has much cheaper licenses for educational institutions, e.g. colleges. It can do linear constraint optimization and mixed integer/linear optimization. I am sure there are open source tools with a lot of overlapping functionality -- Gurobi seems to go the extra mile in terms of taking advantage of multiple cores and/or physical machines, if you ask it to.

i this should work? Meaning that (compile to-compile) will only be invoked once?

(def to-compile "...")
(def my-fn-compiled []
    (let [compiled (compile to-compile)]
    (fn [data] (evaluate compiled data))

I can imagine code doing so, they probably had to account for the two-time eval, and it would probably have been easier for them to do a one time eval.

In that expression (compile to-compile) would only be invoked once regardless of whether you use that, or (compile "...") , not because of the first def, but because the last expression is a def. If you reload the file, it will be re-evaluated of course, but I'm guessing that when you say "will only be invoked once", you mean once per time the form is evaluated.

In a situation like that, some people will write the code like this instead:

(def to-compile "...")
(let [compiled (compile to-compile)]
  (defn my-fn-compiled [data]
    (evaluate compiled data)))

Note that in your code snippet, the [] after my-fn-compiled will be ignored, because it is def my-fn-compiled, not defn my-fn-compiled

What exactly you're trying to do? You don't want compile to re-evaluate for the same data more than once? If so, you can use memoize:

(def compile
  (memoize
    (fn [data]
      {:data data :compiled (compiled data)})))

Another option if you want to only reuse some evaluations of it without def, so assuming you want to do so only locally, just use a Closure:

(let [dc (compile "something")]
  (defn foo
    []
    (doSomething dc)))

That's basically the three options you have. Either cache it in a def, or cache it in a Closure with let, or memoize it.

are you re-using that function? If not, maybe just keep it anonymous as the code is more clear then the name.