Clojurians Log v2

I had a day off which I used to re-learn Clojure. I’m converting some GitHub bots that I wrote in Go to Clojure and it has been pretty interesting so far.

Hi! is there a standard clojure function which does the equivalent of (spec/conform (spec/cat …) […])? This is so useful I assume there must be a similar thing outside of spec that people use.

@denis.baudinot I would say that is the "standard clojure function" for it -- but it depends on exactly what spec functionality you're referring to there...

If you mean "take a list of names and a list of values and make a hash map" then there's zipmap but it doesn't do any conformance on the actual values:

user=> (s/conform (s/cat :a int? :b string?) [42 "Hello"])
{:a 42, :b "Hello"}
user=> (zipmap [:a :b] [42 "Hello"])
{:a 42, :b "Hello"}
user=> (zipmap [:a :b] ["Forty Two" 13])
{:a "Forty Two", :b 13}
user=> 

This is exactly what I meant, thank you for the clarification. (I like the spec version more when looking at this and thinking about it though. Conforming values in general seems to be a really useful thing to do.)

Thanks! It works. Much appreciated.

Hi all. I have a weird failing test that may be the result of AOT compiling and only occurs on a test first run, after deleting the contents of the classes/ directory. I have a defrecord containing numbers, strings and a java.time.LocalDate. Equality testing works fine based on the value of the LocalDate in local testing (even though the objects do not have the same identity) AND works fine on 2nd and subsequent runs of a test that serializes then deserializes the record into a key-value store (mapdb). The problem is on the FIRST run of the test which somehow decides that two LocalDates with the same value (but different identities) are not equal. It feels as if it is something to do with a cache but it simply shows up my ignorance of the process of AOT and its interaction with running with JIT compiled code. EDIT: Actually fascinating as the problem only rears its head if I run tests by executing AOT on the same clj run as the test - and doesn’t happen if I manually run AOT first and THEN the test run.

I have a very beginners, possibly beginner functional, question: I have a collection of a couple hundred things. I have to filter those things using a REST call (to get an answer to a boolean question that I'm filtering on). There's no bulk version of the REST call I need to make, but making a couple of hundred sequential HTTP requests in a row is taking too long (`(filter is-it-a-thing? coll)`). What tool would a more experienced Clojure person reach for in this situation?

Hello! are there any naming conventions for vars in Clojure beside earmuffs *var* . I did stumble upon something like this +sub-name+ and unsubscribe* is it something common or just up to the author?

Looking at clojure.core.reducers/fold and family.

I think +foo+ is not that common. foo* is, usually to indicate a variation or a more low level alternative

+foo+ is often used in other lisps to indicate a constant value, but not really in clojure

I too stupid for fold, but result1 is a lot faster than result2.

user> (def result1 (remove nil? (pmap #(if (is-a-thing? %) % nil) things)))
#'user/result1
user> (def result2 (filter is-a-thing? things))
#'user/result2
user> (time (count result1))
"Elapsed time: 7988.541086 msecs"
73
user> (time (count result2))
"Elapsed time: 29946.487776 msecs"
73
user> 

Or another example:

user> (time (def result1 (into [] (filter is-a-thing? things))))
"Elapsed time: 23648.239951 msecs"
#'user/result1
user> (time (def result2 (into [] (remove nil? (pmap #(if (is-a-thing? %) % nil) things)))))
"Elapsed time: 5181.302774 msecs"
#'user/result2
user> 

need clever way to iterate through a sequence. I have sequence, input-sequence and a function, f, which I can call on each element of the sequence. The problem is that f is expensive to call, so I want to call it as few times as possible. I can suppose that f always returns one of true, false, or :dont-know I want to return false if the sequence is empty, without calling f I want to return true if the f returns true on any element of the sequence. I want to return false if f returns false on every element of the sequence. otherwise I want to return :dont-know If I create a sequence like this (map f input-sequence) this sequence will be lazy. Am I guaranteed that f will only be called when I access that element of the output sequence? I.e., could first check the output sequence to see if it contains true, and if not, then check to see if it only contains false

laziness in clojure doesn't guarantee 1 at a time laziness

Ahh, so I need to do my own interation then to be sure?

but you can (map f (unchunk sequence)) to turn it into 1 at a time

the usual definition of unchunk would be

(defn unchunk [s]
  (when (seq s)
    (lazy-seq
      (cons (first s)
            (unchunk (next s))))))

alternatively you could reduce over the collection with a boolean accumulator, and short circuit using reduced to return true

yes reduce/reduced or even loop/recur will do the trick of course, but the logic is contorted. My suspicion is that the code is more concise If I can just check membership. without resorting to inline iteration.

if I understand correctly, something like (reduce (fn [_ el] (case (f el) true (reduced true) false false :dont-know (reduced :dont-know)) false coll)

I personally find this type of operation more natural to express/understand in terms of folds like above

@bronsa, won't your function return :dont-know if it encounters a :dont-know? It should return true if there is a true anywhere in the list, even after a :dont-know.

ah, yes :)

(reduce (fn [b el] (case (f el) true (reduced true) false b :dont-know :dont-know) false coll) this then I think?

(let [values (map f (unchunked intput-sequence))]
  (cond (some true? values)
        true

        (every? false? values)
        false

        :else
        :dont-know))

¯\(ツ)/¯

it works I guess, tho it's more expensive than the single scan using reduce

FWIW forall would (every? false? values)

and member would be (some true? values)

good suggestions. I normally find forall more readable, but in this case (every? false? values) is more readable.

and them (some? true? values) or (some true? values) I always mix up some vs some?

some is the pair of every? :)

some? is non-nil

bit confusing but if you understand why they're named that way, it makes sense

i.e. some is not a predicate, it doesn't return true/false, but the first non falsey value

in other languages some is better named find-first or things like this

is some the same as (first (filter ...)) ?

no

more like (first (keep (if keep returned on truthy rather than non-nil)

let's say some is (first (filter identity (map f :)

back to the previous question. what's more readable, the use of map depending on the lazy nature, or an explicit but very clever reduce which computes the value without dependence only laziness ?

I personally find reduce significantly more readable here, and not particularly clever/obfuscated

I'll give you that your version is more "declarative"

but this sort of folds is super natural in FP so after a bit of getting used to it, it becomes second nature

but really, it's up to personal preferences if you're not too fussed about the extra linear scans (which you probably shouldn't be)

the one thing I'd say tho, is I'd advise you to prefer functions available in the clojure.core stdlib vs non standard macros that achieve the same thing (like that forall thing)

makes it easier for other people to understand your code

Don't get me wrong. I love reduce, and I use it a lot. However, in this case (maybe I'm wrong, thus asking) am I trying to be too clever. For example, you're first implementation was wrong as well. Is there another bug in there somewhere. For example, does it really do the correct thing on the empty sequence. I suppose in either case I need to write unit tests to make sure, so perhaps corner cases should be considered moot.

well, my first implementation was wrong cause I didn't read the spec well enough :) and re: empty sequence, yes, the behavior of reduce is defined to return the init value if the sequence is empty (from the docs:

If val is supplied, returns the result of applying f to val and the first item in coll, then applying f to that result and the 2nd item, etc. If coll contains no items, returns val and f is not called.

but again, readability is subjective. use whatever feels easier to you, I still find your version very readable

actually I was under the impression that reduce calls the function with 0 arguments if the sequence is empty.

if the init val is not supplied

ahhhhhhhh...

protip: forget about the non-init arity of reduce

in that case I have several pieces of code I can clean up. Because I sometimes pass a mult-arg version of fn to reduce just for the empty-sequence case.

that's something that always annoyed me about clojure's reduce. Now perhaps I don't need to be annoyed.

it's not clear from the doc, whether if I supply a singleton list, AND an init value, whether the function is called. (reduce (fn [acc item] (+ acc (count item))) 0 '("a" "ab"))

it is

If val is supplied, returns the result of applying f to val and the first item in coll

doesnt' matter how many elements are in coll (unless coll is not empty)

hang on, in that example '("a" "ab") is not a singleton list

it has two elements

yes exactly. reduce (as it seems to by experimentation) is supposed to sum the counts. which it does, even if the sequence is a singleton '("a")

yeah sure

It does the right thing, but the docstring could be clearer in my opinion.

clojure docstrings are notoriously terse

but I find this behavior clear from the docstring tbh

that fact that in some cases clojure's reduce calls the function with 0 arguments to get the unit value is bizare and clever. I don't know of other functional languages that do this. well I only know about Common Lisp and Scala.

Cursive and cider both have clojuredocs available and built in for more verbose info with examples

yes, the no-init behavior of reduce is unfortunate and I'm pretty sure if the clojure/core team were to design clojure from scratch, it would not be in the language

I always use the web pages such as. https://clojuredocs.org/clojure.core/reduce

it's an unfortunate wart of clojure being backwards compatibile (which is an asset most of the time)

it is nice for functions like + and *

anyway, I'm happy to realize that I don't need to include the 0-ary version of the function every time when I'm not certain about the length of the input sequence.

thanks for that revelation

😁

QUESTION about the distinct function. The documentation does not define anything about the order of the elements in the returned sequence. Or even whether it always return an = sequence if there are no duplicate elements. Can I depend on the order returned. Or do I need to write my own version of distinct if I care about the order?

A few years ago I worked for a company which maintained a proprietary lisp implementation. At some point we improved the hashing function of the standard hash table to make it faster, or more dispersive. Enough of our customers complained that their code was breaking. Some of their code was depending on the iteration order of the map-over-hash function that we had to revert the changing, leaving in place an inferior function. In retrospect, the original hash iterator should have been intentionally non-deterministic.. But alas....

distinct retains order of the original sequence

thanks @alexmiller is that a feature that's guaranteed ? For example it would be faster (at least on large sequences), if it added the elements to a set, and then extracted them from the set, thus not maintaining the order. Also maintaining the order is not really a specification. For example there are two ways to uniquify (1 2 3 2) maintaining order. either (1 3 2) or (1 2 3) . one maintains left-to-right order and the other maintains right-to-left order.

I had heard that Clojure's behavior of reduce was influenced by Common Lisp's behavior with a similar function.

I would consider it to be implied by the current docstring and likely to break people if it changed

so I can't imagine it would change

http://www.lispworks.com/documentation/HyperSpec/Body/f_reduce.htm

if you want a different behavior, you'll need your own impl

current impl always keeps the first seen element

Thanks for the info.

Returns a lazy sequence of the elements of coll with duplicates removed.
Returns a stateful transducer when no collection is provided.

^^^ @jimka.issy

SBCL common lisp tries to maintain order, and is fast for short lists. Allegro Common Lisp makes no promises about order, and is faster for large lists. Allegro maintainers are more interested in Big Data anyway. So I'm not sure which one is really better. There are lots of small lists in the world.

the Common Lisp reduce function is a lot of functions rolled into one.

If I recall correctly, you can specify right to left traversal, or that only a sub-range be traversed, and lots of stuff. in my opinion it's too much to think about.

I mention it because its documentation explicitly mentions that the function you provide to it can be called with 0 or 2 arguments.

As soon as one element causes the supplied function f to return true, I would guess he wants to avoid calling f again.

Oh yeah, my bad, I see what you mean 👍

As soon as an element is reached that causes f to return true, there is no need to continue evaluating.

if you run (defrecord Foo []) twice, you get two definitions of Foo, and they have the same name and definition, but are not =

creating instances of Foo, then aot compiling its ns, then creating Foo again, will create two totally different Foo

Hey, I want to add a new namespace to my project. I am using leiningen with Intelij. I checked the libary with https://clojars.org/java-http-clj , than added to the :dependencies in the project.clj and it also presented in the Leiningen dependencies tree. But when I am adding it to my core.clj :requires and reload the core file to the REPL I am getting an error of FileNotFound. Is there other place I need to add?

hmm.. i forgot about that part. But now I seem to remember that it is indeed not called with 0 arguments if you provide an :initial-value.

what are you adding to the require part of your namespace?

and want to point you to the documentation of the library: https://github.com/schmee/java-http-clj#examples

The description of the behavior of when the function f is called with 0 args is pretty close between the Common Lisp Hyperspec and Clojure's reduce functoin.

I did use the documentaion of the libray, and yet getting an error only when adding

[java-http-clj.core :as http]

CompilerException java.io.FileNotFoundException: Could not locate clojure/spec/alpha__init.class or clojure/spec/alpha.clj on classpath., compiling:(java_http_clj/core.clj:1:1) 

Can someone confirm if I can invoke (use-fixtures...) multiple times (profitably) in a test? Basically, I want to register two fixtures like

(use-fixtures :once (fn [f] (start-db) (f) (stop-db))  ; wrap the whole suite, I want to get one connection for the test
(use-fixtures :each (fn [f] (f) (clean-up-test-data))) ; after each test, make sure created data is cleaned up

Oh. Yikes. I need to AOT because the class is used by an external java library. But that definitely explains some screwy things I’ve been getting such as “x can’t be cast to x” where x=x after even reloading the REPL.

Best way to find out is to just try it! (Spoilers: yes you can do that)

Thank you for the explanation.

That namespace clojure.spec.alpha requires Clojure 1.9 or later. Are you using Clojure 1.8 or earlier in your project?

Sure, I understand, but I didn't want to go down a road where I start questioning the concept because PEBCAK. Now that I know I'm on the right path, I can proceed with confidence! Thank you

if you need AOT, definitely do it as a separate step and be sure the results of it are loaded before creating instances

and if you need to reload namespaces that use defrecord / deftype etc. you need to also destroy and recreate all those instances

which is part of why stuartsierra/component and weavejester/integrant libraries are so useful - they automate this process

I get that error message you show as well if I try to require the java-http-clj.core namespace from inside of a project that uses Clojure 1.8.0

Ah! That makes sense thank you very much!

@andy.fingerhut update in your project.clj to be [org.clojure/clojure "1.10.0"] than you will not have that error

it doesn't work for the same 'type' of fixture

(use-fixtures :each (fn [f] (println "not here") (f)))
(use-fixtures :each (fn [f] (println "here") (f)))

(deftest a
  (is (= 1 1)))

(use-fixtures :each
              (fn [f] (println "here") (f))
              (fn [f] (println "and here") (f)))

What's a relatively painless way to do a PAAS hosting of a docker container running a JAR built from a Clojure project? Not looking for scale or configurability, just something simple that I can hit with HTTP requests. I looked at Heroku, but looks like a lot to learn and before I start I want to make sure I'm not going down a blind alley, or see if there's a better alternative.

I have a little open source Clojure chat bot running on Heroku, if a worked example is helpful: https://github.com/pmonks/futbot It runs on the “hobby” tier ($7/mth), since I didn’t want it getting quiesced when “inactive” (which it wasn’t, but I was concerned that Heroku wouldn’t detect any activity since the chat bot doesn’t have any traditional web traffic). That said, I don’t use Heroku’s Docker capabilities for this - I just deploy an old-skool “dyno” (which is built by Heroku).

It depends on what you're looking for. I've been using $5 digital ocean servers as well as the equivalent amazon instances and just run the jar directly

amazon also has options like like lightsail and elastic beanstalk

Thanks, it sounds like you and @pmonks just skip the docker step. Maybe I should try without for now. I have very little idea on what the options or best-practices are.

Yeah I’m not going to claim that what I’m doing is best practice, but it works well for me at this tiny “hobby” scale.

I basically don’t think about Heroku much at all, now that it’s setup - it’s all just source control (GitHub) shenanigans, some of which cause a new deployment (i.e. commits to my main branch).

Is there an established naming convention for “pure” namespaces (i.e. those that contain nothing but pure functions) vs “impure” namespaces (that contains state, for example)? If not, any suggestions for such a thing? e.g. a ! suffix for impure namespaces - my.impure.ns! The background is that I (try to) make as many of my namespaces as possible “pure” in this way, with state handling centralised into a few stateful namespaces, and there are times where it would be nice to know which is which, simply by looking at namespace names (e.g. filenames), and also figure out whether I’ve accidentally polluted one of my pure namespaces...

what makes a namespace impure?

e.g. (def state (atom {}))

With code in that ns swap! ing the value of the atom.

I don’t think docker would be necessary for anything that is small to medium sized and/or doesn’t have very frequent updates for java deployments right?

Basically any ns that contains any fns that don’t rely solely on their arguments to do their thing.

using swap! at the top level or just using swap! somewhere inside one of the functions?

The latter.

what if the swap! is using an argument passed to the function and not a global atom?

And to be clear, it’s not limited to atoms.

A function that declares an atom locally (e.g. in a let) could still be pure, no?

I do this for interop with nasty mutable Java code sometimes, for example, but the fn itself is still pure.

also is my assumption correct that the JVM already abstracts from the platform? You can bundle it so it includes all the dependencies in a single jar is what I know. I did my last java production deployment 10ish years ago so idk.

what's the filename look like for a filename namespace with a ! in it?

Yeah for my Heroku I build and run an uberjar. I tried running from source, but for some reason that more than doubled the runtime memory requirements, which blew through the (small, strict) amount of memory Heroku gives you (512MB for a hobby-tier dyno).

Haven’t gotten that far yet, though (in-ns 'my.impure.namespace!) appears to work just fine. I’m open to other suggested conventions, of course - indeed that was my original question! 😉

in-ns is extremely rare in my experience. usually ns is used at the top of namespaces and require is used to reference code in other namespaces

Ok just verified that (ns my.test!) in file src/my/test!.clj works, at least on MacOSX.

I understand that - I’ve rarely if ever used it before. I was simply using that at the REPL to quickly check it. No point creating a foo!.clj file if (in-ns 'foo!) doesn’t even work…

Again, let’s not get hung up on the specific convention of using a ! suffix on the namespace - I’m more interested if anyone has used any convention for indicating pure vs impure namespaces, and if so what that might be.

yea, I'm not sure how that would interact with :gen-class

I'm also not sure pure/impure namespace is a clear distinction

and there's some kinds of impurity that are "acceptable" (ie. interning a var via def)

memoization is another type of impurity that is often "acceptable"

Right - concrete vs abstract side effects, to paraphrase Bertrand Meyer.

Let me ask you though: do you think the notion of a pure fn is, or can be, well-defined?

Because if so, then all I’m referring to by “pure namespace” is a namespace that contains nothing but pure fns (and I’d love to hear of a better label, if such a thing exists!).

I think it can be defined well enough. what kind of impure namespaces do you want? the only kind I can think of is if it's necessary for clojurescript interop

otherwise, I generally expect all namespaces to be pure

if you just make all your namespaces pure, you don't need a naming convention

or at least, idempotent when loaded

another example is when you have a global registry, which can be considered "acceptably" impure in some cases

So here’s one concrete example - not the only one, probably not even a good one - but it’s the one I’m most familiar with. 😉 I have a small chat bot (I happened to link it just above, fwiw) that runs as a standalone command line app. The bootstrap code for that app reads a config file and calls some REST APIs, and then caches that data away (in memory), and starts a bunch of timed jobs (functions that run at specific times throughout the day). Those “job functions”, and all of the underlying functions that they call, are pure - they receive all of the data they need to do their thing as arguments, and they don’t read or persist any state anywhere else. The “bootstrap” code however, is far from pure - it reads all sorts of things from the greater environment (config files, internet services), and then squirrels it away for later. Now I’ve tried hard to keep that “bootstrap” code as small and isolated from the rest of the (pure) codebase as possible, but at times I’ve let some statefulness accidentally creep in, or opened the wrong file because I’ve been away from the code for some time, or whatever. That’s where a naming convention would be helpful.

I agree, but take a look at https://clojurians.slack.com/archives/C053AK3F9/p1605219494143300?thread_ts=1605218686.138600&cid=C053AK3F9 for a concrete example of where impurity necessarily bleeds in.

I would move all the bootstrap code into functions and only call that code from a main function. there are also libraries like https://github.com/stuartsierra/component that can help

then your namespaces are pure again

last time I checked out Heroku, which is a while ago, I got turned off by the fact that you can’t write to files. Most small web applications have some sort of file storage requirement like a SQLite db, some primitive logging or something like that.

Yeah I use mount , which is similar in intent to component, but lighter weight.

But I basically consider the state that mount creates as being equivalent to an atom, and so don’t let those (global) vars creep into my pure functions anywhere.

Instead I pass everything around, and keep references to the mount-managed state to the “impure” namespaces only.

For one thing it makes testing substantially easier (and yes mount supports testing, but it’s even easier to test without having it there at all).

Oh and the impure bootstrap logic is (mostly) within my -main fn (or at least the same main namespace).

But still, I’d like to be able to more quickly / easily differentiate the (impure) main ns from my pure nses.

a function that deals with state doesn't infect a namespace

as long as loading a namespace doesn't have any unacceptably impure side effects, then I would consider the namespace itself pure for all practical purposes. no?

Perhaps “pure” is a misleading label then.

The differentiating criteria is “can I call any fn in this namespace, without having to setup state anywhere else first”.

When using component / mount and similar libraries, fns that directly consume the state created by those libraries no longer meets that criteria. i.e. you have to have those libraries create some form of state first (even if it’s just hardcoded “test data” type state).

ah ok. I was thinking of something different. I have marked functions with side effects using the ! suffix

Right, which is why my first naive thought was to use that same suffix at the namespace level. 😉

Obviously I’d much rather adopt an existing convention, if such a thing exists!

yea, I'm just not sure if namespace, rather than function by function, is the right granularity

like if a namespace contains both, it's still fine using a pure function from that namespace

Good point.

additionally, if in a new version, you want to add a function that is impure to a namespace, what do you do? do you create a new namespace to avoid changing the namespace name? that doesn't seem worth it if it otherwise logically still belongs

My speculative thought would be that that would indicate a design problem.

I'm not opposed to trying new stuff, but just throwing out some considerations

Yep - and I greatly appreciate it!

Yeah could be. My chat bot has (so far) remained mercifully free of any persistent state that it can’t resurrect from 3rd parties, which has greatly simplified the runtime environment.