Clojurians Log v2

transit is designed for wires and should definitely be preferred for that use case

in particular, it's designed to work over json and many languages (critically javascript) have very fast native json parsers. Also, it has caching built in for repeated keys etc so will generally be significantly smaller for common use cases (passing many maps with the same keys) and thus also faster.

edn is designed for places humans read and write data (like config files). transit is also designed to be more extensible in efficient ways - edn has tagged literals but they're not quite as good for crossing machine/language boundaries.

Wow thanks, sorry I somehow missed that.

Ah, it was the point on tagged literals that I was thinking of. Thanks!

https://clojuredocs.org/clojure.spec.alpha/gen > Note that parent generator (in the spec or overrides map) will supersede those of any subtrees. Is there a better way to have a custom generator in a subtree than doing nested gen overrides?

;; Example

;; We love wasteful packaging so we want to create a ::crate that will always
;; contain a ::big-box. We want each ::big-box to contain one or more
;; ::small-box. In each small box, we want to have one Light Bulb.

(s/def ::crate (s/keys :req-un [::big-box]))
(s/def ::big-box (s/coll-of ::small-box :kind vector?))
(s/def ::small-box (s/coll-of string? :kind vector?))

(let [gen-override
      {::big-box
       ;; Override so ::big-box is not empty
       #(gen/not-empty
         (s/gen ::big-box
                ;; And that each ::small-box in ::big-box only has a Light Bulb
                {::small-box (fn [] (s/gen (s/coll-of #{"Light Bulb"}
                                                      :kind vector?
                                                      :count 1)))}))}]
  ;; Generate a ::crate with our nested overrides
  (gen/generate (s/gen ::crate gen-override)))

;; =>
{:big-box
 [["Light Bulb"]
  ["Light Bulb"]
  ["Light Bulb"]
  ["Light Bulb"]
  ["Light Bulb"]
  ["Light Bulb"]
  ["Light Bulb"]
  ["Light Bulb"]]}

Not sure if this is the right place to ask this, but here's an oddity that greatly confuses me.

I understand what is going on here:

> ('foo {'bar 5 'foo 3})
3

But what is up with this behavior:

> ('* 3 2)
2

Is it trying to look up the symbol '* in the "map-like thing" 3, and when it doesn't find it, returns the default value of 2?

This just caused some massive headaches when accidentally wrote '* instead of *', but luckily stumbled into the problem.

Yes, that is the explanation for that behavior.

Why is it happy to treat 3 as a map-like thing without throwing an error? I guess that's the part that really baffles me.

I do not know the reason, but I think it is for the same reason that (get 3 '* 2) is happy to treat 3 as a map-like thing.

It has been that way in Clojure I believe since its beginning, and a JIRA bug was files in 2012 suggesting such expression throw an exception. Rich Hickey responded in 2014 that such a change would be a breaking change, and the Clojure core team values backwards compatibility very highly. I know that doesn't explain why it was like that in Clojure's original versions -- again, I do not know the reason, just giving you a little bit of history that this wasn't a recent thing, nor can I find anything in Clojure FAQ nor http://ClojureDocs.org entry for 'get' that gives a reason: https://clojure.atlassian.net/browse/CLJ-1107

If it's any consolation, I think pretty much everyone new to Clojure asks this... 🙂

Symbols and keywords look themselves up in their argument and it's actually very convenient when writing generic code because you don't have to worry about type checking when you simply get nil from an "inapplicable operation".

It feels like this question should be added to https://clojure.org/guides/faq in some form -- @trhtom do you think you could formulate your question in a form suitable for that page? I'd be happy to help turn it into a contribution for that page (I just recently contributed the ?/`!` FAQ there).

Sure, I'd be happy to.

FWIW, I've been using Clojure for 10 years and just discovered that (get 3 5 :default) is happy to treat 3 as a map-like thing. @andy.fingerhut I appreciate the historical context!

Feel free to DM me to chat about it (tomorrow -- it's a bit late tonight for me to coherently process that).

Looks like this will happen with any type of value that isn't a thing that expects to be used for lookup. My student who ran into this issue found the source code, and if none of those conditions are met, it just returns the notFound value: https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/RT.java#L789-L820

Yup, it's a very deliberate design decision: if the type is something you could "get" a thing from, attempt the lookup (and return notFound if it isn't there), else return notFound.

But it surfaces in a number of expressions and that's the part I'm not sure how to formulate as a "frequently asked question".

The crazy part is that it makes the one-character typo of ('* 3 2) behave in a totally unexpected way. I totally get the two parts making sense independently (symbols can be used as functions for lookup and get returning notFound if you can't get from that type), but the combination is lethal.

As you can see from the JIRA issue, at least some people wish that get being called on a non-associative first arg were considered lethal (i.e. throw an exception instead of returning nil) all by itself, but that is unlikely to ever be changed, given backwards compatibility, plus whatever the original design reason was.

every time I see this familiar question, i remember

(get get get get)  
;;=> #function[clojure.core/get]

I’m looking to build a “job queue”. It’d poll a db for work (I’m doing that with http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/ScheduledExecutorService.html in java), and then fire off work. The work is mostly file imports, that are safe to be run concurrently. What’s the best clojure threading primitive for this type of work?

I care that jobs finish, and their finishing status. but i’m not sure how much more I really care about them while running.

I could see using core.async for it, but I don’t think I have that much of a pipelining need. Futures seem like they might work, but almost seem to barebone — I feel like I’ll be polling them for status, whereas I’d rather react to when they’re done.

@ryan072 I borrowed this bit of code from CLJS to run an amount of tasks in parallel (to fully utilize CPU cores) and wait until they're done: https://github.com/borkdude/clj-kondo/blob/51db4e172478c369208ed90db4b7c29d94a8cbce/src/clj_kondo/impl/core.clj#L209-L233 Not sure if that's similar to your problem

Each of the parallel tasks pick work from a queue, until there is no more work, then they lower some countdown latch.

thanks, that does help. Gives me some confidence on rolling my own, and a some good java primitives to look into

HI, I've got an error saying java.lang.NoClassDefFoundError: manifold/bus/IEventBus when running my backend service using uberjar profile, which configed in project.clj as :profiles {:uberjar {:main hello-ring.main, :aot :all} . Finally I found the reason, the class loader of manifold/bus/IEventBus`` has a hierarchy as below

#object[clojure.lang.DynamicClassLoader 0xc7aac7c clojure.lang.DynamicClassLoader@c7aac7c]
#object[clojure.lang.DynamicClassLoader 0x2186d797 clojure.lang.DynamicClassLoader@2186d797]
#object[sun.misc.Launcher$AppClassLoader 0x23fc625e sun.misc.Launcher$AppClassLoader@23fc625e]
#object[sun.misc.Launcher$ExtClassLoader 0x50313382 sun.misc.Launcher$ExtClassLoader@50313382]

I think you'll need to share more to understand what you're seeing. where do IEventBus and the impl come from? how is this app run? are you using java -jar with the uber jar to launch that main class?

I got this error either run the program through java -jar or through repl .

I'd like to get the line and column number of every element in the below data structure. {:a {:aa 2} :b {:ba 0 :bb 3}} So the first { (the whole map) has coordinates 0 0, the :a has 0, 1 {:aa 2} has 0, 4, 3 has 1, 15. Is there anything in clojure or the ecosystem that will help me do this?

Not directly, but you could write a macro for that that analyzes &form and its metadata:

user=> (defmacro form-meta [] (meta &form))
#'user/form-meta
user=> (form-meta)
{:line 1, :column 1}

My approach would probably be use to use https://clojure.github.io/clojure/clojure.zip-api.html Would you need a need it to return a new data structure or adorn the passed in data?

oh wait, you're looking for line and character positions?

if you use a LineNumberingPushbackReader to read it, you can get some of that from the reader (in metadata)

the maps can carry metadata but keywords and numbers cannot so you won't get those

I believe there are other tokenizing parser libs for Clojure that can though, just not sure which is the best recommendation

Thanks!

But keywords can't have metadata, so any method that relies on it to return position information won't tell you where the keywords are

I wonder if any libraries are able to work around that limitation in some way? I will have to investigate tomorrow. https://github.com/Engelberg/instaparse

presumably, libraries like https://github.com/borkdude/rewrite-edn and https://github.com/lread/rewrite-cljc-playground

The only way I can think of to get position information for things where Clojure does not support metadata, e.g. keywords, numbers, strings, is to return NOT a Clojure collection that is equal to what clojure.edn/read would return, but instead to return some kind of 'parse tree' representation that had an explicit line/column data for each object of the parse tree.

For example, if when reading {:a 2} you tried returning a value x such that (= x {:a 2}) was true, then there is no place "inside" of there to put the line/column info of the number 2. If you instead return a value y such that (= y {:a 2}) was false, you have many choices for how y represents the string that it read and its parts, and many choices for how the line/column info is embedded in there.

@borkdude wrote https://github.com/borkdude/rewrite-edn very recently, so should have fresh on his mind whether it supports line/col info on all sub-parts of the returned data, or not.

@endrebak85 rewrite-edn is based on rewrite-cljc and rewrite-cljc can give you that metadata There is also https://github.com/borkdude/edamame which will parse directly to sexprs with adding location metadata when possible. Using a post-process step you can also have metadata on numbers etc when you coerce them in something that can have metadata. See this test: https://github.com/borkdude/edamame/blob/ba93fcfca1a0fff1f68d5137b98606b82797a17a/test/edamame/core_test.cljc#L306

I'm trying to extend this interface: https://github.com/eclipse/lsp4j/blob/release_0.8.1/org.eclipse.lsp4j/src/main/java/org/eclipse/lsp4j/services/TextDocumentService.java#L91-L94 like so:

(reify
  TextDocumentService
  (completion [position]))

Can't define method not in interfaces: completion

Apparently I wanted proxy, my bad!

Did you need proxy because of the default method?

the completion method in your reify should take 2 args - a "this" arg and then the args on the method

(reify
  TextDocumentService
  (completion [this position]))

you probably don't actually need that arg, so fine to replace this with _ in that case

prior art (defining a type rather than reify though): https://github.com/snoe/clojure-lsp/blob/master/src/clojure_lsp/main.clj#L104

@alexmiller D'oh! yes.

Oh, I didn't know clojure-lsp also used Lsp4j, convenient as a reference!

I think Alex got it, it's because I forgot this

I was going to take a stab at a runtime-LSP for Clojure, but LSP4J feels like the worst of Java 😁 Maybe I'll leave that for a more determined Dominic in the future.

Someone asked me a beginner question that turned out to be deeper than I expected.

(defn a [l]
  (prn "a:" (type l))
  (prn "a:" l))

(defn b [l]
  (prn "b:" (type l))
  (prn "b:" l)
  (a l))

user> (b (list 1 2))
"b:" clojure.lang.PersistentList
"b:" (1 2)
"a:" clojure.lang.PersistentList
"a:" (1 2)
nil

From having implemented small lisps, I can handwave about (1 2) having already been evaluated into an data structure in the host language, but that feels a bit shaky to me.

The list (1 2) is not getting evaluated anywhere in the code. When a is called its argument, the symbol l is evaluated, resulting in the list.

b does not evaluate (list 1 2). the repl does

There are two kinds of evaluation. One is "what does this mean" and the other is "what value does it have". The reader does the former, a function call does the latter.

I would say the reader does not eval and neither does a function call (unless that function is eval)

I think I agree on the former but not the latter. A function doesn't evaluate its arguments. But a function call does IMO. But maybe we mean a bit different things by "a function call".

https://clojure.org/reference/evaluation

That’s the doc I’ve been poring over. “Non-empty Lists are considered calls to either special forms, macros, or functions. A call has the form (operator operands*).” “If the operator is not a special form or macro, the call is considered a function call. Both the operator and the operands (if any) are evaluated, from left to right. The result of the evaluation of the operator is then cast to IFn (the interface representing Clojure functions), and invoke() is called on it, passing the evaluated arguments.” It seems like b is receiving a non-empty list, so it should be evaluated and therefore treated as a function call.

> There are two kinds of evaluation. One is “what does this mean” and the other is “what value does it have”. The reader does the former, a function call does the latter. I’d be interested to see where that’s documented; I don’t feel like I’ve heard of that before.

> A function doesn't evaluate its arguments. But a function call does IMO. But maybe we mean a bit different things by "a function call". I guess that sounds right based on the reference page.

That quote above about non-empty lists talks about list expressions, not values.

It talks about "what it means", not "what its value".

> Non-empty Lists are considered calls That is literal lists, but there is no literal (1 2) in the code.

(def a ())
a

Right, "literal" is a good word, although that evaluation page linked above doesn't mention it.

Hi all. Does anyone know why clojure.data.json/read-str behaves this way? I have this situation where some keys contain multiple slashes. In the first example below, the resulting map is tagged with #:a which appears to come from the key, and the resulting key is missing the a/ prefix. This only happens when the JSON object contains only one key with multiple slashes. In the second example though the object contains two such keys and the behavior changes, it doesn’t add any tag to the resulting map and preserves the form of the original key.

(clojure.data.json/read-str "{\"a/b/c/d\":1}" :key-fn keyword)
;; => #:a{:b/c/d 1}

(clojure.data.json/read-str "{\"a/b/c/d\":1,\"e/f/g/h\":2}" :key-fn keyword)
;; => {:a/b/c/d 1, :e/f/g/h 2}

(cheshire.core/parse-string "{\"a/b/c/d\":1}" true)
;; => #:a{:b/c/d 1}

(cheshire.core/parse-string "{\"a/b/c/d\":1,\"e/f/g/h\":2}" true)
;; => {:a/b/c/d 1, :e/f/g/h 2}

(keyword "a/b/c/d")
;; => :a/b/c/d

In your first example, it appears to be a keyword with namespace a and name b/c/d

@lennart.buit At first I was surprised to see that the parser was taking off the a as a namespace and using it to tag the map. But that doesn’t explain the second example’s results.

this isn't about the parser(s). it's just how the map is being printed out:

user> {:a/b/c/d 1, :e/f/g/h 2}
{:a/b/c/d 1, :e/f/g/h 2}
user> {:a/b/c/d 1, :a/f/g/h 2}
#:a{:b/c/d 1, :f/g/h 2}

the prefix is just a shorthand

Well the #:a{...} syntax denotes all keys share the namespace :a, in your second example it can’t shorten it because there is no common prefix

the key is still the same

user> (get {:a/b/c/d 1, :a/f/g/h 2} :a/b/c/d )
1

Hah! Interesting.

(map namespace (keys ...)) out of the top of my head will tell you what the namespace parts of the keywords are :)

Indeed:

(let [m (cheshire.core/parse-string "{\"a/b/c/d\":1}" true)]
  (keys m))
;; => (:a/b/c/d)

I know that you didn't really ask about it but perhaps you might want reconsider keywordizing such maps where keys contain special symbols like /. Otherwise, it increases the potential to make things worse down the road.

That’s comforting. Now I need to find out why the key :a/b/c/d is getting stored as b/c/d in mongodb.

Oh no. That's exactly what I'm talking about. :)

Because it uses name, I bet.

@p-himik Totally agree. Unfortunately in this particular case I cannot avoid that. 😞

Hmm? Why do you need to keywordize while parsing?

I’m curious whether you all see yourselves as saying the same thing in different words, or whether you think there are real disagreements there. I’m certainly not sure myself. Some of the statements above seem wrong to me, but I may be misinterpreting them.

I definitely don’t feel like I’ve got this nailed down to the point where I can explain it clearly to a beginner.

@lennart.buit It’s a huge map that comes thru a REST endpoint. 99% of the keys are just your good old normal keys but there’s this little map deep inside which contains those bizarre keys with slashes.

I guess I’ll have to give that little map a special treatment—i.e.: converting those keys to strings—before storing in mongo… ugh!

(list 1 2) evaluates to a data structure that looks like (1 2) -- and that's it: it's just data at that point. The call (a l) is calling a and passing in a data structure.

Personally, I don't feel like I have any authority here. So feel free to use my ramblings to adjust your mental models but don't really rely on them. I still feel like Alex Miller is about to come and correct us all at the same time. :) Oh, or Sean Corfield. 👋

There's no additional level of "evaluation" going on because in (a l) the argument is just data already.

The evaluation there is that the local symbol l is "evaluated" (looked up) to get its value, which is a data structure.

That seems like what @p-himik is getting at with the expression/value distinction above.

Or, to put it another way: (b (list 1 2)) evaluates its argument (list 1 2) to get a data structure (1 2) which is passed into b (a l) evaluates its argument l to get the data it is bound to -- the data structure (1 2) which is passed into a

Evaluating a symbol means "look up the symbol's value" -- but there's no "evaluate the thing that the symbol is bound to".

If you had (a (eval l)) then, yes, you'd be evaluating what l is bound to... and that would then evaluate (1 2) and fail...

(let [l (list 1 2)] (println (eval l)))

I think that makes sense to me. It certainly fits with “The expressions exprs [of a fn] are compiled in an environment in which the params are bound to the actual arguments” (from the special forms https://clojure.org/reference/special_forms#fn).

After five years or so of Clojure I’ve gotten to the point where I rarely get confused by any of the specifics (writing macros, obscure core functions), but I periodically go through a fresh stage of confusion about the most basic things 😜

Heh, like noticing how exactly you're breathing. Or walking.

Or just don't keywordize the keys, separate the things that need to be keywordized from the things that don't need that, and keywordize the former manually.

Thanks, y’all!

Ah! Thanks.