Clojurians Log v2

it seems like the instrumentation check my fully qualified key even if not in s/keys

It does

The doc string covers this

https://clojure.org/news/2021/03/18/apis-serving-people-and-programs

(defn destr [& {:keys [a b] :as opts}]
  [a b opts])

(destr :a 1)
->[1 nil {:a 1}]

(destr {:a 1 :b 2})
->[1 2 {:a 1 :b 2}]

that's cool as hecck

fyi, having some problems with the actual release process so not yet available

I get this when trying to run tests in a fresh clj-new library project:

$ clojure -M:test:runner
Error building classpath. Manifest type not detected when finding deps for com.cognitect/test-runner in coordinate {:git/url "git@github.com:cognitect-labs/test-runner.git", :sha "209b64504cb3bd3b99ecfec7937b358a879f55c1"}

I have a very faint memory of that error. Is your clj up to date?

you probably have an empty dir in your gitlibs directory

at ~/.gitlibs/libs/com.cognitect/test.runner/209b64504cb3bd3b99ecfec7937b358a879f55c1

there was some bugs in the recent prereleases that would cause this

you can rm -rf that sha dir and clojure -Sforce -M:test:runner to fix up

Ah, yes, I am still running the prerelease, probably. Sorry for the noice!

in that case, you might want to clean your whole ~/.gitlibs when you update

Yes, that did it. Thanks!

I need some ideas on how to test the output from tap> a bit more reliably than I do now. I first thought I was not going to use CI for the project, but now I am doing that anyway and the tests fails intermittently. The docs for tap> tells me that I am probably tapping while there is no room in the queue. The tests all look similar to this:

(testing "Taps the binding box"
    (let [tapped (atom nil)
          save-tap (fn [v] (reset! tapped v))]
      (add-tap save-tap)
      (is (= [:foo :bar]
             (sut/let> [foo :foo
                        bar :bar]
                       [foo bar])))
      (is (= '[[foo :foo]
               [bar :bar]]
             @tapped))
      (remove-tap save-tap)))

@pez Also, if you make a fresh clj-new project, the test runner should be an https URL:

(! 885)-> clojure -X:new :name pez/example
Generating a project called example based on the 'app' template.
(! 886)-> cat example/deps.edn 
{:paths ["src" "resources"]
 :deps {org.clojure/clojure {:mvn/version "1.10.2"}}
 :aliases
 {:run-m {:main-opts ["-m" "pez.example"]}
  :run-x {:ns-default pez.example
          :exec-fn greet
          :exec-args {:name "Clojure"}}
  :test {:extra-paths ["test"]
         :extra-deps {org.clojure/test.check {:mvn/version "1.1.0"}}}
  :runner
  {:extra-deps {com.cognitect/test-runner
                {:git/url "<https://github.com/cognitect-labs/test-runner>"
                 :sha "b6b3193fcc42659d7e46ecd1884a228993441182"}}
...

Maybe you have a rewrite rule in your Git config?

tap> returns true if it succeeds, else false so you can check the result of calling it.

And I expect there’s some concurrency at play: after you tap> something, it may not run the tap watchers “immediately” @alexmiller?

to solve "eventually" called tap watchers try to replace atom with promise . The assertion should use explicit deref with timeout like (deref tapped 100 :timeout)

Nah, I had just been testing some different variants. 😃

@seancorfield I don’t know how to test the return value from the tap>. It happens inside the macro I am testing…

@nbardiuk it seems it is not eventually called. Rather it is called and then eventually that succeeds. Also needs to work with CLJS, even if I guess I could use a reader conditional for that.

Continuing the thread from #news-and-articles, with clojure 1.11.0 I get:

$ clj -A:clojure-1.11.0
Clojure 1.11.0-alpha1
user=> (let [{:keys [] :as m} {:a 1}] m)
{:a 1}
user=> (let [{:keys [] :as m} [{:a 1}]] m)
[{:a 1}]
user=> (let [{:keys [] :as m} (list {:a 1})] m)
{:a 1}
user=> (let [{:keys [] :as m} (list {:a 1} {:b 2})] m)
{{:a 1} {:b 2}}

The third example was previously not working and now works, but isn't intuitive. Maybe should be seen as a side-effect of supporting maps as kwargs?

can you please move questions like this to ask.clojure ?

Side-effects are bad, looks like a bug to me...

1, 2, and 4 are the same on 1.10.3. 3 is not legal on 1.10.3:

Clojure 1.10.3
user=> (let [{:keys [] :as m} {:a 1}] m)
{:a 1}
user=> (let [{:keys [] :as m} [{:a 1}]] m)
[{:a 1}]
user=> (let [{:keys [] :as m} (list {:a 1})] m)
Execution error (IllegalArgumentException) at user/eval143 (REPL:1).
No value supplied for key: {:a 1}
user=> (let [{:keys [] :as m} (list {:a 1} {:b 2})] m)
{{:a 1} {:b 2}}

So there already was a difference between 2 and 3.

@nilern FYI since you thought it was a bug.

all Clojure-related questions should now be posted to ask.clojure instead of here?

concrete questions that might be bugs can more usefully be asked there where they can be tracked

conversational questions are probably better asked here

This was intended as the latter, just exploring the new patch

The example came more or less directly from the tests

Okay more like it is an opportunity to create bugs by writing a map pattern and giving it a seq where one of those was a mistake and then it just works but ignores the rest of the seq

The tests are there to check certain behaviors do what they're supposed to do. In practice no one directly destructures a seq with an associative form in a let.

Another question about the patch: There was a core function introduced, but never used but anywhere in the tests. However, in destructure this same pattern as the core function was used. Wasn't the new core function used because of performance (inlining), but still exposed for people who want to implement their own kind of destructuring?

I think the function is in the tests no? But yes, your instinct about why it was exposed.

Understood. The test just confused me.

user=> (let [[& {:keys [] :as m}] (list {:a 1})] m)
{:a 1}

yes, it was never used "anywhere but in the tests" (placed the word /but/ in the wrong place)

Thanks for clearing that up.

Also, it wasn't used in destructure not because of the speed of inlining but because emitting a call to a function that only exists in 1.11 isn't backwards compatible.

In fact, I am implementing my own destructure, so I will jump on that new core fn now :)

Good point!

I like it when tools prevent me from shooting myself in the foot but I guess Clojure has always been more in the "power tools" category

What errors are you hoping that Clojure helps you with in this case?

Has anyone been able to make lein ring serve over http/2? Specifically I have a part of my app that is loading a bunch of static files at once and the dev time performance is abysmal

He is concerned that code which used to blow up if you made a mistake now does something, so if you accidentally try to destructure a sequence against a map you get an (unexpected) result now whereas you used to get an exception. Is that accurate @nilern?

this is a tradeoff we have repeatedly considered and made in the design of clojure, in favor of giving users more power (see transducer arities for another example)

@fogus FYI, I backported the new core fn to 1.10 and it seems to work even without the new Java stuff that got added in createAsIfByAssoc. Using it in sci's destructure:

$ ./bb -e '(defn foo [& {:keys [:a :b] :as m}] [a b m]) (foo {:a 1 :b 2})'
[1 2 {:a 1, :b 2}]

I would have expected the change to only affect rest params. But now that I think of it that would be inconsistent, which I don't like either 🤷

Actually this probably is in the right direction. I couldn’t get it to work, because I didn’t really know how to do it. But adding a short pause between tapping and untapping makes it work.

#?(:cljs (set! *exec-tap-fn* (fn [f] (f))))

(def tapped (a/chan (a/sliding-buffer 1)))
(def save-tap (fn [v] (a/offer! tapped v)))

(deftest let>
  (testing "Evaluates as `let`"
    (is (= [:foo :bar] (sut/let> [foo :foo
                                  bar :bar]
                                 [foo bar]))))

  (testing "Taps the binding box"
    (add-tap save-tap)
    (is (= [:foo :bar]
           (sut/let> [foo :foo
                      bar :bar]
                     [foo bar])))
    #?(:clj (a/<!! (a/timeout 10)))
    (is (= '[[foo :foo]
             [bar :bar]]
           (a/poll! tapped)))
    (remove-tap save-tap))
...

Ah I see. Only the ((partial addn 100 :a 1) {:b 2}) tests still fail on Clojure 1.10.x with this backport. Will be automatically fixed for whoever uses sci with Clojure 1.11.1.

(Probably would have worked with the atom as well, but I had already switched to channels and it does make for less boilerplate so I let it be like that.)

Arguably transducers could have lived in separate similarly named functions without losing any power, unless I'm missing something (which is very likely). E.g. mapping, filtering, etc. But that ship has long sailed.

On that note, could you imagine a linter that warned on usages of the lazy arities of those core fns? I find myself looking for those in PRs

you mean, force users to use transducers?

that doesn't seem a good idea?

been away from JSON parsing a bit, but is cheshire still the goto?

err, a goto 8^)

Maybe (->> stuff (map inc) (filter even?) (into #{})) type of thing which I too optimize quite often

https://github.com/metosin/jsonista is faster and the API is less crufty

But Cheshire seems still more common and I am actually working on a PR to it as we speak

Much appreciated! thanks.

Not forcing, just nudging. But at least in our case laziness is rarely if ever justified, and the overhead is a lot less using transducers

data.json is rapidly getting faster thx to @slipset ...

Depending on the use-case, (for rest, payloads around 1k with shallow maps) data.json is at the same speed as at least Cheshire now. When payloads get bigger, data.json suffer a bit, but not all that much. I haven’t worked that much on writes yet, but I’ve gotten some fairly nice speedups there as well. Regarding crufty api, was that for cheshire or data.json, and if data.json, what do you find crufty about the api? I’m curious since I’m working in that area now (not saying the api will change, just interested).

@alexmiller FWIW, our entire test suite passes at work on Clojure 1.11.0 Alpha 1 (on a combination of OpenJDK 8 and OpenJDK 15 for various apps).

Meanwhile @nilern’s PR to speed up encoding of cheshire with 15% was merged :)

by replacing doseq with reduce, haha

On the topic of JSON/jackson, don't the same arguments of trying to avoid jackson apply to transit-java?

for data.json, having no deps is a primary goal. for transit, performance is a higher priority (but I'd love for it to lose it's deps too)

Or are there any plans to move transit to data.json eventually?

well note that transit-java is java and having it depend on data.json (clojure) is maybe weird

For the more significant speedups, all of these libs spend a significant amount of time calling assoc! , so I got some code where I wrap a java map so it looks like a persistent map. I’m currently looking into wrapping java lists so it looks like a persistent vector, but couldn’t find the exact interfaces I needed to implement.

I'm not convinced any of that is a good idea :)

(but for reference, for payloads above 10b data.json (and cheshire) seem to be spending about 60% of their time calling assoc!)

true. Recently jackson was bumped for cheshire related to a security thing. Would transit-java also be open to such a bump if there was a security risk (or just for keeping up with the ecosystem)?

yes, of course

I think the issue with transit-java is that it is on a pretty old version of jackson and there have been some (many?) breaking changes in the intervening time so it's a non-trivial change

if I remember correctly from the last time I looked at it

I meant Cheshire, which has parse-string parse-string-strict, parse-stream, parse-stream-strict etc. and cannot parse InputStream even though Jackson can... while Jsonista just has read-value with protocol dispatch.

but putting this in a transit-java issue is the best place for it (there may be one)

https://github.com/dakrone/cheshire/commit/44f3785201f87942ac84558fbe189fb1bbee276b

@alexmiller FYI I am running cheshire and transit within the same project(s) and never saw an issue so far

well, that's good to know, put all that in a transit-java issue :)

We have exclusions on jackson in our project-clj (for whatever reason)

I expect we’ll take it to production next week…

Ah, the Jackson CVE is only relevant to cbor which isn't used in transit-java, so I would not have a good case to bump it in that lib

But according to their docs, newer minor versions should not break libs using older ones: https://github.com/FasterXML/jackson/wiki/Jackson-Releases#general

(with a caveat about deprecated methods)