Clojurians Log v2

@rahul080327 Do you think we could always add the --compressed flag or would this fail with websites that do not send compressed responses?

We could add it, it should work. My only reason of not adding it would be that there is an implicit behavior and since this is a thin wrapper over curl, its doing more than that and curl users might be surprised. If you ask me i would leave it out. What say?

Ideally this should've been done by curl by inspecting the response header. HTTPie does it

Even web browsers do that too

another reason of not adding would be if later the meaning of --compressed changes based on the curl version, this would start failing for a non obvious reason

@rahul080327 Yep, I agree. - We could support (get ... {:compress true}) to make this a little nicer. - Alternatively we could look at the content-encoding header ourselves and decompress but that may also fall into the category of doing too much.

I wonder if stackexchange should have sent the response in gzip encoding even when we didn't ask for that

https://api.stackexchange.com/docs/compression

> The likelihood of many applications not opting into compression, and being materially worse for it, is unacceptable.

btw, if I start a server with:

$ bb -e '(org.httpkit.server/run-server (constantly {:body "hello"}) {:port 3000}) @(promise)'

yeah curl's --compressed works even if the response isnt. maybe a fn to convert a map of opts -> cli args might be a good addition. But that should be about it, nothing more IMO

hmm, if it "always" works what's the reason not to enable it by default?

is clj-http doing this by default?

(i.e. requesting compressed responses)

i would assume its inspecting the response header like HTTPie

its Apache HTTP client under the hood, maybe thats doing it?

By default, clj-http will add the {"Accept-Encoding" "gzip, deflate"} header to requests, and automatically decompress the resulting gzip or deflate stream if the Content-Encoding header is found on the response. 

yep

we could also do that ourselves and not let curl do that

> If this is undesired, the {:decompress-body false} option can be specified:

but I guess it doesn't hurt to let curl do it

and it seems unlikely to me that --decompress will change its meaning. If curl will change the meaning of its args, then we're doomed anyway in other cases

thats the thing, where do we draw the line between "a wrapper over curl" and "powered by curl".

we're already using these sane defaults:

"curl" "--silent" "--show-error" "--location" "--dump-header"

I would say powered by curl

I think offering sane defaults with the possibility of opting out of it is a better way

yeah if we are sending some flags already as some opinions, then yes we can do that

follow locations is also not enabled by default, but this is the sane default

makes sense

I think curl is conservative because it was growing over time when compression maybe wasn't the default before

or another thought process could be to implement clj-http's API as close as possible and say this is powered by curl. im of the opinion that if we follow one of the standards its more compilant?

I think the more important thing is that it's useful for scripting and doesn't cause any beginner questions. Experts can dig into the docs to disable the defaults. I think this is what clj-http has done too.

clj-http isn't a standard, it's a popular clojure lib that has influenced other libs

it seems org.httpkit is doing this as well: https://github.com/http-kit/http-kit/blob/059deac93b1662077d52e1eb81bef9db8c89d746/src/java/org/httpkit/client/HttpClient.java#L333

right, i guess sticking with some sane defaults which most of us(beginners and others) may use and allow the internal knobs to be turned when needed would be a good choice.

ok, I'll proceed with implementing this and documenting it

Thanks for your input. Btw, I'm keeping some notes about http here: https://github.com/borkdude/babashka/wiki/HTTP-client-and-server-considerations

I am contemplating of building a Java 11 based solution as babashka.http-client similar to what schmee has done.

To have one canonical way of doing http in bb

that would be super awesome. bb setting the standards 😎

on a side note, had great success with java-http-clj. would love to see that being used. pretty much use it in all of my clj needs now 😄

at the moment it's a bit messy having to choose between babashka.curl, org.httpkit or clj-http-lite, slurp, etc. once babashka.http-client is there, the other clients can slowly fade to the background and be deprecated over time. I want to give it some incubation time to flatten out bugs etc. of course

yeah, that one looks great, but isn't yet feature complete.

but it looks like a good start to build babashka.http-client on

yep my vote would be there too

also this is something we made based on OkHTTP: https://github.com/into-docker/unixsocket-http to be able to do UNIX sockets naturally. but quite skeptical for Graal due to the size bloat. Works on it fine though

we can still use babashka.curl for that right?

https://github.com/borkdude/babashka.curl#unix-sockets

yep, in the off chance you dont have/want to install curl

Since babashka.curl is so light-weight, I will forever keep it in bb. curl might cover some weird edge cases that Java doesn't out of the box, like this

this lib is useful for embedding and lean docker deployments more i'd say.

yeah

(deftest compressed-test
  (is (-&gt; (curl/get "<https://api.stackexchange.com/2.2/sites>")
          :body (json/parse-string true) :items))
  (is (thrown?
       Exception
       (-&gt; (curl/get "<https://api.stackexchange.com/2.2/sites>"
                     {:compressed false})
           :body (json/parse-string true) :items))))

### Compression

From babashka 0.2.4 onwards, this library will call `curl` with `--compressed`
by default. To opt out, pass `:compressed false` in the options.

> Since babashka.curl is so light-weight, I will forever keep it in bb Just due to the target audience of bb and the use cases, the likely hood of curl being there along with bb is extremely high 😄

yep. it's also part of the docker image

it's even on windows nowadays, along with tar

windows is the new Mac. (shots fired) 😛

it's kinda true ;)

My next laptop might seriously be Windows, if not Ubuntu

bozhidar is now running Windows as well

yep, was pleasantly surprised seeing the talk

haven't watched it yet. any good?

any new information I should see I mean

Ugh, another github actions instability thing.

https://github.com/borkdude/babashka.curl/runs/1435384230

well some of the lesser known features of CIDER, some Emacs hiccups and the fact that Emacs running on Windows like magic

I kinda found that out myself before bozhidar was on Windows with all the WSL2 stuff

Use DeLaGuardo/setup-clojure@master

is he running emacs on Windows natively or through wsl2?

WSL2

yeah, I love it

I'm eyeing this laptop: https://bestware.com/en/schenker-via-15-pro.html AMD + 64GB... if my current laptop stopped working.

Fix for actions: https://github.com/bob-cd/bob/blob/main/.github/workflows/ci.yml#L33

Fix worked!

## Java 11 client
Java 11 client. Based on <https://github.com/schmee/java-http-clj> minus the specs.
Adds 2.28MB to the binary. See branch `java-11-client`. 

yeah i love the zero dependency aspect of it

This is another one: https://github.com/exoscale/coax

whoops no sorry. This one: https://github.com/exoscale/telex

quite extensive interceptors

All of these things are still moving targets, so babashka.http-client will also probably take another year or so

but the seeds are planted

anyway @yubrshen, I hope we answered your question :)

@borkdude Thanks for taking the time to help me. I'll study the documentation of curl/get to understanding the details of the parameters.

So {:raw-args ["--compressed"]} is only necessary here because stackexchange always sends compressed responses, but curl does not decompress compressed responses automatically. In the next release of babashka I will pass this flag to curl by default, but for now you have to use it like this.