Clojurians Log v2

Hi, i was wondering if anyone had experience using clj-http’s async requests? I am trying to make a request to an endpoint that is cancelled after 2 seconds. If it succeeds, it returns the response body, if it is cancelled or fails it returns nil.

in the [clj-http readme](https://github.com/dakrone/clj-http) there’s a nice example of an async request with cancellation;

(import '(java.util.concurrent TimeoutException TimeUnit))

(let [future (client/get "<http://example.com/slow-url>"
                         {:async true :oncancel #(println "request was cancelled")}
                         #(println :got %) #(println :err %))]
  (try
    (.get future 1 TimeUnit/SECONDS)
    (catch TimeoutException e
      ;; Cancel the request, it's taken too long
      (.cancel future true))))

but the functions do a println, and I want the response body. When I try to just return the response (by switching out #(println :got %) with (fn [res] res)` I only get an apache.http.BasicHTTPResponse, which doesn’t include the body.

My issue seems similar to the issue raised in clj-http here: https://github.com/dakrone/clj-http/issues/512, but the function offered as a solution doesn’t seem to be correct…or like valid clojure: `

I hesitated on asking this in the clj-http github repo just cos this ticket and its pr are over a year old, and figured I may not get too quick a response, and is likely an issue with my understanding more than the library itself…So I wanted to see if any folk here had experience with this. Thank you!

Can anyone recommend a scheduler library (cron-like) that is not abandoned 8 years ago? 😕

I'm browsing through projects on Clojars and so far I mostly see projects without releases, broken documentation and last change 6+ years ago

have you looked at quartzite? https://github.com/michaelklishin/quartzite

I think I just saw a fork that someone picked up again

https://github.com/daniel-zamora/quartzite maybe

I’m thinking I should just build something around java.util.concurrent.ScheduledExecutorService

@webmaster601 I'm pretty sure your callback functions have to return their argument (or a modified version of it). Yours return nil (because println returns nil). You want #(do (println :got %) %) #(do (println :err %) %) -- which is what that issue is suggesting: (fn [_} _) returns its argument.

@st3fan Or just use Quartz itself via interop?

I settled on quartzite .. it it not really a pretty api but it works

@st3fan I noticed that the fork you mention has posted a single snapshot to Clojars under Daniel's personal group and then switched to gov.nasa.earthdata and posted another snapshot, but I didn't see any non-snapshot releases from that fork yet. Just FYI.

Ah, right! I shall try it! I tried something similar to your do function above, but it returned a condensed apache http response, with no response body. I’ll try it again though with some fresh eyes!

yeah messy

I just grabbed clojurewerkz/quartzite instead

I'm learning to use jdbc.next, it seems to me that the following code segment caused the error below:

(let [datasource (jdbc/get-datasource (:db env))]
    (jdbc/execute! datasource ["SELECT 1"])
  ...)

Execution error (ConnectException) at java.net.PlainSocketImpl/socketConnect (PlainSocketImpl.java:-2).
Connection refused (Connection refused)

seancorfield/next.jdbc {:mvn/version "1.1.613"}
org.postgresql/postgresql {:mvn/version "42.2.11"}

:db {:dbtype "postgres"
      :subprotocol "postgresql"
      :subname "postgres"
      :user "postgres"
      :password "scrapsafe"}

@yubrshen For dev work, I use Docker to stand up a local PostgreSQL instance.

Here's what I use for testing clojure.java.jdbc against PG

docker run -p 5432:5432 --name clojure_test -e POSTGRES_PASSWORD=clojure_test -e POSTGRES_USER=clojure_test -d postgres

For testing next.jdbc, I use an Embedded PostgreSQL library (which needs special startup in the application) but the same org.postgresql/postgresql driver (version 42.2.10 right now, against the 12.2.0 embedded database).

If you don't want to use Docker, you'll have to follow the documentation on the PostgreSQL website. I've never installed PG locally tho', I've always used Docker or the embedded server.

@seancorfield Thanks for the prompt help. I'll try the docker first hoping it would work for the purpose of practice of web server application.

@seancorfield Unfortunately, with my Ubuntu in WSL1, docker is quite complicated. I may have to try the "Embedded PostgreSQL library" Could you confirm if this is the link to the library, https://github.com/Bigsy/pg-embedded-clj ? If this is the one, the documentation is very sketchy. If it's possible to show the sample code for setup? Thanks!

Switch to WSL2. It's worth the effort of starting over and creating a new dev env. I run all my Clojure stuff on WSL2. I have Docker for Redis, PostgreSQL, and ElasticSearch. I use VS Code with the remote-wsl2 extension (so it lets you have all your code and processes on WSL2 but you still run the editor itself on Windows). Then I use Xlaunch (VcXsrv) so I can run X11 GUI stuff like Reveal from WSL2.

As for Embedded PG, look in the next.jdbc repo -- the deps.edn file has the deps for the embedded pg library (NOT the one you linked) and the test fixtures code has the startup code for it.

https://github.com/seancorfield/next-jdbc/blob/develop/test/next/jdbc/test_fixtures.clj#L28 (see also the :import at the top of that file)

This is the data source setup for it: https://github.com/seancorfield/next-jdbc/blob/develop/test/next/jdbc/test_fixtures.clj#L130-L132

(if you're using a real PG DB, you'd want to create a pooled datasource instead from the db-spec, using HikariCP or c3p0)

I have a really hard time understanding basic JNI interaction, I really hope someone can give me a hint here: I'm trying to load a native library with System/loadLibrary on an Ubuntu 20.04 system, the lib very much appears to be on the library path, but the JVM stubbornly claims it can't find it. I've made sure the JVM has read and execution access.

try (System/loadLibrary "lapack")

Am doing advent of code but am just wondering why are the first 2 elements produced vectors? Am using {clojure.math.combinatorics]

(take 5 (combo/permuted-combinations data 2))
=> ([1757 1890] [1890 1757] (1757 1750) (1750 1757) (1757 1440))

A question about (ns … (:gen-class …, how would you create a “singleton pattern” where you have a static method which returns an instance of the gen-classed class? Specifically I have failed to figure out a working way of actually instantiating the gen-classed class within the file where the class is defined

ok the above is still of academic interest, but I think I will solve this by moving the relevant functionality to a java class

Anyone else here who's doing #AdventOfCode 2020? https://twitter.com/juniusfree/status/1333727365357207555

I complted the Day1 challenge but I want to know if there's a better approach. https://twitter.com/juniusfree/status/1333727767658053632

There is #adventofcode btw

@zackteo Thanks!

What is the best way to differentiate only the latest content of the email from the context-content(the previous emails auto-added that make up the thread)? Example:

"text" : "On 1\r\nOn 2\r\n\r\nOn 3\r\n\r\n\r\nOn 4\r\n\r\n\r\nOn 6\r\n\r\nOn Tue, Dec 1, 2020 at 4:41 PM Author <x.y@z.com>\r\nwrote:\r\n\r\n> L\r\n> M\r\n>\r\n> N\r\n> where O>P>Q!\r\n>\r\n> Onyx is a pokemon\r\n> On\r\n> yx\r\n>\r\n> On Tue, Dec 1, 2020 at 4:39 PM Author <x.y@z.com>\r\n> wrote:\r\n>\r\n>> L\r\n>> M\r\n>>\r\n>> N\r\n>> where O>P>Q!\r\n>>\r\n>> Onyx is a pokemon\r\n"

(-> notification-text
                                      (str/split #"\r\n\r\nOn ")
                                      (as-> raw (raw 0)))

could you please explain how namespace name depends on directory structure?

I'm absolutely lost

Tools like Leiningen and the Clojure CLI tools have the notion of a class path. I believe both of them by default have the directory named "src" in the class path.

(ns my-app.some.lib.util ,,,)

my-project/
  - src/
    - clj/
      - my_app/
        - some/
          - lib/
            util.clj

but they can be configured to have a different directory, or a list of directories, in the class path, e.g. src/main/clojure, although that is uncommon.

Some characters that are allowed to be in a namespace, like -, must be turned into _ in the corresponding file name. . separators between parts of a namespace must be another level of directory in the file system.

The only other wrinkle is that any dashes in your namespace get mapped to underscores in the directory structure, so my-app in the namespace gets mapped to my_app in the directory name.

I had a go too:

(ns joetague.aoc2020
  (:require [<http://clojure.java.io|clojure.java.io> :as io]
            [clojure.math.combinatorics :as combo]))

(def expense-report (with-open [rdr (io/reader (io/resource "expense_report.txt"))]
                      (doall (mapv #(Integer/parseInt %) (line-seq rdr)))))

(defn line-item-group-matching-sum [col groupsize sum]
  (filter #(= sum (reduce + %)) (combo/combinations col groupsize)))

;; First star
(reduce * (first (line-item-group-matching-sum expense-report 2 2020)))
;; Second star
(reduce * (first (line-item-group-matching-sum expense-report 3 2020)))

I'm not quite sure about my filter because it returns a list (list ()) so thats why I have to do first before my reduce - will see if I can get rid of that

@lkottmann have a look at tick - it uses Java-time and you can use it in both clj and cljs https://github.com/juxt/tick

looks interesting, don't know why I went past it, thanks

usually all class statics would be put into an atom inside a def in the namespace, that can include an instance of the class itself - every def is executed when the namespace itself is loaded

it doesn't seem to support humanizing a time "ago" (i.e: "5 days ago")

but it's definitely more complete than java-time

Oh @lkottmann I'm sorry I completely missed the point of your initial message. Try this https://duckling.wit.ai/

actually I replaced my code with and it's much better already

that duckling lib looks awesome

I never saw it before!

I'm trying to see if it can handle the reverse (a duration to a humanized representation)

mmh it doesn't seem to do that, and it has been deprecated in favor of an haskell version from facebook 😕

@lkottmann sorry I can't help much. I don't think it would be too hard to implement one, though. And at the end you can publish that as a lib 🙂

well, I did but I didn't make into a library

(uses tick)

can I somehow inspect the jvm-opts of my running jvm in the repl?

(System/getProperties) can show you all of the JVM system properties (but not just the ones that were set externally)

seems like my next question then is something like is there a way to set :jvm-opts in deps.edn regardless of an alias?

{:path     ["src"],
 :deps
 {org.clojure/clojure #:mvn{:version "1.10.1"}}
 :jvm-opts ["--some-option=some-value"]}

not currently

you can only include them as part of an alias right now (or on the command line with -J)

as an aside, for a jvm opt, you'd want that to be ["-Dsome-option=some-value"]

thanks for the info. sucks for me, seems like I'll have to figure out how to configure cider to start a repl in a project using command line tools with an alias then or something

you are certainly not the first to walk this path, not sure if this channel is the best place for the question - maybe #cider or #clojure would be better

seems like it thanks a lot 🙂

in case anyone cares: https://github.com/clojure-emacs/cider/issues/2396

That issue is closed with two solutions listed. What problems are you having?

none so far, just thought I'd document this for some reason - not that it's likely someone would be stumbling across this here instead of by googling

yeah and this will disappear in five days or so

@d.eltzner why do you need an alias though? CIDER does a pretty good job of being able to start up automatically

to specify :jvm-opts (see above, apparently that is only possible via aliases)

ah i see

Do folks have any general tips for handling large csvs in clojure? Is it worthwhile to force evaluation of the rows and store in a binding of some kind to access later? Or is it better/more ergonomic to simply read/access the file each time and you can lazily evaluate?

well, i'm not sure how i got here, but i'm starting to feel squeamish about returning functions. because they're not data. i can't "see" what they are - eg #function[workflow.server/eval51723/fn--51726]

is that the right one? lol idk

I always thought it would be cool if functions printed their source code rather than that sort of memory reference

i guess i could return a quoted expression

They’re still data. It’s just not rendered in an easily readable way

Down that path lies madness. Then you have to keep track of quoting and unquoting and evaluating and skipping evaluation

sorry to keep bothering, but I now am using an alias

{:path     ["src"],
 :deps
 {org.clojure/clojure #:mvn{:version "1.10.1"}}
 :aliases
 {:bonkers
  {:jvm-opts []}}}

Also, macros: just functions that run at read-time rather than run-time

You can put a name between fn and the argument list, which will make that output a little clearer.

ooh

user=> (defn foo [n] (fn [x] (* n x)))
#'user/foo
user=> (foo 2)
#object[user$foo$fn__20417 0xb666366 "user$foo$fn__20417@b666366"]
user=> (defn foo [n] (fn times-n [x] (* n x)))
#'user/foo
user=> (foo 2)
#object[user$foo$times_n__20422 0x3113595e "user$foo$times_n__20422@3113595e"]
user=> 

yeah, that's better

It depends a lot on what you’re trying to accomplish in a broader sense. Is it a relatively small CSV that you’ll have to use many times? May as well read it all into memory. Is it a huge CSV that’s too big to read into memory? Maybe process lazily.

@d.eltzner How are you invoking the CLI for that? (and presumably you mean :jvm-opts ["-Dadd-opens=java.base/jdk.internal.ref=ALL-UNNAMED"])

that's not the right syntax for add-opens

--add-opens is a jvm flag (not a jvm system property)

so you'd want something like :jvm-opts ["--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED"]

sorry if I misled you above, I thought you were actually talking about Java system properties

you won't see that in Java system properties - it's used by the jvm itself as it checks module visibility

/usr/local/bin/clojure -Sdeps '{:deps {nrepl {:mvn/version "0.8.3"} refactor-nrepl {:mvn/version "2.5.0"} cider/cider-nrepl {:mvn/version "0.25.5"}}}' -m nrepl.cmdline --middleware '["refactor-nrepl.middleware/wrap-refactor","cider.nrepl/cider-middleware"]' -A:bonkers

I see - still helped a lot in that it helped me along with the jvm-opts so can I see these at runtime? because apparently this still didn't do what I hoped it would

@d.eltzner -A:bonkers needs to be before the "main opts", i.e., before -m nrepl.cmdline etc.

there may be some way via reflection or the module api to introspect those flags, but I don't know

If you like the idea of directly manipulating functions-as-data, quoted expressions and trees of symbols, that’s macros.

Otherwise it will be read as a command-line argument to nrepl.cmdline itself.

😄 yup, I just figured that out. thanks!

maybe like via https://docs.oracle.com/javase/9/docs/api/java/lang/Module.html#isOpen-java.lang.String-

See the thread above -- @d.eltzner had the -A:bonkers flag after the main opts for nREPL.

(I'm assuming a relatively recent CLI version BTW @d.eltzner -- what does clojure -Sdescribe show?)

Some rather interesting information - what would :force and :repro do? and how does the /usr/local deps.edn end up in the config-files? via the :install-dir?

{:version "1.10.1.716"
 :config-files ["/usr/local/lib/clojure/deps.edn" "/home/user/.clojure/deps.edn" "deps.edn" ]
 :config-user "/home/user/.clojure/deps.edn"
 :config-project "deps.edn"
 :install-dir "/usr/local/lib/clojure"
 :config-dir "/home/user/.clojure"
 :cache-dir ".cpcache"
 :force false
 :repro false
 :main-aliases ""
 :repl-aliases ""}

-Sforce forces classpath cache recomputation

thank you for pointing that out

-Srepro omits the ~/.clojure/deps.edn from the deps.edns that are used

these are both doc'ed in clj -h and man clj btw

the :config-files thing is actually a bit of a lie now

i'm pretty new to macros, and the macro club rule #1 tends to dampen my enthusiasm

so it wouldn't have occurred to me.

the first location there is created during installation and is now known as the "root" deps.edn in the docs. in reality it's actually read as a resource from the tools.deps library by clj. the file in that location should be the same file, but it is not actually read from the filesystem anymore (but that is there as some older tools, notably datomic ion dev tools) are still loading it from there

if you have other tools-deps / clj questions, #tools-deps is a better channel for that

yeah, it’s very cool in itself, but practical applications are somewhat limited. Most things can be done more cleanly using mostly functions, with perhaps a small macro for syntactic sugar.

okay I'll keep it in mind! I'm just trying to pick it up as I go at the moment. by the way what I was up to worked out fine eventually, thank you both very much!

The theoretical scope is astounding, though, isn’t it: your macro can take some arbitrary symbolic code as input and rewrite it in any arbitrary way it likes.

The main difference is that functions can’t control whether and how their arguments are evaluated. They don’t get access to the raw symbols, just to the results of evaluating those symbols. Macros get full control at the raw symbol level. This is sadly much less broadly useful than it may seem, though occasionally there are indeed real world use cases where nothing else will do.

Toy example for illustration: you can’t write if-then and similar control constructs directly as a function, only as a macro. As a function, (new-if-fn foo (bar) (baz)) will invoke both (bar) and (baz) before the function even starts. You’d have to do (new-if-fn foo (fn [] (bar)) (fn [] (baz))) to delay evaluation. The macro version (new-if-macro foo (bar) (baz)) just gets the symbols, though, so it can choose to not evaluate the losing branch. The same net effect is achievable both ways, but the syntactic sugar is much nicer with the macro.

@seancorfield Thanks for the generous help! I'll try them.

How would I instantiate the class from the gen-classed namespace for the class? Is that supposed to "just work"?

And thank you for the reply and pointer

the def starts out holding nil, then gets a class instance when the static method is first called, reused afterward - delay / promise can be useful here as they enforce the "only set once" property

the class instance would be made via the constructor of course

For Integrant systems there's https://github.com/troy-west/cronut

@joetague You can check out #adventofcode. I got some great feedback there https://clojurians.slack.com/archives/C0GLTDB2T/p1606822120031800

there's probably no need to keep sending these to the channel. if you have specific questions or are particularly proud you should share them but this is starting to become just a blog feed

Does anyone know a library that can (ascii) tabulate/format data so that i can print pretty reports in the repl or cli? Something like https://pypi.org/project/tabulate/

there's clojure.pprint/print-table for sequences of maps with the same key

oh! cool 🙂

some limitations, but it works for quick and dirty

user=> (print-table [{:a 0 :b 1} {:a 33 :b 12} {:a "OK" :b "bye" :c :hmm}])

| :a |  :b |
|----+-----|
|  0 |   1 |
| 33 |  12 |
| OK | bye |
nil

notice it doesn't attempt to show that :c key

and of course, for other cases, you can probably use group-by / mapcat etc. to make something print-table can consume

interesting

thank you for sharing this, it's quite interesting.

someday i'll have the inspiration to really dig into macros.

that is great - exactly what i need

print table is amazing, I use it all the time. you can also pass in keys to use to fix the "missing :c" problem:

(print-table [:a :b :c] [{:a 0 :b 1} {:a 33 :b 12} {:a "OK" :b "bye" :c :hmm}])

| :a |  :b |   :c |
|----+-----+------|
|  0 |   1 |      |
| 33 |  12 |      |
| OK | bye | :hmm |

> Down that path (syntax quoting) lies madness. Then you have to keep track of quoting and unquoting and evaluating and skipping evaluation it is tricky - i have to deal with this when programmatically building datomic expressions. do you have any thoughts on why datomic would make such extensive use of syntax quoting? would macros fill the same role?

@st3fan another more robust option is using clojure.data.csv to write a csv output, then use a spreadsheet to view the csv

oh, you specifically wanted CLI - so yeah I think you want print-table

Wow, I had no idea print-table was a thing. I’m going to use this all the time!

I’ve never used Datomic beyond a quick tryout, so I’m afraid I don’t know much about why it does anything

I love print-table I use it all the time when I'm REPL'd into production and querying the database 🙂

Are there any other somewhat obscure but highly useful functions like that lurking about?

clojure.set/index is amazing. i see hiredman talk about this one a lot and i think its incredibly useful and underused

Maybe follow https://twitter.com/rcfotd on Twitter? Random Clojure Function Of The Day

there are several

frequencies blew my mind when I found out about it

juxt is a really fun one for making functions with functions

if you'd like to hear more about juxt, check out https://clojuredesign.club/episode/076-multiple-views-on-juxt/ (an episode on my podcast)

Every time I open a new shell: Source: https://github.com/borkdude/babashka/blob/master/examples/random_doc.clj

@dpsutton Sorry about that.

oh no worries. love the enthusiasm. but its just a string of tweets is all

Another fun way to discover new core fns: https://borkdude.github.io/re-find.web/

user=> (->> (all-ns) (mapcat (comp vals ns-publics)) rand-nth meta ((juxt :name :doc)))
[load-file "Sequentially read and evaluate the set of forms contained in the file."]

realistically you probably want the namespace it came from as well...

$ bb -e "(->> (all-ns) (mapcat (comp vals ns-publics)) rand-nth meta ((juxt (comp ns-name :ns) :name :doc)))"
[clojure.set union "Return a set that is the union of the input sets"]

"Fortune, Clojure edition"