Clojurians Log v2

Anyone has an idea how you could detect that *ns*has changed. Or more like, that we left the current namespace?

add-watch only works on root bindings

Nrepl would send back the value of *ns* on every response if I recall

Looks very complicated :thinking_face:

Does anyone here have experience to parameterize a uuid string? I am not being able to make following query in next-jdbc .

["SELECT * FROM sessions WHERE (session_id = ?)" "1594d14d-91f7-4c8e-88cb-b06ca14ada0f"]

ERROR in (logout) (QueryExecutorImpl.java:2433)
Uncaught exception, not in assertion.
expected: nil
  actual: org.postgresql.util.PSQLException: ERROR: syntax error at or near "WHERE"
  Position: 18
 at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse (QueryExecutorImpl.java:2433)
    org.postgresql.core.v3.QueryExecutorImpl.processResults (QueryExecutorImpl.java:2178)
    org.postgresql.core.v3.QueryExecutorImpl.execute (QueryExecutorImpl.java:306)
    org.postgresql.jdbc.PgStatement.executeInternal (PgStatement.java:441)
    org.postgresql.jdbc.PgStatement.execute (PgStatement.java:365)
    org.postgresql.jdbc.PgPreparedStatement.executeWithFlags (PgPreparedStatement.java:155)
    org.postgresql.jdbc.PgPreparedStatement.execute (PgPreparedStatement.java:144)

I think

["SELECT * FROM sessions WHERE (session_id = ?)" (UUID/fromString "1594d14d-91f7-4c8e-88cb-b06ca14ada0f")]

Curious why we need to change a string to uuid before querying against a string column. session_id is a varchar in db. Trying above line I get following error

Uncaught exception, not in assertion.
expected: nil
  actual: org.postgresql.util.PSQLException: ERROR: operator does not exist: character varying = uuid
  Hint: No operator matches the given name and argument types. You might need to add explicit type casts.
  Position: 42
 at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse (QueryExecutorImpl.java:2433)

Ah, it is a varchar? I thought it is an UUID in Postgres, if it isn't then it is just a normal String parameter, isn't it?

It is strange that is says it is syntax error. Does it work if you pass in the full SQL statement without the param? ie. ["SELECT * FROM sessions WHERE session_id = '1594d14d-91f7-4c8e-88cb-b06ca14ada0f'"]

@suren Log all the PG statements (via the server config, not via CLJ) and see what the actual statement is that gets executed.

Hey guys. In this code:

(defn factors [n]
  (filter #(zero? (mod n %)) (range 1 (inc n))))

(defn prime? [n]
  (= (factors n) [1 n]))

(def all-primes
  (filter prime? (range)))

take is lazy.

And filter as well.

To realize the lazy seq, you can do doall.

user=> (do (time (filter odd? (range 1000000))) nil)
"Elapsed time: 0.014845 msecs"
nil
user=> (do (time (doall (filter odd? (range 1000000)))) nil)
"Elapsed time: 45.723745 msecs"
nil

or just use dorun instead of doall?

doall would work but I would use either (last (take n)) or (first (drop n)) since doall holds the seq in memory and you only want the time.

(time (dorun (take 10000 all-primes)))

dorun is better still :)

Thanks, I keep forgetting about it. :)

but in practice you would probably want the nth prime ... dorun is good for just timing it

Yeah, dorun or doall is what I was actually looking for. Thanks everyone :)

just for completeness (time (nth all-primes 10000)) does the job as well and it is shorter 🙂

One more question, if you do

(defn pinc [n] (println "*") (inc n))
(def arr (map pinc (range)))
(take 10 arr)

It is a lazy seq again what you get back.

(take 10 arr) returns the head of a seq. That gets passed to time when you do (time (take 10 arr)) . time prints the elapsed time (a few milliseconds) and returns the head of the seq to the REPL. The REPL prints it out and that's when it gets evaluated.

if you do something larger again ex. (time (take 10000 arr) it might be more obvious that time prints out the Elapsed time first and then the arr is printed out.

Oh, now I get it. Thank you so much! 🙂

no worries, I'm glad I'm making some sense 🙂

@sztamas yup it works if I pass it within the sql statement.

@suren Sorry, was AFK. Next step would be to try it with ["SELECT * FROM sessions WHERE session_id = ?" "1594d14d-91f7-4c8e-88cb-b06ca14ada0f"] . If it fails can we see the code that gets executed and the table definition? Also, what @p-himik suggested (getting the logs of the queries received) makes a lot of sense if it is feasible to get to the DB somehow.

I have a seq of foos that I want to process sequentially and for each foo I might generate zero, one or “a few” bars. The way I’d usually do this is with a (mapcat identity (for …)) just because I like the way for looks — but I’m not so wild about mapcat identity (and the http://chouser.n01se.net/apply-concat/)… Is this a transducer shaped hole?

@orestis I think so:

user=> (into [] (comp (map (fn [x] [x x])) cat) [1 2 3])
[1 1 2 2 3 3]

mapcat can also create a transducer.

user=> (into [] (mapcat (fn [x] [x x])) [1 2 3])

So I guess as a follow up - is it common for internal APIs that implement business logic by returning a transducer? In my case, the logic is all in the function that takes a foo and returns some bars. There will be multiple of those functions (probably a multi method) but I guess they are a little bit concerned about the plumbing since they need to be a bit efficient. I guess what I’m asking is, is it better to return a transducer that can be directly applied to a seq or a function that can only work with the correct plumbing?

what do you mena by "concerned about plumbing"? do you mean tied to particular transducing contexts, or coupled with the shape of the input or something?

Mainly sources and sinks, laziness vs eagerness etc.

@orestis You can also consider an API that returns an eduction which in loosey-goosey terms is a transducer coupled with its input source

this was a good excuse to look a bit at next.jdbc source code, but I don't think anything "special" is happening in there that might cause this error, so I suspect there is something wrong with that SQL and param vector from your initial post. Anyways, you could do the following to see the SQL statement that will be sent to Postgresql for execution:

(import org.postgresql.jdbc.PgPreparedStatement)

(def ps (.prepareStatement (.getConnection YOUR_DATASOURCE) "SELECT * FROM sessions WHERE (session_id = ?)"))
(.setObject ps 1 "1594d14d-91f7-4c8e-88cb-b06ca14ada0f")
(str ps)
"SELECT * FROM sessions WHERE (session_id = '1594d14d-91f7-4c8e-88cb-b06ca14ada0f')" 

Can eduction be further composed into bigger transducers?

Like can you add more transforms to it after the fact?

yes

I see, so they seem the closest thing to lazy-seq.

I'd say it depends, are they going to process the returned collection further? If so, and you care about performance, probably best you return a transducer or an eduction.

Just in case - what PostgreSQL ends up doing can be infinitely complex. It all depends on your schema. E.g. you can have a bazillion INTEAD OF rules and some triggers that end up trying to execute broken SQL that has absolutely nothing to do with the original query.

What makes me especially suspicious is that the error says Position: 18. But the 18th character of that query is in the middle of the sessions word. It doesn't make any sense.

I just want to make Clojure fans aware that in #startup-in-a-month you will be able to follow @andyfry01 picking up Clojure for his project of starting 12 apps in 12 months. As I understand things, he is not completely new to functional, but there sure will be Clojure things he need to figure out. So maybe consider being part of his line of support and helping him create as many success stories for Clojure as possible the following year? ❤️

I guess the semantics I’m debating is should the api itself be concerned with transforming a single element or transforming streams/sequences

Anyone know any good examples of eduction being used like this? I feel like it's the part of transducers i never fully "got". I've re-read the official transducers page and the clojuredocs page for eduction a few times but it's not quite clicking

couldn't you do both with a separate arity in the same way clojure core does?

@jjttjj we use a couple of functions that return an eduction as the result from a sql query. you can do transformations on this, but only until you consume the eduction will the SQL query be executed and the results streamed

I've implemented things as transducers, and then just have another arity that just adds an xs arg and does (into [] (this-fn xf-args) xs) for easier use at the repl with sequences

by transformations you mean like (map my-fn eduction-result) right? like the regular sequence functions?

yes, good points!

anyways, the code above does about what next.jdbc is doing, so if the SELECT string looks legit (and I don't see why it wouldn't) the OP will know that the problem isn't in next.jdbc as originally suspected

@jjttjj that’s an interesting approach, I’ll have a think about it. I need a REPL to try things out :) but it helps discussing here in the abstract too

A random observation (or two): I’m converting an important CI (bash) script, which mainly contains a series of small inline awk scripts, to Clojure (babashka). It is such a bliss being equipped with a REPL! :clojure-spin: Interestingly the loc count more than doubles. I hadn’t expected that.

Use fewer newlines?

As concise as Clojure is, hard to beat awk

Indeed. Back in the days I wrote whole systems in awk. Actually is a thing I like about Clojure. That it somehow reminds me about when I was so productive with awk.

I don’t mind the extra lines at all. For the record. 😃

No I think with further transduce

(into [] (map my-fn) eduction-result) though this is what I'm not sure of. If this will create a single transducer pass over the contained eduction collection or if it will be two pass

Until someone puts awk inside Clojure 😛

@didibus eductions don't do passes until you consume them. you can compose them with other transducers and they will be "merged"

think of an eduction just being a pair of input collection and a transducer

Hey guys, I am seeing strange behavior with tolitius/mount. Is anyone here an expert?

That would be lovely, I think. Crazy, but lovely.

Ok, as long as they are further merged prior to looping

I also wasn't sure if they'd be merged in all context. Like maybe you can:

(->> (eduction (map inc) [1 2 3 4])
  (eduction (filter even?)))

(->> (eduction (map inc) [1 2 3 4])
  (into [] (filter even?)))

@doubleagent it's probably more productive to just post the question - I've solved people's mount problems in the past based on clojure knowledge despite never having used mount

awk is an incredible tool when you have an awk-shaped problem

I would normally do that but taking a different approach since I can't post the code.

And both performs a loop fusion?

#mount ?

This is one such problem, plus the fact that I tend to shape my problems like that. However, I am so happy for babashka helping me to write it in Clojure that I can hardly word it. The requirements list for this pipeline used to stress me out, but now I feel how I am smiling looking at it.

There's probably some utils you could build up to imitate a more awk like workflow, in terms of the read line, process, repeat

that would be a cool library

yes

can you give a clojure (pseudo-code) example of what you would normally do in awk? I am lacking awk knowledge

I fixed the issue but I don't have any explanation for why the issue was happening in the first place.

when faced with this decision (which pops up from time to time) I tend to choose vanilla functions. My reasoning is - which of these is more agnostic and reusable?

(defn logic1 []
  (fn []
    ...))

(defn logic2 []
  (map (fn []
         ...)))

(defn logic3 [xs]
  (map (fn []
         ...)
       xs))

#mount for details

In a way, a vanilla fn is a superset of the other two options. By pickng the vanilla fn, I can choose laziness or transduder, map or filter, a la carte

Neato, used like this eduction is pretty nice actually. Never really thought of using it.

@sztamas @p-himik You guys are right. The issue was with one of my queries. Apparently I was focusing on a wrong query. The query below works fine.

(db/query! (sql "select *"
                       "from sessions s"
                       "inner join users on users.id = s.user_id"
                      (where {:s.session_id session_id})))

(db/query! (sql "UPDATE sessions"
                  (set {:idle_timeout 0})
                  (where {:session-id session-id})))

So, from the top of my mind when I think about what awk is… It has this pattern -> action structure that it applies to all rows of the input. The pattern is a predicate of sorts, having access to the current row as input. The action is code executed when the pattern matches. It also has access to the current row as input. Both the pattern and the action “automatically” has the current row split up into columns. Both also have access to any state set up by actions. Patterns can be of a special variant of FROM, TO, which I think about as two patterns, it will match from when FROM is true until TO is true. (Both the pattern and the action has syntactic convenience access to the current row, but I don’t think it is what makes so much difference.) Pseudocode… I have to think about it a bit…

Sounds like an Advent of Code problem. I bet there's people who did AOC in awk ;)

Haha, yes, Some of the problems would lend themselves to awk. It is a quite full featured language as such, but a bit like early javascript, where sourcing in existing code bases is not really solved. (But there is cat and stuff in the shell that bridges this to some extent.)

If you have experience from Perl you might know about some awk idioms/powers.

Here’s an example which displays some of what I tried to described, and also highlights some things I forgot to mention. It’s from https://stackoverflow.com/a/33110779/44639 and an answer to a question about how to implement tail in awk:

awk -v tail=10 '
  {
    output[NR % tail] = $0
  }
  END {
    if(NR < tail) {
      i = 0
    } else {
      i = NR
    }
    do {
      i = (i + 1) % tail;
      print output[i]
    } while (i != NR % tail)
  }'

> should print the last 10 lines of a text file

cat README.md | bb -io -e '(drop (- (count *input*) 10) *input*)'

Indeed. Babashka has a lot of the awk feeling in my book. Even if awk works on each row at a time, so slurping up the whole input is not usually what you do.

*input* is lazy, but to get the count means that it will be realized fully. how does awk do this?

AWK has three phase: do something at the beginning for each line <process-line>> do something at the end And you can set variables in the begin, that you set in the process phase, and use in the end phase

Yeah. The awk script gets called before the input, once for each input row (where “row” is defined by a separator regex) , and then after the input.

So how does it know the amount of lines at the start, it reads the entire file probably?

It can’t know the amount of lines in the start. This is known only in the action matching the END pattern.

So what does:

{
    output[NR % tail] = $0
  }

tail is the variable fed to this particular script. NR keeps track of the current row number. (Which then in the END pattern is the number of total lines.)

output is an array. An associative array as it happens, because all arrays in awk are like that.

ah, ok, so it assigns the NR mod tail-th element to the current line, so in the end you will have the last 10 lines in the array

Exactly.

Print a random line from a file:

awk 'rand() * NR < 1 { line = $0 } END { print line }' file.name

rand() * NR < 1 is a pattern causing line to be set to the current row ( $0).

This seems to do what awk does then, sort of:

$ cat README.md | bb -io '(first (reduce (fn [[tail nr] line] [(assoc tail (mod nr 10) line) (inc nr)]) [(vec (repeat 10 nil)) 0] *input*))'

Print a random line:

$ cat README.md | bb -io '(rand-nth *input*)'

but yeah, I get awk syntax/semantics more now, thanks, TIL

Cool. The awk script for printing a random line uses a Knuth trick to not realize the whole file. So a reduce might be in place again. Maybe a reduce context is what we can imagine any awk script to “live” in.

Not realize the whole file at once that is.