Clojurians Log v2

I'm trying to programmatically create a union type out of an existing spec and a new spec. i understand this is a complete hack, but i've tried many variations of the following:

(defonce spec-ledger (atom {}))

(defn add-to-ledger!
  [k spec]
  (swap! spec-ledger update k (fnil conj #{}) spec))

(defn union-spec
  [k spec]
  (let [existing (get @spec-ledger k)
        options  (->> spec
                      (conj existing)
                      (map-indexed (fn [i p] [(keyword (str i)) p]))
                      flatten)]
    ;; return quoted form instead of evaluating because this evaluates to an
    ;; object, which can't be "embedded in code"
    `(s/or ~@options)))

(alter-var-root
 #'clojure.spec.alpha/def-impl
 (fn alter-sdef-impl [sdef-impl]
   (fn register-union-spec-instead
     [k _form spec]
     (add-to-ledger! k spec)
     (let [union              (union-spec k spec)
           ;; can't seem to call `s/def` directly here
           ;; but in order to call `s/def-impl`, need to need to generate
           ;; updated `form` and `spec` (which `s/def` would otherwise
           ;; accomplish)
           [_ _ form' spec'] (macroexpand `(s/def ~k ~union))]
       (sdef-impl k (eval form') (eval spec'))))))

are there other strategies (other than getting rid of my existing specs or getting rid of my non-conforming data) that jump out at folks?

What's a good way to track down what line of my code is causing a stackoverflow? I'm only getting it when I work with a really big data set. So I don't think it's an infinite loop as part of some calculation. The biggest problem I'm having debugging it is that the lines in the error are all from clojure namespaces.

LazySeq.java:   42  clojure.lang.LazySeq/sval
              LazySeq.java:   51  clojure.lang.LazySeq/seq
                   RT.java:  531  clojure.lang.RT/seq
                  core.clj:  137  clojure.core/seq
                  core.clj: 2927  clojure.core/drop/step
                  core.clj: 2932  clojure.core/drop/fn
              LazySeq.java:   42  clojure.lang.LazySeq/sval
              LazySeq.java:   51  clojure.lang.LazySeq/seq

(def tightly-packed-trie
  (let [tight-ready-trie
        (->> trie
             (map (fn [[k v]]
                    (let [k (map #(get trie-database %) k)]
                      [k v])))
             (into (trie/make-trie)))]
    #_(tpt/tightly-packed-trie
     tight-ready-trie
     encode-fn
     decode-fn)))

with lazy stuff like this, try mapv instead of map

i've had good experiences getting better stack traces doing so

Hm. I'm iterating over a custom datatype and I guess I'm not implementing what I need to in order to use mapv yet. Method com/owoga/trie/Trie.iterator()Ljava/util/Iterator; is abstract

there are other ways to realize a lazy sequence

doall

Thanks for the idea. I think it's a good lead. I haven't made any progress with it yet though. I noticed http://clojure.github.io/clojure/clojure.stacktrace-api.html and thought I could get all traces rather than just what my editor was showing me. But that's still not showing anything other than core clojure code.

Ah ha ha... -XX:MaxJavaStackTraceDepth=-1

folks are suggesting s/form to go from an object to a form

Bingo. You were right. Adding a doall to a concat fixed it I think. And that stacktrace pointed straight to the line.

newb question: Just getting my feet wet with clojure....every time I run any lein commands, I get this warning: OpenJDK 64-Bit Server VM warning: Options -Xverify:none and -noverify were deprecated in JDK 13 and will likely be removed in a future release. 1. Does this matter? && 2. How do I make it stop?

i've heard folks say never to use concat instead of into for that reason

It doesn't matter a ton at the moment, upgrading your lein version might make it go away

Lein passes (passed?) those options to speed up jvm launching, lein because of the way it works usually ends up launching two

https://stuartsierra.com/2015/04/26/clojure-donts-concat

All of https://stuartsierra.com/tag/dos-and-donts is worth reading.

Hi there! How do I run a python script with sh ?

https://github.com/clj-python/libpython-clj is not better in this way?

I dont know what is the goal at your side, maybe wrong idea

I hope, that is helpful https://stackoverflow.com/questions/61001452/cant-run-sh-python-hello-world-py-from-clojure-repl-but-python3-hello-world

Thanks for the feedback! Due to only calling a python script one time I think its overkill to add another library and would rather try to learn to sh it. I tried the second suggestion

(use '[clojure.java.shell :only [sh]])
(sh "python3" "helloworld.py")

(sh "python3" "helloworld.py")
=>
{:exit 9009,
 :out "",
 :err "Python hittades inte. Kör utan argument för att installera från Microsoft Store eller inaktivera den här genvägen från Inställningar > Hantera alias för körning av program.\r
       "}

I don’t have experience w/ Windows. On mac, works fine:

user=> (use '[clojure.java.shell :only [sh]])
nil
user=> (sh "python3" "hello.py")
{:exit 0, :out "hello world\n", :err ""}
user=> (sh "python" "hello.py")
{:exit 0, :out "hello world\n", :err ""}

Hello! How do I do something like

(take 100 (cycle [{:type :invoke, :f :write, :value (rand-int 5)}
                         (repeat 10 {:f :read})]))

@sebastian.ohlsson.198 have you looked at https://clojuredocs.org/clojure.java.shell/sh ? 🙂

Whoops, I realised you want to specifically call python are you on windows or linux?

Hello! Yes I started looking at the docs but sadly I am no wiser. I am on windows

is your helloworld.py the right path? Perhaps you might want to run (sh "pwd" to check

Bad file location case: different the error message (exit 2). I think, he need to config python3 (maybe alias setup problem) on Windows. Or simply try with `

"python file.py"

Right python might not even be on the path ... :thinking_face:

(sequence
 (comp
  (take 100)
  (mapcat identity))
 (repeatedly
  (fn []
   [{:type :invoke, :f :write, :value (rand-int 5)}
    (repeat 10 {:f :read})])))

@sebastian.ohlsson.198 I found an answer: error 9009 “The python.exe you are using isn’t a real Python executable. It’s just a link to Microsoft Store to download it from there.” maybe true?

Right right! I should use a function and repeatedly

Thanks 🙂

You could also use recursion~

((fn rec []
   (lazy-cat
     [{:type :invoke, :f :write, :value (rand-int 5)}
      (repeat 10 {:f :read})]
     (rec))))

As presented atm, the above is an infinite loop. Also, if you can solve your problem in straightforward way without recursion, you should.

yup, don't use recursion in normal circumstances - I think it's just an interesting thing for beginners to encounter

Also, on the JVM direct recursion can blow up the stack, so be careful with recursion.

If I want to inc the value instead, would recursion be good for that then?

Like

(take 100 ((fn rec [x]
             (lazy-cat
               [{:type :invoke, :f :write, :value x}
                (repeat 10 {:f :read})]
               (rec (inc x)))) 0))

@zackteo Probably not:

(sequence
 (comp
  (take 100)
  (mapcat identity))
 (map
  (fn [n]
   [{:type :invoke, :f :write, :value n}
    (repeat 10 {:f :read})])
  (range)))

If each value depends on the previous ones, you can also look at iterate or reductions

right right :thinking_face:

(do
 ;bad - avoid
 #_(let [f-bad (fn inc-inc [x]
              (if (< 100000 x)
               x
               (inc-inc (inc x))))]
  (f-bad 0))

 ;OK to use
 (let [f-good (fn inc-inc [x]
                (if (< 100000 x)
                 x
                 #(inc-inc (inc x))))]
  (trampoline f-good 0)))

Notice that inc-inc returns a fn in the second case.

so the point is to use trampoline when doing recursion?

Yes… I believe in the example above, you might be able avoid the problem because the recursion is from within lazy-cat… But I am not 100% sure.

https://jakemccrary.com/blog/2010/12/06/trampolining-through-mutual-recursion/

Will look into it 🙂 thanks

is their a way to do destructuring but give unique names to the keys, or should I just use a let? e..g

(defn is-left [{:keys [x y]} {:keys [x y]} {:keys [x y]}]
   ;; takes 3 parameters, all 3 have x and y key. BUt I'd like it to be like:
   ;; {px py} {x0 y0} {x1 y1}
)

(defn is-left [pxy xy0 xy1]
  (let [px (:x pxy)
        py (:y pxy)
        x0 (:x xy0)
        y0 (:y xy0)
     ... so on...]))

There is a way to do destructuring on maps, without using :keys, where you get to pick the local names for the value. See examples here: https://clojure.org/guides/destructuring#_associative_destructuring

:keys is unnecessary, but a nice shorthand when you want the local names to be the same as the keyword without the :

Thank you, that looks like that I want. TIL!

I recently wrote this REPL utility to solve this “problem” 🙂 It will print the destructure form for you (for one map): https://github.com/raspasov/alexandria-clj/blob/main/src/ss/auto_let/core.cljc#L112

(ss.auto-let.core/de
 {:user/id    42
  :message/id 52
  :data       {:comment/id 62}})

(it’s not a library atm, but you can copy the namespace…)

It should do the “right thing” and avoid shadowing of locals for that map.

Hello, do you guys have any tips on how to debug the 'pending puts errror' in core.async for ClojureScript? It happens after several days running on our server on nodejs platform. I understand what is happening, but have trouble identifying the code which is throwing the assertion. Thank you.

Hello guys, I'm trying to make a move file function that is done by copying the file to the new destination and removing the source after copying. However, I get a strange behavior since the io/delete-file function says "Couldn't delete" error so I'm checking the file manually and tried to rename it and it says it's locked since another program is using the file. Is the io/copy function not being able to release the lock to cause this such state? What should I do if so?

(let [input (io/file "foo")]
    (println "contents: " (slurp "foo"))
    (io/copy input (io/file "bar"))
    (io/delete-file input)
    (println "contents: " (slurp "bar")))

returns

contents:  hello

contents:  hello

io/copy doesn’t close any streams have a look at this example - https://clojuredocs.org/clojure.java.io/copy#example-5e008f1ce4b0ca44402ef7fe

i suspect that you are holding two references to the file which is what your problem is

notice i io/copy and then io/delte the same io/file. you make an io/file for the copy, and then make another one for the deletion

and btw, consider using let instead of def withing defn. The difference that def creates global var and let is for making localy scoped references.

@dpsutton am I having a race condition? I can't think of any ways to copy and delete the source file the clojure way

@delaguardo I stand corrected, thank you!

(defn move-file [source destination]
  (with-open [in (io/input-stream source)
              out (io/output-stream destination)]
    (io/copy in out))
  (io/delete-file source))

(spit "/tmp/foo" "lalala")              ; => nil

(slurp "/tmp/foo")                      ; => "lalala"

(move-file "/tmp/foo" "/tmp/bar")       ; => true

(slurp "/tmp/foo")                      ; throws FileNotFoundException

(slurp "/tmp/bar")                      ; => "lalala"

i don't think its a race condition. io/copy loops until it is done. i'd guess you're doing other stuff to the file

I get a lot of help from this channel and just wanted to give a big thank you and show what your help has led to. I just released my first package on Clojars: https://clojars.org/com.owoga/tightly-packed-trie It's an implementation of a https://www.aclweb.org/anthology/W09-1505.pdf and helped me reduce the memory footprint of a language model from several gigabytes to several megabytes (a savings of $15 per month since I can use a $5/mo server rather than a $20/mo server; not a great return on time spent to be fair, but it was fun!). Sending encouragement to those of you new to Clojure and a big shout out to everyone in this channel providing so much help. Special thanks to @seancorfield for https://github.com/seancorfield/depstar and @andy.fingerhut for https://github.com/jafingerhut/cljol.

There is no requirement to do so, but you are certainly welcome to announce your library in the #announcements channel on this Slack if you wish.

@delaguardo I'm still having the issue, the program using the file is OpenJDK Platform binary. I'm using Calva + VSCode how come OpenJDK be using this file?

openjdk is the jvm. clojure runs on the jvm. openjdk is the vm that executes clojure code.

@dpsutton I have followed @delaguardo’s implementation but still no luck :(

can you restart your repl and only execute this code? i'm guessing you've got some open file handlers to it somewhere

sure

nah still the same.. here's what I did: -killed the current repl -started new repl via Calva jack-in -loaded the file and its dependencies -called the function

can you evaluate *e at the repl? I wonder if the stacktrace has more information

seeing that its on the desktop i'm wondering if permissions might come in to play here

:cause "Couldn't delete C:\\Users\\Dave Suico\\Desktop\\desktopCSV3.csv"  :via  [{:type http://java.io.IOException    :message "Couldn't delete C:\\Users\\Dave Suico\\Desktop\\desktopCSV3.csv"    :at [http://clojure.java.io$delete_file invokeStatic "io.clj" 434]}]  :trace  [[http://clojure.java.io$delete_file invokeStatic "io.clj" 434]   [http://clojure.java.io$delete_file doInvoke "io.clj" 430]   [clojure.lang.RestFn invoke "RestFn.java" 410]   [gmaven.csv.main$move_csv_file invokeStatic "main.clj" 43]   [gmaven.csv.main$move_csv_file invoke "main.clj" 36]   [gmaven.csv.main$read_and_move_csv invokeStatic "main.clj" 51]   [gmaven.csv.main$read_and_move_csv invoke "main.clj" 49]   [gmaven.csv.main$eval8153 invokeStatic "NO_SOURCE_FILE" 27]   [gmaven.csv.main$eval8153 invoke "NO_SOURCE_FILE" 27]   [clojure.lang.Compiler eval "Compiler.java" 7177]   [clojure.lang.Compiler eval "Compiler.java" 7132]   [clojure.core$eval invokeStatic "core.clj" 3214]   [clojure.core$eval invoke "core.clj" 3210]   [nrepl.middleware.interruptible_eval$evaluate$fn__960$fn__961 invoke "interruptible_eval.clj" 87]   [clojure.lang.AFn applyToHelper "AFn.java" 152]   [clojure.lang.AFn applyTo "AFn.java" 144]   [clojure.core$apply invokeStatic "core.clj" 665]   [clojure.core$with_bindings_STAR_ invokeStatic "core.clj" 1973]   [clojure.core$with_bindings_STAR_ doInvoke "core.clj" 1973]   [clojure.lang.RestFn invoke "RestFn.java" 425]   [nrepl.middleware.interruptible_eval$evaluate$fn__960 invoke "interruptible_eval.clj" 87]   [clojure.main$repl$read_eval_print__9086$fn__9089 invoke "main.clj" 437]   [clojure.main$repl$read_eval_print__9086 invoke "main.clj" 437]   [clojure.main$repl$fn__9095 invoke "main.clj" 458]   [clojure.main$repl invokeStatic "main.clj" 458]   [clojure.main$repl doInvoke "main.clj" 368]   [clojure.lang.RestFn invoke "RestFn.java" 1523]   [nrepl.middleware.interruptible_eval$evaluate invokeStatic "interruptible_eval.clj" 84]   [nrepl.middleware.interruptible_eval$evaluate invoke "interruptible_eval.clj" 56]   [nrepl.middleware.interruptible_eval$interruptible_eval$fn__991$fn__995 invoke "interruptible_eval.clj" 152]   [clojure.lang.AFn run "AFn.java" 22]   [nrepl.middleware.session$session_exec$main_loop__1058$fn__1062 invoke "session.clj" 202]   [nrepl.middleware.session$session_exec$main_loop__1058 invoke "session.clj" 201]   [clojure.lang.AFn run "AFn.java" 22]   [java.lang.Thread run "Thread.java" 832]]}

will check if I could run this in other drive

oh my apologies, this function actually works

was it just lacking permissions to delete files from the desktop?

@dpsutton actually the move-csv-file works by itself when called alone but now when I call read-and-move-csv the error happens. Your suspicion of 2 handles was right.

don't use def like that. use let

yes yes, I'm still refactoring things

no worries. it's possible that that's preventing garbage collecting since it's still defined and that prevents deletion

done refactoring with let, still the error persist when -reading csv to a variable -moving the csv to another path should I be wrapping these using with-open?

can you post the source of move-csv-file as it currently is?

and i'd restart your repl since you probably still have some defs around

Alright, I'm posting 2 functions

no more defs and I'm restarting my reply every time I call it

oh where did the input streams come from? if you use (io/file source-file) you don't need to worry about the inputstreams

I just copied @delaguardo’s snippet silly me

no worries. the docstring of io/copy says it doesn't close any streams that it didn't open itself. If you provide the file, it will do the stream opening and closing so you don't need to worry about it

so in my case, my csv reader isn't closing my stream?

can you restart your repl and post the source of these two functions again?

and make sure no code is running at startup, no top level forms do work, etc

sure

nevermind the defs, i'll refactor as we speak

I've refactored move-csv-file to this and it is working. Now to test read-and-move-csv

read-and-move-csv doesn't work

ok. so that means one of the two branches csv-to-vector or read-csv must be interacting in a way to prevent it. do you know which branch is taken?

the read-csv function is doing the file reading, the csv-to-vector is only for converting the rows from read-csv if the user opts to

and which branch is being taken?

uhm a library called ultra csv does the actual reading

(if (true? as-vector?) (csv-to-vector source-file) (read-csv source-file)) only one of these two branches is taken. which branch is being taken?

my apologies I may have misled you, allow me to correct. both csv-to-vector and read-csv reference to the same library (ultra-csv)

when you are calling the function read-and-move-csv you are passing in as-vector?. this influences the expression (if (true? as-vector?) (csv-to-vector source-file) (read-csv source-file)) . this form has two branches. which branch is being taken?

the as-vector parameter is a boolean variable. It depends on the client what value to assign in that variable when true, it will take you to csv-to-vector when false, just plain read-csv

i completely understand that

when you are invoking this function and saying that it doesn't work, what value are you passing in for as-vector? and therefore which branch is being taken?

uhm both false and true and both fails

ok. well let's just stay with false and diagnose how read-csv will interact with deleting the file

so we know that we are trying to delete the file after reading it. can you post the docstring for csv-reader/read-csv?

my bad, I should've known this earlier. I am rereading my tasks spec and I just realized that read-csv and move-csv are separate functions and they shouldn't be interacting with each other meaning read-and-move-csv shouldn't exist. .

read-csv and move-csv works when called alone I guess I'm just creating my own problem by putting them together as read-and-move-csv.😅

https://github.com/ngrunwald/ultra-csv/blob/master/test/ultra_csv/core_test.clj#L13

these tests show that there's a close! function that needs to be called. You'll need to ensure that you get all the data from the csv and then close it before trying to delete it. I suspect that's your problem there

this seems like a strange api to me

ahh now it's clear to me, that csv reader is not actually closing the file. Gotta add this closing call to my reader as well. I guess we'll end here thank you very much man I appreciate your efforts!

for sure

i suspect you're going to have to figure out how to get all of the results out of the csv. it's possible it's lazy and gives you results as you need? not sure

yes it gives me the results I need but I have no idea yet how will it behave when dealing with large files or when the next call is move-csv

can you recommend other library for reading csvs?

Clojure data csv is the go to

okay man I'm gonna try that

thanks!

Testing question - I’m trying to test that a method outputs the proper date string.

(t/deftest test-print-period-date
           (t/is (= "2021-W53" (c/print-period-date c/period-week -1 (ti/date-time 2021 01 8))))
           (t/is (= "2020-12" (c/print-period-date c/period-month -1 (ti/date-time 2021 01 8)))))

; expected:
2021-W53
; actual:
(2020-W53)

what does (c/print-period-date c/period-month -1 (ti/date-time 2021 01 8)) output in your repl?

also, i'm seeing an expectation of 2020 as the year but (ti/date-time 2021 ...) if that's your issue?

I get a quoted string back when I run it in REPL

can you paste the output here?

(print-period-date period-week -1 (t/date-time 2021 01 8))
"2020-W53"

(package changes between my test and core NS)

well that's certainly not equal to "2021-W53" which you were expecting

it seems like your repl has some formatting of test results going on

this is what it looks like for me for (is (= "bob" "alice"))

FAIL in (foo) (NO_SOURCE_FILE:1)
expected: (= "bob" "alice")
  actual: (not (= "bob" "alice"))
nil

In Clojure, when I want to check the type of the data I normally do (class foo) but in Clojurescript that is not working, is there something similar in Clojurescript for examine the data, for example if it is a vector, map …

Hi there, can anyone tell me why this

(defmacro no-matching-ctor [q]
  (when (symbol? q)
    [(constantly true)]))

(no-matching-ctor ?e)

1. Caused by java.lang.IllegalArgumentException
   No matching ctor found for class clojure.core$constantly$fn__5690 

(macroexpand '(no-matching-ctor ?e)) 
=> [#function[clojure.core/constantly/fn--5690]]

because you are returning a function object from the macro instead of code that evaluates to the function object

basically https://clojure.atlassian.net/browse/CLJ-1206 but with a slightly different error message

Hi, how can I solve this using next.jdbc.sql/insert-multi! with postgres?

Can't infer the SQL type to use for an instance of java.util.Date. Use
   setObject() with an explicit Types value to specify the type to use.

create table in_sms  (
  in_sms_id serial not null primary key,
  codigo int not null,
  remitente text not null,
  mensaje text not null,
  fechaingreso timestamp without time zone not null,
  unique(codigo)
);

(def input [{:codigo 493
               :remitente "50588997766"
               :mensaje "Loren Ipsum"
               :fechaingreso #inst "2021-04-13T18:32:00"}
              {:codigo 494
               :remitente "50589890909"
               :mensaje "Loren Ipsum"
               :fechaingreso #inst "2021-04-13T18:33:00"}])

(defn insert-incoming-sms!
  "Insert multiple SMS."
  [db sms-coll]
  (let [values (->> sms-coll
                   (mapv vals)
                   (map vec)
                   vec)]
    (sql/insert-multi! db :in_sms
                       [:codigo :remitente :mensaje
                        :fechaingreso]
                       values)))

@orlandomr27 The next.jdbc documentation talks about this… just a sec, let me pull up the link…

https://cljdoc.org/d/com.github.seancorfield/next.jdbc/1.1.646/doc/getting-started#working-with-additional-data-types — “In particular, PostgreSQL does not seem to perform a conversion from java.util.Date to a SQL data type automatically. You can require the next.jdbc.date-time namespace to enable that conversion.”

There’s a lot of documentation for next.jdbc so it can seem a bit overwhelming, but I would recommend reading through the entire thing at least once — and since you are using PostgreSQL, there’s a whole section in the Tips & Tricks page that will help you with various custom PG types.

There’s a #sql channel where it can be easier to get help on JDBC/SQL stuff (since it’s much lower traffic than #beginners — and I have #beginners muted so I don’t always see next.jdbc questions here, but I don’t have #sql muted 🙂 ).

I’m reading and consulting the docs while composing my functions, but somehow I missed this part. Thank you @seancorfield I’m gonna post further questions on #sql channel. By the way! Awesome work with next.jdbc 🙂

Also, be very careful about (mapv vals) over a sequence of hash maps — hash maps are unordered so you will not necessarily get the columns in the order you expect (and you aren’t really guarantee each hash map will return keys in the same order).

The safest thing to do there is (mapv (juxt :codigo :remitente :mensaje :fechaingreso) sms-coll) so you can control the order of the columns.

😲 I wasn’t aware of that. Thanks!

To avoid repetition, you could do:

(let [cols [:codigo :remitente :mensaje :fechaingreso]]
  (sql/insert-multi! db :in_sms cols (mapv (apply juxt cols) sms-coll)))

worked like a charm!