Clojurians Log v2

Thanks, all! I should have mentioned I'd found the community link, but wasn't sure how vetted or up to date it is.

Hi, does anyone know if and how I can extract a list of all required namespaces from a given namespace programmatically in Clojure?

Something like the following:

(ns-requires *ns*) ;; => [clojure.set clojure.string]

@art6661322 a namespace is either already loaded or it is not, but it doesn't matter which namespace did this, so this information isn't saved anywhere I believe. you can inspect the aliases that are valid within a namespace

This can give you some info:

user=> (binding [clojure.core/*loading-verbosely* true] (require '[clojure.set] :reload))
(clojure.core/load "/clojure/set")
nil

Thanks for the help. I'm trying to solve a problem where I want to test if the given namespace loads the proper list, so I can have lein test signal if something weird is going on. We're using a bit of dynamic namespace parsing, so it really matters for us which namespace loads what

for testing you could use with-redefs on require

user=> (with-redefs [require (fn [& args] (prn :req args))] (require 'clojure.set))
:req (clojure.set)
nil

(ns-aliases (find-ns 'foo.bar))

@thheller that's not sufficient here:

user=> (require '[clojure.set])
nil
user=> (ns-aliases *ns*)
{}

true, it is not 100%. most of the time things will have :as aliases though so should be good enough

Not for my case, unfortunately

What you can also do as a test is, require a namespace and test if after doing that, the other namespaces have been loaded?

hmm yeah then it gets tricky, especially if there are some namespaces that are AOT compiled since those won't show up in regular require logs

unless you temporarily re-define require using with-redefs

once it enters AOT loading it won't use require anymore so redef does nothing

I think I can parse out the require list or go with the with-redefs solution

true

Hi, following my previous post. I re-did evaluations with criterium. 16ns for the java method, vs 64ns for the native clojure one. Is there any reason why clojure does not implement rand-intthat way ? https://gist.github.com/a30660391c68255e65ff60970e706c22

@caumond consider using a gist for long code fragments

I think this would be a good question for http://ask.clojure.org as well

Try to benchmark java version when you create seed each time. Notice that clojure version probably allocates it since it is not an input argument

ok, I'm just trying to learn what a gist is first, (;p

or pastebin :)

I did it !

yay :)

Seed is expected to be done once ! If I re-init seed each time, I'll have always the same result

And I followed the question to http://ask.clojure.org following your recommandation. Thx.

Procrastination alert! I can implement this operation

[{:key :foo :value 1} {:key :bar :value 2}] => {:foo 1, :bar 2}

(reduce #(assoc %1 (:key %2) (:value %2)) {} coll)

(into {} (map (juxt :key :value) coll))

last one is good except it could be in transducer form:

(into {} (map (juxt :key :value)) coll)

what do you think is wrong with it?

I have a data reader #object defined. This works:

(def x #object[org.joda.time.DateTime 0x3ff80a66 "2021-07-04T09:54:54.937Z"])

(identity #object[org.joda.time.DateTime 0x3ff80a66 "2021-07-04T09:54:54.937Z"])

not sure. i think it has something todo with the fact this the expression is saying ‘add this value under this key’, but in my head i’m thinking ‘add as this key’. i suppose (juxt :key identity) would express that. thanks!

I’ve just type hinted the return value of a protocol function with a class. It’s a common pattern in our code to call this function inside a with-open. However it seems this is strictly speaking a breaking change, because consuming namespaces now need to import the class that the type hint relates to, is that right?!

I know get compile errors at call sites that call (with-open [(myprotocol/open-the-thing ,,,)] ,,,)

What does #object read into ?

I don’t know why. Why does it matter?

Sorry @alexmiller I don't understand. Why the performance is important? the seed?

What's the reason behind the whole thread?

I know about this article that advocates for passing system as a first argument for a protocol impl: https://puredanger.github.io/tech.puredanger.com/2014/01/03/clojure-dependency-injection/

There's some other article that says this for functions that does IO?

To make your IO execution to not depend on global variables and to pass the dependency in a explicit way (a.k.a. as a first argument of a function)

I've been programming a lot in this way since I began on Clojure and "it just felt right", how can I explain that it gets your code better more than simply saying that It's easier to debug / It's easier to mock?

@d.ian.b do you mean passing the component system as the first arg?

Yeah, some form as component system, can be a map of system stuff (as right now I'm using https://github.com/piotr-yuxuan/closeable-map to make the system component)

Note that component itself discourages it: https://github.com/stuartsierra/component#do-not-pass-the-system-around

(although I have to admit that this is quite a common pattern in the codebase I'm working on 😅 )

yeah, but right now I'm not using component itself, if I have to make IO to a database I can do something like

(defn my-insertion-db! [{::config/keys [datasource]} data]
  (log "some data!" (pr-str data))
  (->> data
       some-transformation
       (jdbc/execute! datasource))

and for testing I can do something like

(with-open [system (-> config make-dev-system!)]
  (my-insertion-db! system data))

it's basically the same thing, just a data structure with components

No function should take the entire system as an argument => I know that sometimes you would prefer to pass part of the system (the part that your IO needs to use to handle it) as the argument of your function

but destructuring seems to fit very well for these stuff

but the thing I don't like is to have the system as a global variable (people are arguing that we need to make functions that has global vars embedded and returns a function that does the IO, because is simple to mock using with-redefs)

I agree, pass arguments in favor of globals

because polutes functions to pass around system for every stuff (I've said not every stuff, just IO stuff haha)

I think is more simple / clean to have just a map 😄

the idea is that instead of my-insertion-db receiving system it receives the datasource itself?

yes

only what the function needs I guess

but this is just an opinion, do what you want

Just the public APIs generally needs the system-map, other private functions downstream generaly will get only what it needs

1. I feel like a newbie so I may have done something fundamentally wrong 2. If that implementation is quicker and equivalent (what I think), it may be an improvement, something like :

(def ^java.util.Random seed (java.util.Random.))

(defn rand-int-alternative
  "Returns a random integer between 0 (inclusive) and n (exclusive)."
  {:added "1.0"
   :static true}
  [n] (.nextInt seed n))

Ahhh, I guess that it happens because destructuring a map is so easy 😄

yeah, I generally agree on pass only what the function needs, but destructuring helps a lot with this on most part of the time 😄

> but destructuring seems to fit very well for these stuff Exactly!

as the author, I think "advocates" is a strong description - this is an idea, it may be useful, or it may not :)

If you need more context I am trying to port an operations research tool I built years ago in C++. That kind of algorithms are combinatorial optimization, with a huge number of iterations on int-array. I start with some benchmarking between java and clojure and find out the global execution time gap was due to that rand-intfunction. That kind of use case are demanding from a time execution prospective (billion of iterations are expected).

I'm struggling to find this on clojure 😞, it has for Elixir though https://dashbit.co/blog/mocks-and-explicit-contracts

thanks, that was the context I was looking for. I don't think anything is preventing you from using your alternative method. I don't know why it's implemented the way it is. I'll make a ticket out of it when I get a chance.

I'm trying to use Apache Tika to detect a file type. I'm getting some reflection errors I can't seem to figure out, and I haven't done much Java interop prior to this. Here's what I have:

(with-open [xin (<http://clojure.java.io/input-stream|clojure.java.io/input-stream> (get-in params [:uploaded_file :bytes]))
                            tika (Tika.)
                            ftype (org.apache.tika.Tika/.detect tika)]
                  (prn "tika ----&gt;" ftype))

To your question about how to explain the value proposition of passing arguments vs. depending on globals: It sounds like you're trying to convince colleagues about the benefits of pure functions. Unfortunately, you'll probably have to tailor your arguments to your audience so your pitch will resonate with their specific experiences. Some arguments you could you could be: parallelization of tests - without global state you can safely parallelize your tests. transparency - global state is an additional, silent, side-channel input. It doesn't come through the front door like other function arguments. So the developer who picks up the test/code needs to know all of the relevant context to figure out how the function will behave. If you use pure functions and you know that the behavior only depends on the arguments supplied to the function, I've found it's much easier to understand and easier for other developers to understand.

So the first complaint is 'reference to field close can't be resolved' and then 'reference to field detect on org.apache.tika.Tika can't be resolved'

I assume this is either an import issue or a namespace issue

(.detect tika)

That was my first attempt, and I got: 'No matching field found: close for class org.apache.tika.Tika'

(defn joda-time-data-reader [[class-symbol _objid date-time-string :as value]]
  (if (= "org.joda.time.DateTime" (name class-symbol))
    (tf/parse date-time-string)
    (throw (ex-info "Unsupported #object instance" {:value value}))))

I'm conflating 2 issues, probably because I tried a plain 'let' in place of 'with-open' to see if I could get past the 'close' error

this is because you have this object in your with-open

org.joda.time.Datetime

in what way does it "not work" ?

bindings in with-open are called with close at the end

When I put

(def x #object[org.joda.time.DateTime 0x3ff80a66 "2021-07-04T09:54:54.937Z"])

(defn y [] #object[org.joda.time.DateTime 0x3ff80a66 "2021-07-04T09:54:54.937Z"])

Syntax error compiling fn* at (src/.../plans/presentation.clj:324:1).
Can't embed object in code, maybe print-dup not defined: 2021-07-04T09:54:54.937Z

Oh, silly me. A separate 'let'

Similarly if I just put like (identity #object….) in namespace top-level and try to load it, same error

yes, I agree this is not a critical issue. Thanks for your confirmation to use the 'alternative method'. It's all what I need.

seems odd that def parses differently than other expressions

And now it cannot resolve the 'detect'...

Which is why I tried the full namespace... let's try that again...

Hmm. So the 'close' is resolved, but not 'detect'

just try it one expression at a time

Got it!

(.detect tika xin)

Thanks, @borkdude

right, so the data reader is reading into an object. that object can't be represented in bytecode as a value, so you can't compile the function like that. def is different case

you can probably add a printer for this type of object though and get it to work

you do that by extending the print-method and print-dup multimethods

you then have roundtripping of read/print which are complementary operations

I see, thank you

Also if you define a print operation for these values, then it would be preferable for the reader to not read #object, but rather read something like #joda/datetime

for sure

This will prevent issues with other objects being read as date times.

Does anyone know a file-system backed TTL cache? I'm communicating with an API with limits with a process that will stop/start on my machine. I've not found a java thing surprisingly!

@dominicm Do you mean for http caching, etags etc?

@rickmoynihan I can hit an api 20 times a day, and I only want to do that every hour to make sure I don't deplete that. I want to make sure while I'm developing I don't deplete it either by restarting my jvm a few times.

something like https://github.com/oliyh/martian/tree/master/vcr ?

There are some other similar projects “vcr” will probably help you mind them.

@rickmoynihan vcr is more about testing. I am looking for something that expires after 1 hour, and to fetch a new one when that happens.

redis?

A filesystem would be great, this project will have no other dbs or anything.

postgres with a timestamp column?

or sqlite for that matter

could you use varnish as a proxy?

@borkdude I mean, if I'm gonna roll my own I'll probably just use duragent and the local-file thing 😛

yeah, it doesn't seem that much work to be honest

Feels like one of those obvious things though, I thought it would already exist.

I am not sure I understand why it works: ns A has a macro that outputs a call to another macro from ns B ; ns C calls that macro but does not require ns B ; all is cool

(defmacro macro-a [x] `(B/macro-b ~x))

just use macroexpand to see what happens?

when macro-a expands it will make the symbol B/macro-b fully qualified

and if the namespace with that symbol is loaded by any other namespace it will be resolved correctly

Right, I might have been confused on that last point in my CLJC project. Indeed, ns B was being loaded in Clojure but not in CLJS, so it was misbehaving in CLJS. Make sense!

Just being lazy instead of translating this but anyone know of a clojure bigdec natural log implementation in the wild, like this java one https://stackoverflow.com/questions/739532/logarithm-of-a-bigdecimal

this relation seems really handy: log(AB) = log(A) + log(B). This might get into things like numerical stability and errors related to that

That relation is how slide rules work

(juxt :key idenity) creates a different output though

whenever I find myself writing (juxt ... identity) I reach for Medley's index-by

Is there a way to have the std{out,err} from shell/sh sent somewhere in real-time? Now I only get :out and :err info after the job is done 🙂

I believe Java ProcessBuilder class can enable you to do that. I don't think that the built-in-to-Clojure shell/sh can do that.

@endrebak With babashka.process (https://github.com/babashka/process) this is very easy:

@(babashka.process/process ["ls"] {:inherit true})

https://docs.oracle.com/javase/7/docs/api/java/lang/ProcessBuilder.html

yeah I have a snippet doing that with ProcessBuilder:

(let [pb (ProcessBuilder. ["lua"])
      p (.start pb)
      out (.getOutputStream p)
      in (.getInputStream p)]
  (spit out "print \"hello from lua\"\n")
  (slurp in))

of course for real interactivity you have to use something other than slurp / spit (and that means more complexity) - I'm sure there's some good libs that simplify that streaming IO stuff out there

sounds like you want the stdout / stderr for monitoring? you should be able to .read from the output stream and error stream in a loop and trigger a callback for each line / chunk / whatever

you can also use processbuilder to stream to a file or pipe instead of a stream, and to join the streams if that simplifies your use case as well

and vanilla clojure / interop version of the same:

(doto (ProcessBuilder. ["ls"]) (.inheritIO) (.start))

Cool! I'll have to look more into babashka. But let's say I have multiple processes running at the same time in futures. Can each of them send data to a tap or something with inherit? So I get messages like:

jobid1 :stdout "message from job1"
jobid3 :stderr "this job failed"
... 

inherit just sends to *out* and *err*- see my reply in the other thread for using the streams with a callback

inherit doesn't send to *out*, it sends to System/out

oh, good catch

Thanks to both of you 🙂

I am using my software to run multiple (shell) jobs concurrently and want output from them ASAP and continually so I can see if anything goes wrong. So yes, I guess monitoring is the word.

@endrebak Example with babashka.process (which you can use as a regular Clojure library):

(require '[babashka.process :refer [process]])
(process ["foo"] {:inherit true})
(process ["bar"] {:inherit true})
.. wait .. e.g. @(promise)

they mentioned they don't want the output to go to stdio of the repl

> and want output from them ASAP and continually so I can see if anything goes wrong oh, I thought he/she/they did say that here

But if multiple jobs send to stdout concurrently they might overlap. Also, I want to send some to the front-end and so on so it would be best to send it to a tap or something 🙂

just make a loop that reads the output line by line, prints it and does whatever else with it

This is all new to me. Sounds like it is not that hard then. Thanks!

I have a similar example to that here: https://book.babashka.org/#child_processes

Here is a simple example that reads the output from yes:

(require '[babashka.process :as p :refer [process]]
         '[<http://clojure.java.io|clojure.java.io> :as io])

(def yes (process ["yes"] {:err :inherit
                           :shutdown p/destroy}))

(with-open [rdr (io/reader (:out yes))]
  (binding [*in* rdr]
    (loop []
      (let [line (read-line)]
        (println :line line))
      (recur))))

why does evaluating an empty anonymous function using the #() syntax yield an empty list? I expected... maybe nil?

user=> (#())
()
user=> ((fn []))
nil

See what the reader returns. ‘#()

I see! nice!

Learned that one from noisesmith.

is reducing a list of maps a bad idea? I'm trying to write a reducer function, but it seem problematic. I want string values to turn into a list of strings, list values to be a flattened list, but picking the right incantation (concat cons conj, etc.) seems challenging.

@joel380 when in doubt, loop it out

skip all them helpers and just write a big ol' loop and recur

A transducer with a mapcat would work a treat on that