Clojurians Log v2
clojure
Why does lein repl throws Execution error (ClassNotFoundException) at <http://java.net|java.net>.URLClassLoader/findClass (URLClassLoader.java:466). when I try to run a function in my project?
Impossible to answer without knowing more about exactly what you are trying to do and exactly what the error is (i.e., more of a stacktrace etc).
That was it. Had to make a little java shim that setContextClassLoader before loading Clojure. Is there any way to do this with just clojure?
Got it working. Previously I used
(ns app.db)
(get-posts)Execution error (ClassNotFoundException) at java.net.URLClassLoader/findClass (URLClassLoader.java:466).(use 'app.db)
(get-posts)You should probably use (require 'app.db) and then (app.db/get-posts)
use is discouraged (and isn't available in ClojureScript as I recall).
(use 'app.db) is the same as (require '[app.db :refer :all]) which, again, is discouraged.
(require '[app.db :as db]) followed by (db/get-posts) would be preferable -- always use an alias rather than referring in names.
(ns app.db) will create an empty namespace called app.db so that's almost certainly not what you want in the REPL.
Thanks, I will keep that in mind.
As I said (ns app.db) creates an empty namespace. Nothing defined in it.
It sounds from this that you're fairly new to Clojure?
Yup you are right. I am trying to jump to Clojure landscape.
I'd recommend asking questions in #beginners -- you'll find people are more patient there and less likely to assume knowledge you might not have yet.
The experienced folks in #beginners have opted in to helping new folks in great detail.
Hmm sweet. I will jump to that channel.
Is there a Clojure/(Script) library for using Secure Remote Password Protocol? I know there's some Java/(Script) ones, but curious if there's something that exists for Clojure already or if I'm gonna have to define a wrapper around existing ones.
Hello,
Does any use Google BigQuery API with in Clojure?
I am trying to query some insert jobs in a non-US non-EU region but when I do a get call to fetch the status of the job I get
404 Job not found
This is my job spec which has location set in it
(defn- load-job-spec [table]
(let [{:keys [project-id dataset-id table-name table-suffix]} table
table-id (str table-name "$" table-suffix)
time-string (->> (LocalDateTime/now) (.format (DateTimeFormatter/ofPattern "yyyy-MM-dd_HH-mm-ss")))
job-id (str table-name "-" table-suffix "-" time-string)
job-reference (doto (JobReference.) (.setLocation "australia-southeast1") (.setJobId job-id) )
table-reference (doto (TableReference.)
(.setProjectId project-id)
(.setDatasetId dataset-id)
(.setTableId table-id))
config-load (doto (JobConfigurationLoad.)
(.setDestinationTable table-reference)
(.setEncoding "UTF-8")
(.setWriteDisposition "WRITE_TRUNCATE")
(.setSkipLeadingRows (Integer. 1)) ;; CSV headers
(.setMaxBadRecords (Integer. 0)) ;; Failure on garbled records
(.setAllowQuotedNewlines (Boolean/TRUE)))
job-config (doto (JobConfiguration.) (.setLoad config-load))
job (doto (Job.)
(.setJobReference job-reference)
(.setConfiguration job-config))]
job))(defn load-file-bigquery [fullpath]
(let [name (s/replace fullpath #"^.*4244_" "")
name (s/replace (s/lower-case name) #"_\d{6}.csv$" "")
name-parts (s/split name #"_")
table-suffix (last name-parts)
table-name (s/join "_" (butlast name-parts))
content (FileContent. "application/octet-stream" (File. fullpath))
project-id (cfg/config :project-id)
job-spec (load-job-spec {:project-id project-id
:dataset-id (cfg/config :dataset-id)
:table-name table-name
:table-suffix table-suffix})
job-id (->> job-spec (.getJobReference) (.getJobId))
_ (log/info "BigQuery Load Job : " job-id " : " fullpath)
bq @bigquery-conn ;; Blocks until initialised
job-obj (->> bq
(.jobs)
(#(.insert % project-id job-spec content))
(.execute)
)
status (->> job-obj (.getStatus) )]
(log/info "job-obj outside "job-obj)
(loop [status status] ;; Waiting until successfully processed
(log/info job-id " : " (->> status (.getState)))
(if (= "DONE" (->> status (.getState)))
(do (log/info "Status seems done?")
(if-let [errors (.getErrors status)]
(do
(log/info "seems like we have errors")
(vec (map #(.getMessage %) errors)))
nil))
(do
(Thread/sleep 3000)
(recur (->> bq
(.jobs)
(#(.get % project-id job-id))
(.execute)
(.getStatus))
))))))Job job = <http://bigquery.jobs|bigquery.jobs>().insert(PROJECT_ID, runJob).execute();
String status = job.getStatus().getState();
while(!status.equalsIgnoreCase("DONE")) {
status = <http://bigquery.jobs|bigquery.jobs>().get(PROJECT_ID, job.getId()).execute().getStatus().getState();
System.out.println("Status: " + status);
Thread.wait(1000);
}
= is not the same thing as the equalsignorecase method
My guess is you have some error creating the job, but you never print errors outside before starting to loop
sorry for confusing post, i put the java code just as an equivalent
job runs ok, however I cannot get the status unless I pass the location to get call
(->> bq
(.jobs)
(#(.get % project-id job-id))
(.execute)
(.getStatus))
How come (let [foo "bar"] (eval (read-string "foo"))) doesn’t work? How can I get eval to pick up locally scoped bindings?
Alright, I found this, which says that I either need to def or declare my bindings/vars: https://stackoverflow.com/a/6221829
Unfortunately, defing doesn’t seem to work either… Or, it does when I do it in the REPL, but not in my actual namespace. Any ideas why?
You’ll need a macro that captures lexical environment and feeds it into eval
Hello everyone, I'm using the jackdaw library for connecting to Kafka I want to use a key.serializer and value.serializer for Kafka producer other than Kafka's own serializers. this is my code for the producer
(def serialize nippy/freeze)
(def Kafka-Edn-Serializer
(reify Serializer
(serialize [this topic headers data]
(serialize data))
(serialize [this topic data]
(nippy/freeze data))
(configure [this _ _])
(close [this])))
(def producer-config
{"bootstrap.servers" "localhost:9092"
"key.serializer" (-> Kafka-Edn-Serializer
.getClass
.getName)
"value.serializer" (-> Kafka-Edn-Serializer
.getClass
.getName)
"acks" "all"
"client.id" "foo"})
(defn -main []
(jc/producer producer-config)
)lein run to run this I got the following error
Invalid value producer_example$reify__12915 for configuration key.serializer: Class producer_example$reify__12915 could not be found.makes sense, thanks 🙂
Have you tried using deftype instead of reify?
I think reify doesn't create an actual type, unless aot compiled.
I will try it thanks
Hello all, are you using lein to create templates for base project? Is there other way instead of lein to do it?
mkdir src && echo '{}' > deps.edn 😁
ah, you asked about creating templates, not a particular template to use instead of lein..
I am using https://github.com/uswitch/opencensus-clojure tracing library which follows the opencensus standard. Here is my code:
(def routes ["/" [["ping" {:get (constantly {:status 200
:body "pong"})}]
["foo" {:post (->> foo-handler
(trace/span "foo-file-handler"))}]
[true (constantly {:status 404})]]])
(m/defstate server
:start (jetty/run-jetty (bd/make-handler routes) {:port 8080
:host "localhost"
:join? false})
:stop (.stop ^Server server))
(m/defstate tracer
:start (jaegar/report "foo-service")
:stop (jaegar/shutdown))My bad, the traces are probabilistic as mentioned in the README of the library. I was using the default probability which less than 0.1 (I guess). I changed the probability to 1. Now the traces show up
For calls to assoc with multiple key-value pairs, would it make sense to use a transient? (In this code): https://github.com/clojure/clojure/blob/ee3553362de9bc3bfd18d4b0b3381e3483c2a34c/src/clj/clojure/core.clj#L192-L199
Maybe not, because not everything passed to assoc will be a IEditableCollection
☝️
@vlaaad yes, creating templates for use as starting of a new project
Are there idioms in clojure which allow me to partially process a seq and return the rest of the sequence without iterating over it twice (as with split-at)? It is easy to write myself but wondering if I am missing something?
Maybe you could us reduce in combination with reduced which terminates reduce loops. Then you can add both the processed items results and the rest of the sequence.
@emccue coming out of a an iterator-seq I have three types of objects, which are not interleaved. the first type of objects I need to put in some kind of map, while the rest is being processed using earlier created map and are processed with side effects. Naturally partition-by would be useful but it keeps the head of the first stream alive and thus I run into out of heap. For me it is important to not walk this long stream multiple times while also not keeping any heads alive while processing them. a custom loop recur works for the moment but feels non-idiomatic.
@oscarlinusericsson I haven't thought about reduced interface, I will play with this idea a bit. Thanks!
The simplest cases of loop/recur do usually map to a call to reduce
so
(let [long-lazy-seq ...
data-from-before {...}
process! (partial do-side-effect! data-from-before)]
(loop [remaining long-lazy-seq
collected []]
(if (empty? remaining)
collected
(if (predicate? (first remaining))
(do (process! (first remaining))
(recur (rest remaining) collected))
(recur (rest remaining)
(conj collected (first remaining)))))))so this (which is what it sounds like you have)
can map to a separate reducing function (which is kinda hard to name, but c'est la vie)
(let [long-lazy-seq ...
data-from-before {...}
process! (partial do-side-effect! data-from-before)
reducing-function (fn [collected item]
(if (predicate? item)
(do (process! item)
collected)
(conj collected item)))]
(reduce reducing-function [] long-lazy-seq))so roughly this - reduced isn't super helpful except to terminate early, which it doesn't sound like you want to do
Hm, thanks. My code indeed looks a bit like your earlier one. I wanted to actually have some kind of split in between when it operated on the first types of objects and the second types, which seems hard to do in reduce (e.g. using a transient to construct the map of the first elements).
While I generally prefer map/reduce, in this case something like:
(let [[first-elem-map r] (do-while is-first-elem? process r)
[_ r] (do-while (is-second-elem? side-effect-on-second-fn r)
_ (do-while (last-element? side-effect-on-thid-fn))]
...)I usually don't like reduce and prefer the explicit loop, but thats me. You can maybe handle the different cases via a dispatch thing
so a multimethod or just a condition on what function to call
but conceptually you need to handle each item in the same "place"
splitting a lazy sequence into logical parts necessitates that at least one will hold on to the head while the one you choose to evaluate first is running
you can maybe "queue up" side effects with filter
(let [long-lazy-seq ...
data-from-before {...}
process! (partial do-side-effect! data-from-before)]
(->> long-lazy-seq
(filter (fn [item] (if (predicate? item)
(do (process! item) false)
true)))
(reduce (fn [collected remaining-item]
(conj collected remaining-item)
[])))but uhh, beauty is in the eye of the beholder
I think I stay with my do-while approach, which looks like that:
(defn do-while [pred f s coll]
(let [c (first coll)]
(if (pred c)
(recur pred f (f s c) (rest coll))
[s coll])))I tried the multimethod approch as well, but this loop might process a lot of items and I am unsure if I add just too much overhead in that case.
The filter stuff looks like a neat idea, but I agree, it feels weird to do anything else in filter than filtering.
How can I run a shell command in the foreground (i.e. with stdin/-out/-err attached) in Clojure? Ideally I’d like the command to believe it’s in a TTY as well.
there is a clojure.java.shell namespace with some support for this (not sure about tty)
lately though I've actually been using Java interop directly to the newer Java ProcessBuilder api
that has full support for a variety of options of handling in/out/err
seconding that ^
java >=9 give you a completion callback too
hmm.. how would I attach stdin/out/err with clojure.java.shell?
if you (doc clojure.java.shell/sh) you can read all the options
but I find ProcessBuilder is actually easier to use
I did look at that doc, but AFAICT it doesn’t support what I’m trying to do
if you want more clojure-y flavor, there is stuff like https://github.com/clj-commons/conch
if you're doing a lot of this stuff, that can be pretty nice too
I’m also not sure if there’s an easy way to run in the foreground with the standard streams hooked up using the ProcessBuilder API. I tried Googling for it but the answers I found all involve manually redirecting output from/to the process (e.g. https://stackoverflow.com/a/17393606), which seems a bit cumbersome
I also tried using conch, but couldn’t figure out how to get it working. The closest I could get was doing something like:
(require '[me.raynes.conch.low-level :as sh])
(def proc (sh/run "bash"))
(future (sh/stream-to-out proc :out))
(future (binding [*out* *err*] (sh/stream-to-out proc :err)))
(sh/feed-from proc *in*)Isn't that because *in* and *out* are buffered streams?
Doing unbuffered console I/O in Java is kind of a pain, as I recall...
Yeah that’s probably it
look at inheritIO options on process builder
@ghadi heh, I actually just tried that 🙂 basically doing this: https://stackoverflow.com/a/49706743 works, except that I can’t see my output
e.g. running echo hello in my bash proc works, but I can’t see what I’m typing:
$ (def pb (java.lang.ProcessBuilder. (into-array String ["bash"])))
$ (.inheritIO pb)
$ (.waitFor (.start pb))
bash-3.2$ hello
bash-3.2$perhaps this could be because I’m in a REPL though..
Yeah, that was it 🙂 It works when I use lein trampoline run
Thanks!
@alexmiller Hi Alex, any plans about the next edition Clojure Applied and/or Programming Clojure?
hasn't really been much new since 3ed of Programming Clojure so no plans there
funny enough, we just passed the 5 yr anniversary for Clojure Applied and I was talking to the publisher about it this week. considering whether to do a new ed
anything you'd be burning to have included? :)
@alexmiller Would a new Ed of C.A. deprioritize the use of records somewhat? I recall you mentioning something about that.
(and could it also normalize the use of qualified keywords in hash maps from the get-go? 🙂 )
yes on maps vs records, dunno on qualified keywords
I have a list of ideas already, just wondering what others would be interested in :)
having just gone through the whole book again, I think probably 85% is still pretty solid. I mostly have stuff I'd want to add to it (spec - would replace the schema use for validation and expand generative testing in ch 8), clj stuff, would like to add stuff on macros and java interop
big plus on generative testing
(to add to that, what I often struggle with in generative testing is how to define my properties)
(or, what, ‘good’ properties are)
So I read through the generative testing section again. I have a hard time pinpointing what my exact issue is when I said that I find it difficult to define properties. I think the section is very clear, and definitely hints to what I need. But, I don’t have the intuition when to choose property based testing except for some set patterns (serialize/deserialize springs to mind). Another problem I experience in practice is that properties I do come op with are very broad (serialize -> deserialize = identity) which I struggle to properly assess how useful those are (in a proof burden kinda sense, this generatively tested property ‘buys’ me the assurance that the system does or does not do X). Or, I have trouble finding properties underlying implementations (say, the textbook example of a sort algorithm guarantees orderness with ‘orderness’ being the property and ‘quick-sort’ being a way to get to that) which leads to a feeling of testing by ‘asserting the implementation’ (if you get what I mean). So I guess that most places I read about property based testing explain me how (if I may use an analogy) a screwdriver works, show me examples of screws, but so far, I don’t feel like I have a grasp on starting to realise that a thing is a screw. (Does that make sense?)
(Not all about the book, btw, I’m just trying to … also for myself, pinpoint what makes it hard for me to grasp property based testing)
(I guess you heard this a million times, but the last listing on page 154 is duplicated on 155, or actually the other way around)
it does make sense - I think it is truly difficult to discern what properties are useful to validate. I spent a lot of time thinking about it when writing that chapter. spec has highlighted an additional way to think about it in terms of the in/out of a function too and that's what I'd like to add (although spec 2 is likely to going to change in practice quite a bit)
and yes, there are a number of copy edit issues in the book. we've got lots of errata to go through
I would think that we’d need to tease the subject a part a bit. I would imagine that the job of Clojure Applied would be to showcase how property based testing could/should be used in Clojure. Writing extensively on how to do property based testing in the general case would probably be outside the scope of C A? Oscar Wickström has a book on this https://leanpub.com/property-based-testing-in-a-screencast-editor and did a talk on it at flatMap(Oslo) in 2019 https://2019.flatmap.no/talks/wickstrom in which, IIRC, there were som links to other resources on property based testing.
That sounds interesting, I’ll definitely watch the talk
the only hard parts are defining properties and defining generators :)
also the only parts :)
I took my best swing at this in the current edition, would be curious if you've read it and what you thought
https://grep.app/search?q=prop/for-all is an assist. and looking for test.check :as is another good search term
Its at the office, I happen to go there tomorrow, I’ll read through it
Yeah I found a blog from the F# community
oh thats also clever!
This one: https://fsharpforfunandprofit.com/posts/property-based-testing/
the existing book covers generative testing with test.check, I'd mostly be looking to augment that by talking about how spec plays into it
uh no, there was another with pretty pictures
Ah yes, this one: https://fsharpforfunandprofit.com/posts/property-based-testing-2/
Yeah, +💯 on using Spec instead of Schema and adding more generative testing stuff. I've been working my way through @ericnormand’s Property-Based Testing courses online and finding lots more places to use generative testing than I imagined possible!