Clojurians Log v2

Having some issues implementing something using core.async. What I want is to use go blocks to do a bunch of HTTP requests and process the responses. I'm approaching this coming from golang which may be where my issues are originating. The way i'm approaching this is to have a pool of go blocks trying to read from a channel which is feeding them urls, and then returning results over another channel. The issue im having is that I can't ensure all go blocks are finished working.

Because it would have to be at that exact same relative path in resources and your :exclude would still exclude it.

(defn worker [urls-chan results-chan]
  (async/go (while true
              (let [url (async/<! urls-chan)
                    result (is-vulnerable? url)]
                (if (and (not (nil? result)) (not (empty? result)))
                  (async/>! results-chan (string/join "\n" result)))))))

(defn -main
  "I don't do a whole lot ... yet."
  [& args]
  (let [urls-chan (async/chan 100)
        results-chan (async/chan 100)]
    (dotimes [_ 10]
      (worker urls-chan results-chan))
    (async/go (while true
          (println (async/<! results-chan))))
  (doseq [ln (line-seq (java.io.BufferedReader. *in*))]
    (async/>!! urls-chan ln))))

yes… so I don’t have more ideas

Anyone have an example of java.util.logging formatter in Clojure? The best to print custom JSON :)

https://gist.github.com/hiredman/64bc7ee3e89dbdb3bb2d92c6bddf1ff6 is a mini logging library built on java.util.logging that logs clojure data

thank you, I check this one!

don't use while true

loop only when you take a non-nil from urls-chan

close urls-chan when you are done

in general using go blocks as workers is not useful

since they run on a limited size threadpool (usually capped at 8 threads)

something like async/pipeline-blocking is likely much more of what you want

Hmm yeah potentially, in go you would use go routines even though they are also capped by physical threads, but there's a runtime for switching while they're sleeping

Figured it was the same here

Managed to get this now:

(defn worker [urls-chan results-chan]
  (async/go (loop []
              (let [url (async/<! urls-chan)]
                (if (not (nil? url))
                    (let [result (is-vulnerable? url)]
                      (println url)
                      (if (and (not (nil? result)) (not (empty? result)))
                        (async/>! results-chan (string/join "\n" result)))
                      (recur)))))))

(defn printer [results-chan]
  (async/go (loop []
              (let [result (async/<! results-chan)]
                (if (not (nil? result))
                  (do
                    (println result)
                    (recur)))))))
(defn -main
  [& args]
  (let [urls-chan (async/chan 100)
        results-chan (async/chan)
        worker-chans (vec [])
        printer-chans (vec [])]
    (dotimes [_ 10]
      (conj worker-chans (worker urls-chan results-chan)))
    (conj printer-chans (printer results-chan))
    (doseq [ln (line-seq (java.io.BufferedReader. *in*))]
      (async/>!! urls-chan ln))
    (async/close! urls-chan)
    (mapv #(async/<!! %) worker-chans)
    (async/close! results-chan)
    (mapv #(async/<!! %) printer-chans)))

It's not working though haha, it was almost working a min ago

my initial suspicion is you are doing some blocking operations in go blocks, which is gumming up the threadpool

no, the go macro is just a macro

it is a syntax transformation

it can't turn your blocking io (or blocking on promises, locks, etc) in to not blocking

Current issue is still that it's just exiting before doing anything 😞

Doesn't even take a single value from url-chan in the worker

@cgboal521 what does is-vulnerable? return? A boolean? When I tested it I was getting an error from calling empty? on result with result being a boolean

Think i've figured out the issue, just way too tired haha

I can post the full code if you like, but nah it was a boolean now it returns a string

btw if you don't have it already it might help to have an uncaught exception handler, like this:

;;<https://stuartsierra.com/2015/05/27/clojure-uncaught-exceptions>
;; Assuming require [clojure.tools.logging :as log]
(Thread/setDefaultUncaughtExceptionHandler
 (reify Thread$UncaughtExceptionHandler
   (uncaughtException [_ thread ex]
     (log/error ex "Uncaught exception on" (.getName thread)))))

(ns the-great-escape.core
  (:require
   [clj-http.client :as client]
   [hickory.core :as html]
   [hickory.select :as s]
   [clojure.core.async :as async]
   [clojure.string :as string]
   [lambdaisland.uri :refer [uri]])
  (:gen-class))

(defn doc->hickory [doc]
  (-> (html/parse doc)
      (html/as-hickory)))

(defn fetch-and-parse-page [url]
  (-> (client/get url
                  {:cookie-policy :none
                   :redirect-strategy :none})
      (:body)
      (doc->hickory)))

(defn extract-asset-tags [body]
  (s/select (s/or
             (s/attr :href)
             (s/attr :src))
            body))

(defn extract-href-urls [tags]
  (map #(get-in %1 [:attrs :href]) tags))

(defn extract-src-urls [tags]
  (map #(get-in %1 [:attrs :src]) tags))

(defn extract-asset-urls [tags]
  (->> (extract-href-urls tags)
       (filter #(not (nil? %1)))))

(defn parse-urls [urls]
  (map uri urls))

(defn local-asset? [base-domain {:keys [host]}]
  (or (nil? host) (= base-domain host)))

(defn filter-local-paths [base-domain parsed-urls]
    (->> (filter (partial local-asset? base-domain) parsed-urls)
         (map :path)
         (filter #(not (nil? %1)))
         ))

(defn url->parts-map [url]
  (let [parts (string/split url #"/")]
    {:dir (str "/" (second parts))
     :path (str "/" (string/join "/" (drop 2 parts)))}))

(defn urls->parts-maps [urls]
  (map url->parts-map urls))
(defn get-unique-dirs [parts-map]
  (map #(first (val %1)) (group-by :dir parts-map)))

(defn filter-bad-traversals [parts-map]
  (filter #(and (not= (:dir %1) "/") (not= (:path %1) "/")) parts-map))

(defn build-original-url [base-url parts-map]
  (str (assoc base-url :path (str (:dir parts-map) (:path parts-map)))))

(defn build-traversal-url [base-url parts-map]
  (str (assoc base-url :path (str (:dir parts-map) ".." (:dir parts-map) (:path parts-map)))))

(defn request-and-hash [url]
  (-> (client/get url {:redirect-strategy :none
                       :cookie-policy :none})
      (:body)
      (hash)))


(defn doc->candidates [base-domain doc]
  (->> doc
       (extract-asset-tags)
       (extract-asset-urls)
       (parse-urls)
       (filter-local-paths base-domain)
       (urls->parts-maps)
       (get-unique-dirs)
       (filter-bad-traversals)))

(defn traversal-comparison [base-url candidate]
  (try
    (let [baseline (request-and-hash (build-original-url base-url candidate))
          traversal (request-and-hash (build-traversal-url base-url candidate))
          blank (hash "")]
      (if (not= baseline blank)
        (if (= baseline traversal)
          (build-traversal-url base-url candidate))))
    (catch Exception _
        nil)))

(defn is-vulnerable? [url]
  (try 
  (let [url (uri url)
        doc (fetch-and-parse-page (str url))
        candidates (doc->candidates (:host url) doc)]
    (filter #(not (nil? %1)) (map #(traversal-comparison url %1) candidates)))
  (catch Exception _
    nil)))


(defn stdin-urls []
  (doseq [ln (line-seq (java.io.BufferedReader. *in*))]
    (println ln)))

(defn worker [urls-chan results-chan]
  (async/go (loop []
              (let [url (async/<! urls-chan)]
                (if (some? url)
                    (let [result (is-vulnerable? url)]
                      (if (and (not (nil? result)) (not (empty? result)))
                        (async/>! results-chan (string/join "\n" result)))
                      (recur)))))))

(defn printer [results-chan]
  (async/go (loop []
              (let [result (async/<! results-chan)]
                (if (some? result)
                  (do
                    (println result)
                    (recur)))))))

(defn spawn-workers [urls-chan results-chan n]
  (loop [workers []
         x 0]
    (if (= x n)
      (identity workers)
      (recur (conj workers (worker urls-chan results-chan)) (inc x)))))

(defn -main
  [& args]
  (let [urls-chan (async/chan 100)
        results-chan (async/chan 100)
        worker-chans (spawn-workers urls-chan results-chan 10)
        printer-chans (into [] (printer results-chan))]
    (doseq [ln (line-seq (java.io.BufferedReader. *in*))]
      (async/>!! urls-chan ln))
    (async/close! urls-chan)
    (mapv #(async/<!! %) worker-chans)
    (async/close! results-chan)
    (mapv #(async/<!! %) printer-chans)))

urls over stdin

Don't worry about it though, think i'm close to figuring it out

not yet

tools.build is coming...

(defn worker [urls-chan results-chan]
  (async/go (loop []
              (let [url (async/<! urls-chan)]
                (if (some? url)
                    (let [result (is-vulnerable? url)]
                      (if (and (not (nil? result)) (not (empty? result)))
                        (async/>! results-chan (string/join "\n" result)))
                      (recur)))))))

(defn printer [results-chan]
  (async/go (loop []
              (let [result (async/<! results-chan)]
                (if (some? result)
                  (do
                    (println result)
                    (recur)))))))

(defn spawn-workers [urls-chan results-chan n]
  (loop [workers []
         x 0]
    (if (= x n)
      (identity workers)
      (recur (conj workers (worker urls-chan results-chan)) (inc x)))))

(defn -main
  [& args]
  (let [urls-chan (async/chan 100)
        results-chan (async/chan 100)
        worker-chans (spawn-workers urls-chan results-chan 10)
        printer-chans (conj [] (printer results-chan))]
    (doseq [ln (line-seq (java.io.BufferedReader. *in*))]
      (async/>!! urls-chan ln))
    (async/close! urls-chan)
    (mapv #(async/<!! %) worker-chans)
    (async/close! results-chan)
    (mapv #(async/<!! %) printer-chans)))

That works

Oh cool, ty

on http://clojure.org, the section about typehinting (in the weird characters guide) in the reader gives the following example for typehinting: (def ^Integer five 5). Yet all over Clojure's source code, ^long is used (although possibly in different contexts, typehinting locals and return types for functions). When I typehint a simple var like that if i use ^Long i get its tag as java.lang.Long and when typehinted as ^long i get #function[clojure.core/long]. Are these things semantically different? One obviously wrong?

they are semantically different

https://clojure.org/reference/java_interop#TypeAliases gives some background.

Java has 7 primitive types with lowercase letters - int, long, etc

it also has object-based wrappers around those primitive types java.lang.Integer, java.lang.Long, etc

in many places these will automatically converted if needed so that hides a lot of mismatches

I'd say that particular example is confusing

by default, Clojure will read integer numbers as java.lang.Long objects

so ^Long there would probably make more sense, but this is case is probably not a good place to really need/want that

I think what really confused me was when i type hinted the var with ^long, I would get errors of > Syntax error (IllegalArgumentException) compiling . at (REPL:1:31). > Unable to resolve classname: clojure.core$long@67ec8477 when the compiler was using that typehint.

and that made me want to find some proper documentation instead of just the kinda rule of thumb nature of my knowledge of type hinting

(def ^long year-threshold 20)
#'user/year-threshold
(def ^:private past-threshold (.. (LocalDateTime/now)
                                  (minus year-threshold ChronoUnit/YEARS)
                                  (toInstant ZoneOffset/UTC)
                                  (getEpochSecond)))
Syntax error (IllegalArgumentException) compiling . at (REPL:1:31).
Unable to resolve classname: clojure.core$long@67ec8477

I believe this is related: https://ask.clojure.org/index.php/4272/primitive-hints-function-names-should-print-error-message

can you type hint a def? I'm reading https://clojure.org/reference/java_interop#typehints > They can be placed on function parameters, let-bound names, var names (when defined), and expressions but I'm not sure

or atleast the part discussing {:tag 'long}

I’m using the monger database, which is showing the results from the find function. It’s showing the results as empty even though I have items in my database. Here’s the code:

(ns humboiserver.db.core
  (:require
   [monger.core :as mg]
   [monger.gridfs :as gfs]
   [monger.query :as q]
   [monger.collection :as mc]
   [cheshire.core :as json]
   )
  (:import
   org.bson.types.ObjectId))

(let [{:keys [conn db]}
      (mg/connect-via-uri “<mongodb+srv://user>:<mailto:pass@cluster0.ww5gh.mongodb.net|pass@cluster0.ww5gh.mongodb.net>/&lt;dbname&gt;?retryWrites=true&amp;w=majority”)]
  (defn find [coll query] (mc/find-maps db coll query))

(db/find “featured” {})

ah ok, @lasse.maatta’s link seems to show that you can.

you can type hint a def, you just have to be aware that type hints on the var (as in def, but also the function name in defn) are evaluated, so ^long will evaluate to the long function object, which is no good. defn type hints on the arg vector and in the arg vector are handled by defn macro and are not evaluated

ah ok. that's the secret sauce that i was missing in documentation. I didn't understand why inline typehints could be ^long but on a var like that had to be ^Long. I thought i remembered that the clojure.core/long function was identified with the `'long' and classname typehint, but that might be cljs or might just be me just completely making stuff up

thanks as always for your answers @alexmiller

Hello! Is there an equivalent to a tap function in clojure? Something that does this: (fn [f x] (f x) x) ?

It would be most useful for stuff like (tap println (do-some-stuff data)) and is a little easier to read and write than what I usually do: (let [x (do-some-stuff data)] (println x) x).

@gerome.bochmann Hi! Maybe using doto, a little bit like in this example: https://twitter.com/lambdaisland/status/1338427872617250816

Oh! This is awesome! Thanks! @admin055

Hi! I am not sure this question belongs here, so if you think there’s a better channel please tell me. Anyways, for all mac users, how do you switch jdk globally - not limited to current shell. I can’t seem to find a one script solution i.e. switch jdk 11 that sets the jdk globally.

I have set JAVA_HOME in ~/.zshrc.

I see but say you change JAVA_HOME in current shell, then when a new shell is opened it will default to whatever is in ~/.zshrc. To fix this I resorted to some emacs helpers that allow me to manage the jdk version emacs uses from within emacs itself.. but yeah ideally it would be great to be able to do this in a shell. Thanks for the response anyways 🙂

Maybe sdkman would help?

Or is that shell only?

I am just guessing but if you set java path in project.clj/profile.clj cider should pick that version provided you are using lein

@lucio Check out jenv

awesome it seems it does exactly what I need. Will try it out now.. @borkdude thanks!

It kinda helps but from the docs it doesn’t seem you can change a default besides right after installing a new jdk through the prompt 🙂

@lucio I’ll vouch for jenv too.

Java file foo/bar/Class.java is foo.bar.Class. Clojure

(ns foo.bar)
(def custom (proxy [SimpleFormatter] []
                (format [^LogRecord record]
                  (println record)
                  "bar")))

(import '(java.util.logging ConsoleHandler
                            Logger
                            LogRecord
                            SimpleFormatter
                            Level))

(defonce ^Logger logger (doto (Logger/getLogger "clojure")
                          (.setUseParentHandlers false)
                          (.addHandler
                           (doto (ConsoleHandler.)
                             (.setLevel Level/ALL)
                             (.setFormatter
                              (proxy [SimpleFormatter] []
                                (format [^LogRecord record]
                                  (let [sb (StringBuilder.)]
                                    (.append sb "#:log{")
                                    (.append sb ":z ")
                                    (.append sb (pr-str (str (java.time.Instant/ofEpochMilli (.getMillis record)))))
                                    ;; (.append sb " :b ")
                                    ;; (.append sb (format "%02d" (.getSequenceNumber record)))
                                    ;; (.append sb " :c ")
                                    ;; (.append sb (format "%02d" (.getThreadID record)))
                                    (.append sb " :v :")
                                    (.append sb (.toLowerCase (.getName (.getLevel record))))
                                    (.append sb " :n ")
                                    (.append sb (.getSourceClassName record))
                                    (.append sb " :l ")
                                    (.append sb (.getSourceMethodName record))
                                    (.append sb " :m ")
                                    (.append sb (pr-str (.getMessage record)))
                                    (doseq [p (seq (.getParameters record))
                                            :when (map? p)
                                            [k v] (seq p)]
                                      (doto sb
                                        (.append " ")
                                        (cond->
                                            (namespace k) (.append (pr-str k))
                                            (not (namespace k)) (-> (.append ":_/") (.append (name k))))
                                        (.append " ")
                                        (.append (pr-str v))))
                                    (when-let [t (.getThrown record)]
                                      (.append sb " :thrown \n")
                                      (.append sb (pr-str t)))
                                    (.append sb "}\n")
                                    (str sb)))))))))

(doseq [level '[severe warning info config fine finer finest]]
  (intern *ns*
          level
          (fn
            ([&form &env msg]
             (let [ns (name (ns-name *ns*))]
               `(.logp logger ~(symbol "java.util.logging.Level" (.toUpperCase (name level))) ~ns ~(str (:line (meta &form))) (print-str ~msg))))
            ([&form &env msg parameters]
             (let [ns (name (ns-name *ns*))]
               `(let [p# ~parameters]
                  (if (map? p#)
                    (.logp  logger
                            ~(symbol "java.util.logging.Level" (.toUpperCase (name level)))
                            ~ns
                            ~(str (:line (meta &form)))
                            (print-str ~msg)
                            p#)
                    (.logp  logger
                            ~(symbol "java.util.logging.Level" (.toUpperCase (name level)))
                            ~ns
                            ~(str (:line (meta &form)))
                            (print-str ~msg)
                            ^Throwable p#)))))))
  (.setMacro ^clojure.lang.Var (ns-resolve *ns* level)))

i modified that snippet so it uses the fully qualified java.util.logging.Level/ stuff so you can use it from other namespaces

Just in case you still need to access Clojure vars from Java: https://clojure.org/reference/java_interop#_calling_clojure_from_java

@dpsutton Do you know how to refer to this from configuration file for JUL? I know how to achieve this in pure Clojure / Java, but I wanted to use config file to point the formatter.

But I don’t know how to point to this heh

java.util.logging.ConsoleHandler.formatter = api.logs.mycustom <- exactly this line of config

no i was just playing around with it this morning. I think the point of it is that it uses JUL and ignores the config as it makes its own logger named clojure and its configured inline there

maybe I will end with this. I wanted to have only formatter in Clojure and set everything in config file

but I don’t know how to refer to the class made by proxy in clj file heh

you want to use this from java?

not sure what you mean

why are you writing a line like this: java.util.logging.ConsoleHandler.formatter = api.logs.mycustom ?

(ns foo.bar)
(def custom (proxy [SimpleFormatter] []
                (format [^LogRecord record]
                  (println record)
                  "bar")))

JUL.properties is config file for java.util.logging

like log4j2.properties

dunno sorry

no problem

you can use config file by "-Djava.util.logging.config.file=JUL.properties"

so then I could only use Clojure for Formatter code

And keep everything native

i think this snippet is kinda intended to just be evaluated in a random namespace and used there almost transiently. its a quick in the repl thing and not really what you're looking for. and for your usecase i think you'd need to aot the file for it to work as that classfile needs to exist already (but i'm not sure about that)

hmm

BTW What / Why are you doing in (doseq [level '[severe warning info config fine finer finest]] ?

Why not just overwrite default root logger?

(let [root-logger (.getLogger (LogManager/getLogManager) "")
        formatter (proxy [SimpleFormatter] []
                    (format [^LogRecord record]
                      (println record)
                      "bar"))
        console-handler (doto (ConsoleHandler.)
                          ;(.setUseParentHandlers false)
                          (.setFormatter formatter))]

    (doseq [handler (.getHandlers root-logger)]
      (.removeHandler root-logger handler))

    (doto root-logger
      (.setLevel Level/INFO)
      (.addHandler console-handler)))

Do you understand when use Formatter vs SimpleFormatter?

I know this isn't a very scientific benchmark, but I get the same-ish numbers with just nth and a self inlined version (.nth ^clojure.lang.Indexed [0 1] 1) here:

user=> (time (dotimes [_ 100000000000] (.nth ^clojure.lang.Indexed [0 1] 1)))
"Elapsed time: 1937.860586 msecs"
nil
user=> (time (dotimes [_ 100000000000] (nth [0 1] 1)))
"Elapsed time: 1901.539964 msecs"

just based on the implementation, I'm not sure how much extra it's doing:

(defn nth
  "Returns the value at the index. get returns nil if index out of
  bounds, nth throws an exception unless not-found is supplied.  nth
  also works for strings, Java arrays, regex Matchers and Lists, and,
  in O(n) time, for sequences."
  {:inline (fn  [c i & nf] `(. clojure.lang.RT (nth ~c ~i ~@nf)))
   :inline-arities #{2 3}
   :added "1.0"}
  ([coll index] (. clojure.lang.RT (nth coll index)))
  ([coll index not-found] (. clojure.lang.RT (nth coll index not-found))))

how does it compare to the (. clojure.lang.RT (nth ~c ~i ~@nf)) version?

I mean the RT#nth version is doing something extra

I don’t have time to dig into it but maybe ^clojure.lang.Indexed make a difference?

why do you think one version is doing something extra?

@hiredman it's doing this instance check right? https://github.com/clojure/clojure/blob/0df3d8e2e27fb06fa53398754cac2be4878b12d1/src/jvm/clojure/lang/RT.java#L895

that kind of thing is, to make a crazily broad claim, basically free

Right. And in clojure itself - less free but maybe not so bad either?

(instance? ...) I mean

it is complicated, but very often exactly the same

cool

an instance check compiles to a single jvm instruction in java, in clojure we have a function instance?, and calling that would be more expensive, but several features of clojure (definline, and compiler intrinsics) will usually turn it into a single instruction

I don't see any :inline stuff on instance? itself though

(the compiler recognizes some static methods and replaces them with bytecode directly instead of a method call)

right. clever :)

how does it compare to (.get ^List [0 1] 1) ?

getting something from a list is O(n) right

indexed is way faster

but it depends on what the List implementation is

ah, actually it isn't definline and the intrinsics stuff

in Clojure a list is a linked list, with O(n) access - but in Java it could also be an ArrayList which probably is indexed

the compiler special cases known calls to instance?

that explains why it's so blazing fast

if it sees a call where the first thing is a reference to the var clojure.core/instance? it does the single bytecode instruction (with maybe an extra instruction to box the boolean)

that must predate the more general pattern of using definline to emit a static method call, and the compiler having a mapping between certain static methods and the bytecode to emit instead

https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Intrinsics.java

it is important to keep in mind instance? is not the same thing as satisfies?

yeah - satisfies? can be slow I've heard

https://clojure.atlassian.net/browse/CLJ-1814

reminds me to remove a call to satisfies? from a patch