Clojurians Log v2

Caused by: java.lang.ClassNotFoundException: clojure.core.async.Mutex issue showing up with :dependencies [[org.clojure/clojure "1.9.0"] [org.clojure/core.async "0.4.500"] [me.raynes/conch "0.8.0"] [com.draines/postal "2.0.3"] [propertea "1.3.1"] [instaparse "1.4.10"] [com.taoensso/timbre "4.10.0"]] :source-paths ["src" "src/main/clojure"])

Hello guys, I am experimenting with pipelines in core.async and I ran into the following problem, the call to (<!! (async/into [] output-channel)) (last line) just blocks indefinitely and never returns. The idea is to put passwords from filel to a channel and then pass it to a http-get function which performs an API call and puts result into the channel. Than the ouput has a filter transducer to filter outputs

(defn- create-passwords-channel
  "Creates a channel based on the a lazy sequence from a file"
  []
  (let [c (chan 50)]
    (go
      (with-open [lines (-> (io/reader "passwords.txt")
                            line-seq)]
        (doseq [ln lines]
          (>! c ln))))
    c))

(defn http-get
  "Tries to call a login API with a password and saves result into the channel "
  [password channel]
  (http/get "<http://localhost:5000/auth/login>"
            {:body {:timeout 200
                    :username "<mailto:admin@temify.com|admin@temify.com>"
                    :password password}}
            (fn [{:keys [status]}] // callback
              (put! channel {:status status
                             :password password})
              (close! channel)))
  channel)

(defn dictionary-attack
  []
  (let [input-channel (create-passwords-channel)
        output-channel (chan 10 (filter #(= (:status %) 200)))]
    (pipeline-async 10 output-channel http-get input-channel)
    (&lt;!! (async/into [] output-channel))))

you should never block a go thread

which means, don't do i/o in it

using pipeline-async correctly is subtle

in this case you are not actually running the http/get in an async way

so you will break things

pipeline-blocking is somewhat harder to get wrong

pipeline-blocking is like, "I have this blocking process and I would like to run it as a pipeline", pipeline-async is "I have this async process and I would like to run it as a pipeline"

they don't inherently change the nature of your process, using pipeline-async doesn't make your code asynchronous

Thanks for response. Actually I thought I can se http-kit client in an async manner, like they are describing in the docs. https://www.http-kit.org/client.html#async

So I have to use pipeline-blocking, right?

that might be ok, then as long as they do all io stuff on their own threadpool

async/into doesn't do anything until the input channel is closed

and I guess the io alex was referring to was reading the password file, not the http requests

So actually is there a better way how to put content of a file to a channel? 😕

sure, use thread

use thread instead of go

but create-passwords-channel missing async/close! as @markmarkmark suggested

But that should come after calling pipeline-async, right? In order to have the channel opened for processing

close the producer after submitting all the items to channel

(defn- create-passwords-channel
  "Creates a channel based on the a lazy sequence from a file"
  []
  (let [c (chan 50)]
    (thread      (with-open [lines (-> (io/reader "passwords.txt")
                            line-seq)]
        (doseq [ln lines]
          (>!! c ln))
         (close! c)))
    c))

sorry I messed up the formatting, but you get the gist

there may still be pending vals on a channel after it closes

ah, now it works perfectly, even with the pipeline-async. Thanks a lot, great!