Clojurians Log v2

what is the best practice to keep a thread running inside a with-open block

You'd have to wait for it to complete inside the with-open block.

Otherwise, the resource the thread is using could be closed before it was done.

I'm aware of that, the question was more on which mechanism to use for the waiting

i'm spawning 2 threads inside the with-open

future + deref?

There are lots of ways to do it. Hard to say what's best without knowing a bit more about exactly what you're doing.

i'm opening a stream, reading from it into a buffer on one thread, and the other is locked waiting for a signal to start processing from the buffer

future/`deref` is a simple but fairly uncontrolled way to do it. Using Java threads directly or an executor or... all sorts of possibilities depending on how much control you wanted over this.

control in terms of?

these threads should be running in the background

with no interaction other than the data they're receiving and processing

Per the docstring: "Sets ns to the namespace named by the symbol, creating it if needed."

So it will create a completely empty namespace if it doesn't already exist. It won't have any clojure.core functions referred in.

Is there a usecase for that? I'd hate to be derailed by an errant typo

Ahh, hmm so how does it differ from ns then?

ns does a whole bunch of stuff: creates the ns if needed, refer in clojure.core and a bunch of other stuff, requires specified namespaces, imports specified classes, etc.

in-ns is very low-level.

Wait, you meant *ns* right?

I see, I see

I mean the ns macro. Not *ns*. The latter is a dynamic Var.

user=> (in-ns 'foo.bar)
#object[clojure.lang.Namespace 0xdf5f5c0 "foo.bar"]
foo.bar=> (ns-publics *ns*)
Syntax error compiling at (REPL:1:1).
Unable to resolve symbol: ns-publics in this context
foo.bar=> (clojure.core/ns-publics *ns*)
Syntax error compiling at (REPL:1:1).
Unable to resolve symbol: *ns* in this context
foo.bar=> (clojure.core/ns-publics clojure.core/*ns*)
{}
foo.bar=> (clojure.core/ns-refers clojure.core/*ns*)
{}
foo.bar=> (clojure.core/ns-aliases clojure.core/*ns*)
{}
foo.bar=> 

BTW I saw your post earlier this year on whether or not switching to deps.edn is a good idea and it's the main reason I'm trying to do it now. Thanks!

See how I have to specify clojure.core/ in front of various symbol names? Normally -- when you use the ns macro -- those are automatically referred in.

I mean in your first reply, since *ns* appears as ns

Oh, this? Per the docstring: "Sets `*ns*` to the namespace named by the symbol, creating it if needed." Yeah, I just pasted in the docstring and forgot it would auto-format that!

(defn watch-collection [collection-name signal]
  (println (str "Waiting for changes in " collection-name))
  (with-open [watch-cursor (.watch (get-collection collection-name))]
    (let [buffered-channel (async/chan (async/sliding-buffer 100))]
      (map deref [(future (-> watch-cursor
                              .iterator
                              iterator-seq
                              (map #(async/>! buffered-channel %))))
                  (future (do (async/<! signal)
                              (loop []
                                (when-let [change (async/<! buffered-channel)]
                                  (println change)
                                  (recur)))))]))))

map is lazy so that is not safe.

need a dorun after

Safer to use run! there since you want the side-effects.

ah yes

(run! defer [...])

Just checked ClojureDocs, I should probably check it more often. In any case, thanks for the detailed responses! Really starting to get the hang of this thing.

other than that

would something like the above work?

Don't forget you can use doc and source right there in your REPL.

No. You cannot do <! outside a go block

Or >!

ah right

i'm calling (go (watch-collection

doseq + !>

should the go go inside the function body?

or use another mechanism

go blocks only work on immediate lexical scope; the call inside #() is a new scope

Oh yeah, that's really helpful. Should probably get used to them.

Consider letting the caller of this function supply a channel onto which changes are published

(I always cringe a bit when I see beginners trying to use core.async 👀 -- it is really not easy to learn/use and there are so many ways to shoot yourself in the foot with it...)

🙃

It's always felt like one of those "Here Be Dragons!" things in Clojure to me...

I have doc bound to ctrl-; d and source bound to ctrl-; S in my editor so they are easily available 🙂

@ghadi but both the pushing and pulling from the channel are happening inside this function

(aside from helping beginners with REPL sessions like the above, I never actually type into my REPL -- I always type into a source file, sometimes in a comment form, and then eval code directly into the REPL from the editor)

I am reading on my phone and missed the context... sorry! What is it you are doing?

@ghadi i'm opening a stream, reading from it into a buffer on one thread, and the other is locked waiting for a signal to start processing from the buffer

and I need to keep the threads alive inside the with-open block

@seancorfield well I'm not one to shy away from a new challenge 🙂 that's why I came to clojure

@ghadi my current attempt

(defn watch-collection [collection-name signal]
  (println "Start watching" collection-name)
  (with-open [watch-cursor (.watch (get-collection collection-name))]
    (let [buffered-channel (async/chan (async/sliding-buffer 1000))] ;buffer size might need tweaking
      (run! #(async/<!! %) [(async/thread
                              (->> watch-cursor
                                   .iterator
                                   iterator-seq
                                   (run! #(async/>!! buffered-channel %))))
                            (async/thread
                              (async/<!! signal)
                              (println (str "Waiting for changes in " collection-name))
                              (loop []
                                (when-let [change (async/<!! buffered-channel)]
                                  (println collection-name change)
                                  (recur))))]))))

I need help, For example, if my sequence is '((if a b)(if a c), and i want to find "if" and then "b" and want to return true because i found "b" with a "if", would i use some or filter?

(filter #(and (sequential? %) (= (first %) 'if) (= (second %) expr)) kb)

What do you mean by "third index of a sequence"?

Sequences are not indexed.

(if a b), i want to look at b

Instead of (second %) you want (nth % 2)

Or (second (rest %)) would also work

I was doing (nth 2 %) the whole time

my bad

a lot of the functions that manipulate colls take them as the last arg

@ntrut you need to get used to doing things like (doc nth)

@joao.galrito No, sequence functions take the sequence as the last argument. Collection functions take the collection as the first argument.

oh ok

I still mix the two a bit

Most of the time, if the first argument is a collection, the function will have some specific guarantees about performance and/or return type.

If the last argument is a sequence, and you pass a collection, it will be coerced to a sequence first. For example (map inc [1 2 3]) will call seq on the vector and then process it as a sequence.

(filter #(and (sequential? %) (= (first %) 'if) (= (second (rest %) ) 'b)) '(if a b))

@seancorfield thanks for the help btw

i owe you guys

Because you're passing an expression instead of a list of expressions (again 🙂 )

filter expecs ( (if a b) ) remember @ntrut?

it's calling sequential? on 'if, then 'a, then 'b

Oh i see my mistake now

@ntrut Some suggested viewing for you https://www.youtube.com/watch?v=Qx0-pViyIDU and https://www.youtube.com/watch?v=FihU5JxmnBg -- two talks on debugging techniques by Stuart Halloway of Cognitect. I think you'll find they'll help you figure out some of the problems you are running into.

(you might also want to track down Stu's "REPL-Driven Development" talk as well)

I don't see anything that's wrong. Do you observe your (println "Inserted " (count operations)) output?

> anyway in my case I can take advantage of parallelism so ended up using `pmap` and `dorun` Looks like you worked it out!

Maybe you want a default result:

(defn judge [x] 
  (get {true 1 false 0} x -1))

maps have the same interface as get) ({true 1 false 0} x -1) also will give you default value

Thank you @delaguardo, I have read from somewhere that only get can append a default value.

Hi all. Any recommendations for simple websockets client and server implementations in clojure on the jvm for both client and server? Nothing fancy, no need for fallbacks etc. Needed as a way of linking two servers to exchange information where firewalls block other types of socket and bidirectional flow would be better than polling.

https://github.com/ptaoussanis/sente

never used it, just saw it yesterday somewhere

not sure if that's what you need

Thanks I have looked at it but as I’m not an expert in this, on first inspection it has more functionality than I think I need. Of course, in its favour, it does seem to be the top hit on search - and is maintained, and the API looks good!

in the README there's also a couple other alternatives he recommends

Thanks. I’ve seen a variety of options including sente, direct http-kit usage - although the examples / docs use now deprecated methods, immutant, aleph, and I’ve looked at wrapping java libraries directly (e.g. Java-WebSocket) which actually I think was just added to sente for the java client bit. Up until then, it looks as if the main tooling was cljs for client and clj for server, because of the browser focus, but websockets now being used more and falling back to AJAX (as per sente) and starting connection using HTTP POST isn’t necessary for my use case. Hence just asking for advice. Thank you

Is there an any type in lacinia that I can use while defining the schema?

when using Cursive, do I only get method name resolution if I import the corresponding classes?

Are there any simple examples of using reitit, ring and http-kit where some routes are regular routes and a websocket route?

Not sure if you'll find an exact match, but the reitit repository provides plenty of examples: https://github.com/metosin/reitit/tree/master/examples

May be I am not seeing properly. But none of them are with websockets. Is reitit capable of routing websocket requests? Thank you.

@wls What do you use for websockets? Which routing library and server are you using? Thank you.

haven't seen a public example, but we have been using reitit + ws since day1. Just mount a ws-endpoint into a route like you would do with any other routing lib, like compojure. Sente has examples of this

Thank you @ikitommi. Can you please see the ws route below. I understand that the route is not set up to handle websocket connections. Also when I visit using the url using a regular browser window, I know it will fails. But I expect the httpkit/websocket? to work as described on page <https://http-kit.github.io/http-kit/org.httpkit.server.html#var-websocket.3F> Would that now print out`false`. But instead I get an error saying: java.lang.IllegalArgumentException: No implementation of method: :websocket? of protocol: #'org.httpkit.server/Channel found for class: clojure.lang.PersistentHashMap. I am very new to clojure. Just trying to figure out what to use for webapps. Thank you. My code:

(ns sampleapp 
  (:require
   [reitit.ring :as ring]
   [ring.middleware.params :as params]
   [ring.middleware.multipart-params :as multparams]
   [org.httpkit.server :as httpkit]
  ;  [ring.adapter.jetty :as jetty]
  ;  [ring.adapter.jetty9 :as jetty]
    )
  (:gen-class))

(defn wrap [handler val]
  (fn [request]
    (let [currval (get request :val [])]
      (handler (assoc request :val (conj currval val))))))



(defn jerryhandler [req]
  (println (str "Wrapped value: " (:val req)))
  {:status 200
   :headers {"Content-type" "text/plain"}
   :body (str "Hello Jerry!!\nI got values " (:val req))})

(defn busterhandler [req]
  (println (str "Parameters: " (:params req)))
  {:status 200
   :headers {"Content-type" "text/plain"}
   :body "Hello Buster!!"})

(defn fileupload [req]
  (println (str "File info: " (get-in req [:params "file"])))
  (let [fileobj (get-in req [:params "file" :tempfile])
        pathtofile (.getAbsolutePath fileobj)
        contents (slurp fileobj)]
    (println contents)
    (println pathtofile)
    {:status 200
     :headers {"Content-type" "text/plain"}
     :body "File uploaded..."})
  )

(defn wshandler [req] 
  (println (httpkit/websocket? req))
  {:status 200 :headers {"Content-type" "text/plain" :body "Not valid ws request..."}})

(def router
  (ring/router 
   [["/hellojerry" {:get {:middleware [[wrap 1] [wrap 2]] :handler jerryhandler}}]
    ["/hellobuster" {:get (params/wrap-params busterhandler)}]
    ["/fileupload" {:post (multparams/wrap-multipart-params fileupload)}]
    ["/ws" wshandler]]
   ))

(def app
  (ring/ring-handler
   router
   (constantly {:status 404 :body "Not found!"}))
  )


(defn -main
  [& args]
  (httpkit/run-server app {:port 3000})
  ; (jetty/run-jetty app {:port 3000 :h2? true}))
)

  

Does https://github.com/ptaoussanis/sente require using it for both server and client, or can you use it only on the server with standard Javascript websocket on the client? Thank you.

I believe you need to pull out the channel linked to the request to get web-sockets with http-kit. There seems to be an example here: https://http-kit.github.io/server.html#channel

Thank you @ikitommi. I believe the linked approach is deprecated. However, the example given here: https://http-kit.github.io/http-kit/org.httpkit.server.html#var-as-channel, works well. It appears that my mistake was using (httpkit/websocket?, when, in fact, I was supposed to be using just (`:websocket?` ).

Compojure is the most common library for routing on the server side. reitit is a new project that takes a data driven approach to routing and other aspects of webapps. Its very interesting, but has more of a learning curve. Http-kit can do we sockets, although not something I've used recently

I'm not generally against having helper functions that provide a way to explicitly list what the common operations are, but I would warn against pretending that it "encapsulates" your data. I also wouldn't start with defining getters/setters and would only consider adding them on a case by case basis. In my experience, getters/setters provide little benefit for data at rest or being sent over the network which is the important part to get right for maintaining long running programs and systems To address "what keys?" and "what entity?", I would just use spec. As he mentions in the talk, it's meant to address these problems. The big difference between spec's approach and defining getters/setters is that spec is data focused and getters/setters are operations focused. The differences between a data specification and an operational specification are really critical. Most programs would be better served by focusing on the data than on the operations. Depending on your domain, it may be useful to use something like specter or getters/setters to help declutter deeply nested code. In my experience, there are many domains where an extra layer of abstraction is unnecessary. I really wish computer science would be renamed data-ology.

i think this is an interesting discussion. i don't have anything to add except to note that re-frame basically has getters and setters on a big map (`app-db`) in the form of subscriptions and events. for a while i really felt like i was back in oop land, writing a getter and setter for each piece of data in app-db. in that context, i do think there's something to be said for separating out the data from "exactly where in this huge nested map does it belong"

i've run into many snags trying to reorganize the structure of app-db and unknowingly breaking code that depended on that exact configuration

I think if you're running into "what's the shape of the data supposed to be?", I would reach for spec or defining the data specification rather than getters/setters

it would be great to hear other's thoughts on the subject though

Hi Mark, I am very new to clojure, so this is not coming from any experience in production. But I am in the same boat as you. My experiments suggest that reitit ring and http-kit can make for a fairly easy setup.

there are examples for connecting other types of infrastructure https://github.com/theasp/sente-nodejs-example

if you wanted a non-clojure or non-clojurescript participant, you'd have to conform to the message passing. It looks like they do have some emulation on web sockets to make up for web sockets short comings, so that may be a little tricky

@curiouslearn, I'm using aleph.http/websocket-connection ... I had to do a little fiddling, though.

Thanks. I have an aleph client and server talking nicely to one another - but there’s a lot of new stuff/patterns to learn for me to fully understand how it’s working! My comparison is a golang based library : http://nhooyr.io/websocket

Thank you @jr0cket. Appreciate the input. Also, thank you @wls. Will look into it; although it looks difficult; I have no idea what a manifold is.

What is a way that i can check if there is two "nots" in my sequence like this '(not (not b)), so therefore, if there is two "not" like this example.

(not(not a)) would be true

(not a) would be false

I would define a predicate that returned true if an expression was (not X) and then you can do (and (has-not? expr) (has-not? (second expr))) if you make has-not? robust (check for sequential expr and then check first item is 'not)

Manifold is a very interesting approach, especially when you dont want to be tied to the classic web server approach. However, there is more of a learning curve compared to ring and compojure. If you are creating API's then there is the compojure-api, https://github.com/metosin/compojure-api/ which includes swagger for documenting the API automatically. Luminus is a very useful template for web apps and full stack apps https://luminusweb.com/

clojure collections are not falsey. in another dynamic language i might test for the presence or absence of a given value of an unknown type with if . is there a conventional way to do this in clojure? so far i have (or (nil? x) (and (seqable? x) (empty? x)))

I think you can just do (if (empty? x) ...)

should i use (seq x) instead of (empty? x) ?

that throws on integers

i'm reminded of the little schemer where the first question you always ask is atom?...

indeed. what are you trying to do?

i'm writing a general purpose "is this var value missing?" predicate for use with values of an unknown type

maybe i shouldn't be doing that...

i could do additional work to bring through the type, i suppose

@jr0cket, thank you very much for the pointers. I bought the second edition of the Luminusweb book a couple of years ago. However, it has too much magic for my taste. I usually don't like when lot of files are created for me in the source code. Will look into compojure-api. But so far reitit with http-kit seems a good option. I hope that http-kit continues to be maintained.

trying to define a universal "missing?" predicate always reminds of https://bugs.python.org/issue13936

ok, maybe i should rework such that i'm starting with a map where absence is represented by a missing key

@jr0cket, Thank you so much for this. I was looking for a better repl than the standard one. I believe your deps.edn has that. Need to copy from there into Sean's that I downloaded 🙂. Will look into Kaocha too. Never heard of that before.

that would be more clojuristic

maybe

basically, it's highly context dependent and it might be ok if you're defining the context, but it's trouble if you're defining the context for someone else and may be confusing for others to have to figure out what your definition of "missing" is

I much prefer clojure's approach compared to something like javascript: > All values are truthy unless they are defined as https://developer.mozilla.org/en-US/docs/Glossary/Falsy (i.e., except for false, 0, -0, `0n`, "", null, undefined, and NaN). 🤢

https://www.youtube.com/watch?v=YR5WdGrpoug seems apropos

yes