Clojurians Log v2

I’m sending a request with the following params {:user/foo 1 :user/bar 2}, however, in the request map in the server, this map is converted to: {:foo 1 :bar 2} . Is there a way to preserve the namespacing?

How exactly are you sending the request to your Ring server?

And what middleware do you have active in your Ring server?

Are you sending that hash map as text, JSON, EDN, Transit?

@seancorfield I’m using this library to send the request: https://github.com/day8/re-frame-http-fx This is my :http-xhrio map:

{:method :get
   :uri "/my/uri"
   :params {:user/foo 1 :user/bar 2}
   :on-success [:on-success]
   :on-failure [:on-failure]
   :response-format (edn/edn-response-format)
   :format (edn/edn-request-format)
   }  

{:middleware [#(wrap-keyword-params % {:parse-namespaces? true})
                 middleware/wrap-formats
                 wrap-multipart-params
                 ]}                

Can you check in your browser dev tools that the URL being sent to the server actually has the qualified query parameter names? Since you’re using a GET you should see /my/uri?user/foo=1&user/bar=2 being sent. Or possibly with : in front of user in each slot?

(I would expect POST params to be sent as qualified names I think, although you may have to ask in #re-frame to be sure of that)

Looking at wrap-keyword-params with :parse-namespaces? true it looks like it should preserve the qualifier in any string params that arrive at the server. But I’d probably add some debugging at the outermost layer of your middleware to print the whole Ring request to be certain that you are getting the raw string data at the server as you expect. You don’t have wrap-params in your middleware stack so the :query-params and :form-params may not be processed into the common :params slot that wrap-keyword-params expects (middleware is hard — which is why I always use ring-defaults to try to get everything all in one place).

OK, good that confirms the browser is sending what you expect. Now add some debugging on the server side to see what is in the raw Ring request at the outermost layer.

@seancorfield does ring-defaults come with qualified keywords?

What is the outermost layer? Since I’m using the luminus framework, ring is a complete black-box to me

This is why I do not recommend Luminus to beginners 😞

I’m not even sure if #(wrap-keyword-params % {:parse-namespaces? true}) is the correct way to specify that middleware

would finding out about the outermost layer entail a print statement somewhere in the app-routes function in handler.clj?:

(mount/defstate app-routes
  :start
  (ring/ring-handler
    (ring/router
      [(home-routes)])
    (ring/routes
      (ring/create-resource-handler
        {:path "/"})
      (wrap-content-type
        (wrap-webjars (constantly nil)))
      (ring/create-default-handler
        {:not-found
         (constantly (error-page {:status 404, :title "404 - Page not found"}))
         :method-not-allowed
         (constantly (error-page {:status 405, :title "405 - Not allowed"}))
         :not-acceptable
         (constantly (error-page {:status 406, :title "406 - Not acceptable"}))}))))

Oh, Luminus uses Mount as well 😞 Something else I advise beginners to avoid.

Okay, so your debugging would need to be in the form of a middleware function that you would write that would need to go in that :middleware vector. I’m not sure which end because I don’t know what order Luminus wraps the middleware, so I’d probably add it to both ends.

(defn debug [h] (fn [req] (println req) (h req)))

So you’d have:

{:middleware [debug
                 #(wrap-keyword-params % {:parse-namespaces? true})
                 middleware/wrap-formats
                 wrap-multipart-params
                 debug
                 ]}   

That should let you see what’s coming into the server at one end and what’s going into your handler at the other end.

I’m not seeing the print statements when the request comes in

Did you restart the server with that new middleware in place?

Upon restarting the server, I do see the logs. And the params don’t have qualified keywords on both ends

OK, I just confirmed in the REPL that you need wrap-params as well as wrap-keyword-params:

dev=> ((p/wrap-params (kp/wrap-keyword-params identity {:parse-namespaces? true})) {:query-string "user/foo=1&user/bar=2"})
{:query-string "user/foo=1&user/bar=2", :form-params {}, :params #:user{:foo "1", :bar "2"}, :query-params {"user/foo" "1", "user/bar" "2"}}
dev=> ((kp/wrap-keyword-params identity {:parse-namespaces? true}) {:query-string "user/foo=1&user/bar=2"})
{:query-string "user/foo=1&user/bar=2", :params nil}

p is an alias for ring.middleware.wrap-params, kp is ring.middleware.wrap-keyword-params

Here’s my complete REPL session where I debugged this so you can see how I did it:

dev=> (require '[ring.middleware.keyword-params :as kp])
nil
dev=> ((kp/wrap-keyword-params identity {:parse-namespaces? true}) {:query-params {"user/foo" "1" "user/bar" "2"}})
{:query-params {"user/foo" "1", "user/bar" "2"}, :params nil}
dev=> (require '[ring.middleware.params :as p])
nil
dev=> ((kp/wrap-keyword-params (p/wrap-params identity) {:parse-namespaces? true}) {:query-params {"user/foo" "1" "user/bar" "2"}})
{:query-params {"user/foo" "1", "user/bar" "2"}, :params {}, :form-params {}}
dev=> ((p/wrap-params (kp/wrap-keyword-params identity {:parse-namespaces? true})) {:query-params {"user/foo" "1" "user/bar" "2"}})
{:query-params {"user/foo" "1", "user/bar" "2"}, :form-params {}, :params {}}
dev=> ((p/wrap-params (kp/wrap-keyword-params identity {:parse-namespaces? true})) {:query-string "user/foo=1&user/bar=2"})
{:query-string "user/foo=1&user/bar=2", :form-params {}, :params #:user{:foo "1", :bar "2"}, :query-params {"user/foo" "1", "user/bar" "2"}}
dev=> ((kp/wrap-keyword-params identity {:parse-namespaces? true}) {:query-string "user/foo=1&user/bar=2"})
{:query-string "user/foo=1&user/bar=2", :params nil}

But that’s how you can debug the pieces via the REPL. identity is just taking the place of a handler function that does nothing (just returns the input Ring request) so it’s easier to debug.

And just to show what happens if you have them the other way round in the middleware stack:

dev=> ((kp/wrap-keyword-params (p/wrap-params identity) {:parse-namespaces? true}) {:query-string "user/foo=1&user/bar=2"})
{:query-string "user/foo=1&user/bar=2", :params {"user/foo" "1", "user/bar" "2"}, :form-params {}, :query-params {"user/foo" "1", "user/bar" "2"}}

Having middleware in the wrong order is often the same as not having it at all:

dev=> ((p/wrap-params identity) {:query-string "user/foo=1&user/bar=2"})
{:query-string "user/foo=1&user/bar=2", :form-params {}, :params {"user/foo" "1", "user/bar" "2"}, :query-params {"user/foo" "1", "user/bar" "2"}}

As for ring-defaults, yes, you can have it parse namespaces but that isn’t the default — and it’s not entirely obvious how to tell it to do it. You need to pass something like (assoc-in api-defaults [:params :keywordize] {:parse-namespaces? true}) I think, instead of just api-defaults (you may want to tweak other settings too — it’s all just data).

The nice thing about ring-defaults is it provides out-of-the-box stacks of middleware in the correct order for both API style apps and website style apps (the latter has CSRF enabled by default and a few other differences), and you can also control how HTTP vs HTTPS behaves and several other useful things very easily.

But all this is why I recommend beginners avoid “frameworks” like Luminus and start with just Ring at first and learn how Ring works on its own, and then either learn Compojure or Reitit for routing (mapping URLs to handlers), and then learn Component or Integrant for managing the start/stop lifecycle of components (web servers, database connection pools, anything that has some process to “start” it up and another process to “stop” it when you’re done).

@seancorfield I tried:

{:middleware [debug
                 wrap-params
                 #(wrap-keyword-params % {:parse-namespaces? true})
                 middleware/wrap-formats
                 wrap-multipart-params
                 debug
                 ]}

{:middleware [debug
                 #(wrap-keyword-params % {:parse-namespaces? true})
                 wrap-params
                 middleware/wrap-formats
                 wrap-multipart-params
                 debug
                 ]}

Hard to be sure — as I said, I’ve no idea how Luminus rolls that middleware vector up.

Here’s the order that ring-defaults composes them: https://github.com/ring-clojure/ring-defaults/blob/master/src/ring/middleware/defaults.clj#L98-L117

That suggests you probably want wrap-params at the end (after wrap-multipart-params).

@seancorfield, in my request, I checked {…:data {:middleware …}} and found this:

{:middleware [#function[humboiserver.routes.home/debug] #function[ring.middleware.params/wrap-params] #function[humboiserver.routes.home/home-routes/fn--19313] #function[humboiserver.middleware/wrap-formats] #function[ring.middleware.multipart-params/wrap-multipart-params] #function[humboiserver.routes.home/debug]]

It seems like wrap-keyword-params isn’t being applied

No, actually: #function[humboiserver.routes.home/home-routes/fn--19313] is that

#(..) is an anonymous fn so, yes, #function[humboiserver.routes.home/home-routes/fn--19313] is that 🙂

As I said above, looking at ring-defaults, I think you want:

{:middleware [debug
                 #(wrap-keyword-params % {:parse-namespaces? true})
                 middleware/wrap-formats
                 wrap-multipart-params
                 wrap-params
                 debug
                 ]}

BTW, the wrap-formats middleware there is specific to Luminus and is a conditional wrapper around this library: https://github.com/metosin/muuntaja

(just so you understand how many “moving parts” are being pulled in by Luminus)

@seancorfield I tried this order but still no qualified keywords

And you restarted your server again?

yes

Then maybe try in #luminus — if this was plain ol’ Ring, what I showed above would solve this.

But, seriously, consider abandoning Luminus for now and learn the basics so you can debug this sort of stuff yourself.

@seancorfield the incoming :query-string doesn’t have qualified namespaces in the log. I mean, it’s just foo=1&bar=2 and not user/foo=1&user/bar=2

Since you’re using Luminus, I have no idea at this point.

My advice is to build a very simple pure Ring server and make sure you can get it working with that.

@seancorfield In pure ring, is it true that the query-string always contains the qualified keywords whether or not “wrap-keyword-params parse-namespace true” and “wrap params” middleware are applied?

@seancorfield, I confirmed that it doesn’t have to do with #luminus . I do get qualified keywords when I make the request from the browser with “user/foo=1…“, but it doesn’t work with the reframe http library

When I asked about what browser devtools showed, at the beginning of this discussion, I meant while you were using the re-frame app — that was the first thing you needed to verify 😞

I hope that the REPL session showing how to debug middleware was useful?

@seancorfield yes it was useful thanks 🙏