Clojurians Log v2

@borkdude hey, we were talk about exceptions on web page in yada before somewhere. So how do you deal with it https://clojurians.slack.com/archives/C0702A7SB/p1547229018025500 ?

It is happening for me for all types of “errors”

this is basically what I do: https://gist.github.com/borkdude/441379ec59d8da7f72348453ee15f32a note: this code is almost 2 years old and yada might have better ways right now

But is it only for definend content-type?

?

You can add :show-stack-traces? true to your yada resource map

I don’t see in this code option to turn off it for all errors. I see you use defmethod render-error "text/plain"

Or of course, :show-stack-traces? false

@malcolmsparks woo how can I know about that?

I turn off stacktraces in prod.

all the schema is defined in yada.schema. It's a single ns that defines all possible things you can have in a yada/resource

there should definitely be a documentation chapter explaining it all

Maybe I miss something big, but I don’t see it

but it's good to know where the actual source is because documentation can be out of date

that was added 23rd of May 2017, I probably wrote this code before it

if you want to do something to multiple resources at once, the standard answer is to clojure.walk/postwalk your resources

you can add some namespaced entries in your resources if you need to filter them, etc.

@malcolmsparks we have a common resource model that we merge with overridable data per resource, that also works

yes, that too - it's data

So I have to add this to all resource definition? No way to set it when start yada?

no - everything is at a resource level - this is intentional

sometimes you want different policies for different resources

and yada discourages putting policies on the global or 'route prefix' level

This is partly to make resources self-contained (which can help when testing them individually).

It's also about making it easier to reason about resources on their own

sure I understand it, but this one I would like to see as global conf 🙂 So I will ended up with function wrapper to create resource without stacktrace like everybody 😉

But I understand the concept

@kwladyka like I said, you can make a “global conf” by using a global map with your default and then merge differences in per resource

Do you have other cool tips like https://github.com/juxt/yada/blob/master/src/yada/schema.clj for me? 🙂

@kwladyka in our global resource map there’s for example auth stuff

I've begun to work more actively on yada lately, and I'm trying to add documentation to each feature as I commit. But I do have some outstanding documentation to author.

One good general tip is to read the source code.

Also the Edge app is a good example

Once you understand how yada works it's not that frightening. The overall design is quite simple once you understand it - it's a series of functions that are called in a sequence, acting on a single map (called the yada context)

once the context is fully built, the response is ready to be returned

The trick is that there is enough data in the yada context to figure out things like content negotiation, conditional responses and so on.

agree, as a feedback (please think about it positive! I have good intention!) in my feeling doc is too long without right value. It could be 20% of that length with the same value or better. I mean there is a lot of content which doesn’t give me any improvement.

thanks for the feedback

we are working on some Edge tutorials which may be easier to follow

I understand, it is in progress

@malcolmsparks have you thought about swapping bidi for e.g. reitit to make it work with spec?

But I really like idea about yada and I am full of hope 🙂

@borkdude I have looked at reitit and really like it - however, it does have some constraints such as segmenting on / only.

There is some integration between bidi and yada to makes URI forming 'easier' which I dislike

you dislike easier?

@borkdude Why reitit use [reitit.coercion.spec] instead of core one? Sorry I am asking, but I don’t know this library. Is it possible to use it with [clojure.spec.alpha :as s]?

I don’t know reitit at all.

oh ok 🙂

@borkdude there are alternative ways of doing bidi url forming without going via the yada ctx, and I think the use of the yada ctx is a bit of an anti-pattern. I think bidi and yada should work more independently. I don't think there's any reason today why yada and reitit can't work well together. It's not really a case of 'swapping' bidi for reitit, just to support both and give the user the choice.

Edge could have some examples where reitit is used instead of bidi.

yeah. but yada also is coupled with schema right now, right?

you can just turn that off by not using it I guess

but then the schema/spec is not part of the resource anymore

I was thinking of swapping out schema for clojure.spec when clojure.spec first came out. But I have since reconsidered. Schema plays 2 roles in yada - the first is by checking the data given to yada resources. The second is for parameters and Swagger. I don't think clojure.spec is a good choice for either.

We could take schema out of yada, but then you'd lose the shorthands and they're quite handy. Not having coercions would make yada resources quite a bit more verbose.

makes sense

I do think that the use of schema in yada parameters is a bit of a problem. I would prefer schema just be an internal thing really.

For parameters, I think that parameter validation (to produce 400 errors) is really a separate concern and not one that yada should be involved with. The declaration of parameters isn't part of the HTTP standards and was really driven by Swagger. I used to think it was a big advantage to have Swagger 'built-in' but now I think it would be fine if all the Swagger stuff were available as an extension and not part of yada core. That would reduce the dependencies and overall size of yada a lot.

(I still think parameters validation is important, but something where there should be more options rather than 'built-in')

@malcolmsparks > I don’t think clojure.spec is a good choice for either. Why?

1. https://stackoverflow.com/questions/45188850/is-use-of-clojure-spec-for-coercion-idiomatic

2. I don't think clojure.spec is designed for validation of web forms - there are more purpose-built libraries for that

Hmm I did exactly such library haha https://github.com/kwladyka/form-validator-cljs 😛

Form validator based on clojure spec

It seems to be fine, but maybe I miss something :thinking_face:

I really want to use the same code for validation on BE and FE.

@kwladyka you should not run spec against data from the Internet until you have validated it.

Additionally, if you want to spec that :password and :password-repeat are equal, you cannot put the error message on :password-repeat, only on the whole form.

Can you expand that?

It can be done, because you can recognise spec name (s/def ::foo/bar ...)

But fn for that purpose are probably better

I mean I use spec + fn for not standard things and it works without any issue.

But the biggest advantage is I can use spec from BE on FE and be sure everything is consistent

Writing different code validation on FE and BE twice sounds bad for me

But if you can five me further input which will change my view it will be great!

But I agree spec needs support of fn for coercion

Or check things like “this username already exist”

But in practice you don’t send :password-repeat to BE, it is checking only on FE

So you don’t want to have :passowrd-repeat in main spec anyway

@malcolmsparks {:show-stack-traces? false} how can I add this to for example HTTP 500? Do I have to define all HTTP responses for that purpose?

ok this was bad example probably

because 500 is about resources itself

Oh I see, if yada doesn’t find any resources it returns 204 without stacktrace

ok, never mind, I see how it works 👍

It's a security risk, due to not being able to constrain the registry to only safe ones

What do you mean?

I don't understand. How does this work?

While you can get reason of fail like [::form ::form-map ::password ::password-not-empty] you can get [::form ::form-map ::password-repeat ::password-not-empty] instead

so you know error is about password-repeat

all is about spec namig

code example?

If a library contains a slow spec, for example, let's say it takes 1s per character. Then I can send a long string with that key, even though your endpoint doesn't accept it.

E. G. {:ring/method "longstring"}, I can use that to easily dos your site

Traditionally this happens with password hashing and not limiting string length

:via spec key is the point of get this info

But my spec is on ::form

mmm but that is why spec are? To check lenght of that string before hashing

(ns form-validator-doc.spec
  (:require [cljs.spec.alpha :as s]
            [clojure.test.check.generators]))

(s/def ::email (s/and string? (partial re-matches #"^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,63}$")))

(s/def ::password-not-empty not-empty)
(s/def ::password-length #(<= 6 (count %)))
(s/def ::password (s/and string? ::password-not-empty ::password-length))

(s/def ::form (s/keys :req-un [::email ::password]
                      :opt-un [::password-repeat]))

if you create (s/def ::password-repeat ...) you can get for example [::form ::form-map ::password-repeat ::password-not-empty]

The same situation with Schema. App has to check it in right moment.

But you are not validating that the fields are equal

it can be done in spec!

but i prefer fn

But the :via will not point at the right field

yes, then it is more complex. That is why I use spec + fn

Most of the validation can be done by spec

But sure, you can do anything if you add extra validation, but then you have to put custom logic on top

You lose the benefits of spec

In that specific example not, because I don’t even want to have :password-repeat as part of my deep logic. It is not something what I will send to BE. It is only for UI purpose.

How would you do it better? spec + Schema? It sounds like doing work twice.

and issues about consistency

But I would love to know how you are doing that 🙂

I like vlad for forms

The messages are good by default

lib looks like not maintained anymore

Anyway it is very interesting discussion! 🙂

I have to say after second try of yada I like it even more

https://github.com/juxt/yada/blob/master/src/yada/schema.clj this is very helpful

It's a finished library pretty much

Schema use a restricted map of validations, spec uses an open map

interesting fact: after change liberator to yada I removed about half dependencies and about half code

*with the same effect

not sure what it means

I had the same experience with an http-kit/ring based stack. I really like the batteries included of yada. I don’t mind the dependencies as long as it’s cohesive

https://twitter.com/borkdude/status/857979807358910464

Do you know best practice example for yada + lacinia? Something what can inspire me or boost my process of learning how to glue things

lacinia does have a lib for pedestal. I’m not sure what exactly it’s doing and I never used it in anger

it is just graphql library

I did try lacinia once I must say, but we dropped it because the GraphQL standard was too narrow for us

My goal is to make GraphQL API with option to login by web browser + by token for integrations

There is a lot of decisions to make

maybe Edge uses GraphQL

https://github.com/juxt/edge/blob/master/phonebook-graphql/src/edge/phonebook/graphql.clj

Generally when join to project this part is already done. But now I start something from 0 and I have to decide about everything. It is biggest challenge, than I thought 🙂

the example I just posted uses lacinia + yada

cool, thx

Maybe I am not sure what is the best way for auth

maybe auth should have its own endpoint?

I have no good idea how to do it using only GraphQL in secure way. I need this POST REST API method to login (get token based on email and password)

Maybe it should be like that or maybe it should be moved to GraphQL mutations, but then it start to confuse me how to implement it

yes, this is exactly how I am doing it now

but there is not a lot of people with who I can do brainstorm about that topic

actually I don’t know people personally with deep experience about it I think

system have REST + GraphQL, because they exited before etc.

maybe there’s a #graphql channel

yes, I know that channel

but there is not a lot of system which based only on GraphQL without REST API I think, so people don’t have this experience

including me 😛

Me neither 😉

And there are some resources like download PDF which are not GraphQL… ok it is off topic now 🙂

GraphQL is designed for fetching nested data, to prevent unnecessary data sent from the sever, right? What stops you from making extra REST points together with it, that do not have that problem?

Do you know any place to talk about architecture of systems in that context?

It is something deeper than only GraphQL

yes, it could be done, but still there is a lot of decision to make. Like for example about auth.

It is about decisions 🙂

For example I started with sessions in Redis, but now I change concept to not use sessions at all

instead I want to send token in each API call

How to run yada to use easy parameters from POST? curl -vs -H "Accept: application/json" --data 'query="{ game_by_id(id: "1237") { name designers { name }}}"' -X POST -i <http://localhost:3000/authentication> <- for example I want to read query parameter form POST method.

Should I use ring middlewares?

@kwladyka I can give you an example of yada + lacinia when I'm at the computer

We do authentication in http (session / token) and authorization in lacinia obviously

https://blog.apollographql.com/a-guide-to-authentication-in-graphql-e002a4039d1

This might help you decide

Thanks, example of code will definitely boost me

It's just much more convenient to do authentication outside of graphql because of all the web standards like cookies

But that also means you need at least one non graphql endpoint for login e.g.

Gotta go, remind me later if I forget 😊

Different topic: how do you make POST request (yada/response-for authentication :post "/authentication" {:body "{\"email\": \"<mailto:foo@example.com|foo@example.com>\", \"password\": \"qwaszx\"}"}) ? I am trying to use response-for but I can’t guess the right way 🙂

{:status 400,
 :headers {"content-length" "67", "content-type" "text/plain;charset=utf-8"},
 :body "No body present but body is expected for request.\r\n\r\n{:status 400}\n"}

curl --data 'email=foo@example.com&amp;password=qwaszx' -X POST -i <http://localhost:3000/authentication>                        Fri Jan 18 23:08:35 2019
HTTP/1.1 400 Bad Request
Content-Length: 129
Content-Type: text/plain;charset=utf-8
Server: Aleph/0.4.4
Connection: Keep-Alive
Date: Fri, 18 Jan 2019 22:09:11 GMT

Bad form fields

{:status 400,
 :error
 {:email missing-required-key,
  :password missing-required-key,
  nil disallowed-key}}

Seriously I have no idea how to use POST in yada after a few hours of trying 😕 I will appreciate example of code.

there is no example in tests https://github.com/juxt/yada/blob/ba10db43e1a332f0bfd352cd8ea39f176190c4ff/test/yada/request_for_test.clj

And for some reason it doesn’t work also with curl

:parameters {:form {:email String
                                 :password String}}

{:status 400,
 :error
 {:email missing-required-key,
  :password missing-required-key,
  nil disallowed-key}}

Is it possible to use peridot with yada? How?

@kwladyka I have this in my tests:

(defn request [method url clj-body]
  {:method method
   :content-type "application/json"
   :accept "application/json"
   :as :json
   :url url
   :body (cheshire/encode clj-body)
   :throw-exceptions false})

(defmacro with-server
  "Runs resource in server and defines url for use in body"
  [url-bind resource & body]
  `(with-level :level/off "yada.handler"
     (let [resource# ~resource
           vmodel# (vhosts-model [:* ["/api/foo" resource#]])
           listener# (y/listener vmodel#)
           port# (:port listener#)
           close# (:close listener#)
           ~url-bind (str "<http://localhost:>" port# "/api/foo")]
       ~@body
       (close#))))

(deftest schema-validation-error-test
  (er/define-error-renderers true)
  (with-server url
    (y/resource {:methods
                 {:post {:parameters {:body {:foo s/Str}}
                         :response (fn [ctx] {:a 1})
                         :produces #{"application/json"}
                         :consumes #{"application/json"}}}})
    (let [response (client/request
                    (request :post url {}))
          body (cheshire/decode (str (:body response)) true)]
      (is (= "missing-required-key" (-&gt; body :error :foo)))
      (is (= "Schema validation error" (-&gt; body :message))))))

Hmm I will try to translate it to (response-for ... ) yada fn

Do you have any idea why https://clojurians.slack.com/archives/C0702A7SB/p1547849446131300 ?

It looks like yada doesn’t see parameters

I don’t use response-for, because it has some limitations, I can’t remember which, maybe to do with not all interceptors being present

it’s been 2 years since I wrote this code. This code only tests some error logic. Most of my other API tests I run completely outside of yada, as integration tests

the with-server macro starts a real aleph server, so it’s more like what you would normally see

oh, I committed it to yada itself too: https://github.com/juxt/yada/blob/ba10db43e1a332f0bfd352cd8ea39f176190c4ff/src/yada/test.clj#L40

you can see its use here: https://github.com/juxt/yada/blob/ba10db43e1a332f0bfd352cd8ea39f176190c4ff/test/yada/classpath_resource_test.clj#L49

Ok I have to do small steps. Any idea why yada doesn’t see POST parameters at all?

(def authentication
  (build-resource
    {:id :authentication
     :consumes #{"application/x-www-form-urlencoded"}
     ; :access-control
     :methods {:post {:produces "application/json"
                      :parameters {:form {:email String
                                          :password String}}
                      :response (fn [{:keys [parameters] :as ctx}]
                                  (println (pr-str ctx))
                                  (println (pr-str parameters))
                                  ;(println (pr-str form))
                                  (let [{:keys [email password]} (:body parameters)]
                                    {:token "foo"
                                     :email email
                                     :password password}))}}}))

curl -d 'email=foo@example.com&amp;password=qwaszx' -X POST -i <http://localhost:3000/authentication>
HTTP/1.1 400 Bad Request
Content-Length: 129
Content-Type: text/plain;charset=utf-8
Server: Aleph/0.4.4
Connection: Keep-Alive
Date: Fri, 18 Jan 2019 23:11:04 GMT

Bad form fields

{:status 400,
 :error
 {:email missing-required-key,
  :password missing-required-key,
  nil disallowed-key}}

yeah, you need get rid of the :form key in your schema

I think… let me check

what should be there instead?

But the point is (println (pr-str ctx)) doesn’t show me any parameters

So something is wrong

I have {:parameters {:body …}}

I was trying with that too

oh I see you’re dealing with a form, let me check if I have such an example

Do you do something extra to let yada see POST parameters in request?

It doesn’t work for me with any example. I don’t have parameters in`ctx`

can you try with a different consumes thing

e.g. application/json and then post a JSON body

:parameters is {}

sure

the same with JSON

:parameters is {}

ok but :body should be ok now, let me check

ok it works good for JSON

but for POST application/x-www-form-urlencoded so typical form it doesn’t work

oh I see I have to have parameters in map to have them in ctx. So for POST it has to be different way than :form and :body hmm

But I have no idea how to set it then

hmm my guess is yada doesn’t know how to handle application/x-www-form-urlencoded

this is super weird:

curl -H "Content-Type: application/x-www-form-urlencoded" -X POST -i <http://localhost:3000/authentication>                  Sat Jan 19 00:35:45 2019
HTTP/1.1 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Length: 45
Content-Type: application/json
Server: Aleph/0.4.4
Connection: Keep-Alive
Date: Fri, 18 Jan 2019 23:36:06 GMT

{"token":"foo","email":null,"password":null}

curl -d 'email=foo@example.com&amp;password=qwaszx' -X POST -i <http://localhost:3000/authentication>                            Sat Jan 19 00:36:06 2019
HTTP/1.1 400 Bad Request
Content-Length: 129
Content-Type: text/plain;charset=utf-8
Server: Aleph/0.4.4
Connection: Keep-Alive
Date: Fri, 18 Jan 2019 23:36:56 GMT

Bad form fields

{:status 400,
 :error
 {:email missing-required-key,
  :password missing-required-key,
  nil disallowed-key}}

it could be that yada doesn’t coerce the body against the schema in that content-type

for this setting

{:form {:email String
                                          :password String}}

is that documented somewhere?

What exactly?

form urlencoded post body

no but there are examples in the doc about POST

so it should work

here’s a test: https://github.com/juxt/yada/blob/master/test/yada/form_test.clj

Looking on this test I don’t understand why it doesn’t work for me

also, I can’t tell what your build-resource function returns. I’m off to bed. good luck 🙂

#yada.resource.Resource{:id :authentication,
                        :consumes [{:media-type #yada.media_type.MediaTypeMap{:name "application/x-www-form-urlencoded",
                                                                              :type "application",
                                                                              :subtype "x-www-form-urlencoded",
                                                                              :parameters {},
                                                                              :quality 1.0}}
                                   {:media-type #yada.media_type.MediaTypeMap{:name "application/json",
                                                                              :type "application",
                                                                              :subtype "json",
                                                                              :parameters {},
                                                                              :quality 1.0}}
                                   {:media-type #yada.media_type.MediaTypeMap{:name "application/edn",
                                                                              :type "application",
                                                                              :subtype "edn",
                                                                              :parameters {},
                                                                              :quality 1.0}}],
                        :methods {:post {:response #object[api.core$eval40522$fn__40524
                                                           0x41e5148f
                                                           "api.core$eval40522$fn__40524@41e5148f"],
                                         :produces [{:media-type #yada.media_type.MediaTypeMap{:name "application/json",
                                                                                               :type "application",
                                                                                               :subtype "json",
                                                                                               :parameters {},
                                                                                               :quality 1.0}}],
                                         :parameters {:form {:email java.lang.String, :password java.lang.String}}}},
                        :show-stack-traces? false}

I add there only :show-strack-traces? false

try moving the consumes into the post map, like in the test

I was trying that one too

I give up. I am going sleep.

Thank you for help!