Clojurians Log v2
architecture
I have an architecture question about the interaction of the component library and web frameworks like ring, pedestal etc. Usually web requests will need access to pretty much every component under the sun, on top of request-specific information (like who the user is etc). How is this to be modelled without having to pass in the entire system under a key in the request map?
We do the same with component — subsystems with dependencies, and it’s working well. I’m more asking about “don’t pass the system around” which is a common advice…
Same here. For example, our web server component usually has dependencies on db and monitoring components only.
Or maybe decompose a service layer into components and pass only needed dependencies per a service (component). It’s similar as you do in Spring etc. But I don’t know if this approach would be better/readable.
In our current code we’re creating a map with namespaced keys that correspond to various components that the requests need, and we pull those out in our handlers. But as gradually add more and more components, we have to expose those too, so it eventually degenerates to be a renamed version of our main component system.
> Usually web requests will need access to pretty much every component under the sun This strikes me as the root cause. It doesn't seem normal to me that every handler needs every component. That could mean any of these two: * there's a somewhat excessive amount of low-level components * maybe: create a higher-level component for them * or: don't create as many components - sometimes direct access can be OK * the app intends to be highly modularized, but it breaks its own module boundaries * this happens quite often which trying to use somehing like the Polylith but without any automatic means of ensuring that modules have a tractable dependency graph
With Integrant you can decompose a system into components and define dependencies for each component. A component could be a subset of routes.
I see some interesting points here, which leads me to ask: how many components do your apps have? It would fun to see the results of (keys system) to see what kind of dimensions people use...
In our case we have roughly a dozen components (afk so I can’t run that code atm) - and our web server component needs: • the multi tenant support • The mongo connection factory • The Postgres connection factory • The solr instance • The notification component • The email component • Some config stuff
So some real low level stuff isn’t used from the web (eg we have some queue component that isn’t exposed)
I’m using integrant, but aside from that my components look like:
• postgres connection
• app config
• firebase system
• stripe system
However, I will need to add my own email and other components as I go, so I would imagine it to include similar ones to what you have. The stripe and firebase ones are components because they require a “state object” which throws warnings/errors if you try to restart the whole system so I manage their lifecycles via integrant.
And I guess you have to pass all these to the web component, right?
I have to pass postgres and app config to my HTTP Handler component (web component), yes.
And then as I add things, I would likely have to pass those as well, so I can see where your coming from, but haven’t reached it as of yet.
> I’m more asking about “don’t pass the system around” which is a common advice… That question was the interesting one to me as well. I would like to see an example of an alternative approach. I figured what I do is fine (for now) because it’s not a map with random things that I pass in, but a curated map….which maybe is the same thing, but with more discipline? 😆
Does integrant not have dependency settings like Component?
As in, I can define which parts of the overall system my component will need in the system definition
Yes, it does (if i’m following your meaning).
of course, it's possible to have a component (say web handler), which require everything - but you never pass the whole system explicitly
having a bunch of keys in a map that you don’t care about isn’t an issue. it can, but doesn’t necessarily lead to some other problems like: • since every request may interact with any key, reasoning about the system becomes strained • producing information for the request that may not be needed may introduce noticeable overhead • others? It might be useful to further clarify the problem.
Another underlying problem might be that requests are strongly coupled to specific implementations of resources they need. For example, rather than just requiring some necessary user info, they are instead interacting with a mongo connection.
This is a good example of a problem I see a lot in the wild.
If I remember correctly, we have 31 components in our System. But the web server (and all remaining components) only depends on the DB and Monitoring for now. I think there’s nothing you can do if there’s a dependency between things. You can re-think if you really need a dependency. Our project is doing mostly data pipelines so they are more or less independent from each other. We are now breaking them into different “services”.
Yes, we use protocols for “interfaces” that are mature and we know every component is using them to avoid spreading internal details.
However, we still have some details for some things that are not very well mature yet. We prefer to feel the pain first and then refactor later. For that reason, we have integration tests which cover the interaction between components.
We have an "application" component which is reused across most of our web apps (which also have a "web server" component and a few others). That "application" component contains database (multiple connection pools), environment, caches, ElasticSearch, Redis, template engine (wrapper around Selmer that installs tags and filters at startup), and "SDKs" for communication with some of our subprojects, plus a few other things. Then there are a handful of other components that are used in some apps but not others (e.g., Microsoft Active Directory, email-based error logging, "presence" -- for tracking online members on our sites).
We do not try to create interfaces/protocols/APIs that hide all the implementation details except where we genuinely have multiple implementations.
we kind of lean on existing interfaces/protocols, some of our components implement clojure.lang.IFn, so you can look up a configuration value by calling the configuration component as a function
presence implements a few protocols from core.async
the nice thing about re-using interfaces and protocols is you can get mocks/stubs in tests "for free", so like if we need to mock the configuration component in a test for some reason, because it acts like a function, you can just use a function
Yeah, implementing IFn on a component is a nice affordance -- and work is where I got the idea to do that for next.jdbc.connection/component 🙂
Interesting that we never run into 'too many dependencies passed in' problem in our applications - perhaps it's because we run a SOA?
So I guess it’s the application component that in the end acts as the “orchestrator” and gets everything else as a dependency I guess, right @seancorfield ?
I’m curious to see how component is used without records/protocols, is it just a map with dependencies then?
@orestis Several apps have that application component as a subcomponent, but that's where most of our system dependencies live because that's our "core" system across most of our apps.
When I said no "interfaces/protocols/APIs", I meant we don't write those to wrap subsystems -- which you see as examples sometimes of "how to use Component". I'm not talking about the two lifecycle methods of Component itself.
That said, we do have non-record implementations of Component's lifecycle. And next.jdbc works that way too.
See https://github.com/seancorfield/next-jdbc/blob/develop/src/next/jdbc/connection.clj#L310-L318
My original question came from an argument at work - why have a protocol and a record where you could just have a function?
So it's an empty hash map with a start function and then a function with a stop function attached.
Especially in the case of non-stateful components where you just need the dependencies
Component requires associativity for things with dependencies.
I asked Stuart Sierra about enhancing Component to run dependencies via metadata and he felt it was too narrow a need (because only a few things can carry metadata that aren't already associative). So, if you need dependencies, you need a hash map or a record (but you don't need the protocol implementation if the component has no lifecycle). And if you don't need dependencies, you can have anything that carries metadata.
See https://github.com/stuartsierra/component/issues/67 for his response.
To rephrase: most protocols have “this” as their first argument. Is there any point in making a protocol + implementation (reify or record) vs just defining a “public” function that takes “this” as a first argument where “this” is expected to be a hash map with dependencies?
one way to think of component is has a system for building a graph of closures
The caller shouldn’t care because to the caller the dependencies are opaque - record or hashmap makes little difference
for protocols it is usually to support multiple implementations
Right, so if you don’t care for that, you don’t really need protocols. Tests can always redef the public function if they need to stub out stuff.
they can, but redef is fairly brittle
a def is a global thing, so is a redef, so if you have two instances of a component, redefing the public function to behave differently for one of them becomes a chore
That’s true, but then you are back at square one if you want testability, right? A protocol or multimethod...
Or that IFn trick which sounds interesting
multimethods implementations are also global
at my last job some people really hated records and protocols, so we had what was basically a fork of component where the lifecycle protocol was replaced with multimethods
it is ok, but you can't just (reify ...) up something that those multimethods will do the right thing with in the middle of a test if you need to stub/mock
It’s just annoying that re-evaluating protocols breaks existing implementations and you have to reload everything.
our protocols are pretty all defined in a namespace like *.protocols which pretty much only contain protocols and almost never change (don't need re-evaluating)
which is a common pattern you can see https://github.com/clojure/core.async/blob/master/src/main/clojure/clojure/core/async/impl/protocols.clj and https://github.com/clojure/core.logic/blob/master/src/main/clojure/clojure/core/logic/protocols.clj
if you use multimethods, I recommend doing something similar, pull the defmultis (the interface definition) into a distinct file
(I wish I'd followed that pattern more rigidly with next.jdbc -- I mistakenly put a few protocols in with other code!)
I do like the idea of a Component-variant that uses just functions (with metadata for dependencies as well as lifecycle hooks).
no always, but often enough mixing implementations and interfaces leads to pain
https://gist.github.com/hiredman/075b45eaeb01e4b526ce6f8854685487#file-ur-component-clj-L1-L10
My takeaway from this is that if you want to be able to stub out things protocols are needed. Put them in another file. See if you can implement IFn then the protocol is just a function which can easily be stubbed.
Perhaps if a record cannot be avoided, defer all the logic in a plain function and implement the protocol by just forwarding the calls to the plain function, this way you can REPL away without having to tear down your system.
"plain function" is doing a lot of hand wavy work there
alternative is :extend-via-metadata
Oh you said they hated records and protocols.
Well, you can get rid of the record bit.
ClojureScript does it this way I think, a lot of protocols that just forward the call to a similarly named function...
Oh, I stand corrected. I wonder why’s the case; perhaps a GCC optimisation?
I don’t hate anything 😅 just trying to figure out stuff, without cargo culting some approaches blindly.
About the IFn approach, is that where your protocol would have a single function so it’s replaced by a record that implements IFn?
It doesn't need to be "a record that implements IFn" -- it can just be a function.
It only needs to be a record if you have dependencies and you want it to implement IFn as well.
Right, I meant that to the caller the component is a function. If it’s a plain one or an IFn record or a closure is an implementation detail...
I’m off to bed, I’ll revisit all this tomorrow in front of a computer. Thanks for a nice chat!
clojurescript does the reverse of this, a lot of functions that forward the call on to a protocol function