Clojurians Log v2

afaii components do consume other components, and that seems to be a big point with the architecture.

I’ve read some documentation, watching talks and also listened to some podcast episodes regarding Polylith. The biggest value proposition for me is that they have solved a number of problems (monorepo-structure, testing, growing the system, optimize for RDD) and created a ready to use solution with documentation. I haven’t tried it out yet but I will do a poc soon (polylith vs a deps.edn monorepo). I am also Interested in @smith.adriane question about keeping things in sync. I guess you need to have backward compatibility in mind and maybe the polylith-tool can help with that.

Thanks for your interest in Polylith @smith.adriane. I will try to answer your questions. “Generally, it seems like polylith advocates for” 1) “A mono repo” Answer: Yes. The reason is that it allows us to support incremental builds and it also keeps the whole codebase consistent which is important because components are used across projects. 2) “A separation between projects and the artifacts they produce.” Answer: Polylith doesn’t care about how you build your artifacts from each project (or what language you use, but we currently use Clojure). The poly tool uses tools.deps that has a deps.edn file per project. You are free to put aliases there that helps you build the project and/or use external build solutions. 3) Building applications (bases?) on top of interfaces Answer: Each project has a deps.edn file. In this file you will list all the bases, components and libraries that are included in the project. We are currently working on [issue 66](https://github.com/polyfy/polylith/issues/66) where each component and base will have its own deps.edn file where it specifies its src, resources, test directories and other dependencies. Each project also has a deps.edn file where components and bases are listed as :local/root entries and a list of the libraries it uses (this is not released yet - work in progress) . “However, those principles are pretty general and many companies already do that.” Answer: You are right that mono repos, projects and interfaces are existing concepts. One big difference here is the level of reuse and flexibility you get by having small Lego-like building blocks that can be reused across projects. In the end you will end up with https://polylith.gitbook.io/polylith/architecture/simplicity systems made of small Lego-like bricks. Bases “just” delegate to components. Depending on the definition “just” 1) Delegation is mechanical. Bases are boilerplate and really shouldn’t exist Answer: Bases are the “interface” between the outside world and the components. Separating bases into their own concept have many benefits, and allows you to change how your functionality is exposed by just replacing a base (for example by switching from a “REST API” base to a “lambda function” base). 2) Delegation is not mechanical. It’s not really “just” since the non mechanical delegation can be a source of incidental complexity. Answer: We try to keep different concepts separated, which makes things easier to reason about. Bases are no exception. All dependencies points in the same direction in a Polylith codebase which stops you from introducing circular dependencies, which would otherwise be possible when not using Polylith (e.g. that service A calls service B and that B calls A). If you summarise the cost of an interface, you will realise that they actually save you time and makes your system(s) simpler. “It makes sense that components don’t consume other components (or do they?), but it might make sense for bases to consume other bases. Since bases are compositions of components, it seems like it might be reasonable to build larger bases from simpler bases.” Answer: Components and bases only know about interfaces. It’s only when you put them together into projects that you decide which concrete components to use for each interface. Components are allowed to use/call libraries and interfaces. The same idea goes for libraries, that they are allowed to use/call other libraries. If you disallowed that for libraries, you would end up with a lot of code duplication. The same principle goes for components. The main idea here is that both components and bases only depend on interfaces, not concrete implementations. We mention in the documentation that a project can be made by several bases, but the normal use case is that e.g. a service contains only one base, and that a library doesn’t contain any bases at all, just components. The benefit of putting the business logic and infrastructural code into components is that they are not just reusable but also replaceable, just like Lego-bricks. You don’t get replaceability with bases, so this restriction is there to help you make better design decision and decouple your system. 1) How are interfaces implemented by separate components kept in sync Answer: This is done by you as a developer, but with help from the poly tool. If you have two components that “implements” the same interface, and you add a new function to one of the component, you also need to add a function with the same name and signature to the other component’s interface. If you forget, the poly tool will remind you to do so. 2) What are the guidelines for writing good interfaces? Answer: Follow the same guidelines as you would normally do for an interface (e.g. for an interface in OO). The end of the https://github.com/polyfy/polylith#interface section lists a number of things you can do with interfaces. 3) What if the interface provided isn’t a good interface? Answer: What would you do if you had a class and it’s interface wasn’t a good interface? Probably redesign the interface, by making sure that each function has a descriptive name, and that the class only exposes what it needs to expose, and so on. “Building complex functionality out of simpler pieces is good, but it seems like a missing piece is clear guidelines for keeping bases, components, and libraries simple so that they can be easily composed. An example of the type of guidelines I’m describing is Erlang’s OTP, http://erlang.org/doc/design_principles/users_guide.html.” Answer: This Erlang guideline looks like a nice piece of work. I will have a look and see if it can help us improve our documentation. In the Polylith tool documentation we have some tips on how to name things, how to think about interfaces, and so on. We also have an https://github.com/furkan3ayraktar/clojure-polylith-realworld-example-app that shows how it could turn out in a real application. What is best can differ from codebase to codebase, so my advice to you is to try it out yourself, where a good start is https://github.com/polyfy/polylith#interface. You can always ask questions to the Polylith team in the #polylith channel. We try to help out as soon as we can. Regard, Joakim Tengstrand

Thanks for the detailed response! My hesitation with Polylith is that my impression of the pitch is "If you use Polylith, you end up with simple, composable, and reusable lego bricks." Based on the documentation I've seen and the presentation, the focus seems to be mostly about where code is located. From the perspective of a potential user, having guidelines for where to put code is the easy part (Note: I'm not trying to trivialize the effort it takes to design a holistic system and all the tooling, documentation, and support code that goes along with it). In my experience, designing simple, composable, and reusable systems is never easy and knowing where to put code is only small part. I haven't seen much documentation that talks about: • how to design good interfaces • how to grow/extend interfaces • how does Polylith allow you to upgrade a system while it's running? • how to deal with version dependencies (component A depends on version X, but component B depends on version Y) • how to handle state These are some of the hard problems I would want addressed if I were to consider something like Polylith. It's possible that Polylith has good answers to all of these problems, but the fact that the pitch omits these hard problems and makes it sound like these problems are easy makes me uninterested in reading more to find out. > Components are inherently simple and easy to reason about; they are just code, have a clear interface, and hide their implementation. This is an example of the type of statement that is a red flag for me. If I were to adopt something like Polylith, I want to trust that the design is realistic about what the hard problems are and how they are addressed. A good comparison for Polylith might be Erlang + OTP. Armstrong's thesis is one of the best examples for describing this type of holistic design, http://www.cs.otago.ac.nz/coursework/cosc461/armstrong_thesis_2003.pdf. > When we program we want to structure the code into “difficult” and “easy” modules. The difficult modules should be few and written by expert programmers. The easy modules should be many and written by less experienced programmers. In addition to clearly describing the problem, the paper also does of a good job of highlighting what the hard problems are and how they should be addressed.

I think my biggest “objections” to Polylith are the proscribed naming -- interface.clj and api.clj — and the stricture that those namespaces contain only single-line delegation functions to the same function names in (one or more) implementation namespaces. It seems to me like you would get all the benefits of reusable components by just using the name of the component (or base) for the .clj entry point and having the “API” of the component being the public functions (which should be kept very simple, I agree). Having lots of files with the same name is a pain when you have a lot of tabs open in your editor (with Polylith you’d have lots of interface.clj files and probably quite a few core.clj based on the examples given).

If I see components/foobar/src/com/acme/foobar.clj I can assume that’s the “interface” without needing an artificial .interface added to the namespace/path. And the implementation can be in files like components/foobar/src/com/acme/foobar/quux.clj (so you could mandate that only one .clj exists in the “top-level” of a component’s code to make that distinction even clearer).

Let’s start by answering @seancorfield’s “objections” first. The api.clj is just an example for a base. You can use any name of the namespaces in a base. When you create a base, it will be create a core.clj namespace for you, that you can rename to whatever you want. It’s just a starting point. An example could be the https://github.com/polyfy/polylith/tree/master/bases in the Polylith workspace. They expose a command line interface, and are quite tiny, a few lines of code in a single core namespace. In the realworld example app the https://github.com/furkan3ayraktar/clojure-polylith-realworld-example-app/tree/master/bases/rest-api/src/clojure/realworld/rest_api base consists of four namespaces api, handler, main, and middleware. We haven’t experienced the interfaces as a problem. The answer to the question “What’s your experience of working with Polylith in practice?” in the https://polylith.gitbook.io/polylith/conclusion/faq section may give you an idea of how it is to work with Polylith in a production system. If you think a single namespace is enough, it’s perfectly valid to put all the implementing code for a component in the interface namespace. I understand what you suggest with having only one namespace at the top level and be able to give it any name, and in that way indicate that this is the interface of the component. I guess that could also work. I can discuss it with the team and maybe we could support this behaviour too. It wouldn’t be too much work to implement. Now the default is to use interfaces as default (actually, this is also configurable, so that you can change it to e.g. ifc) but by setting a flag in the config, it could maybe be changed to use the behaviour that you suggest.

OK. “example” is not the way that api.clj and interface.clj come across in the docs/videos. Could I ask that you please don’t perpetuate the core.clj naming default? Leiningen did it just to avoid the single-segment namespace issue and Phil has said that he would not do that if he could do things over (he’d force you to use a qualified or multi-segment name — like clj-new does).

Do you mean it’s bad practice to use the corenamespace in general?

Yeah, it’s an artifact of history and not a good idea really.

I would prefer a structure of components/database/src/clojure/realworld/database.clj and then components/database/src/clojure/realworld/database/db.clj and components/database/src/clojure/realworld/database/schema.clj

The nice thing about that would be that there’s then a single top-level .clj file that is obviously the entry point rather than an artificial name, and all the implementation code is in a folder below that level.

And it avoids having an editor full of tabs that all say interface.clj 🙂

Ok, so for example, this https://github.com/clojure/clojure/blob/master/src/clj/clojure/core.clj namespace should have the name clojure instead?

Huh?

I think you have valid objections here, so I will take this back to the team and discuss it further! 🙂

Leiningen likely took the default from clojure.core but that makes more sense for a language than it does for any “general” code.

Sure, I get the idea. It can be a configuration thing, but we need to discuss it in the team first.

Yup. I’m not expecting you to just change stuff immediately and arbitrarily because some random Clojurian suggests it 🙂

I’d be interested to hear more justification around the “interface” containing only one-line delegation functions to other namespaces, if you have time?

We welcome all ideas that can improve Polylith, and if you find anything else that you feel can be improved, don’t hesitate to reach out to us.

Sure.

In an app like ours, a component could have a pretty large “API” (e.g., there’s a lot of member-related stuff for our dating sites) so that would be a lot of functions that needed to have “duplicate” definitions in two namespaces (the delegation function in interface.clj and the implementation function elsewhere). That seems overly-restrictive to me — and I haven’t yet found a good justification/explanation in your docs/videos. At least, not one I find convincing.

You are not forced to delegate to other implementing namespaces from the “interface” namespace. It’s just a pattern that makes the interface really clean (in the same way that interfaces in the OO world don’t contain any (or just little) implementation code). In some situations I put the implementation in the interface if the implementations are tiny (e.g. mostly one liners).

OK. Again, not clear that is even an option based on docs/videos 🙂

I’m going to switch to the #polylith channel at this point, to share more specifics about stuff…

We could add that you are not forced to delegate from the interfaces to the documentation. Okay, will just finish with this: The idea behind Polylith is not to make the life harder for developer by adding restrictions, but instead to support a good separation between different concepts and only add restrictions when it adds a lot of value. One “restriction” is the directory structure, but that really makes sense to me and adds a lot of values. The other restriction is that we separate components from bases. That also makes a lot of sense, because it helps you replace bases (the public API) without affecting the components + that they takes care of different concepts (exposing a public API / packaging other functionality). The last restriction is that we force you to have (at least) one namespace that serves as the interface, and that components and bases are only allowed to access interface namespaces (and libraries of course).

Hi again @smith.adriane, how to design good interfaces Answer: Polylith gives you the “tool” here, but it’s up to you to decide what is a good or bad interface. My best advice here is that you have a look at the https://github.com/furkan3ayraktar/clojure-polylith-realworld-example-app and the https://github.com/polyfy/polylith itself to get some answers/inspiration. how to grow/extend interfaces Answer: I normally add one more function at a time to an interface when I need more functionality. I also change the name of a function when I found a better name. I use different techniques to improve the readability of the interface which you can read about in the end of the https://github.com/polyfy/polylith#interface. When a part within a component can be used somewhere else, I extract it to a new component to get rid of code duplication. In that case the functions that previously lived in the first component’s interface will now live in the new one. In general, try to communicate what the interface does and/or is as clearly as possible. We mention some of this in the documentation. how does Polylith allow you to upgrade a system while it’s running? Answer: Polylith doesn’t help you with that. Polylith helps you with a lot like separating development from production, but it’s not what e.g. Spring is for Java. We write about it in the https://polylith.gitbook.io/polylith/conclusion/faq “Spring is a framework with a lot of ready-to-use functionality. Polylith is much simpler and doesn’t provide any ready-to-use functionality...“. (except maybe incremental builds) how to deal with version dependencies (component A depends on version X, but component B depends on version Y) Answer: All code in the workspace, including components, always use the latest version of the code. One way to put different versions of the code is to divide the interface into sub namespaces, e.g. mycomponent.interface.v2 and then delegate to different implementing namespaces within that component. how to handle state Answer: The short answer is that this is also handled by you as a developer, by using an existing library or tool. In the https://github.com/polyfy/polylith#profile we say: This example was quite simple, but if our project is more complicated, we may want to manage state during development with a tool like https://github.com/tolitius/mount or we could create our own helper functions that we put in the dev.lisa namespace, which can help us switch profiles by using a library like https://github.com/clojure/tools.namespace. “If I were to adopt something like Polylith, I want to trust that the design is realistic about what the hard problems are and how they are addressed.” Answer: We try to explain this by comparing Polylith with other architectures and also by listing the problems it solves in https://polylith.gitbook.io/polylith/conclusion/advantages-of-polylith. We also try to explain why we have made those decisions in the https://polylith.gitbook.io/polylith/architecture/simplicity section. I agree though that we can be clearer about what problems it solves in the beginning of the high-level documentation, because sometimes people stop reading after one or two minutes! When we program we want to structure the code into “difficult” and “easy” modules. The difficult modules should be few and written by expert programmers. The easy modules should be many and written by less experienced programmers. Answer: You are free to organise your components bases, and projects in any way you like in Polylith. Because it’s so easy to refactor a Polylith codebase, it’s also easy to adjust the design while you go, without painting yourself into a corner. If you prefer to divide the codebase into “difficult” and “easy” components, that’s fine, but we don’t have strong opinions about this, because people have different perspectives on what is good or bad practice/design. /Joakim

I'll just say that I view this as a big benefit of a monorepo: > how to deal with version dependencies (component A depends on version X, but component B depends on version Y) > Answer: > All code in the workspace, including components, always use the latest version of the code.

I totally agree. The monorepo guarantees this and you don’t have the “library hell” problem that you sometimes can get into when libraries are not 100% backward compatible. The test suit is also always running against the latest code, which is not the case in some codbases that I have worked in, where you have tens or hundreds of services using different versions of the same internal library.

When using libraries as a way to share internal code across services, you can get into problems when people change functionality in one library, creates a new version of it and then only use the new version in “their” service. Six months later when another team decides to update their service to the latest version (maybe they needed to add some new functionality to that library) the changes that the other team introduced affects “their code” (monorepos also fights this problem with their/mine mentality) and now you need to go to the other team and ask them to fix the problem or you have to fix it yourself.

My 2c, having not dug too deeply (watched the video): Assuming we're talking about a Clojure system, I'm not really seeing what Polylith offers me other than a set of code organization rules that I could also (arguably more easily) implement with Clojure namespaces: put components under project.components.* with public functions as its interface (as @seancorfield suggested above), and put bases in project.bases.* (although, at this point the bases segment becomes kinda superfluous). To get the same Lego-block composability you can implement the same interface in different components (ns's), and swap out the ns's as necessary. If you require a more formally defined interface, it seems like we're talking protocols. Having an (automatically) enforced organizational scheme certainly has merit, because it removes that "freedom" from individual devs. It's like enforcing clj-fmt formatting in CI; you don't need to think/worry/argue about formatting ever again, because it's fixed and taken care of for you. It seems like Polylith possibly provides a similar kind of stability for code layout. OTOH it looks like this pattern adds a lot of overhead and boilerplate, where there is much less with "normal" namespaces.

@clojurians-slack100 Yeah, this is part of what I’ve been discussing with @tengstrand — Polylith comes across as a set of naming conventions and directory structures but now that I’ve spent some time with it, I can see it’s more than that: it’s just not being communicated very well…

It took me a while, using the poly tool, to see what they’ve really achieved here — and the structured naming is just a small part of what makes the tooling possible… and it’s actually the tooling that I think is more important (and it needs certain conventions in order to work).

Good to hear! Would you now say you're now convinced that the benefits outweigh the added overhead?

I wouldn’t say I was convinced of the “whole plan” but there are interesting elements that are clearly useful.

Nice. Is there something specific you'd recommend I look at for those elements, or is using poly the best way?

The incremental testing is very well thought out. The whole tracking of dependencies is pretty good. The constraints that poly checks about how bases and components (and, I think, projects) all interact with each other is consistent and well thought-out. It's funny, we had a couple of SDK subprojects and we initially mandated a bunch of rules about which subprojects could depend on each other in order to keep separation of concerns -- and Polylith formalizes that and enforces it beyond what we thought was useful.

At work, we've developed the incremental testing and the coverage testing (i.e., testing every component necessary when we build a specific project) -- but in a somewhat ad hoc way, because our repo doesn't have the same level of structure as Polylith.

What I'm still trying to ascertain is exactly which parts of Polylith are "important" in their own right and which are just "ceremony" 🙂

Sounds interesting. Thanks for the details.

Good morning guys! I’m glad to see that @seancorfield appreciates the poly tool. Let’s start trying to answer the last question: what parts of Polylith are important and what are just “cermony”. The short answer is that all parts are needed. • libraryused in bricks (components and bases) and projects to reuse globally shared functionality. • interfaceenables us to swap out implementations (components) in our projects. • component a piece of composable and reusable functionality that implements an interface • base exposes a public API and is the bridge between the consumer/user and the service/tool. A base is often the base of an artifact (project/service/tool/artifact) and allows us to change how our functionality is exposed in production by choosing what base to use for each project. • projects it’s in the projects we decide what concrete building blocks (bases, components, and libraries) to use in each deployable (e.g. service/tool/library) and are used in our test/stage/production environments. • development we separate the development project from other projects and the reason is that we want to support the best possible development experience as possible, a single REPL. The profile s will also help out and is part of the dev experience. • workspace is needed to support incremental builds, allows us to easily share code between projects, allows us to have a single development project that is always up to date with the latest code, and guarantees that we always use the latest code in our deployables. And yes, the workspace directory structure, where every concept is structured consistently allows tooling to be created like the poly tool by using https://en.wikipedia.org/wiki/Convention_over_configuration. The outcome from all this is that you can postpone how things are executed in production and instead focusing on the domain in dev, and postpone (or change) decisions on how to execute things in production without affecting the development experience. #polylith

I also want to elaborate on “…and the structured naming is just a small part of what makes the tooling possible… and it’s actually the tooling that I think is more important”. We started out using Polylith without having a tool and after working with Polylith for years, I just want to say that the tooling support is not the important thing here, it’s the Lego-like way of working with code and the way you solve architectural problems using Polylith. The tooling support is very convenient and shortens the feedback loop, and allows other tooling to be build around it (e.g. build tools, code generation of documentation, and so on…) where tools.deps is a really good fit for achieving that. We also believe that using a https://martinfowler.com/bliki/UbiquitousLanguage.html for software design has a huge value when it comes to communicating ideas and to e.g. understand a new codebase and to have clear definitions for all the above concepts shouldn’t be underestimated.

We have now updated the first page of the high-level documentation. Please have a look and give us feedback if you like @smith.adriane and @seancorfield. We have also updated the https://polylith.gitbook.io/polylith/conclusion/faq with several of the questions from my answers here (at the end of the FAQ).

I think those changes definitely help.

Okay, cool. We plan to add what it’s not also.

This discussion was great to read, thanks for everybody involved, as a Lead Dev of my current company I am also considering Polylith, I found actually hard to give the code base some structure (partly because of its young age and startupy nature). We settled for the classic Web->Services->Repository structure but I don't think it sits really well with Clojure can offer, I cannot describe it better than "it feels too OO". Will read more but wanted to say thank you for the nice tooling and ideas:smile:

Thank you for the kind words @richiardiandrea and good luck with your Polylith journey!

Thanks for being invited! 🙂