Clojurians Log v2

I am tagging the code after each complete step of the migration so folks can “follow along at home”.

Interesting

I see that the deps.edn is now scattered across various components and bases - with quite a bit of duplication, i.e., same library's, versions etc. Not sure how I feel about that.

People have been using scripts to use the same version in every deps.edn in a monorepo. I think there is one written in bb.

I see. Would you have a reference?

Wow, this is really good news! Initiatives like this is what we need right now, so that people more easily can adopt Polylith. Thanks for contributing to the Polylith echo system @seancorfield!

As I’m migrating the usermanager example, I had a “lightbulb” moment about components that interact with a database: they do not need to specify the database driver as a dependency! The :dev alias can specify what database you want to use for the REPL, the :test alias can specify what database you want to use for tests, and the project deps.edn can specify what database you want to use for “production” (and can have additional tests — potentially slow ones — that verify system behavior against that database). The usermanager tests have generally used an in-memory H2 database for speed (and that it automatically starts in a “clean” state).

I tend now to use test containers for database stuff. Whilst H2 is very good at looking like Postgresql, it's still not the real deal 🙂 So I spin up a real postgresql instance during my tests (using testcontainers clj) to ensure that everything is honkydory

(esp useful when doing postgis stuff...)

Right, but for REPL/dev/test you want near immediate feedback. For full CI you can afford a slower cycle and a real database.

agreed, although for local dev, I usually have a real postgresql instance running anyway (either in a container, or on a server I can access)

Me too, although it’s MySQL, but it’s really too slow for immediate feedback so isolating those dependencies and swapping them out for faster options when you can is definitely valuable.

yes.

it would be nice if the polylith tool was made into a binary, using graalvm, so that it can be installed and used like babashka, clojure-lsp, joker etc...

@dharrigan You can install it via brew etc — I assumed it was a binary? But I kinda like that you can use it via just a :git/url dep so you can tie the tool to a specific version easily as part of the project.

It is not a native binary.

In the previous https://github.com/tengstrand/lein-polylith version of the tool we had a https://github.com/tengstrand/lein-polylith#sync command that took the library versions from development (`./environments/development/project.clj`) and made sure that all project config files used the same version of all the libraries, picked from that config file. In the new tools.deps version we went with another approach and decided that all dependencies should be defined in each project and to leave them out in the components and bases. This solved the library version problems, but added another problem, and that was to know what libraries was used in each component and base, so that the usage could be viewed by the libscommand. To solve that problem, we introduced the`:ns-to-lib`mapping. I was quite happy with this solution in the beginning, but over time it felt like a burden to keep it up to date. The old sync command made a lot of “magic” also. We are not big fans of magic, so we decided to have the rule that the new tools.deps based tool should only report if the workspace went into an invalid state, and that you had to update the files yourself manually. That has worked out quite well so far. We are now working on solving https://github.com/polyfy/polylith/issues/66 in a separate branch where we have removed :ns-to-lib and introduced separate deps.edn files for each brick (to get rid of relative paths, which is now deprecated in tools.deps). When you run the libs command you will get a sorted list of the libraries that is used, and it’s quite easy to spot if you use more than one version of the same library. To improve on this, we could add warnings (which could be turned on/off) if more than one version of a library is used across bricks. I have other ideas also on how to solve this, but I think this could be a good starting point.

Yes, we will look into that, but it’s further down on the priority list.

@tengstrand I definitely approve of the “no magic” choice.

At work, we “duplicate” nearly all our external dependencies from our subprojects into our root deps.edn file but not the versions — because we have a :defaults alias with all of the versions pinned in :override-deps — and then each subproject has its specific set of dependencies with {} for the version. But that also means the “build” parts have to use that same :defaults alias so everything is pretty tightly coupled. I don’t think there’s really a good solution for t.d.a-based apps yet, given the restriction to three deps files (system, user, project) and the lack of any sort of “include” or external reference.

I really wish t.d.a has the ability to include other edn files, and support env vars, like $HOME

As noted in the side thread above, using :local/roots for dev-time deps is problematic because of the transitive dependency staleness issue, although it does address the duplication of external dependencies. But that transitive dep issue is a bear and has bitten us several times (we’ve taken to simply deleting .cpcache from the repo root at times — and that’s what we do in our build/deploy script, to avoid stale dependencies.

@dharrigan Anything that can vary what a deps.edn file “means” is in conflict with being able to cache the dependencies — you’d have to make the cache conditional on everything that could vary: it would add a lot of complexity (and probably slow things down).

I still think there’s a good argument for providing a fourth deps.edn file so it would be system, user, repo, project (or some similar naming) but I think the core team is aligning around the idea of a repo “root” deps.edn file with either paths or (local) deps for subprojects — i.e., variants of either what we do at work or what Polylith is doing.

It’s a hard problem and pretty much all the options have downsides.

I’m curious as to what @tengstrand and @furkan3ayraktar et al do when they are adding new dependencies to a project while developing? Do you just restart your :dev REPL each time?

(I’m used to working with the add-lib3 branch of t.d.a which Alex has been keeping relatively up-to-date, and then using clojure.tools.deps.alpha.repl/add-libs to load new deps into the REPL so I don’t have to restart)

I’ve tried to use add-lib3 but had an issue with some other dependencies and gave up. I should give it a try again. At the moment I restart my REPL when I add a new dependency. Usually I add them to my :dev REPL first and start working with it right away. When I’m comfortable with it, I can add it to the project deps.edn or in the new case, to the component’s deps.edn.

Right now, the latest add-lib3 is from October, 2020 but there are no conflicts with master. I’ve asked Alex to bring it up to date again but I’ve been using that version for months with no problems. My REPLs nearly all include that branch and they mostly run for weeks (sometimes months: my HoneySQL V2 REPL has been running since January 31st at this point).

Also, maybe good to mention, I use https://github.com/clojure/tools.namespace/blob/master/src/main/clojure/clojure/tools/namespace/repl.clj during development time to do a clean start when needed.

Ugh! I’m so against the t.n.r approach 😞

I think reload/refresh is a bad way to work: better to develop clean working practices where you just don’t need those.

Sorry. I just see so many beginners pick up on it and then get into trouble when the “magic” doesn’t work.

This used to be such a silent chan, look at it now; hard to keep up 🙂

I’ve been curious about your no-reload /refresh workflow for a long while, @seancorfield . It is time I dig deeper there.

@seancorfield Yeah, I see what you mean. I guess I need some more time to develop those dev time best practices! How do you clean up your REPL state in case you removed some of the previously executed statements?

Do you mean like a function definition that you deleted from the source code?

Yeap, a function or a def for example

I find it very rare that I need to — mostly leaving old def’s in place is harmless, except for old tests, but you can just (def old-name nil) in a (comment ..) block or (ns-unmap *ns* 'old-name) if you really need to.

Or — and this is the most “nuclear” I ever get — remove-ns followed by the hot key to “load file” again. But that’s also really rare.

Sounds like I have similar habits as you there already, @seancorfield. 😃

how might polylith fit into an environment where certain projects (bases?) require more restricted access but depend on shared components? for example, Base A is a project shared by all engineers and it makes use of Component B. Base Z is a restricted project and accessible to only a few engineers but also depends on Component B. (my apologies if that's a silly question, we're only just getting started with polylith)

But my normal working practice is eval-top-block for every change I make, without even saving the files. I often work for quite a while with several “dirty” files in VS Code before I go around and save my files. And I’ll be running tests all the time too (I have a hot key bound to code that looks for the matching test ns to the one I’m in and requires it and runs its tests) so I can just stay in a source file, while I’m editing it, and eval a form and then run the tests, whenever I feel like, without switching contexts.

@joshkh Can you elaborate on how a project would be “restricted”?

(in a monorepo, I’d imagine all engineers have access to all code)

I can try to answer @joshkh. A solution could be to create a separate workspace for the team with restrictions. Then you can create a symbolic link to the componentsdirectory for the other project. Make sure they are cloned so they have the same parent directory on your local disk (you need to know where the other workspace is located to get the symbolic link to work). If you only need your base in that separate workspace, you should be fine. You should also create a project that includes the base and the components from the other workspace. Note that this is a “hack” to keep the development experience that Polylith provides. If you don’t think you need the fast feedback loop in development, there are other options too, for example to depend on the components by selecting the git hashes which would probably be a better choice than to create libraries as a way to share the code across the two workspaces. I haven’t tried this out myself, but my gut feeling is that it should work quite well.

that's very helpful, thanks @tengstrand! we'll play around with your symlink/hash suggestions and see which best fits our situation. we're really curious to see how we practically benefit from a monorepo 🙂

Good luck with that!

Just a thought (that might have been floated before and shown not worth considering): What if the interface was a file named like the component, one level up in the directory structure? So, for the RealWorld example and article, which now looks like so:

components/article/src/clojure/realworld/
- article/
  - core.clj
  - interface.clj
  ...

components/article/src/clojure/realworld/
- article.clj <- the interface
- article/
  - core.clj
  ...

Interesting idea! I kind of like it @pez. Need to think more about, but it looks nice! :-)

I think as long as we add the functionality to be able to have component name as the name of the interface namespace, this should work out of the box. Developers are pretty flexible in structuring the internals of a component.

I'm not sure about that tbh, taking a lesson from the go world, it advises people to avoid "stutter" for packages, i.e., "clojure.realworld.article.article" is just repetition and doesn't add anything extra meaning or clarity.

relevant section here:

Avoid stutter. Since client code uses the package name as a prefix when referring to the package contents, the names for those contents need not repeat the package name. The HTTP server provided by the http package is called Server, not HTTPServer. Client code refers to this type as http.Server, so there is no ambiguity.

Well, the use of something else than core.clj is entirely up the user anyway. It is not really part of the idea. Haha.

The problem is that sub namespaces for the interfaces will not work, because they will be treated as implementation namespaces, e.g. if util.date is part of the interface in com.mycompany.util.date where dateis the sub namespace of the interface, but part of the interface, then we can’t distinguish between implementing namespaces and the interface.

indeed. I always consider "core" to be a library-centric term. when writing applications, I favour using "main".

could of course call it "interface.clj" then "impl.clj" 🙂

Makes sense @tengstrand.

interface/impl is a nice pair of names.

Today we use the naming convention com.mycompany.article.interface for the interface and then we could of course easily change the implementing namespace (that is created when you create a new brick, but that can be renamed) from core to something else, e.g. main, implor something else, e.g. com.mycompany.article.main or com.mycompany.article.impl.

This could even be a configuration in the workspace.ednfile (which replaces the :polylith section in today’s ./deps.edn in the next version).

Yeah, at least now I know why the interface part of the name is needed. 😃 I think I will try to go with impl once I get to actually setting a project up.

But because it’s so easy to rename the namespace names, I think adding a comment in the implementing namespace that it can be renamed is probably enough.

@tengstrand This is exactly what I was suggesting at the weekend — but I think we were not on the same page while discussing this. I agree with @pez that this would be a good alternative (I would like it much better than the current structure, as it matches what we tend to do already at work).

This was also what I was hinting at in my post on ClojureVerse — because I thought it was what you’d already discussed with the team…

However, it doesn’t seem it is really an option, if I understood the further comments correctly?

At work, we tend to use a form of naming like:

subproject/src/com/acme/
- subproject.clj
- subproject/
  - impl.clj
  - more_impl.clj

Yes, it looked promising, but unfortunately, it will not work (see my answer further down). I discussed with the team after I spoke with you, and it was after that we decided to support your idea, but to keep the default we have today.

I understand that’s a structural change, not just a naming change but I think Polylith could do it without conflict/ambiguity.

But we also support having sub namespaces as part of the interface, see the https://github.com/polyfy/polylith/tree/master/components/util/src/polylith/clj/core/util/interface component, and in your example, Polylith will treat impl and more-impl as part of the interface (because they live under a package with the same name as the interface).

But you you can of course put your implementing code in any other structure under subproject/src/com/acme.

No, it will not work out, because if you put other namespaces under subproject/src/com/acmethat belong to subprojectit will be treated as other bricks.

How would impl and more-impl be considered component interfaces when they don’t match the declared interface naming convention?

In your example, the subproject.cljis the interface of the component, right?

If you had :interface-ns :component-name then components/comp-name/src/com/acme/comp_name.clj would be the interface (named to match the component) and anything else in that components/comp-name/src/com/acme/ tree would be implementation for that component.

If you had :interface-ns "fixed-name" then components/comp-name/src/com/acme/comp_name/fixed_name.clj would be the interface.

But you forget that we support sub namespaces today, and we still need to support that even if we support the :interface-ns :component-name option. Today com.acme.comp-name.interfaceand com.acme.comp-name.interface.abcwill both belong to the interface.

In that case com.acme.comp-nameand com.acme.comp-name.impwould therefore be part of the interface, otherwise we have to introduce something like com.acme.comp-name.interface.abcto support sub namespaces for interfaces.

That is possible thought, so I’m not saying it would be a bad idea.

Sub namespaces are not used very often in interfaces (e.g. 1 out of 25 components in the Polylith workspace) so this could actually be the way to go, and maybe even be the default (we need to think more about it first).

Oh, I get it now! Sorry, I did not understand what you meant about subnamespaces being interfaces — I did not know you could put additional “interface” namespaces under a directory named for the :interface-ns! Is that in the documentation anywhere?

Yes, in the interfaces section.

OK, will go read that again…

It’s in the (long) README but it’s not mentioned in the GitBook, yes? I can see that now…

So… maybe I’ll go with components/comp-name/src/com/acme/comp-name/interface/some-file.clj for my interfaces… that’s what I’m understanding the README to say will work?

(although I’d probably use api for :interface-ns at this point)

That’s an interesting alternative. So a component may have multiple interfaces in a way (multiple namespaces). Would you consider multi-namespace interfaces an indication that maybe the component is doing too much and should be split into separate components?

I write in the documentation that it can be an indication that a component does too much. That’s why I only use it in the util component, as a way to group util functions around different areas, like string manipulations, time helper functions, and so on. The alternative was to create five components or to put all the functions in the same interface namespace, which I didn’t like. We also say in the doc that you could put e.g. spec definitions as a sub interface, to make it easier to find and to make the interface more cohesive.

We put the specs in the interface.spec sub namespace in the realworld example app.

I realised that we still need the :interface-nsas it is today, giving the name of the interface to support sub namespaces in the interface. We need to figure out another property to specify which naming convention to use for the interfaces (as today, or what you suggest @seancorfield).

Thanks. Food for thought. Naming (and structure) is hard 🙂

Yes, one of the hardest thing in this industry!

I posted a correction in https://clojureverse.org/t/on-polylith-from-amazing-ideas-thread/7410 also.

I watched James Trunk’s video about Polylith last night and he illustrated the LEGO-brickyness very nicely. Made me understand why base is named as it is. Could be that some of those illustrations should be used early in the documentation? https://www.youtube.com/watch?v=Y3FfLq8QATY

It’s a beautiful coincidence(?) that his talk on Clojure seems to be spreading like wildfire the last few days. 😃

I haven’t noticed, but he is an excellent communicator so I’m not surprised. People get more and more interested in FP and also hopefully in Clojure!

Might as well post the link to that here for context: https://www.youtube.com/watch?v=C-kF25fWTO8

You are right @pez that we think of a base as something that stays at the bottom of the artifact we build. We realised though that the Polylith building blocks (libraries, components, and bases) are even simpler than LEGO bricks. One way to illustrate this is that it’s enough to put all building blocks you need into one place (a project) and they will automatically “connect” (kind of magic)! I still think the “old way” of illustrating the building blocks has value, but that metaphor was also more complicated than the thing we tried to explain, which was the reason we abandoned it.

Maybe then that base is no longer the right name? Just thinking out loud here. OTOH, there is a point with using a name different from most concepts the users might bring with them. So to not mislead the perception.

@tengstrand Moving on to a component/architecture Q: as I’m refactoring the usermanager example, it has an “application state” Component (as in Stuart Sierra’s lib) that currently does “nothing” beyond track that the app is running or not — it’s an example placeholder to show users how they could handle some overall application state that has a start/stop lifecycle. That’s currently in the web base but it isn’t related to the web “API” aspect of the code. It explicitly depends on a :database Component. I’m planning to turn the :database Component and some of its associated code into a (Polylith) component, and I think the “application state” Component should probably also be a Polylith component — thoughts?

Is this code that is only used at development time?

No.

This is separate from the WebServer (SS) Component, BTW — which I’ll also turn into a (P) component because it is independent of any particular base: it’s a generic “this is how a web server starts/stops” component, reusable across any number of bases and/or projects.

Okay. Then it sounds like it should go into a component. If you have code that you don’t need in production, like convenient functions that start/stop servers, then an option is to put that code under the development project which indicates that it will never be shipped to production and that it’s only used for development.

Right, and I do have some expressions in development/src/dev.clj for “manually” starting and stopping the overall system for REPL usage: https://github.com/seancorfield/usermanager-example/blob/polylith/development/src/dev.clj

When creating new test stub files, it would be great if, instead of just requiring clojure.test, those stubs also required the namespace that is supposed to be tested. That provides a quick sanity check to catch syntax errors and typos in the newly added base/component.

For example:

(ns usermanager.web-server.interface-test
  (:require [clojure.test :as test :refer [deftest is testing]]
            [usermanager.web-server.interface :as sut]))

(I don’t like :refer :all so even in tests I tend to use aliases and explicit :refer clauses)

Is there a way to display the libs used by the :test alias via the poly command? It seems to only display libs used by bricks and projects.

(my usermanager app uses H2 in-memory for testing and it doesn’t show up under poly libs or poly libs :all)

We don’t show test dependencies at the moment. If we want to show them, we should decide if we want to se only the test deps or both at the same time and wether we should show in some way if it’s used only by the tests of not. I haven’t thought about a good solution for these things.

Any thoughts on my comment about tests above?

(if folks have suggestions about Polylith, what is your preferred process for handling them? discuss here, issues on GH, some other venue)

Yeah, sorry, we play poker at work (online) right now, so I’m a bit distracted! I think it depends, but I like communicating with folks here. Often it’s a good way to sort things out. Video can be even better sometimes, but text has the advantage that the questions and the answers can be read by everybody (at least for a while if you run the free version of Slack). But I’m fine with Issues at Github also, I don’t have a strong opinion. Texting here is a bit faster that GH issues.

I’m not a big fan of :refer :alleither. It’s just that I have found that a lot of people use it when they write tests, so I tried to conform to that. If there are good reasons to change, then I think we should. I like the idea to include the namespace that are under test, so I will take a note and look into it. Good suggestions!

“It’s just that I have found that a lot of people use it are lazy when they write tests” 🙂

On testing, is there a preference around testing components against their interface vs testing against their implementation? (I would expect a preference for testing against the interface, but I’m curious what sorts of tests you would do against the implementation instead in some situations?)

Okay, this wasn’t so easy to answer! I think we should use the same principles as we do when we test code in general. I like to have at least one happy path test and one or more “unhappy” paths for each function against the interface. If the code has many paths through the call chain, or if some of the functions are complex, then I sometimes add tests for functions that live in the implementation. When I do that I normally add these to a corresponding test namespace. I don’t have any strict rules here, but if a component has function f1 in its interface and delegates to f2 in its implementation, then I always tries to test f1 instead of f2.

Cool. Based on the Testing section in the README, that’s what I expected your answer to be but it didn’t seem to be explicit in the docs.

I asked because the usermanager originally had tests against the “model” — the implementation — and as I refactored it into components, I switched the tests to the interface (and discovered I’d missed one function from the interface because it was only used in one of the tests — but should have been part of the interface).

Ok, cool.

I’d love to get some review/feedback on where I am with the usermanager refactor: https://github.com/seancorfield/usermanager-example/tree/polylith — I think it’s “done” although perhaps I could split out the department handling and user handling into two components, and rename usermanager to addressbook perhaps as the overall coordinating component? But then that components interface would mostly be implemented by calls into either the department interface or the user interface — would that be weird, having a component interface delegating to two other interfaces for most functions? Anyways, any and all feedback appreciated.