Clojurians Log v2

@rickmoynihan The profiles are traversed in dependency order. This is deterministic, and unrelated to the order the profile keys are given.

Hi 🙂 It’s deterministic, but when you add new profiles the ordering is highly non intuitive. Is there a reason they’re not in profile-key order?

Because modules (of which profiles are a type) have their own ordering.

So you can explicitly say “I want profile A to be applied before profile B”

how do you state that?

Add a key like: :profile/a {:duct.core/requires #ig/ref :profile/b}

So a reference will make one profile dependent on another.

If you want a soft dependency - where you want A to depend on B, but only if B exists, then you can use a refset

:profile/a {:duct.core/requires #ig/refset :profile/b}

There’s nothing particularly special about :duct.core/requires, other than it’s cleaned out of the profile when it’s merged in.

I should probably make that key public, as right now it’s just an internal detail.

As an example, this ensures modules come after profiles:

(defmethod ig/prep-key :duct/module [_ profile]
  (assoc profile :duct.core/requires (ig/refset :duct/profile)))

ahh ok makes some sense — I saw that code, but didn’t quite know what it was doing. I’ll give it a try. Though I still question why profiles are modules… They don’t typically depend on each other in the same way as modules… i.e. each module may define subsets (potentially overlapping but not necesssarily) of the keys in the final system. And they’re not actually modules in the hierarchy (isa? :duct/profile :duct/module) ;; => false. Wouldn’t it be better and more direct to merge profiles in profile key order?

So when designing Duct there were three options. The current design, where a profile is a type of module. A design where modules and profiles are separate things, but at the same layer, or a design where modules are inside profiles, and normal configuration is inside modules.

The problem with placing modules and profiles at the same layer, if they’re not the same thing, is how to resolve references between the two.

ok makes sense… I was expecting profiles were design 2.

ok makes sense. I guess I don’t use modules very much, aside from the ring wrappers.

If another layer is added, then that makes things more verbose. And options 2 and 3 are also more complex, in that there are more things to consider.

However, I recognize that the first option is conceptually hard to get to grips with at first.

Yeah, I think I just expected because profiles were a sequence for that to determine the meta merge order.

They should probably be documented as a set, to make things clear.

it turns out we’ve been using duct for years with custom profiles, and getting the merge semantics we wanted by accident. Until now.

It isn’t the most intuitive thing, I agree. More documentation coupled with using a set instead of a vector would help.

It does allow for some pretty sophisticated dependency resolution though.

And I wanted to get away from explicit ordering in favour of using dependency order for keys.

Ok, this makes a lot more sense to me now, as I hadn’t thought about modules being inside profiles. Thanks again for the answers; and super sorry if my questions sounded critical. We’re huge fans of the work you do, particularly now in integrant and duct.

:man-bowing:

Criticism is good 🙂

I’m still not entirely sure if I’ve made the right decision. The current design trades ease for simplicity, perhaps more than any other library I’ve designed. In terms of moving parts it has very little there, but conceptually it’s not as intuitive as I’d like.

While we generally talk about making things simple rather than easy, making something easy is still a good design goal to have. Sometimes things can be both simple and easy.

But Duct’s modules and profiles currently have a learning curve to them. I need to clear some time to improve the documentation.

yeah we’ve spoken about this before… it’s a tricky balance. > The current design trades ease for simplicity, I thought it was more the other way around… in that duct/modules are trying to apply ease to the bare simplicity of integrant.

Maybe there’s a bit of that, too.

I suspect this is the root cause of your uneasiness about the design of duct, and likely the source of my issues with its design (compared to integrant)… i.e. when something is simple it’s far easier to know it’s correct, that it is self consistent, and achieves it’s goals; you can objectively talk about it. When it’s trying to be easy, well you’ve moved into more subjective territory.

Though I appreciate there is a simplicitly to duct… i.e. the meta-system is just an integrant system, profiles and modules are (almost) the same; etc… but I think perhaps the motivation for duct modules lies in trying to be easy rather than simple.

Yes - certainly the modules are something I’ve wondered about.

Like whether or not a set of defaults to be merged in would deal with most eventualities.

Yeah those are tough decisions

FWIW most of those defaults seem pretty sane; I don’t think we’ve tweaked that many

But they do allow for more involved defaults, and there’s a nice parallel we can draw between higher level functions, and higher level configurations, which is what modules are.

yeah they’re the macros of duct edn… but with global scope

One issue with the profiles relying on :duct.core/requires is that it’s optional; and that it seems easy to accidentally end up in implicit/accidental ordering territory simply by missing one. This is a little awkward if different environments the app is deployed in, have different profile chains, e.g. you have various levels of dev/prod override. You want to make sure you’ve specified a total ordering/precedence but there’s no guarantee you have, and you’re not accidentally just getting the right result.

You can always set the key via the prep-key step. Or enforce it though the pre-init-spec. Ideally you also shouldn’t be thinking in terms of total ordering, but what profiles make changes that are dependent on each other.

That said, that’s easier to do with modules than profiles.

> Ideally you also shouldn’t be thinking in terms of total ordering, but what profiles make changes that are dependent on each other. I don’t know, I think ideally I want to say: - in dev merge in this order [:duct.profile/base :project.profile/customer :duct.profile/dev :project.profile/customer-dev :project.profile/customer-dev-local] - in test merge in this order [ ,,, ] - in prod mege in this order [:duct.profile/base :project.profile/customer :project.profile/customer-prod :project.profile/customer-prod-local] as a somewhat simplified example of what I actually want. I don’t really want to think about dependencies between profiles, because any key/config could go in any profile depending on circumstance. So the dependencies are only really there to force the ordering.

Hm. I can see the reasoning.

Though I’m not sure of the solution 🙂

(defn apply-profiles [meta-sys profiles]
  (reduce (fn [system profile]
            (mm/meta-merge system (profile meta-sys))) {} profiles))

(defn build-system [uber-config project-key profiles]
  (let [common-modules (-> (duct/resource "modules.edn")
                                               (duct/read-config default-readers))]
               
    (-> (merge {:duct.profile/base (apply-profiles uber-config profiles)}
               common-modules)
       (project/assoc-project-injector project-key))))

then exec that with duct

which is admittedly a hack.

In order for profiles to be ordered, and the configuration still to make sense, it would need to be redesigned. Specifically, profiles would probably need to be separated from modules, which means either adding a new layer, or redesigning modules…

Or perhaps I could add something in where the profile ordering is used when the dependency order is ambiguous…

oh that might work

and be backwards compatible… maybe. :thinking_face:

I’m not sure if the latter is a smart backward compatible move, or a cheap workaround. Perhaps it depends on how good my marketing department is 😉

On second thoughts it won’t be backwards compatbile, if someone is getting an arbitrary ordering right now that is accidentally the right one.

It might be a good idea to expose an accidental ordering.

But I’d still like to think about it a little before committing to anything.

(Incidentally my hack above is essentially the extra layer of system you talk about.)

for sure, it’d be a pretty fundamental and scary change if it wasn’t well thought out.

I think I’m solid on the Integrant part, and solid on the profiles. I like how they’re keys, and I like the simplicity of having #duct/include.

I’m less sold on the modules part. I like the concept, but I’m not sure how they work in practice.

I’d have to say that matches my experience with duct/integrant. Though given that profiles are modules I’m not sure I’m sold on profiles… I’m sold on them with the semantics I expect 😉 (and thought we were getting).

Yes, that’s what I meant about the profiles. 🙂

I like having something that gives me good defaults. I recently started an application and being able to just dump in a :duct.module/sql felt very nice.

Yeah profiles I think are essentially just override chains that control the order of meta-merge

> I like having something that gives me good defaults. I definitely agree that it’s really nice to get good defaults… but I’ve often wondered if those defaults might not be better as just raw integrant config, in a jar… and then if you stray from the defaults, you either stomp the defaults via the meta-merge; or if it’s more fundamental just copy the config from the jar into your app and modify.

I admit I worry about over-using meta-merge.

when you have a hammer 🔨

Right 🙂

I think I’m less sold on the use of modules as data macros… though having said that we have a subset of elaborate/fiddly config which could really use something module like simplifying it for ease… so I totally get the desire for them.

An alternative to modules might be tagged readers — no idea what a design based on those might look like though. I guess it would lexically scope the module’s field of influence somewhat.

The introduction of prep-key in Integrant has lessened the need a little, as they’re effectively local macros.

I’ve definitely overused it in the past; though I think meta-merge is great for config when combined with profiles. I also think I prefer meta-merge to using something more granular like aero; aero tends to snowball tagged readers, and become much harder to reason about than ig + meta-merge.

Yeah prep-key is a nice addition; I’ve used it a little; though I am wary of it too as it can obscure dependencies… e.g. returning ig/refs in a prep-key can make things hard to debug; though it can also be super handy.

Yeah. I admit I’ve had a few times where I’ve decided to remove prep-key.

Like should I use it for adding the logger automatically, or do that explicitly. It’s a hard decision.

agreed… though I also avoid the duct logging stuff; simply because the raw JVM logging stuff is simpler with fewer layers of configuration. Sure it’s not pure, and doesn’t work with clojure data; but I’d rather get logging out of my 3rd party deps consistently and use one logger/config everywhere.

I think Timbre can intercept logging… that said, I take your point. I prefer to be pure where I can be.

Also logging from 3rd party deps is usually plaintext.

And ideally I want to be logging data.

Yeah don’t get me wrong, I’d love to do it ducts way; it’s more principled… but I found the practicalities were more important.

Understandable.

Thanks for the feedback, btw.

Pleasure. Thanks for listening 🙂

I think we have quite an interesting duct app; in that we’re multi-tenant (i.e. multiple client configurations) in one app; with corresponding dev/prod/test overrides etc, some localised to specific customers, others going across customers etc. So we’ve had to stray a little out of the duct mold/template a little to achieve it; but it’s holding together really well.

That’s good to hear.

we have a few nice tweaks… such as (reset :customer-name) that let you switch customers live at the repl; customers have different classpaths configured through tools.deps that are all merged together in a :prj/all alias for the repls classpaths; so essentially in dev all assets are on the classpath (which lets the branding etc swap when you change customers). Then some tests iterate over all configured customers and check things like their systems all start, and have a home page that 200s. So being data driven enables quite a lot; we have some macros to start systems in tests and they put in scope both the prepped config and the system, so your test can in principle do things like check every configured route returns a 200 for every customers app.

(different customers can have different subsets of routing configured)

Another weirdness/dimension to profiles as they currently are is that they can exist in hierarchies… so does the line (exec-config config [:duct.profile/prod]) mean start all profiles which isa? :duct.profile/prod? If that’s the case and you were to derive profiles from that key; how would you guarantee a merge ordering across that refset?

I’m guessing the answer is don’t do that 🙂

Yes, it will start all derived profiles. The ordering is always the same - dependency order followed by alphanumeric ordering of keys to tiebreak.

If I want configuration specified in the :duct.profile/local profile to merge over configuration specified in a :duct.profile/test profile, how do I tell Duct that? If local has configuration: {:foo {:a false}} and test: {:foo {:a true}} Then I want the resulting config to be: {:foo {:a false}} Which I think means that I need to have :local depend on :test, so I added: :duct.core/requires #ig/ref :duct.profile/test to my local.edn but the value is still true I am eventually calling:

(-> (duct/resource "config.edn")
    (duct/read-config)
    (duct/prep-config [:duct.profile/test :duct.profile/local]))