Clojurians Log v2

Hi, How can I use data collected by CLI args with duct config defined in edn file? With duct config as a data-structure is easy as I can inject the values into the map but I don't know how to do it if the config is sourced from edn.

@lambder You can either update your -main method to update the configuration before it's prepped, or - preferably - use environment variables instead of CLI args and take advantage of the #duct/env reader tag.

can I set env vars in jvm?

I'm thinking about writing my own data reader similar to #duct/env but sourcing info from java properties, which can be set during runtime

for the first advice, how do I update the configuration?

what value placeholders would you suggest in the edn?

@lambder Take a look at the generated -main function. The configuration is read in, then prepped and executed, but after it's read in and before it's prepped you can modify it as you want.

I don't know why you'd want to set variables during runtime. That seems a bad idea.

so I use org.clojure/tools.cli before I construct duct config

I want to some of the cli args to be injected to duct

it's too late to set the env vars

I could set the java props tho

Yes, if you really want to do that. Are you using 0.10 or 0.11-beta?

[duct/core "0.6.2"]

@lambder So your -main function should look like:

(defn -main [& args]
  (let [keys (or (duct/parse-keys args) [:duct/daemon])]
    (-> (duct/read-config (io/resource "bar/config.edn"))
        (duct/prep keys)
        (duct/exec keys))))

You can alter the configuration with whatever function you want after read-config

Though I'd advise sticking to environment variables.

the keys are the list of ig keys . correct?

Yep

what rather than selecting the keys I want to take a http port form cli?

I know I can use env var for this.

I'm using this as an illustration of the case only.

Okay

so, how can I inject http port form cli to duct?

It's up to you. You can pass it as a raw argument or you can define a flag to take it, like -p 3000.

sure

but how to pass it to duct conf?

The Duct configuration is just a map, returned by read-config

So you can use assoc-in as normal

Or whatever you want.

Ok, I've managed to implement what I wanted. I'm sharing here just in case you want to include it in the duct framework.

(defmacro local 
  "A string reader providing values of local bindings"
  []
  (let [lvals (vec (keys &env))
        lvars (mapv keyword (keys &env))]
    `(let [local-vars-map# (into {}
                     (map vector
                          ~lvars
                          ~lvals))]
       (fn [v#]
         ((keyword v#) local-vars-map#)))))

(let [a 123
      b "xyz"]
  (duct/read-config
    (java.io.StringReader. "{:foo  {:x #ig/val a}
                             :bar  {:x #ig/val b}}")
    {'ig/val (local)}))

instead of symbols the (local) string reader cold take keywords or both

e.g

(let [a 123
      b "xyz"]
  (duct/read-config
    (java.io.StringReader. "{:foo  {:x #ig/val :a}
                             :bar  {:x #ig/val :b}}")
    {'ig/val (local)}))

@weavejester I use [duct/core "0.6.2"] which seems to be latest. My print reader works with duct/read-config but if my config includes other one by :duct.core/include it looks like my reader is no longer being applied for the included edn. Any idea how to fix it?

@lambder The readers should be passed straight through to any includes. It might be something weird with the macro you're using, though it produces a closure at the end of it so it should be safe.

I think this is something you'll need to investigate yourself if you want to head down this route. If it turns out to be a bug in duct/core, send me an issue report and I'll put together a fix.

I'd also be interested in knowing why you're taking this approach to begin with 🙂

I take this approach to have the full duct configs in edn files. The injected dynamic values (established in runtime before the duct system is initialised) are used in the duct config on many levels (in the sense of nested maps).

What are the dynamic values for?

Transforming that would require post or pre walking the data structure and determining what should be repalaced.

the dynamic values are coming from the user

imagine the user runs it as a command line providing the dynamic values as args

I could convert the loaded duct conf as you suggested but is a but cumbersome

with my approach (local bindings) it is resolved automatically

I create my conf by:

(let [insert-record        (:insert-record opts)]
                              (duct/read-config
                                (io/resource duct-config ) {'local (u/local)}))

which works

if the duct-config is a string to my prod duct conf

if I point it to my dev one (dev includes prod)

the duct claims there is no #local reader

Okay, and the 'local isn't being transferred to files included with :duct.core/include?

correct

ah

one sec

I just killed my REPL

no

it still is not working

I've ended up to repeat all the config from prod in dev (except the difference)

Are you passing the readers to the prep function as well as read-config?

no, just to read-config

should I do with prep as well?

if yes how can I supply the readers and require prep for all the keys, not just selected ones?

@weavejester

prep is what loads the include files, so you need to supply the readers to that if you want the include files to have readers.

@lambder You should just need to supply your reader function to both the read-config and prep stage.

The next version of Duct makes this process simpler.

(defn prep
  "Prep a configuration, ready to be initiated. Key namespaces are loaded,
  resources included, and modules applied.

  Like `init` and other Integrant functions, `prep` can be limited to a subset
  of keys. The keys supplied indicate which keys will be later initiated, and
  therefore which keys to load. Modules are always loaded and applied.

  A map of options may also be supplied. Currently the only supported option is
  `:readers`, which should contain a map of data readers that will be used when
  reading configurations imported through `:duct.core/include`. "
  ([config]
   (prep config nil))
  ([config keys]
   (prep config keys {}))
  ([config keys {:keys [readers] :or {readers {}}}]
   (-> config
       (apply-includes (memoize #(read-config % readers)))
       (doto (ig/load-namespaces [:duct/module]))
       (apply-modules)
       (doto (as-> cfg (if keys
                         (ig/load-namespaces cfg keys)
                         (ig/load-namespaces cfg)))))))

if I call (prep conf :reders my-readers)

which arity will be called?

@weavejester

You mean: (prep conf {:readers my-readers})?

yes

that would be the 2-arity right?

Yes, because there are 2 arguments.

which is `([config keys] (prep config keys {}))`

that would break

Yeah, so just supply the keys, or nil if you want the default.

Though -main supplies [:duct/daemon] as its default, mind.

ok, that was what I meant how do I require all the keys

many thanks @weavejester

I believe just supply nil, though usually you just want [:duct/daemon] if you're running from -main, and nil if you're running from the REPL.

I run it a bit diferently

I construct a duct system

and then I have my own main driver which picks different duct components from the system

but that is a different story 😉

It sounds like you're doing some sort of customer/user-specific sharding.

@weavejester thank to your advices everything works perfect now. Thanks!

@lambder excellent news