Clojurians Log v2

Sorry for the confusion. This is not my module, I'm just using it to learn duct + datomic integration. Although the main config.edn has these variable being pulled from enviroment variables for release, my local dev setup has them defined in dev/resources/local.edn file. But for some reason, they are not being merged with the config.edn file and so during module initialisation, the module get a config with nil values for everything except for :server-type.

Can you share your full local.edn and config.edn files?

Sure...here's my config.edn, local.edn and output after running (prep) command. Thank you for helping me out 🙂

@dionysius.almeida Ok, i think I know what your problem is. If you have a look at https://github.com/duct-framework/duct/wiki/Configuration#duct-base-profiles-and-modules you'll see that there are three parts to prep the configuration. Duct profiles (like :duct.profile/local, the one that deals with local.edn) are merged in the base config. And the above link states that "The base config is a map of data inside of the :duct.profile/base key". So Duct reads config.edn, takes the data from the :duct.profile/base key and creates the base config. Next it takes all the profiles keys from config.edn and merges the associated data with the base config. That's why you see there is a :duct.module/datomic key in your config var (the one you show in the repl-output.txt file). And it has the exact same values that you configured in local.edn. Finally it takes all the modules keys from config.edn and runs the configuration through them. But the module key that you are trying to "inject" from local.edn is not in config.edn! It's only in the configuration merged so far, but not in config.edn. So Duct doesn't know about it and doesn't try to run the configuration through it. The only Datomic module key it knows about it the one in config.edn. And that one uses the settings from environment variables. And as you said, they are not defined, so they end up being nil. One (simplified, but still mostly true) way to see it is that Duct modules need to live outside the :duct.profile/base and :duct.profile/* keys for them to be processed. In this particular case I don't see and easy way around it, as the Datomic module expects the configuration settings to be passed in as the configuration map for its Integrant key. And there is no way (that I know of) to specify the configuration settings in the base profile (so we can override them from dev.edn or local.edn) and ig/ref to their key from the Datomic module Integrant key. But if you modified the Datomic module to look for the configuration settings in the config map instead of passing them as the options argument to the init-key multimethod, you could achieve what you want. If you wanted to make the modification a bit more generic, instead of hardcoding the key to look for in the config map, you could pass that as a configuration option to the init-key multimethod. Something along these lines:

(ns duct.module.datomic
  (:require [duct.core :as core]
            [integrant.core :as ig]))

(defn- get-environment [config options]
  (:environment options (:duct.core/environment config :production)))

(def ^:private logger-configs
  {:production
   {:duct.logger.timbre/cast {}
    :duct.logger/timbre
    ^:demote {:appenders ^:displace {:duct.logger.timbre/cast (ig/ref :duct.logger.timbre/cast)}}}})

(defmethod ig/init-key :duct.module/datomic [_ options]
  (fn [config]
    (core/merge-configs
     config
     {:duct.database/datomic (ig/ref (:settings-key options))}
     {:duct.migrator.ragtime/datomic
      ^:demote {:database   (ig/ref :duct.database/datomic)
                :logger     (ig/ref :duct/logger)
                :migrations []}}
     (logger-configs (get-environment config options)))))

{:duct.profile/base

 {:duct.core/project-ns duct-redis-example

  :duct.handler/root
  {:router #ig/ref :duct.router/reitit
   ;; use this for applying middleware to entire root handler
   ;; :middleware [#ig/ref :duct-redis-example.handler.middleware/wrap-admin]
   }
  :duct.router/reitit
  {:routes
   ["/"
    ["example" {:handler #ig/ref :duct-redis-example.handler/example}]
    ["datomic-example" {:handler #ig/ref :duct-redis-example.handler/datomic-example}]
    ["admin" {:handler #ig/ref :duct-redis-example.handler/example
              :middleware [:wrap-admin ]}]
    ]
   :reitit.ring/opts
   {:reitit.middleware/registry
    {:wrap-admin #ig/ref :duct-redis-example.handler.middleware/wrap-admin}}
   }
  :duct-redis-example.handler.middleware/wrap-admin {}
  :duct-redis-example.handler/example {:db #ig/ref :duct.database.redis/carmine}
  :duct-redis-example.handler/datomic-example {:db #ig/ref :duct.database/datomic }

  :duct.server.http/jetty {:port #duct/env ["SERVER_PORT" Int]}

  :duct.database.redis/carmine
  {:spec
   {:host #duct/env "REDIS_HOST",
    :port #duct/env ["REDIS_PORT" Int]}
   }

  [:duct/const :datomic/settings]
  {
   :server-type :peer-server,
   :secret #duct/env "DATOMIC_DB_PASS",
   :access-key #duct/env "DATOMIC_DB_ACCESS_KEY",
   :endpoint #duct/env "DATOMIC_HOST",
   :validate-hostnames false
   :database #duct/env "DATOMIC_DB_NAME"
   }
 }

 :duct.profile/dev   #duct/include "dev"
 :duct.profile/local #duct/include "local"
 :duct.profile/prod  {}

 :duct.module/logging {}
 :duct.module.web/api {}
 :duct.module/datomic {:settings-key :datomic/settings}
 }

{[:duct/const :datomic/settings]
 {
  :server-type :client-server-local-value,
  :secret :secret-local-value
  :access-key :access-key-local-value
  :endpoint :endpoint-local-value
  :validate-hostnames :validate-hostnames-local-value
  :database :database-local-value
  }
 }

Thank you for the detailed explanation. I did figure out that the way modules are initialised is different from other keys, but this explanation now help me understand why. Since, this is not my module, I don't think i would be able to change it. I also found a way to deal this with....instead of adding my configuration in :duct.module/datomic, I define the configuration in :duct.database/datomic key in base profile and then override it in local.edn file using this same key. But your solution is something I will keep in mind as I'm trying to write a module to use datomic pro which uses a URI string, instead of a map to connect to datomic transactor and not peer server. This module uses client API library and not peer library api, so i might put toghether something which works for peer library. Thank you again for your time and have a nice weekend 🙂