Clojurians Log v2

How can I set a dev-only JDBC database url when using the duct sql module? My dev.edn looks as follows:

{:duct.database.sql/hikaricp
 {:jdbc-database-url "jdbc:<postgresql://postgres:postgres@localhost:5432/postgres%22|postgresql://postgres:postgres@localhost:5432/postgres">}}

:duct.module/sql {}

Execution error (IllegalArgumentException) at clojure.java.jdbc/get-connection (jdbc.clj:292).
db-spec null is missing a required parameter

So it seems that only the base profile key is loaded, hence the dev profile is not being expanded at all.

I don't know how to resolve this. I followed the documentation steps to the letter but it does not work for me. Are the docs out of date?

(doto (read-config) ig/load-namespaces)
=>
#:duct.profile{:base {:duct.profile/prod {},
                      :duct.profile/dev {:duct.core/environment :development,
                                         :duct.database.sql/hikaricp {:jdbc-database-url "jdbc:<postgresql://postgres:postgres@localhost:5432/postgres%22|postgresql://postgres:postgres@localhost:5432/postgres">}},
                      :duct.profile/local {},
                      :duct.module/logging {},
                      :duct.module/sql {},
                      :duct.module.web/api {},
                      :duct.core/project-ns person-ld-api}}

@sgerguri I’m not sure if it helps you but I was able to fix the connection issue by writing dev.edn like : `{:duct.database/sql {:connection-uri “jdbc:<postgresql://localhost/db_dev>” :username “postgres_user” :password “db_pw”} } `

But if I try to build it, I get basically the same thing back:

(duct/build-config (doto (read-config) ig/load-namespaces) [:duct.profile/dev])
=>
{:duct.profile/prod {},
 :duct.profile/dev {:duct.core/environment :development,
                    :duct.database.sql/hikaricp {:jdbc-database-url "jdbc:<postgresql://postgres:postgres@localhost:5432/postgres%22|postgresql://postgres:postgres@localhost:5432/postgres">}},
 :duct.profile/local {},
 :duct.module/logging {},
 :duct.module/sql {},
 :duct.module.web/api {},
 :duct.core/project-ns person-ld-api}

My base config.edn looks as follows:

{:duct.profile/base
 {:duct.core/project-ns person-ld-api

  :duct.profile/dev     #duct/include "dev"
  :duct.profile/local   #duct/include "local"
  :duct.profile/prod    {}

  :duct.module/logging  {}
  :duct.module.web/api  {}
  :duct.module/sql      {}
  }
 }

A measure of progress - this seems to work just fine on another duct project that I just created. The difference between these two repos is probably just a week and a half, tops. I checked the configs, versions of dependencies etc. and I don't see any effective difference, yet the earlier repo simply fails to properly initialise the configuration, while the latter one succeeds. Any hints on what I could be missing?

:jdbc-database-url => :jdbc-url?

Right, something that is not showing in this snippet is that I am using migrations, specifically:

:duct.migrator/ragtime
 {:migrations [#ig/ref :person-ld-api/create-database
               #ig/ref :person-ld-api/create-table]}

 [:duct.migrator.ragtime/sql :person-ld-api/create-database]
 #duct/include "migrations/001-create-db.edn"

 [:duct.migrator.ragtime/sql :person-ld-api/create-table]
 #duct/include "migrations/002-create-table.edn"

https://github.com/duct-framework/database.sql.hikaricp/blob/master/src/duct/database/sql/hikaricp.clj#L41

I guess I am using #duct/include incorrectly - I want to simply refer to another EDN file and have it be included in here.

True, @jahson - I have since fixed that, however, and the migrations are what break it.

It seems that if one uses duct with ragtime one is effectively forced out of the edn migration files - either everything ends up being included in the config.edn file, or one has to make sql up/down files and refer to those. Neither feels like a particularly great solution.

:duct.migrator/ragtime
 {:database #ig/ref :xxx.database/master
  :migrations-table "xxx_migrations"
  :migrations [#ig/ref :xxx.migration/create-table1]}

[:duct.migrator.ragtime/sql :xxx.migration/create-table1]
{:up ["CREATE TABLE table1;
 :down ["DROP TABLE table1"]}

The files I am referring to via the #duct/include have basically

{:up ["CREATE TABLE table1;"]
 :down ["DROP TABLE table1;"]}

I just thought I could make config.edn more concise by simply referring to these through that include.

I think there is nothing wrong with #duct/include

What is happening? Exceptions?

(prep)
Execution error (IllegalArgumentException) at clojure.java.jdbc/get-connection (jdbc.clj:292).
db-spec null is missing a required parameter

*e?

ExceptionInfo Error on key :duct.migrator/ragtime when building system 
Caused by:
IllegalArgumentException db-spec null is missing a required parameter
	clojure.java.jdbc/get-connection (jdbc.clj:292)
	clojure.java.jdbc/get-connection (jdbc.clj:176)
	ragtime.jdbc/get-table-metadata (jdbc.clj:27)
	ragtime.jdbc/get-table-metadata (jdbc.clj:24)
	ragtime.jdbc/table-exists? (jdbc.clj:38)
	ragtime.jdbc/table-exists? (jdbc.clj:36)
	ragtime.jdbc/ensure-migrations-table-exists (jdbc.clj:41)
	ragtime.jdbc/ensure-migrations-table-exists (jdbc.clj:40)
	ragtime.jdbc.SqlDatabase (jdbc.clj:61)
	ragtime.core/migrate-all (core.clj:57)
	ragtime.core/migrate-all (core.clj:33)
	duct.migrator.ragtime/migrate (ragtime.clj:76)

Interesting. I thik the issue is here https://github.com/duct-framework/migrator.ragtime/blob/master/src/duct/migrator/ragtime.clj#L63 It is happening because you should explicitly set :database I think.

{:duct.migrator/ragtime
 {:database   #ig/ref :duct.database/sql
  :logger     #ig/ref :duct/logger
  :strategy   :rebase
  :migrations [#ig/ref :foo.migration/create-foo-table]}}

I'm not sure that's it - I have database settings in my dev.edn.

I am using the sql duct module, see https://github.com/duct-framework/module.sql/blob/master/src/duct/module/sql.clj#L23.

My dev.edn looks as follows:

{:duct.database/sql
 {:connection-uri "jdbc:<postgresql://localhost:5432/postgres>"
  :username "PLACEHOLDER"
  :password "PLACEHOLDER"}}

You should provide :database for :duct.migrator/ragtime

Yes - and I am through the dev profile.

That section is demoted in the module, so it gets overriden if there is a value for it.

Can you try to use an explicit :database to check if it will work?

I can - will update here later after I've tried.

Sorry if I sound a little bit rude — I not a native nor fluent english speaker.

Not at all - and I hope I did not come across that way either. I am just trying it out.

Hi, can I help in any way?

Ah, the man himself!

Yes please, let me summarise.

Basically, I set up a duct project with DB and routes, and was struggling to get the DB layer set up.

I set the project up again and am seeing success in instantiating hikari but the moment I specify some DB migrations it all blows up.

Could you post your config file, and the exception that occurs?

Contents of config.edn:

{:duct.profile/base
 {:duct.core/project-ns ld-api

  :duct.router/cascading
  []}

 :duct.profile/dev   #duct/include "dev"
 :duct.profile/local #duct/include "local"
 :duct.profile/prod  {}

 :duct.module/logging {}
 :duct.module.web/api {}
 :duct.module/sql {}

 ;; == DB migrations ==

 :duct.migrator/ragtime
 {:migrations [#ig/ref :ld-api/create-database
               #ig/ref :ld-api/create-table]}

 [:duct.migrator.ragtime/sql :ld-api/create-database]
 #duct/include "migrations/001-create-db.edn"

 [:duct.migrator.ragtime/sql :ld-api/create-table]
 #duct/include "migrations/002-create-table.edn"
 }

Contents of dev.edn:

{:duct.database/sql
 {:jdbc-url "jdbc:<postgresql://localhost:5432/postgres>"
  :username "PLACEHOLDER"
  :password "PLACEHOLDER"}}

Error:

(prep)
Execution error (IllegalArgumentException) at clojure.java.jdbc/get-connection (jdbc.clj:292).
db-spec null is missing a required parameter

The above happens when calling (prep) after calling (dev).

Hm. What happens if you change :duct.database/sql to :duct.database.sql/hikaricp in your dev.edn?

The same.

Well at least that's consistent 🙂

Oh, I see a problem.

True, though that provides little comfort to me. 😄

Your migrator keys are outside the base profile.

You should dump them under :duct.profile/base, or create a new profile for them.

🤦 :picard-facepalm:

It's an easy mistake to make, because in older versions that would have worked. But there were too many issues with putting modules and non-modules in the same configuration.

So non-module (component) keys need to be under a profile.

Which is just a module that merges its contents into the configuration.

Unfortunately, now the config isn't being expanded properly.

(pprint config)
{[:duct.migrator.ragtime/sql :ld-api/create-person-table] {:up ["CREATE TABLE person (...);"],
                                                           :down ["DROP TABLE person;"]},
 :duct.profile/prod {:duct.core/requires #ig/refset :duct.profile/base,
                     :duct.core/environment :production},
 :duct.profile/dev {:duct.database.sql/hikaricp {:jdbc-url "jdbc:<postgresql://localhost:5432/postgres>",
                                                 :username "PLACEHOLDER",
                                                 :password "PLACEHOLDER"},
                    :duct.core/requires #ig/refset :duct.profile/base,
                    :duct.core/environment :development},
 :duct.migrator/ragtime {:migrations [#ig/ref :ld-api/create-database
                                      #ig/ref :ld-api/create-person-table]},
 :duct.router/cascading [],
 :duct.profile/local {:duct.core/requires #ig/refset :duct.profile/base},
 :duct.module/logging {:duct.core/requires #ig/refset :duct/profile},
 :duct.module/sql {:duct.core/requires #ig/refset :duct/profile},
 [:duct.migrator.ragtime/sql :ld-api/create-database] {:up ["CREATE DATABASE ld;"],
                                                       :down ["DROP DATABASE ld;"]},
 :duct.module.web/api {:duct.core/requires #ig/refset :duct/profile},
 :duct.core/project-ns ld-api}

That looks pretty mixed up... sorry, can you post your config.edn again?

{:duct.profile/base
 {:duct.core/project-ns ld-api

  :duct.router/cascading
                        []

  :duct.profile/dev     #duct/include "dev"
  :duct.profile/local   #duct/include "local"
  :duct.profile/prod    {}

  :duct.module/logging  {}
  :duct.module.web/api  {}
  :duct.module/sql      {}

  ;; == DB migrations ==

  :duct.migrator/ragtime
                        {:migrations [#ig/ref :ld-api/create-database
                                      #ig/ref :ld-api/create-person-table]}

  [:duct.migrator.ragtime/sql :ld-api/create-database]
                        #duct/include "migrations/001-create-db.edn"

  [:duct.migrator.ragtime/sql :ld-api/create-person-table]
                        #duct/include "migrations/002-create-person-table.edn"}
 }

Right. So you need to put component keys in profiles, module keys outside.

So:

{:duct.profile/base
 {:duct.core/project-ns ld-api
  :duct.router/cascading []

  :duct.migrator/ragtime
  {:migrations [#ig/ref :ld-api/create-database
                #ig/ref :ld-api/create-person-table]}

  [:duct.migrator.ragtime/sql :ld-api/create-database]
  #duct/include "migrations/001-create-db.edn"

  [:duct.migrator.ragtime/sql :ld-api/create-person-table]
  #duct/include "migrations/002-create-person-table.edn"}

 :duct.profile/dev     #duct/include "dev"
 :duct.profile/local   #duct/include "local"
 :duct.profile/prod    {}

 :duct.module/logging  {}
 :duct.module.web/api  {}
 :duct.module/sql      {}}

Ah, should've paid closer attention to https://github.com/duct-framework/docs/blob/master/GUIDE.rst. I went by what the READMEs and Wiki of duct framework and modules said, and mashed it up together based on that.

I'm working on some better docs that should make things clearer

Happy to help out once I've wrapped my head around stuff comfortably. So I have a question.

Sure

What if I want to override some component-level stuff from a non-base profile?

Technically, the module keys are still part of the base profile.

So does that mean that components can go into any profile (nested in base) and it will all work, right?

Can you give an example of what you mean?

For example, could my dev.edn file look like the following?

{:duct.database.sql/hikaricp
 {:jdbc-url "jdbc:<postgresql://localhost:5432/postgres>"
  :username "postgres"
  :password "postgres"}

:duct.migrator/ragtime
{:migrations [...]}

...
}

Yes, that's right.

Perfect, that makes sense now.

The only things that currently can't be put in profiles are modules.

Thanks a lot, James. I might try my hand at a few PRs that hopefully clear up how things should be used now that I have a better understanding of it.

No problem. Let me know if you have any more questions.