diff --git a/docs/how-to/configure-the-database.md b/docs/how-to/configure-the-database.md index f2d8d1cf..0f206a8d 100644 --- a/docs/how-to/configure-the-database.md +++ b/docs/how-to/configure-the-database.md @@ -46,8 +46,30 @@ The pool starts empty and grows to the size as connections are needed. ## One database, several applications Each application can keep its tables in a Postgres schema of its own, so two applications can -share the dev database without their drift bleeding into each other. Two overrides on `Main`, -told once each: +share the dev database without their drift bleeding into each other. Name the schema on `Main`: + +```scala +object Main extends EezoApp { + + override def databaseSchema: String = "bookshelf" + // ... +} +``` + +That one override is enough. Every connection the pool opens creates the schema when it is +missing and sets `search_path` to it, because `search_path` is per connection, so the tables +`sync` creates, the migration ledger and your own queries all land in `bookshelf`. The schema +commands read the catalog under the same name, so `status` compares your model against the right +tables. The examples under `examples/` each do this. + +The schema is created only when the catalog says it is missing, so a production role that owns an +existing schema but may not create one on the database still connects. `search_path` is the schema +alone: objects in `public`, such as an extension's types and functions, are not on it. An +application that needs those too sets the path itself in `databaseInit`, which runs after the +schema step: `set search_path to bookshelf, public`. + +`databaseInit` is for any other per connection setting, such as `application_name` or a statement +timeout. It runs after the schema step on every pooled connection and needs no `super` call: ```scala import java.sql.Connection @@ -58,20 +80,13 @@ object Main extends EezoApp { override def databaseInit: Connection => Unit = { c => val st = c.createStatement() - try { - st.execute("""create schema if not exists "bookshelf"""") - st.execute("""set search_path to "bookshelf"""") - } finally st.close() + try st.execute("set application_name to 'bookshelf'") + finally st.close() } // ... } ``` -`databaseInit` runs on every pooled connection, because `search_path` is per connection. -`databaseSchema` points the schema commands' catalog reads at the same name, so `status` compares -your model against the right tables. Both halves are needed; the examples under `examples/` each -do this. - ## In tests Point a second `DbApp` at a throwaway Postgres, such as one from testcontainers, by overriding diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index 1441b099..95333327 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -49,19 +49,22 @@ overridden `boot` that still wants to serve calls it. |---|---|---| | `schema: Schema` | abstract | per command | | `boot(): Unit` | abstract on `DbApp`; `EezoApp` supplies the http layer's | the program | -| `databaseSchema: String` | `"public"` | by the commands that introspect | +| `databaseSchema: String` | `"public"` | on every physical connection the pool opens, and by the commands that introspect | | `databaseUrl: String` | from the environment, above | when a `Database` is built | | `databaseUser: String` | from the environment | same | | `databasePassword: String` | from the environment | same | | `databasePoolSize: Int` | from the environment, 10 | same | | `databaseAcquireTimeout: Duration` | from the environment, 5 s | same | -| `databaseInit: Connection -> Unit` | nothing | on every physical connection the pool opens | +| `databaseInit: Connection -> Unit` | nothing | on every physical connection the pool opens, after the schema step | A `Database` is built per command and per program run, installed around it, and closed after. -`databaseInit` is where `search_path`, `application_name` and statement timeouts go. An -application that keeps its tables in a schema other than `public` sets both `databaseSchema` -and a `databaseInit` that sets `search_path`, because the first is what the drift commands -read and the second is what every query runs under. +`databaseSchema` is where the application's tables live and what the drift commands read. Any +name but `public` makes every physical connection create that schema when the catalog says it is +missing and set `search_path` to it alone, so the DDL, the migration ledger and every query land +where `status` looks; `public` runs nothing and leaves Postgres's own `search_path`. The name is +quoted as an identifier, so it is used exactly as written. `databaseInit` is where +`application_name`, statement timeouts and any other per connection setting go; it runs after the +schema step and needs no `super` call to keep it. ## `LiveApp` diff --git a/examples/blog/README.md b/examples/blog/README.md index 4f25db31..77f5815c 100644 --- a/examples/blog/README.md +++ b/examples/blog/README.md @@ -18,8 +18,8 @@ where they came from and wraps only the derived ones in `Route.under("/admin")`, routes and the URLs the pages emit together; `app/Index.scala` explains the one link that has to name `/admin` by hand. -The blog keeps its table in a Postgres schema named `blog` (`databaseSchema` plus the `databaseInit` -hook), so it shares the dev database with the other examples. +The blog keeps its table in a Postgres schema named `blog` (`databaseSchema` in `Main.scala`, the +one override that does it), so it shares the dev database with the other examples. ```bash docker run -d --name eezo-pg -p 5442:5432 \ diff --git a/examples/blog/src/main/scala/CreateUser.scala b/examples/blog/src/main/scala/CreateUser.scala index 12d33052..b6dedaa1 100644 --- a/examples/blog/src/main/scala/CreateUser.scala +++ b/examples/blog/src/main/scala/CreateUser.scala @@ -1,5 +1,3 @@ -import java.sql.Connection - import io.eezo.auth.Password import io.eezo.core.Id import io.eezo.db.* @@ -34,14 +32,6 @@ object CreateUser extends DbApp { override def databaseSchema: String = "blog" - override def databaseInit: Connection => Unit = { c => - val st = c.createStatement() - try { - st.execute("""create schema if not exists "blog"""") - st.execute("""set search_path to "blog"""") - } finally st.close() - } - override def boot(): Unit = { val email = required("EEZO_BLOG_EMAIL") val password = required(PasswordVariable) diff --git a/examples/blog/src/main/scala/Main.scala b/examples/blog/src/main/scala/Main.scala index 2b90bcc3..9ab3e09e 100644 --- a/examples/blog/src/main/scala/Main.scala +++ b/examples/blog/src/main/scala/Main.scala @@ -1,5 +1,3 @@ -import java.sql.Connection - import io.eezo.EezoApp import io.eezo.db.Schema import io.eezo.generated.Routes @@ -45,20 +43,12 @@ object Main extends EezoApp { override def schema: Schema = AppSchema /** This app keeps its table in a Postgres schema of its own, so it can share the dev database - * with the other examples without their drift bleeding into each other. Two halves, told once - * each: the init hook runs on **every** pooled connection (search_path is per connection), and - * `databaseSchema` points the drift commands' catalog reads at the same name. + * with the other examples without their drift bleeding into each other. Naming it is enough: + * eezo creates the schema and sets search_path to it on **every** pooled connection, because + * search_path is per connection, and the drift commands read the same name. */ override def databaseSchema: String = "blog" - override def databaseInit: Connection => Unit = { c => - val st = c.createStatement() - try { - st.execute("""create schema if not exists "blog"""") - st.execute("""set search_path to "blog"""") - } finally st.close() - } - override def routes: RouteTable = { val table = Routes.table() val (derived, handwritten) = table.routes.partition(_.provenance == Provenance.Derived) diff --git a/examples/blog/src/test/scala/PostOwnershipSuite.scala b/examples/blog/src/test/scala/PostOwnershipSuite.scala index e5d390f1..937da796 100644 --- a/examples/blog/src/test/scala/PostOwnershipSuite.scala +++ b/examples/blog/src/test/scala/PostOwnershipSuite.scala @@ -1,6 +1,5 @@ import java.net.URLEncoder import java.nio.charset.StandardCharsets -import java.sql.Connection import java.util.UUID import io.eezo.core.Id @@ -94,17 +93,16 @@ class PostOwnershipSuite extends munit.FunSuite { * * A second `DbApp` for the same reason `CreateUser` is one: `withDatabase` is `protected`, so * installing a `Database` is something only a subclass can ask for, and a test is a front end - * like any other. The schema, the per connection hook and `AppSchema` are `Main`'s own values - * rather than copies, so a change to where the blog keeps its tables moves this with it. + * like any other. The schema and `AppSchema` are `Main`'s own values rather than copies, so a + * change to where the blog keeps its tables moves this with it. */ private object BlogDb extends DbApp { - override def schema: Schema = Main.schema - override def databaseUrl: String = postgres.getJdbcUrl - override def databaseUser: String = postgres.getUsername - override def databasePassword: String = postgres.getPassword - override def databaseSchema: String = Main.databaseSchema - override def databaseInit: Connection => Unit = Main.databaseInit - override def boot(): Unit = () + override def schema: Schema = Main.schema + override def databaseUrl: String = postgres.getJdbcUrl + override def databaseUser: String = postgres.getUsername + override def databasePassword: String = postgres.getPassword + override def databaseSchema: String = Main.databaseSchema + override def boot(): Unit = () /** The one thing this object exists to publish: a `Database` installed around a block, which is * what `sbt run` does around `boot` and what every handler below reaches through. diff --git a/examples/reminders/README.md b/examples/reminders/README.md index 2f606fb6..d5f79f3a 100644 --- a/examples/reminders/README.md +++ b/examples/reminders/README.md @@ -12,9 +12,9 @@ deliver every reminder that is due and not yet sent, and mark it sent, in one `t is a line on stdout. `DbApp` brings `main`, the `Database` it installs around `boot`, and the eight schema commands. There is no `EezoPlugin` on this project, because there are no routes to generate. -The job keeps its table in a Postgres schema named `reminders` (`databaseSchema` plus the -`databaseInit` hook; `Main.scala` says why both halves exist), so it shares the dev database with -the other examples without their drift bleeding into each other. +The job keeps its table in a Postgres schema named `reminders` (`databaseSchema` in `Main.scala`, +the one override that does it), so it shares the dev database with the other examples without +their drift bleeding into each other. ```bash docker run -d --name eezo-pg -p 5442:5432 \ diff --git a/examples/reminders/src/main/scala/Main.scala b/examples/reminders/src/main/scala/Main.scala index 027d450b..6c080ed9 100644 --- a/examples/reminders/src/main/scala/Main.scala +++ b/examples/reminders/src/main/scala/Main.scala @@ -1,4 +1,3 @@ -import java.sql.Connection import java.time.LocalDate import io.eezo.db.* @@ -25,20 +24,12 @@ object Main extends DbApp { override def schema: Schema = AppSchema /** This app keeps its table in a Postgres schema of its own, so it can share the dev database - * with the other examples without their drift bleeding into each other. Two halves, told once - * each: the init hook runs on **every** pooled connection (search_path is per connection), and - * `databaseSchema` points the drift commands' catalog reads at the same name. + * with the other examples without their drift bleeding into each other. Naming it is enough: + * eezo creates the schema and sets search_path to it on **every** pooled connection, because + * search_path is per connection, and the drift commands read the same name. */ override def databaseSchema: String = "reminders" - override def databaseInit: Connection => Unit = { c => - val st = c.createStatement() - try { - st.execute("""create schema if not exists "reminders"""") - st.execute("""set search_path to "reminders"""") - } finally st.close() - } - override def boot(): Unit = { val reminders = Table[Reminder] val today = LocalDate.now() diff --git a/examples/shop/src/main/scala/CreateUser.scala b/examples/shop/src/main/scala/CreateUser.scala index 5921f263..b33a329b 100644 --- a/examples/shop/src/main/scala/CreateUser.scala +++ b/examples/shop/src/main/scala/CreateUser.scala @@ -1,5 +1,3 @@ -import java.sql.Connection - import io.eezo.auth.Password import io.eezo.core.Id import io.eezo.db.* @@ -15,14 +13,6 @@ object CreateUser extends DbApp { /** Same Postgres schema as `Main`, so the account lands where the app reads it. */ override def databaseSchema: String = "shop" - override def databaseInit: Connection => Unit = { c => - val st = c.createStatement() - try { - st.execute("""create schema if not exists "shop"""") - st.execute("""set search_path to "shop"""") - } finally st.close() - } - override def boot(): Unit = { val email = sys.env("SHOP_EMAIL") val hashed = Field[Password] diff --git a/examples/shop/src/main/scala/Main.scala b/examples/shop/src/main/scala/Main.scala index 301561c9..88055c5c 100644 --- a/examples/shop/src/main/scala/Main.scala +++ b/examples/shop/src/main/scala/Main.scala @@ -1,5 +1,3 @@ -import java.sql.Connection - import io.eezo.EezoApp import io.eezo.db.Schema import io.eezo.generated.Routes @@ -12,17 +10,9 @@ object Main extends EezoApp { override def layout: Layout = views.Layout /** The shop keeps its tables in a Postgres schema of its own, so it shares the dev database with - * the other examples without their drift bleeding into each other. The init hook runs on every - * pooled connection, because search_path is per connection, and `databaseSchema` points the - * drift commands at the same name. + * the other examples without their drift bleeding into each other. Naming it is enough: eezo + * creates the schema and sets search_path to it on every pooled connection, and the drift + * commands read the same name. */ override def databaseSchema: String = "shop" - - override def databaseInit: Connection => Unit = { c => - val st = c.createStatement() - try { - st.execute("""create schema if not exists "shop"""") - st.execute("""set search_path to "shop"""") - } finally st.close() - } } diff --git a/examples/shop/src/test/scala/ApiSuite.scala b/examples/shop/src/test/scala/ApiSuite.scala index 1129f3be..1399fb18 100644 --- a/examples/shop/src/test/scala/ApiSuite.scala +++ b/examples/shop/src/test/scala/ApiSuite.scala @@ -1,6 +1,5 @@ import java.net.URLEncoder import java.nio.charset.StandardCharsets -import java.sql.Connection import io.eezo.auth.{Password, Token} import io.eezo.core.Id @@ -49,18 +48,16 @@ class ApiSuite extends munit.FunSuite { started } - /** The shop's own database settings pointed at the container. `Main`'s schema name and per - * connection hook rather than copies, so the routes read the `shop` search path they read in - * production. + /** The shop's own database settings pointed at the container. `Main`'s schema name rather than a + * copy, so the routes read the `shop` search path they read in production. */ private object ShopDb extends DbApp { - override def schema: Schema = Main.schema - override def databaseUrl: String = postgres.getJdbcUrl - override def databaseUser: String = postgres.getUsername - override def databasePassword: String = postgres.getPassword - override def databaseSchema: String = Main.databaseSchema - override def databaseInit: Connection => Unit = Main.databaseInit - override def boot(): Unit = () + override def schema: Schema = Main.schema + override def databaseUrl: String = postgres.getJdbcUrl + override def databaseUser: String = postgres.getUsername + override def databasePassword: String = postgres.getPassword + override def databaseSchema: String = Main.databaseSchema + override def boot(): Unit = () def around[A](body: => A): A = withDatabase(body) } diff --git a/examples/todo/README.md b/examples/todo/README.md index 38048528..689570d3 100644 --- a/examples/todo/README.md +++ b/examples/todo/README.md @@ -1,4 +1,4 @@ -# The todo app — a guided tour of the eezo CLI +# The todo app: a guided tour of the eezo CLI | | | |---|---| @@ -11,7 +11,7 @@ routes, and a Postgres schema of its own. Everything the CLI currently does can this directory. Every output block below was captured from a real run. The Store↔Table integration landed (`JdbcStore`). A model -deriving `Table` — like `Todo` — now gets derived CRUD backed by Postgres, so todos you create in +deriving `Table`, like `Todo`, now gets derived CRUD backed by Postgres, so todos you create in the browser survive restarts, and the migrate-before-serve order in this guide is load-bearing: browse `/todos` before the table exists and you'll meet a real database error. @@ -38,24 +38,24 @@ sbt The app serves on **8090** (`Main.scala` overrides `port` so the tour never collides with whatever your machine runs on 8080) and keeps its tables in a Postgres schema named `todo` -(`databaseSchema` + the `databaseInit` hook — see `Main.scala` for why both halves exist). +(`databaseSchema` in `Main.scala`, the one override that does it). ## 1. The code -- `src/main/scala/models/Todo.scala` — the model. `derives Table, Form, Resource` is the whole +- `src/main/scala/models/Todo.scala`: the model. `derives Table, Form, Resource` is the whole declaration: table, HTML form, seven CRUD routes. -- `src/main/scala/AppSchema.scala` — registers the table and adds what types alone cannot say: +- `src/main/scala/AppSchema.scala`: registers the table and adds what types alone cannot say: `table[Todo].index(_.done).unique(_.title)`. The selectors are typed; rename a field and the index breaks at compile time. -- `src/main/scala/app/Index.scala` — `GET /` by filename convention. A handler is a plain +- `src/main/scala/app/Index.scala`: `GET /` by filename convention. A handler is a plain `Request => Response` on a virtual thread. -- `src/main/scala/app/Health.scala` — a custom name mounts a GET at its own segment: +- `src/main/scala/app/Health.scala`: a custom name mounts a GET at its own segment: `GET /health`. -- `src/main/scala/Main.scala` — the entry point: name `schema` and `routes`, done. `main` is +- `src/main/scala/Main.scala`: the entry point; name `schema` and `routes`, done. `main` is inherited from `EezoApp` and dispatches. `EezoApp` is the umbrella's trait, both layers stacked; an application with one layer extends that layer's trait instead (`../hello`, `../reminders`). -## 2. The route table — no database needed +## 2. The route table, no database needed ``` sbt:examples> todo/run routes @@ -75,7 +75,7 @@ sbt:examples> todo/run routes ``` Handwritten routes first (most-static-first, so `/todos/new` beats `/todos/:id`), then the seven -derived ones. This command never opens a database connection — stop Postgres and it still answers. +derived ones. This command never opens a database connection: stop Postgres and it still answers. `todo/eezoRoutes` is the same thing as an sbt task. ## 3. Status → freeze → migrate: the reviewed path @@ -94,9 +94,9 @@ sbt:examples> todo/run db status + index uq_todo_title on todo ``` -(The task "fails" — drift is exit code 1 on purpose, so CI can gate on it.) +(The task "fails": drift is exit code 1 on purpose, so CI can gate on it.) -Freeze it as the first migration. Everything after `freeze` that is not a flag is the name — no +Freeze it as the first migration. Everything after `freeze` that is not a flag is the name, no quoting needed: ``` @@ -113,7 +113,7 @@ sbt:examples> todo/run db freeze initial schema wrote db/migrations/0001_initial_schema.sql ``` -Open `todo/db/migrations/0001_initial_schema.sql` — committed SQL with a fingerprint header (edit +Open `todo/db/migrations/0001_initial_schema.sql`: committed SQL with a fingerprint header (edit it by hand and `migrate` will refuse, telling you to change the model and re-freeze instead). `todo/db/schema.json` is the snapshot the *next* freeze diffs against. @@ -158,7 +158,7 @@ sbt:examples> todo/run ``` Open . Click through: the homepage (try `/?name=you`), `/health`, and -`/todos` — create a couple of todos, edit one, delete one. That whole CRUD flow is derived from +`/todos`: create a couple of todos, edit one, delete one. That whole CRUD flow is derived from the case class; the form's optional `notes` and the `done` checkbox come from `Option[String]` and `Boolean`. Ctrl+C (or `enter` in the sbt shell) stops it. @@ -172,9 +172,9 @@ sbt:todo> eezoDev First run compiles and starts the app (`eezo dev: started Main`), with the drift check and route listing on. Now, **without stopping anything**: -1. Edit `app/Index.scala` — change the `h1` text — and save. +1. Edit `app/Index.scala`, change the `h1` text, and save. 2. Watch the console: recompile, restart. -3. Refresh — your change is live. Budget is well under a second of +3. Refresh : your change is live. Budget is well under a second of compile plus ~a quarter second of JVM boot. Break the file on purpose (delete a `)`) and save: the compile fails, **the old server keeps @@ -190,7 +190,7 @@ With `eezoDev` still running, add a field to `Todo`: priority: Option[Int] ``` -Save. The restarted server prints a warning but serves — a column the database lacks breaks +Save. The restarted server prints a warning but serves: a column the database lacks breaks nothing until code touches it: ``` @@ -214,14 +214,13 @@ refuses to route: every path on answers **503** with the - two buttons: **apply to the dev database** (sync, no migration written) or **freeze as a migration and apply it** (name field + your decisions → `000N_*.sql` + applied) -Pick either. The page re-checks on every refresh, so it flips to *"resolved ✓ — save any file to -restart"* — save a file (or press `enter` and rerun `eezoDev`) and the app serves again. If an -action fails, the database's own error renders on the page — for example, adding a non-null -column over existing rows is a `[risky]` change that Postgres may refuse; that refusal is the -flag doing its job. +Pick either. The page re-checks on every refresh, so it flips to *"resolved ✓"*; save a file (or +press `enter` and rerun `eezoDev`) and the app serves again. If an action fails, the database's +own error renders on the page: for example, adding a non-null column over existing rows is a +`[risky]` change that Postgres may refuse; that refusal is the flag doing its job. The decisions are on the page rather than in the terminal because under `eezoDev` the forked app -shares stdin with sbt's watch — a console prompt would race it for every keystroke. +shares stdin with sbt's watch, and a console prompt would race it for every keystroke. ## 8. Start over @@ -253,8 +252,8 @@ drop + sync when you don't care about migrations.) `eezo new myapp` scaffolds one against the locally published eezo, and inside it `eezo routes`, `eezo db status --json`, `eezo dev` forward to the same dispatch these `todo/run ...` commands hit. -**For agents and scripts:** every command takes `--json` — same exit codes, machine-readable -body, each change carrying its `sql` alongside the `destructive`/`risky` classification. Try -`todo/run db status --json`. `freeze` swaps its prompt for a policy flag: `--accept-all` or -`--skip-destructive` (bare `--json` implies the latter, so a machine caller never hangs on a -prompt). +**For agents and scripts:** every command takes `--json`, with the same exit codes and a +machine-readable body, each change carrying its `sql` alongside the `destructive`/`risky` +classification. Try `todo/run db status --json`. `freeze` swaps its prompt for a policy flag: +`--accept-all` or `--skip-destructive` (bare `--json` implies the latter, so a machine caller +never hangs on a prompt). diff --git a/examples/todo/src/main/scala/Main.scala b/examples/todo/src/main/scala/Main.scala index 3a70967e..f6ea5b0e 100644 --- a/examples/todo/src/main/scala/Main.scala +++ b/examples/todo/src/main/scala/Main.scala @@ -1,5 +1,3 @@ -import java.sql.Connection - import io.eezo.EezoApp import io.eezo.generated.Routes import io.eezo.db.Schema @@ -21,17 +19,9 @@ object Main extends EezoApp { override def port: Int = 8090 /** This app keeps its tables in a Postgres schema of its own, so it can share the dev database - * with other examples without their drift bleeding into each other. Two halves, told once each: - * the init hook runs on **every** pooled connection (search_path is per-connection), and - * `databaseSchema` points the drift commands' catalog reads at the same name. + * with other examples without their drift bleeding into each other. Naming it is enough: eezo + * creates the schema and sets search_path to it on **every** pooled connection, because + * search_path is per connection, and the drift commands read the same name. */ override def databaseSchema: String = "todo" - - override def databaseInit: Connection => Unit = { c => - val st = c.createStatement() - try { - st.execute("""create schema if not exists "todo"""") - st.execute("""set search_path to "todo"""") - } finally st.close() - } } diff --git a/modules/db/src/main/scala/io/eezo/db/DbApp.scala b/modules/db/src/main/scala/io/eezo/db/DbApp.scala index a2e1f951..0d8e0657 100644 --- a/modules/db/src/main/scala/io/eezo/db/DbApp.scala +++ b/modules/db/src/main/scala/io/eezo/db/DbApp.scala @@ -48,11 +48,6 @@ trait DbApp extends Dispatch with DbInit { */ def schema: Schema - /** The Postgres schema the drift commands introspect. `search_path` is set per connection in - * [[databaseInit]]; this is the same name, told to the side that reads catalogs. - */ - def databaseSchema: String = "public" - /** The program: what `sbt run` does, once, with a `Database` installed. Abstract, because a * database only application is its program. The umbrella supplies the http layer's default, * which is to serve. diff --git a/modules/db/src/main/scala/io/eezo/db/DbInit.scala b/modules/db/src/main/scala/io/eezo/db/DbInit.scala index 617f0311..0cb46c11 100644 --- a/modules/db/src/main/scala/io/eezo/db/DbInit.scala +++ b/modules/db/src/main/scala/io/eezo/db/DbInit.scala @@ -1,20 +1,22 @@ package io.eezo.db import io.eezo.db.engine.Database +import io.eezo.db.schema.Ddl import java.net.{URI, URLDecoder} -import java.sql.Connection +import java.sql.{Connection, SQLException} import java.time.Duration -/** How an application names its database: the connection settings and the per connection hook. +/** How an application names its database: the connection settings, the Postgres schema it lives in, + * and the per connection hook. * * Split out of [[DbApp]] so that a front end which is not the application's entry point (a deploy * runner, the testkit) can build the same `Database` from the same settings without inheriting * `boot`. `DbApp` extends this, and the umbrella's `EezoApp` through it; users override members * here and never name the trait. * - * Named for [[databaseInit]], its one member that is not a string, because it is the one people - * get wrong; see its comment. + * Named for what happens on every connection the pool opens, because that is the part people get + * wrong: a setting made on one connection is missing from the next. See [[databaseSchema]]. * * **Two spellings of the environment, one mechanism.** `EEZO_DB_URL`/`EEZO_DB_USER`/ * `EEZO_DB_PASS` are eezo's own and always win. When they are absent and `DATABASE_URL` is present @@ -61,12 +63,30 @@ trait DbInit { def databaseAcquireTimeout: Duration = DbInit.acquireTimeout(sys.env.get("EEZO_DB_ACQUIRE_TIMEOUT")) - /** Run on **every** connection the pool creates, not once on a borrowed one (DESIGN §8.7). + /** The Postgres schema the application lives in: where its tables are created and where the drift + * commands read the catalog. Declared here, where the `Database` is built, because the name has + * to reach the connections as well as the catalog reads, and a front end that builds a + * `Database` through this trait must land in the same schema as the application. * - * This is where `search_path`, `application_name` and statement timeouts belong. Without it an - * application that works in a non-`public` schema silently splits in two: its own connections - * see one schema and eezo's see another, which shows up as a migration that appears not to have - * run. + * Naming it is enough. Any name but `public` makes every physical connection create the schema + * when it is missing and set `search_path` to it, before [[databaseInit]] runs, so the DDL the + * commands apply, the migration ledger and the application's own SQL all land where `status` + * looks. With the setup left to the application, a name set here without the matching hook split + * the application in two: tables created in `public`, the catalog read in the named schema, and + * `status` reporting the whole model missing after a `sync --apply` that succeeded. + * + * `public`, the default, runs nothing, so the connection keeps Postgres's own `search_path`. + */ + def databaseSchema: String = "public" + + /** Run on **every** connection the pool creates, not once on a borrowed one (DESIGN §8.7), + * because the settings it exists for are per connection: a statement run once would reach only + * the connection that ran it. + * + * This is where `application_name`, statement timeouts and any other per connection setting + * belong. The schema is not among them: [[databaseSchema]] sets `search_path` on its own, ahead + * of this hook, so an override here needs no `super` call to keep the application in its schema, + * and one that sets `search_path` on purpose still has the last word. */ def databaseInit: Connection -> Unit = _ => () @@ -75,7 +95,7 @@ trait DbInit { databaseUrl, databaseUser, databasePassword, - databaseInit, + DbInit.connectionInit(databaseSchema, databaseInit), databasePoolSize, databaseAcquireTimeout ) @@ -83,6 +103,54 @@ trait DbInit { object DbInit { + /** The one hook the pool runs: the schema step `databaseSchema` implies, then the application's + * own. Composed here rather than offered as `databaseInit`'s default, because a default would be + * lost by every override that forgot `super`, which is the split this exists to prevent. + */ + private[eezo] def connectionInit(schema: String, init: Connection -> Unit): Connection -> Unit = + if (schema == "public") init + else + c => { + enterSchema(c, schema) + init(c) + } + + /** Creates the schema only when the catalog says it is missing, because Postgres checks the + * CREATE privilege on the database before it honours `if not exists`: a least privileged role + * that owns an existing schema would otherwise fail every connection the pool opens. + * + * Two connections that both found it missing still race, on a fresh database where the pool + * fills or several processes boot at once, and `if not exists` does not settle that race: the + * loser fails on the catalog's unique index. Its failure means the schema is there, which is all + * this step needs, so it goes on to `search_path`; any other failure is a real one and fails the + * connection. + */ + private def enterSchema(c: Connection, schema: String): Unit = { + val name = Ddl.q(schema) + val st = c.createStatement() + try { + if (!schemaExists(c, schema)) + try st.execute(s"create schema if not exists $name"): Unit + catch { case e: SQLException if CreatedMeanwhile(e.getSQLState) => () } + st.execute(s"set search_path to $name"): Unit + } finally st.close() + } + + /** `unique_violation`, what Postgres answers the loser of the race, and `duplicate_schema`, which + * says the same thing in its own words. + */ + private val CreatedMeanwhile: Set[String | Null] = Set("23505", "42P06") + + private def schemaExists(c: Connection, schema: String): Boolean = { + val ps = c.prepareStatement("select 1 from pg_namespace where nspname = ?") + try { + ps.setString(1, schema) + val rs = ps.executeQuery() + try rs.next() + finally rs.close() + } finally ps.close() + } + private[eezo] val DefaultPoolSize: Int = 10 /** A value HikariCP would refuse, below one, is as malformed as one that is not a number: either diff --git a/modules/db/src/main/scala/io/eezo/db/schema/Ddl.scala b/modules/db/src/main/scala/io/eezo/db/schema/Ddl.scala index 8faad70d..6080574a 100644 --- a/modules/db/src/main/scala/io/eezo/db/schema/Ddl.scala +++ b/modules/db/src/main/scala/io/eezo/db/schema/Ddl.scala @@ -8,7 +8,11 @@ package io.eezo.db.schema */ object Ddl { - private def q(s: String): String = "\"" + s.replace("\"", "\"\"") + "\"" + /** A name as a Postgres quoted identifier, its own double quotes doubled. Shared with + * [[io.eezo.db.DbInit]], whose schema statements run on every connection, where a stray quote + * would fail them all or run more than the one statement meant. + */ + private[db] def q(s: String): String = "\"" + s.replace("\"", "\"\"") + "\"" private def checkName(table: String, column: String, expr: String): String = { val h = Integer.toHexString(expr.hashCode & 0xffffff) diff --git a/modules/db/src/test/scala/io/eezo/db/DbAppSuite.scala b/modules/db/src/test/scala/io/eezo/db/DbAppSuite.scala index ab2584dd..602b3800 100644 --- a/modules/db/src/test/scala/io/eezo/db/DbAppSuite.scala +++ b/modules/db/src/test/scala/io/eezo/db/DbAppSuite.scala @@ -29,8 +29,6 @@ class DbAppSuite extends PgSuite { override def databaseUser: String = Pg.username override def databasePassword: String = Pg.password - override def databaseInit: Connection -> Unit = Pg.searchPath(pgSchema) - override def boot(): Unit = { ran = true sawDatabase = read { Commands.status(schema, databaseSchema) }.changes.nonEmpty @@ -39,6 +37,96 @@ class DbAppSuite extends PgSuite { private def installed: Boolean = Installed.installed + /** An application that names its Postgres schema and nothing else: no `databaseInit`, so whatever + * puts its tables in that schema is the framework's doing. The schema is one that does not exist + * yet, so the framework has to create it, and the test drops it after. + */ + private class Named(name: String, user: String = Pg.username, password: String = Pg.password) + extends DbApp { + override def schema: Schema = Library + override def databaseSchema: String = name + + override def databaseUrl: String = Pg.jdbcUrl + override def databaseUser: String = user + override def databasePassword: String = password + + override def boot(): Unit = () + } + + /** The same application with a hook of its own for something unrelated to the schema, written + * without `super`, the way an application adds `application_name` and forgets the rest. + */ + private class NamedWithHook(name: String) extends Named(name) { + override def databaseInit: Connection -> Unit = c => { + val st = c.createStatement() + try st.execute("set application_name to 'eezo-dbappsuite'"): Unit + finally st.close() + } + } + + /** The schemas that hold a table called `table`, read from the catalog. */ + private def schemasHolding(table: String): Set[String] = { + val ps = + db.prepareStatement("select table_schema from information_schema.tables where table_name = ?") + try { + ps.setString(1, table) + val rs = ps.executeQuery() + try Iterator.continually(rs).takeWhile(_.next()).map(_.getString(1)).toSet + finally rs.close() + } finally ps.close() + } + + /** Runs `f` with `name` absent before and dropped after, so no later suite sees its tables. */ + private def inFreshSchema(name: String)(f: => Unit): Unit = { + val drop = s"""drop schema if exists "${name.replace("\"", "\"\"")}" cascade""" + exec(drop) + try f + finally exec(drop) + } + + test("databaseSchema alone puts the tables in that schema, creating it, and status agrees") { + val name = pgSchema + "_own" + inFreshSchema(name) { + val app = new Named(name) + assertEquals(captured(app.run(List("db", "sync", "--apply")))._1, 0) + assertEquals(schemasHolding("author"), Set(name)) + val (code, out, _) = captured(app.run(List("db", "status"))) + assertEquals(code, 0, out) + } + } + + test("a role that owns its schema but cannot create one on the database still connects") { + val name = pgSchema + "_owned" + val role = pgSchema + "_role" + inFreshSchema(name) { + exec( + s"""drop role if exists "$role"""", + s"""create role "$role" login password 'owner'""", + s"""create schema "$name" authorization "$role"""" + ) + try { + val app = new Named(name, user = role, password = "owner") + assertEquals(captured(app.run(List("db", "sync", "--apply")))._1, 0) + assertEquals(schemasHolding("author"), Set(name)) + } finally { + exec(s"""drop schema if exists "$name" cascade""", s"""drop role if exists "$role"""") + } + } + } + + test( + "a databaseInit of the application's own, without super, keeps the tables in databaseSchema" + ) { + val name = pgSchema + "_hook" + inFreshSchema(name) { + val app = new NamedWithHook(name) + assertEquals(captured(app.run(List("db", "sync", "--apply")))._1, 0) + assertEquals(schemasHolding("author"), Set(name)) + val (code, out, _) = captured(app.run(List("db", "status"))) + assertEquals(code, 0, out) + } + } + test("no arguments is the program, run with a Database installed and uninstalled after") { val app = new Job assert(!installed) diff --git a/modules/db/src/test/scala/io/eezo/db/DbInitSuite.scala b/modules/db/src/test/scala/io/eezo/db/DbInitSuite.scala index 4a25eb0e..741c0b1a 100644 --- a/modules/db/src/test/scala/io/eezo/db/DbInitSuite.scala +++ b/modules/db/src/test/scala/io/eezo/db/DbInitSuite.scala @@ -2,13 +2,137 @@ package io.eezo.db import munit.FunSuite +import java.lang.reflect.{InvocationHandler, Method, Proxy} +import java.sql.{Connection, PreparedStatement, ResultSet, SQLException, Statement} import java.time.Duration +import scala.collection.mutable.ListBuffer /** The `DATABASE_URL` fallback parser: the one secret a PaaS injects, split into the three values - * JDBC wants. Pure, so a plain suite. + * JDBC wants; and the schema step the pool runs on every connection, read off a connection that + * records what it is asked. Pure, so a plain suite. */ class DbInitSuite extends FunSuite { + /** A connection with no database behind it: it writes down every call made on it and every + * statement executed through it, and answers the catalog lookup as `schemaExists` says. With + * `createFails`, the `create schema` statement fails with that SQLState, the way Postgres + * answers the loser of two connections that both found the schema missing. + */ + private final class Recording(schemaExists: Boolean, createFails: Option[String] = None) { + val calls: ListBuffer[String] = ListBuffer.empty + val executed: ListBuffer[String] = ListBuffer.empty + + private def proxy[T]( + cls: Class[T] + )(answer: (Method, Array[AnyRef] | Null) => AnyRef | Null): T = + cls.cast( + Proxy.newProxyInstance( + getClass.getClassLoader, + Array[Class[?]](cls), + new InvocationHandler { + def invoke(p: AnyRef, m: Method, args: Array[AnyRef] | Null): AnyRef | Null = + answer(m, args) + } + ) + ) + + private val resultSet: ResultSet = proxy(classOf[ResultSet]) { (m, _) => + if (m.getName == "next") java.lang.Boolean.valueOf(schemaExists) else null + } + + private val prepared: PreparedStatement = proxy(classOf[PreparedStatement]) { (m, _) => + if (m.getName == "executeQuery") resultSet else null + } + + private val statement: Statement = proxy(classOf[Statement]) { (m, args) => + if (m.getName == "execute") { + val sql = String.valueOf(args.nn(0)) + executed += sql + createFails match { + case Some(state) if sql.startsWith("create schema") => + throw new SQLException("duplicate key value violates unique constraint", state) + case _ => java.lang.Boolean.FALSE + } + } else null + } + + val connection: Connection = proxy(classOf[Connection]) { (m, _) => + calls += m.getName + m.getName match { + case "createStatement" => statement + case "prepareStatement" => prepared + case _ => null + } + } + } + + test( + "the default public schema runs nothing on the connection, so its search_path is Postgres's" + ) { + val c = Recording(schemaExists = true) + DbInit.connectionInit("public", _ => ())(c.connection) + assertEquals(c.calls.toList, Nil) + } + + test("a named schema is created when missing, then becomes the search_path, before the hook") { + val c = Recording(schemaExists = false) + DbInit.connectionInit( + "shop", + conn => conn.createStatement().execute("set application_name to 'x'"): Unit + )( + c.connection + ) + assertEquals( + c.executed.toList, + List( + """create schema if not exists "shop"""", + """set search_path to "shop"""", + "set application_name to 'x'" + ) + ) + } + + test( + "a schema that exists is not created again, so a role without CREATE on the database connects" + ) { + val c = Recording(schemaExists = true) + DbInit.connectionInit("shop", _ => ())(c.connection) + assertEquals(c.executed.toList, List("""set search_path to "shop"""")) + } + + test( + "losing the race to create the schema to another connection still enters it, either SQLState" + ) { + for (state <- List("23505", "42P06")) { + val c = Recording(schemaExists = false, createFails = Some(state)) + DbInit.connectionInit("shop", _ => ())(c.connection) + assertEquals( + c.executed.toList, + List("""create schema if not exists "shop"""", """set search_path to "shop""""), + state + ) + } + } + + test("any other failure to create the schema fails the connection") { + val c = Recording(schemaExists = false, createFails = Some("42501")) + val e = intercept[SQLException](DbInit.connectionInit("shop", _ => ())(c.connection)) + assertEquals(e.getSQLState, "42501") + assertEquals(c.executed.toList, List("""create schema if not exists "shop"""")) + } + + test("a double quote in the schema name is doubled, so the name stays one identifier") { + val c = Recording(schemaExists = false) + DbInit.connectionInit("""a"; drop table x; --""", _ => ())(c.connection) + assertEquals( + c.executed.toList, + List( + """create schema if not exists "a""; drop table x; --"""", + """set search_path to "a""; drop table x; --"""" + ) + ) + } + private def parsed(url: String) = DbInit.parseDatabaseUrl(url).getOrElse(fail(s"expected $url to parse")) diff --git a/modules/db/src/test/scala/io/eezo/db/support/Pg.scala b/modules/db/src/test/scala/io/eezo/db/support/Pg.scala index 77a25e9f..af0ddb95 100644 --- a/modules/db/src/test/scala/io/eezo/db/support/Pg.scala +++ b/modules/db/src/test/scala/io/eezo/db/support/Pg.scala @@ -57,8 +57,9 @@ object Pg { acquireTimeout = DbInit.DefaultAcquireTimeout ) - /** The connection init hook that isolates a suite by schema: [[database]]'s, and the one a suite - * that builds its own `Database` through `DbInit` names as `databaseInit`. + /** The connection init hook that isolates a suite by schema: [[database]]'s. A suite that builds + * its own `Database` through `DbInit` names the schema as `databaseSchema` instead, which sets + * the same `search_path` on every connection. */ def searchPath(schema: String): Connection -> Unit = c => { val st = c.createStatement() diff --git a/modules/example/src/test/scala/example/Tour.scala b/modules/example/src/test/scala/example/Tour.scala index acfd6a45..1e0be3e2 100644 --- a/modules/example/src/test/scala/example/Tour.scala +++ b/modules/example/src/test/scala/example/Tour.scala @@ -48,21 +48,11 @@ object Tour extends DbApp { override def databaseUser: String = container.getUsername override def databasePassword: String = container.getPassword - /** The tour works in its own schema, so eezo's connections have to be told about it too; - * otherwise the migrator writes to `public` while the drift checks read `eezo_tour`, and a - * replay looks like it never happened. DESIGN §8.7. + /** The tour works in its own schema, and naming it is all eezo needs: every connection the pool + * opens creates it if it is missing and sets `search_path` to it, so the migrator writes where + * the drift checks read. DESIGN §8.7. */ - // `Connection => Unit`, not `->`: this module does not enable capture checking, so eezo's pure - // arrow presents here as an ordinary function type (research/capture-checking.md §6.3). - override def databaseInit: Connection => Unit = { c => - val st = c.createStatement() - try st.execute(s"""create schema if not exists "$Schema0"""") - catch { case NonFatal(_) => () } - finally st.close() - val st2 = c.createStatement() - try st2.execute(s"""set search_path to "$Schema0"""") - finally st2.close() - } + override def databaseSchema: String = Schema0 private var paused = true @@ -405,8 +395,8 @@ object Tour extends DbApp { private def prepare(c: Connection): Unit = { header() exec(c, s"""drop schema if exists "$Schema0" cascade""") - // `if not exists`: the pool fills in the background while this runs, and each new connection's - // `databaseInit` may create the schema between the drop and this line. + // `if not exists`: the pool fills in the background while this runs, and each new connection + // creates the schema `databaseSchema` names when it is missing, maybe between the drop and here. exec(c, s"""create schema if not exists "$Schema0"""") exec(c, s"""set search_path to "$Schema0"""") note(s"Working in schema `$Schema0` on ${Db.url}, so nothing else is touched.")