From 9c8c008d1ec421bde87538ce950210c7e6334157 Mon Sep 17 00:00:00 2001 From: Alistair Young Date: Sat, 31 Oct 2026 15:42:31 -0500 Subject: [PATCH 1/4] Record PostgreSQL extensions, and restore-test pgvector sets in pgvector's image Postgres manifests now list each database's extensions. A restore checks the destination has them all before changing anything, instead of failing part-way through pg_restore; restore-test reports a missing extension as an error; and scripts/restore-test.sh picks pgvector/pgvector:pg as the scratch image for sets that use vector, since the stock image only has contrib extensions. Co-Authored-By: Claude Opus 5.5 --- README.md | 2 +- docs/restore.md | 8 ++++-- scripts/restore-test.sh | 14 +++++++---- src/dbbackup/cli.py | 4 +++ src/dbbackup/engines/postgres.py | 13 +++++++++- src/dbbackup/restoretest.py | 7 ++++-- tests/integration/seed/postgres/01-seed.sql | 2 ++ tests/integration/test_end_to_end.py | 2 ++ tests/test_engines.py | 16 +++++++++++- tests/test_restoretest.py | 27 ++++++++++++++++++--- 10 files changed, 79 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index 944abe8..149ba9e 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,7 @@ A containerized database backup solution: logical backups of any number of **Mar - **Isolates failures.** One server's failure never stops the others. A target is pruned only after one of its own backups succeeds, and the newest good set is always kept. - **Reports to MQTT,** with Home Assistant discovery. Failures, and backups that have *stopped happening*, can raise [Alert Redux](https://github.com/arkane-systems/ha-alert-redux) alerts; see [docs/home-assistant.md](docs/home-assistant.md). - **Restores with one command,** `dbbackup restore`. Every set can also be restored with stock client tools; see [docs/restore.md](docs/restore.md). -- **Tests restores locally.** `scripts/restore-test.sh` restores sets into throwaway Docker servers of the matching version, and checks every database, table, collection, user and role against what the backup recorded. +- **Tests restores locally.** `scripts/restore-test.sh` restores sets into throwaway Docker servers of the matching version, and checks every database, table, collection, extension, user and role against what the backup recorded. PostgreSQL sets that use pgvector are tested in pgvector's image. Supported servers: MariaDB 11.4+, PostgreSQL 18+, MongoDB 8.3+ (as a replica set, for `--oplog`). Older versions may well work: the tool logs a warning and carries on. The image ships the PostgreSQL 18 client, the MariaDB 11.8 client and MongoDB Database Tools 100.19. It is built for linux/amd64 only, because MongoDB publishes its Debian tools packages only for x86-64. diff --git a/docs/restore.md b/docs/restore.md index c9f20bc..742462b 100644 --- a/docs/restore.md +++ b/docs/restore.md @@ -49,6 +49,7 @@ dbbackup restore [--set latest|] [-d ...] \ - **Existing databases.** The tool refuses to overwrite a database that already exists unless you pass `--force`. - **Existing accounts are left alone.** Users and roles that already exist on the destination keep their passwords and privileges. That includes the account you restore as, and a fresh server's own `root` and `healthcheck` accounts. Missing users and roles are created with their backed-up passwords and grants. - **The set is checked first.** The tool re-checks every file's checksum before it restores anything. +- **PostgreSQL extensions must be installed.** A destination without an extension the backup uses (pgvector's `vector`, say) is refused before anything is restored. The manifest's `inventory` lists each database's `extensions`, and `dbbackup inspect` shows them. ### From Kubernetes @@ -97,6 +98,8 @@ Use client tools at least as new as the server that was backed up. The manifest' A set contains `globals.sql.zst` (roles and tablespaces) and one `.pgdump/` directory per database. Each directory is a `pg_dump --format=directory` archive. +The new server needs every extension the databases use installed, such as pgvector (the `pgvector/pgvector` images have it). The manifest's `inventory` lists them per database. A missing one makes `pg_restore` fail on `CREATE EXTENSION`, and then on everything that depends on it. + 1. **Roles first.** Restored objects are owned by these roles and granted to them. ```bash @@ -182,7 +185,7 @@ scripts/restore-test.sh --set 20261009T021500Z /mnt/db-backups mariadb-main For each target, the script: -1. Reads the set's manifest, using the db-backup image, and starts a throwaway scratch server matching the backed-up server's version: `postgres:`, `mariadb:` or `mongo:`. MongoDB runs as a single-node replica set, so the oplog is replayed as it would be in production. Each scratch server has its own compose project and network, and nothing is published on the host. +1. Reads the set's manifest, using the db-backup image, and starts a throwaway scratch server matching the backed-up server's version: `postgres:`, `mariadb:` or `mongo:`. PostgreSQL sets that use pgvector get `pgvector/pgvector:pg` instead, since the stock image only has the contrib extensions. MongoDB runs as a single-node replica set, so the oplog is replayed as it would be in production. Each scratch server has its own compose project and network, and nothing is published on the host. 2. Runs `dbbackup restore-test`, which: - re-checks every file's checksum, and the dumps' integrity - restores the set exactly as `dbbackup restore` would, users and roles included @@ -194,6 +197,7 @@ The comparison treats these as **errors**: - a database missing from the restored server - a missing table, view, materialized view or collection - a missing user or role +- a missing PostgreSQL extension - an object restored as a different kind (say, a view that came back as a table) Row-count differences are only **warnings**. The restored server is always counted exactly; what it's compared with depends on the target's `exact_counts` setting: @@ -211,7 +215,7 @@ The run ends with a summary table, including how long each restore took, which i | `-t`, `--tmpfs` | Keep the scratch servers' data in RAM. Faster, but the restored data must fit in memory. | | `-k`, `--keep` | Leave each scratch server running afterwards, to look around in. The script prints how to remove it. | -To use a scratch image other than the one matching the recorded version, set `SCRATCH_IMAGE_POSTGRES`, `SCRATCH_IMAGE_MARIADB` or `SCRATCH_IMAGE_MONGODB`. For example, `SCRATCH_IMAGE_POSTGRES=postgres:19` rehearses an upgrade. +To use a scratch image other than the one matching the recorded version, set `SCRATCH_IMAGE_POSTGRES`, `SCRATCH_IMAGE_MARIADB` or `SCRATCH_IMAGE_MONGODB`. For example, `SCRATCH_IMAGE_POSTGRES=postgres:19` rehearses an upgrade. A set that uses another extension the stock image lacks, such as PostGIS, needs an image that has it (`SCRATCH_IMAGE_POSTGRES=postgis/postgis:18-3.6`): without one, the restore stops before it starts, naming the missing extensions. The backups are mounted read-only, and are read as the backup root's owner (`pg_dump`'s directory archives are private to their owner). diff --git a/scripts/restore-test.sh b/scripts/restore-test.sh index 9883450..bf5a2b9 100755 --- a/scripts/restore-test.sh +++ b/scripts/restore-test.sh @@ -21,7 +21,8 @@ # -h, --help show this help # # Scratch images follow the recorded server version (postgres:, -# mariadb:, mongo:); override them with +# mariadb:, mongo:), and PostgreSQL sets that use +# pgvector get pgvector/pgvector:pg; override them with # SCRATCH_IMAGE_POSTGRES, SCRATCH_IMAGE_MARIADB or SCRATCH_IMAGE_MONGODB. # # Exits non-zero if any target fails. @@ -105,7 +106,7 @@ for target in "${targets[@]}"; do cleanup continue fi - DBB_TYPE="" DBB_SET="" DBB_SERVER_VERSION="" + DBB_TYPE="" DBB_SET="" DBB_SERVER_VERSION="" DBB_EXTENSIONS="" while IFS= read -r line; do [[ $line =~ ^DBB_[A-Z_]+= ]] && eval "$line" # values are shell-quoted by dbbackup done <<<"$info" @@ -115,7 +116,10 @@ for target in "${targets[@]}"; do minor=${version#*.}; minor=${minor%%.*} case $DBB_TYPE in postgres) - scratch_image=${SCRATCH_IMAGE_POSTGRES:-postgres:$major} + # The stock image has only the contrib extensions; pgvector's image adds vector. + default_image=postgres:$major + [[ " $DBB_EXTENSIONS " == *" vector "* ]] && default_image=pgvector/pgvector:pg$major + scratch_image=${SCRATCH_IMAGE_POSTGRES:-$default_image} conn=(--host scratch-postgres --username postgres --password-env SCRATCH_PASSWORD) ;; mariadb) scratch_image=${SCRATCH_IMAGE_MARIADB:-mariadb:$major.$minor} @@ -172,9 +176,9 @@ for target in "${targets[@]}"; do done echo -printf '%-24s %-17s %-16s %-36s %s\n' TARGET SET SCRATCH RESULT RESTORE +printf '%-24s %-17s %-24s %-36s %s\n' TARGET SET SCRATCH RESULT RESTORE for row in "${results[@]}"; do IFS='|' read -r t s i r d <<<"$row" - printf '%-24s %-17s %-16s %-36s %s\n' "$t" "$s" "$i" "$r" "$d" + printf '%-24s %-17s %-24s %-36s %s\n' "$t" "$s" "$i" "$r" "$d" done exit $failed diff --git a/src/dbbackup/cli.py b/src/dbbackup/cli.py index fa5f735..e10e085 100644 --- a/src/dbbackup/cli.py +++ b/src/dbbackup/cli.py @@ -247,6 +247,7 @@ def cmd_inspect(args) -> int: if not bset.ok: raise StorageError(f"{bset.path} has no valid manifest") m = bset.manifest + extensions = sorted({ext for db in m.get("inventory", {}).get("databases", {}).values() for ext in db.get("extensions", {})}) if args.format == "json": print(json.dumps({k: v for k, v in m.items() if k != "files"}, indent=2, sort_keys=True)) elif args.format == "env": @@ -257,6 +258,7 @@ def cmd_inspect(args) -> int: "DBB_SERVER_VERSION": m.get("server_version", ""), "DBB_DATABASES": str(len(m.get("databases", []))), "DBB_SIZE_BYTES": str(m.get("size_bytes", "")), + "DBB_EXTENSIONS": " ".join(extensions), } for key, value in values.items(): print(f"{key}={shlex.quote(value)}") @@ -264,6 +266,8 @@ def cmd_inspect(args) -> int: print(f"target: {m['target']} ({m['type']} {m.get('server_version', '?')}, {m.get('host') or 'URI'})") print(f"set: {bset.name} finished {m.get('finished')} ({human_size(m.get('size_bytes'))}, {m.get('duration_s')}s)") print(f"contents: {', '.join(m.get('databases', [])) or '(no databases)'}") + if extensions: + print(f"extensions: {', '.join(extensions)}") for warning in m.get("warnings", []): print(f"warning: {warning}") return EXIT_OK diff --git a/src/dbbackup/engines/postgres.py b/src/dbbackup/engines/postgres.py index f5699ab..4d16901 100644 --- a/src/dbbackup/engines/postgres.py +++ b/src/dbbackup/engines/postgres.py @@ -131,7 +131,9 @@ def inventory(self, databases: list[str], *, exact: bool = False) -> dict: sql = " UNION ALL ".join(f"SELECT {i}, count(*) FROM {qualified}" for i, (_, qualified) in enumerate(countable)) for i, count in self.query(sql, db=db): objects[countable[int(i)][0]]["rows"] = int(count) - result[db] = {"objects": objects} + # A restore needs these installed on the destination (pgvector, PostGIS, ...). + extensions = dict(self.query("SELECT extname, extversion FROM pg_extension ORDER BY extname", db=db)) + result[db] = {"objects": objects, "extensions": extensions} users = [r[0] for r in self.query("SELECT rolname FROM pg_roles WHERE rolname !~ '^pg_' ORDER BY rolname")] return {"counts": "exact" if exact else "estimated", "databases": result, "users": users} @@ -194,6 +196,15 @@ def restore(self, set_dir: Path, manifest: dict, databases: list[str], *, force: if clashes and not force: raise EngineError(f"database(s) already exist on {self.conn.host}: {', '.join(clashes)} (use --force to replace them)") + # Fail before changing anything, rather than part-way through pg_restore. + recorded = manifest.get("inventory", {}).get("databases", {}) + needed = {ext for db in databases for ext in recorded.get(db, {}).get("extensions", {})} + unavailable = needed - {r[0] for r in self.query("SELECT name FROM pg_available_extensions")} + if unavailable: + raise EngineError( + f"{self.conn.host} doesn't have extension(s) the backup uses: {', '.join(sorted(unavailable))} (install them first)" + ) + if include_globals: # Roles first: restored objects are owned by, and granted to, them. log.info("%s: restoring roles and tablespaces", self.target.name) diff --git a/src/dbbackup/restoretest.py b/src/dbbackup/restoretest.py index e6c35f0..a914ffb 100644 --- a/src/dbbackup/restoretest.py +++ b/src/dbbackup/restoretest.py @@ -1,7 +1,7 @@ """Restore testing: restore a set into a scratch server, then check the result against the inventory its manifest recorded at backup time. -Anything missing (a database, table, view, collection, user or role) is an error. +Anything missing (a database, table, view, collection, extension, user or role) is an error. Row-count differences are only warnings. The restored side is always counted exactly (it's a scratch server); what it's compared with depends on the backup: @@ -62,7 +62,10 @@ def compare_inventory( if db not in present: result.errors.append(f"database {db} is missing") continue - restored = actual.get("databases", {}).get(db, {}).get("objects", {}) + restored_db = actual.get("databases", {}).get(db, {}) + for ext in sorted(set(info.get("extensions", {})) - set(restored_db.get("extensions", {}))): + result.errors.append(f"{db}: extension {ext} is missing") + restored = restored_db.get("objects", {}) for name, obj in sorted(info.get("objects", {}).items()): result.objects_checked += 1 got = restored.get(name) diff --git a/tests/integration/seed/postgres/01-seed.sql b/tests/integration/seed/postgres/01-seed.sql index 8bfcff0..05c8f26 100644 --- a/tests/integration/seed/postgres/01-seed.sql +++ b/tests/integration/seed/postgres/01-seed.sql @@ -14,6 +14,7 @@ CREATE DATABASE "odd.name/db" OWNER app_owner; CREATE DATABASE scratch; \connect app1 +CREATE EXTENSION pg_trgm; SET ROLE app_owner; CREATE SCHEMA billing; CREATE TABLE public.customers (id serial PRIMARY KEY, name text NOT NULL, created timestamptz DEFAULT now()); @@ -25,6 +26,7 @@ CREATE TABLE billing.invoices ( ); INSERT INTO billing.invoices (customer_id, amount) SELECT (g % 5000) + 1, g * 1.5 FROM generate_series(1, 20000) g; CREATE INDEX invoices_customer ON billing.invoices (customer_id); +CREATE INDEX customers_name_trgm ON public.customers USING gin (name gin_trgm_ops); CREATE VIEW billing.big_invoices AS SELECT * FROM billing.invoices WHERE amount > 10000; CREATE MATERIALIZED VIEW billing.totals AS SELECT customer_id, sum(amount) AS total FROM billing.invoices GROUP BY customer_id; CREATE FUNCTION billing.invoice_count(c int) RETURNS bigint LANGUAGE sql AS $$ SELECT count(*) FROM billing.invoices WHERE customer_id = c $$; diff --git a/tests/integration/test_end_to_end.py b/tests/integration/test_end_to_end.py index 0199ecc..906f6c1 100644 --- a/tests/integration/test_end_to_end.py +++ b/tests/integration/test_end_to_end.py @@ -150,6 +150,7 @@ def test_postgres_manifest(): objects = m["inventory"]["databases"]["app1"]["objects"] assert objects["billing.invoices"] == {"kind": "table", "rows": 20000} assert objects["billing.big_invoices"]["kind"] == "view" + assert set(m["inventory"]["databases"]["app1"]["extensions"]) == {"pg_trgm", "plpgsql"} assert {"app_owner", "app_reader", "reporting"} <= set(m["inventory"]["users"]) assert m["warnings"] == [] @@ -220,6 +221,7 @@ def test_restore_postgres(): RESTORE_PG, "SET ROLE reporting; SELECT count(*) FROM billing.totals", db="app1", user="app_reader", password="reader-pw" ) == ["5000"] assert pg(RESTORE_PG, "SHOW work_mem", db="app1") == ["8MB"] + assert pg(RESTORE_PG, "SELECT extname FROM pg_extension WHERE extname = 'pg_trgm'", db="app1") == ["pg_trgm"] # ...and the restoring account's own password was left alone (pg() connects with it). assert pg(RESTORE_PG, "SELECT 1") == ["1"] diff --git a/tests/test_engines.py b/tests/test_engines.py index 37b9e6e..d5480ae 100644 --- a/tests/test_engines.py +++ b/tests/test_engines.py @@ -9,7 +9,7 @@ from dbbackup import proc from dbbackup.config import parse_config from dbbackup.engines import engine_for -from dbbackup.engines.base import safe_filename, version_tuple +from dbbackup.engines.base import EngineError, safe_filename, version_tuple from dbbackup.engines.mariadb import option_file_value, statement_account, without_accounts from dbbackup.engines.mongodb import _ns_escape from dbbackup.engines.postgres import conninfo_dbname, without_roles @@ -216,5 +216,19 @@ def test_select_databases(monkeypatch): engine.select_databases() +def test_postgres_restore_refuses_a_server_without_the_extensions(monkeypatch, tmp_path): + rec = Recorder(monkeypatch) + engine = engine_for(target("postgres")) + monkeypatch.setattr(type(engine), "list_databases", lambda self: ["postgres"]) + monkeypatch.setattr(type(engine), "query", lambda self, sql, db=None, **v: [["plpgsql"], ["pg_trgm"]]) + manifest = { + "contents": {"globals": "globals.sql.zst", "databases": {"app": "app.pgdump"}}, + "inventory": {"databases": {"app": {"extensions": {"plpgsql": "1.0", "vector": "0.8.1"}}}}, + } + with pytest.raises(EngineError, match="doesn't have extension\\(s\\) the backup uses: vector"): + engine.restore(tmp_path, manifest, ["app"], force=False, include_globals=True) + assert rec.calls == [] # nothing restored + + def test_ns_escape(): assert _ns_escape("a*b$c") == "a\\*b\\$c" diff --git a/tests/test_restoretest.py b/tests/test_restoretest.py index fbc3490..73cdbc4 100644 --- a/tests/test_restoretest.py +++ b/tests/test_restoretest.py @@ -14,7 +14,8 @@ "public.customers": {"kind": "table", "rows": 5000}, "public.big_customers": {"kind": "view", "rows": None}, "public.unanalyzed": {"kind": "table", "rows": None}, - } + }, + "extensions": {"plpgsql": "1.0", "vector": "0.8.1"}, }, "logs": {"objects": {"events": {"kind": "collection", "rows": 2000}}}, }, @@ -31,7 +32,8 @@ def restored(**overrides): "public.big_customers": {"kind": "view", "rows": None}, "public.unanalyzed": {"kind": "table", "rows": 12}, "public.extra": {"kind": "table", "rows": 1}, - } + }, + "extensions": {"plpgsql": "1.0", "vector": "0.8.2"}, }, "logs": {"objects": {"events": {"kind": "collection", "rows": 2000}}}, }, @@ -58,6 +60,12 @@ def test_missing_database_object_and_user_are_errors(): ] +def test_missing_extension_is_an_error(): + actual = restored() + del actual["databases"]["app"]["extensions"]["vector"] + assert compare_inventory(EXPECTED, actual, {"app", "logs"}).errors == ["app: extension vector is missing"] + + def test_kind_change_is_an_error(): actual = restored() actual["databases"]["app"]["objects"]["public.big_customers"] = {"kind": "table", "rows": 0} @@ -112,7 +120,17 @@ def make_set(tmp_path, target="pg-main", ttype="postgres"): when = datetime(2026, 10, 9, 2, 15, tzinfo=UTC) partial = store.begin(target, when) (partial / "globals.sql.zst").write_bytes(b"x") - manifest = {"status": "ok", "target": target, "type": ttype, "server_version": "18.6", "databases": ["app", "it's"], "size_bytes": 1} + manifest = { + "status": "ok", + "target": target, + "type": ttype, + "server_version": "18.6", + "databases": ["app", "it's"], + "size_bytes": 1, + "inventory": { + "databases": {"app": {"extensions": {"plpgsql": "1.0", "vector": "0.8.1"}}, "it's": {"extensions": {"plpgsql": "1.0"}}} + }, + } return store.finalize(partial, manifest) @@ -136,13 +154,14 @@ def test_inspect_env_output_is_shell_safe(tmp_path, capsys): bset = make_set(tmp_path) assert main(["inspect", str(tmp_path / "pg-main"), "--format", "env"]) == 0 values = dict(line.split("=", 1) for line in capsys.readouterr().out.splitlines()) - assert {k: shlex.split(v)[0] for k, v in values.items()} == { + assert {k: " ".join(shlex.split(v)) for k, v in values.items()} == { "DBB_TARGET": "pg-main", "DBB_TYPE": "postgres", "DBB_SET": bset.name, "DBB_SERVER_VERSION": "18.6", "DBB_DATABASES": "2", "DBB_SIZE_BYTES": "1", + "DBB_EXTENSIONS": "plpgsql vector", } From 1650dd1da07c7669fd09ce8445b9e849fea4e3b5 Mon Sep 17 00:00:00 2001 From: Alistair Young Date: Sat, 31 Oct 2026 15:49:23 -0500 Subject: [PATCH 2/4] Cover restoring sets made before extensions were recorded; release 0.3.0 Sets from 0.2.x and earlier have no extensions in their inventory. They restore and restore-test as before: there's no extension preflight or comparison, and the stock scratch image is used. Add unit tests for that, and say so in the restore docs. Checked end to end: sets made by the published 0.2.0 image, one with pg_trgm in use, pass scripts/restore-test.sh with this code. Co-Authored-By: Claude Opus 5.5 --- deploy/k8s/kustomization.yaml | 2 +- docs/restore.md | 6 +++--- pyproject.toml | 2 +- src/dbbackup/__init__.py | 2 +- tests/test_engines.py | 10 ++++++++++ tests/test_restoretest.py | 18 +++++++++++++++++- 6 files changed, 33 insertions(+), 7 deletions(-) diff --git a/deploy/k8s/kustomization.yaml b/deploy/k8s/kustomization.yaml index 42b2893..bd7bb99 100644 --- a/deploy/k8s/kustomization.yaml +++ b/deploy/k8s/kustomization.yaml @@ -23,7 +23,7 @@ configMapGenerator: images: - name: ghcr.io/arkane-systems/db-backup - newTag: "0.2.0" + newTag: "0.3.0" labels: - pairs: diff --git a/docs/restore.md b/docs/restore.md index 742462b..d96ec41 100644 --- a/docs/restore.md +++ b/docs/restore.md @@ -49,7 +49,7 @@ dbbackup restore [--set latest|] [-d ...] \ - **Existing databases.** The tool refuses to overwrite a database that already exists unless you pass `--force`. - **Existing accounts are left alone.** Users and roles that already exist on the destination keep their passwords and privileges. That includes the account you restore as, and a fresh server's own `root` and `healthcheck` accounts. Missing users and roles are created with their backed-up passwords and grants. - **The set is checked first.** The tool re-checks every file's checksum before it restores anything. -- **PostgreSQL extensions must be installed.** A destination without an extension the backup uses (pgvector's `vector`, say) is refused before anything is restored. The manifest's `inventory` lists each database's `extensions`, and `dbbackup inspect` shows them. +- **PostgreSQL extensions must be installed.** A destination without an extension the backup uses (pgvector's `vector`, say) is refused before anything is restored. The manifest's `inventory` lists each database's `extensions`, and `dbbackup inspect` shows them. Sets made by 0.2.x and earlier didn't record extensions, so they restore without this check, as they always did. ### From Kubernetes @@ -69,7 +69,7 @@ spec: securityContext: {runAsNonRoot: true, runAsUser: 1004, runAsGroup: 1004} containers: - name: restore - image: ghcr.io/arkane-systems/db-backup:0.2.0 + image: ghcr.io/arkane-systems/db-backup:0.3.0 args: [restore, postgres-main, --host, new-postgres.example.lan, --username, postgres, --password-env, ADMIN_PASSWORD] env: - name: ADMIN_PASSWORD @@ -86,7 +86,7 @@ spec: ```bash docker run --rm -v /mnt/db-backups:/backups:ro -v $PWD/config.yaml:/etc/dbbackup/config.yaml:ro \ - -e ADMIN_PASSWORD ghcr.io/arkane-systems/db-backup:0.2.0 \ + -e ADMIN_PASSWORD ghcr.io/arkane-systems/db-backup:0.3.0 \ restore mariadb-main --host new-mariadb.example.lan --username root --password-env ADMIN_PASSWORD ``` diff --git a/pyproject.toml b/pyproject.toml index 64f40b4..446ed92 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "dbbackup" -version = "0.2.0" +version = "0.3.0" description = "Containerized logical backups of MariaDB, PostgreSQL and MongoDB servers, with GFS retention" readme = "README.md" license = "MIT" diff --git a/src/dbbackup/__init__.py b/src/dbbackup/__init__.py index c49409c..5ac089a 100644 --- a/src/dbbackup/__init__.py +++ b/src/dbbackup/__init__.py @@ -1,3 +1,3 @@ """Logical backups of MariaDB, PostgreSQL and MongoDB servers, with GFS retention.""" -__version__ = "0.2.0" +__version__ = "0.3.0" diff --git a/tests/test_engines.py b/tests/test_engines.py index d5480ae..0a0ab72 100644 --- a/tests/test_engines.py +++ b/tests/test_engines.py @@ -230,5 +230,15 @@ def test_postgres_restore_refuses_a_server_without_the_extensions(monkeypatch, t assert rec.calls == [] # nothing restored +def test_postgres_restore_of_a_set_without_recorded_extensions(monkeypatch, tmp_path): + rec = Recorder(monkeypatch) + engine = engine_for(target("postgres")) + monkeypatch.setattr(type(engine), "list_databases", lambda self: ["postgres"]) + monkeypatch.setattr(type(engine), "query", lambda self, sql, db=None, **v: []) # not even plpgsql + manifest = {"contents": {"globals": "globals.sql.zst", "databases": {"app": "app.pgdump"}}, "inventory": {"databases": {"app": {}}}} + engine.restore(tmp_path, manifest, ["app"], force=False, include_globals=False) + assert [cmd[0] for cmd, _env in rec.calls] == ["pg_restore", "vacuumdb"] + + def test_ns_escape(): assert _ns_escape("a*b$c") == "a\\*b\\$c" diff --git a/tests/test_restoretest.py b/tests/test_restoretest.py index 73cdbc4..fd2548f 100644 --- a/tests/test_restoretest.py +++ b/tests/test_restoretest.py @@ -115,7 +115,7 @@ def test_users_not_listable_on_either_side(): assert result.errors == [] and result.warnings -def make_set(tmp_path, target="pg-main", ttype="postgres"): +def make_set(tmp_path, target="pg-main", ttype="postgres", *, extensions=True): store = Store(tmp_path) when = datetime(2026, 10, 9, 2, 15, tzinfo=UTC) partial = store.begin(target, when) @@ -131,6 +131,8 @@ def make_set(tmp_path, target="pg-main", ttype="postgres"): "databases": {"app": {"extensions": {"plpgsql": "1.0", "vector": "0.8.1"}}, "it's": {"extensions": {"plpgsql": "1.0"}}} }, } + if not extensions: # as made by 0.2.x and earlier + del manifest["inventory"] return store.finalize(partial, manifest) @@ -165,6 +167,20 @@ def test_inspect_env_output_is_shell_safe(tmp_path, capsys): } +def test_inspect_sets_without_recorded_extensions(tmp_path, capsys): + make_set(tmp_path, extensions=False) + assert main(["inspect", str(tmp_path / "pg-main"), "--format", "env"]) == 0 + assert "DBB_EXTENSIONS=''" in capsys.readouterr().out.splitlines() + assert main(["inspect", str(tmp_path / "pg-main")]) == 0 + assert "extensions:" not in capsys.readouterr().out + + +def test_extensions_not_compared_when_the_backup_recorded_none(): + expected = {"databases": {"app": {"objects": {}}}, "users": None} + actual = {"databases": {"app": {"objects": {}, "extensions": {"plpgsql": "1.0"}}}, "users": None} + assert compare_inventory(expected, actual, {"app"}).errors == [] + + def test_restore_test_needs_a_scratch_server(tmp_path): make_set(tmp_path) assert main(["restore-test", str(tmp_path / "pg-main")]) == 2 From 33d9b480c0d9a9b4072ad6b24a50526ca73d45f8 Mon Sep 17 00:00:00 2001 From: Alistair Young Date: Sat, 31 Oct 2026 15:53:56 -0500 Subject: [PATCH 3/4] Run CI's integration job on ubuntu-24.04, where MongoDB 8.x can start ubuntu-latest is now ubuntu-26.04, whose kernel is in the 6.19 to 7.0.13 range that MongoDB 8.x refuses to start on (SERVER-121912: its TCMalloc breaks with those kernels' rseq changes; 7.0.14 and 7.1 restore the old behaviour). Pin the integration job to ubuntu-24.04 until the newer runner has a fixed kernel. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/ci.yml | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 3e15078..c31946e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -23,7 +23,10 @@ jobs: - run: make test PYTHON=python integration: - runs-on: ubuntu-latest + # MongoDB 8.x refuses to start on Linux 6.19 to 7.0.13 (SERVER-121912), which + # includes the ubuntu-26.04 (ubuntu-latest) runner's kernel. Go back to + # ubuntu-latest once that runner has 7.0.14+ or 7.1+. + runs-on: ubuntu-24.04 needs: test steps: - uses: actions/checkout@v4 From 7904b9213445574dd07c7d1c4b6dd14f26fa25dd Mon Sep 17 00:00:00 2001 From: Alistair Young Date: Sat, 31 Oct 2026 16:03:28 -0500 Subject: [PATCH 4/4] Run CI's integration job on ubuntu-22.04 instead The ubuntu-24.04 runner now has the same 7.0.0 kernel as ubuntu-26.04, so MongoDB 8.x refuses to start there too. ubuntu-22.04 still has 6.8. Its runners are being retired by April 2027, so this is a stopgap until ubuntu-latest has kernel 7.0.14+ or 7.1+. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/ci.yml | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c31946e..6b9df3c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -23,10 +23,11 @@ jobs: - run: make test PYTHON=python integration: - # MongoDB 8.x refuses to start on Linux 6.19 to 7.0.13 (SERVER-121912), which - # includes the ubuntu-26.04 (ubuntu-latest) runner's kernel. Go back to - # ubuntu-latest once that runner has 7.0.14+ or 7.1+. - runs-on: ubuntu-24.04 + # MongoDB 8.x refuses to start on Linux 6.19 to 7.0.13 (SERVER-121912). The + # ubuntu-24.04 and ubuntu-26.04 runners both have 7.0.0; ubuntu-22.04 has 6.8. + # Go back to ubuntu-latest once it has 7.0.14+ or 7.1+, and before 22.04's + # runners are retired (April 2027). + runs-on: ubuntu-22.04 needs: test steps: - uses: actions/checkout@v4