Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,11 @@ jobs:
- run: make test PYTHON=python

integration:
runs-on: ubuntu-latest
# MongoDB 8.x refuses to start on Linux 6.19 to 7.0.13 (SERVER-121912). The
# ubuntu-24.04 and ubuntu-26.04 runners both have 7.0.0; ubuntu-22.04 has 6.8.
# Go back to ubuntu-latest once it has 7.0.14+ or 7.1+, and before 22.04's
# runners are retired (April 2027).
runs-on: ubuntu-22.04
needs: test
steps:
- uses: actions/checkout@v4
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ A containerized database backup solution: logical backups of any number of **Mar
- **Isolates failures.** One server's failure never stops the others. A target is pruned only after one of its own backups succeeds, and the newest good set is always kept.
- **Reports to MQTT,** with Home Assistant discovery. Failures, and backups that have *stopped happening*, can raise [Alert Redux](https://github.com/arkane-systems/ha-alert-redux) alerts; see [docs/home-assistant.md](docs/home-assistant.md).
- **Restores with one command,** `dbbackup restore`. Every set can also be restored with stock client tools; see [docs/restore.md](docs/restore.md).
- **Tests restores locally.** `scripts/restore-test.sh` restores sets into throwaway Docker servers of the matching version, and checks every database, table, collection, user and role against what the backup recorded.
- **Tests restores locally.** `scripts/restore-test.sh` restores sets into throwaway Docker servers of the matching version, and checks every database, table, collection, extension, user and role against what the backup recorded. PostgreSQL sets that use pgvector are tested in pgvector's image.

Supported servers: MariaDB 11.4+, PostgreSQL 18+, MongoDB 8.3+ (as a replica set, for `--oplog`). Older versions may well work: the tool logs a warning and carries on. The image ships the PostgreSQL 18 client, the MariaDB 11.8 client and MongoDB Database Tools 100.19. It is built for linux/amd64 only, because MongoDB publishes its Debian tools packages only for x86-64.

Expand Down
2 changes: 1 addition & 1 deletion deploy/k8s/kustomization.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ configMapGenerator:

images:
- name: ghcr.io/arkane-systems/db-backup
newTag: "0.2.0"
newTag: "0.3.0"

labels:
- pairs:
Expand Down
12 changes: 8 additions & 4 deletions docs/restore.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ dbbackup restore <target> [--set latest|<timestamp>] [-d <database> ...] \
- **Existing databases.** The tool refuses to overwrite a database that already exists unless you pass `--force`.
- **Existing accounts are left alone.** Users and roles that already exist on the destination keep their passwords and privileges. That includes the account you restore as, and a fresh server's own `root` and `healthcheck` accounts. Missing users and roles are created with their backed-up passwords and grants.
- **The set is checked first.** The tool re-checks every file's checksum before it restores anything.
- **PostgreSQL extensions must be installed.** A destination without an extension the backup uses (pgvector's `vector`, say) is refused before anything is restored. The manifest's `inventory` lists each database's `extensions`, and `dbbackup inspect` shows them. Sets made by 0.2.x and earlier didn't record extensions, so they restore without this check, as they always did.

### From Kubernetes

Expand All @@ -68,7 +69,7 @@ spec:
securityContext: {runAsNonRoot: true, runAsUser: 1004, runAsGroup: 1004}
containers:
- name: restore
image: ghcr.io/arkane-systems/db-backup:0.2.0
image: ghcr.io/arkane-systems/db-backup:0.3.0
args: [restore, postgres-main, --host, new-postgres.example.lan, --username, postgres, --password-env, ADMIN_PASSWORD]
env:
- name: ADMIN_PASSWORD
Expand All @@ -85,7 +86,7 @@ spec:

```bash
docker run --rm -v /mnt/db-backups:/backups:ro -v $PWD/config.yaml:/etc/dbbackup/config.yaml:ro \
-e ADMIN_PASSWORD ghcr.io/arkane-systems/db-backup:0.2.0 \
-e ADMIN_PASSWORD ghcr.io/arkane-systems/db-backup:0.3.0 \
restore mariadb-main --host new-mariadb.example.lan --username root --password-env ADMIN_PASSWORD
```

Expand All @@ -97,6 +98,8 @@ Use client tools at least as new as the server that was backed up. The manifest'

A set contains `globals.sql.zst` (roles and tablespaces) and one `<db>.pgdump/` directory per database. Each directory is a `pg_dump --format=directory` archive.

The new server needs every extension the databases use installed, such as pgvector (the `pgvector/pgvector` images have it). The manifest's `inventory` lists them per database. A missing one makes `pg_restore` fail on `CREATE EXTENSION`, and then on everything that depends on it.

1. **Roles first.** Restored objects are owned by these roles and granted to them.

```bash
Expand Down Expand Up @@ -182,7 +185,7 @@ scripts/restore-test.sh --set 20261009T021500Z /mnt/db-backups mariadb-main

For each target, the script:

1. Reads the set's manifest, using the db-backup image, and starts a throwaway scratch server matching the backed-up server's version: `postgres:<major>`, `mariadb:<major.minor>` or `mongo:<major.minor>`. MongoDB runs as a single-node replica set, so the oplog is replayed as it would be in production. Each scratch server has its own compose project and network, and nothing is published on the host.
1. Reads the set's manifest, using the db-backup image, and starts a throwaway scratch server matching the backed-up server's version: `postgres:<major>`, `mariadb:<major.minor>` or `mongo:<major.minor>`. PostgreSQL sets that use pgvector get `pgvector/pgvector:pg<major>` instead, since the stock image only has the contrib extensions. MongoDB runs as a single-node replica set, so the oplog is replayed as it would be in production. Each scratch server has its own compose project and network, and nothing is published on the host.
2. Runs `dbbackup restore-test`, which:
- re-checks every file's checksum, and the dumps' integrity
- restores the set exactly as `dbbackup restore` would, users and roles included
Expand All @@ -194,6 +197,7 @@ The comparison treats these as **errors**:
- a database missing from the restored server
- a missing table, view, materialized view or collection
- a missing user or role
- a missing PostgreSQL extension
- an object restored as a different kind (say, a view that came back as a table)

Row-count differences are only **warnings**. The restored server is always counted exactly; what it's compared with depends on the target's `exact_counts` setting:
Expand All @@ -211,7 +215,7 @@ The run ends with a summary table, including how long each restore took, which i
| `-t`, `--tmpfs` | Keep the scratch servers' data in RAM. Faster, but the restored data must fit in memory. |
| `-k`, `--keep` | Leave each scratch server running afterwards, to look around in. The script prints how to remove it. |

To use a scratch image other than the one matching the recorded version, set `SCRATCH_IMAGE_POSTGRES`, `SCRATCH_IMAGE_MARIADB` or `SCRATCH_IMAGE_MONGODB`. For example, `SCRATCH_IMAGE_POSTGRES=postgres:19` rehearses an upgrade.
To use a scratch image other than the one matching the recorded version, set `SCRATCH_IMAGE_POSTGRES`, `SCRATCH_IMAGE_MARIADB` or `SCRATCH_IMAGE_MONGODB`. For example, `SCRATCH_IMAGE_POSTGRES=postgres:19` rehearses an upgrade. A set that uses another extension the stock image lacks, such as PostGIS, needs an image that has it (`SCRATCH_IMAGE_POSTGRES=postgis/postgis:18-3.6`): without one, the restore stops before it starts, naming the missing extensions.

The backups are mounted read-only, and are read as the backup root's owner (`pg_dump`'s directory archives are private to their owner).

Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"

[project]
name = "dbbackup"
version = "0.2.0"
version = "0.3.0"
description = "Containerized logical backups of MariaDB, PostgreSQL and MongoDB servers, with GFS retention"
readme = "README.md"
license = "MIT"
Expand Down
14 changes: 9 additions & 5 deletions scripts/restore-test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,8 @@
# -h, --help show this help
#
# Scratch images follow the recorded server version (postgres:<major>,
# mariadb:<major.minor>, mongo:<major.minor>); override them with
# mariadb:<major.minor>, mongo:<major.minor>), and PostgreSQL sets that use
# pgvector get pgvector/pgvector:pg<major>; override them with
# SCRATCH_IMAGE_POSTGRES, SCRATCH_IMAGE_MARIADB or SCRATCH_IMAGE_MONGODB.
#
# Exits non-zero if any target fails.
Expand Down Expand Up @@ -105,7 +106,7 @@ for target in "${targets[@]}"; do
cleanup
continue
fi
DBB_TYPE="" DBB_SET="" DBB_SERVER_VERSION=""
DBB_TYPE="" DBB_SET="" DBB_SERVER_VERSION="" DBB_EXTENSIONS=""
while IFS= read -r line; do
[[ $line =~ ^DBB_[A-Z_]+= ]] && eval "$line" # values are shell-quoted by dbbackup
done <<<"$info"
Expand All @@ -115,7 +116,10 @@ for target in "${targets[@]}"; do
minor=${version#*.}; minor=${minor%%.*}
case $DBB_TYPE in
postgres)
scratch_image=${SCRATCH_IMAGE_POSTGRES:-postgres:$major}
# The stock image has only the contrib extensions; pgvector's image adds vector.
default_image=postgres:$major
[[ " $DBB_EXTENSIONS " == *" vector "* ]] && default_image=pgvector/pgvector:pg$major
scratch_image=${SCRATCH_IMAGE_POSTGRES:-$default_image}
conn=(--host scratch-postgres --username postgres --password-env SCRATCH_PASSWORD) ;;
mariadb)
scratch_image=${SCRATCH_IMAGE_MARIADB:-mariadb:$major.$minor}
Expand Down Expand Up @@ -172,9 +176,9 @@ for target in "${targets[@]}"; do
done

echo
printf '%-24s %-17s %-16s %-36s %s\n' TARGET SET SCRATCH RESULT RESTORE
printf '%-24s %-17s %-24s %-36s %s\n' TARGET SET SCRATCH RESULT RESTORE
for row in "${results[@]}"; do
IFS='|' read -r t s i r d <<<"$row"
printf '%-24s %-17s %-16s %-36s %s\n' "$t" "$s" "$i" "$r" "$d"
printf '%-24s %-17s %-24s %-36s %s\n' "$t" "$s" "$i" "$r" "$d"
done
exit $failed
2 changes: 1 addition & 1 deletion src/dbbackup/__init__.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
"""Logical backups of MariaDB, PostgreSQL and MongoDB servers, with GFS retention."""

__version__ = "0.2.0"
__version__ = "0.3.0"
4 changes: 4 additions & 0 deletions src/dbbackup/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -247,6 +247,7 @@ def cmd_inspect(args) -> int:
if not bset.ok:
raise StorageError(f"{bset.path} has no valid manifest")
m = bset.manifest
extensions = sorted({ext for db in m.get("inventory", {}).get("databases", {}).values() for ext in db.get("extensions", {})})
if args.format == "json":
print(json.dumps({k: v for k, v in m.items() if k != "files"}, indent=2, sort_keys=True))
elif args.format == "env":
Expand All @@ -257,13 +258,16 @@ def cmd_inspect(args) -> int:
"DBB_SERVER_VERSION": m.get("server_version", ""),
"DBB_DATABASES": str(len(m.get("databases", []))),
"DBB_SIZE_BYTES": str(m.get("size_bytes", "")),
"DBB_EXTENSIONS": " ".join(extensions),
}
for key, value in values.items():
print(f"{key}={shlex.quote(value)}")
else:
print(f"target: {m['target']} ({m['type']} {m.get('server_version', '?')}, {m.get('host') or 'URI'})")
print(f"set: {bset.name} finished {m.get('finished')} ({human_size(m.get('size_bytes'))}, {m.get('duration_s')}s)")
print(f"contents: {', '.join(m.get('databases', [])) or '(no databases)'}")
if extensions:
print(f"extensions: {', '.join(extensions)}")
for warning in m.get("warnings", []):
print(f"warning: {warning}")
return EXIT_OK
Expand Down
13 changes: 12 additions & 1 deletion src/dbbackup/engines/postgres.py
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,9 @@ def inventory(self, databases: list[str], *, exact: bool = False) -> dict:
sql = " UNION ALL ".join(f"SELECT {i}, count(*) FROM {qualified}" for i, (_, qualified) in enumerate(countable))
for i, count in self.query(sql, db=db):
objects[countable[int(i)][0]]["rows"] = int(count)
result[db] = {"objects": objects}
# A restore needs these installed on the destination (pgvector, PostGIS, ...).
extensions = dict(self.query("SELECT extname, extversion FROM pg_extension ORDER BY extname", db=db))
result[db] = {"objects": objects, "extensions": extensions}
users = [r[0] for r in self.query("SELECT rolname FROM pg_roles WHERE rolname !~ '^pg_' ORDER BY rolname")]
return {"counts": "exact" if exact else "estimated", "databases": result, "users": users}

Expand Down Expand Up @@ -194,6 +196,15 @@ def restore(self, set_dir: Path, manifest: dict, databases: list[str], *, force:
if clashes and not force:
raise EngineError(f"database(s) already exist on {self.conn.host}: {', '.join(clashes)} (use --force to replace them)")

# Fail before changing anything, rather than part-way through pg_restore.
recorded = manifest.get("inventory", {}).get("databases", {})
needed = {ext for db in databases for ext in recorded.get(db, {}).get("extensions", {})}
unavailable = needed - {r[0] for r in self.query("SELECT name FROM pg_available_extensions")}
if unavailable:
raise EngineError(
f"{self.conn.host} doesn't have extension(s) the backup uses: {', '.join(sorted(unavailable))} (install them first)"
)

if include_globals:
# Roles first: restored objects are owned by, and granted to, them.
log.info("%s: restoring roles and tablespaces", self.target.name)
Expand Down
7 changes: 5 additions & 2 deletions src/dbbackup/restoretest.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
"""Restore testing: restore a set into a scratch server, then check the result
against the inventory its manifest recorded at backup time.

Anything missing (a database, table, view, collection, user or role) is an error.
Anything missing (a database, table, view, collection, extension, user or role) is an error.
Row-count differences are only warnings. The restored side is always counted
exactly (it's a scratch server); what it's compared with depends on the backup:

Expand Down Expand Up @@ -62,7 +62,10 @@ def compare_inventory(
if db not in present:
result.errors.append(f"database {db} is missing")
continue
restored = actual.get("databases", {}).get(db, {}).get("objects", {})
restored_db = actual.get("databases", {}).get(db, {})
for ext in sorted(set(info.get("extensions", {})) - set(restored_db.get("extensions", {}))):
result.errors.append(f"{db}: extension {ext} is missing")
restored = restored_db.get("objects", {})
for name, obj in sorted(info.get("objects", {}).items()):
result.objects_checked += 1
got = restored.get(name)
Expand Down
2 changes: 2 additions & 0 deletions tests/integration/seed/postgres/01-seed.sql
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ CREATE DATABASE "odd.name/db" OWNER app_owner;
CREATE DATABASE scratch;

\connect app1
CREATE EXTENSION pg_trgm;
SET ROLE app_owner;
CREATE SCHEMA billing;
CREATE TABLE public.customers (id serial PRIMARY KEY, name text NOT NULL, created timestamptz DEFAULT now());
Expand All @@ -25,6 +26,7 @@ CREATE TABLE billing.invoices (
);
INSERT INTO billing.invoices (customer_id, amount) SELECT (g % 5000) + 1, g * 1.5 FROM generate_series(1, 20000) g;
CREATE INDEX invoices_customer ON billing.invoices (customer_id);
CREATE INDEX customers_name_trgm ON public.customers USING gin (name gin_trgm_ops);
CREATE VIEW billing.big_invoices AS SELECT * FROM billing.invoices WHERE amount > 10000;
CREATE MATERIALIZED VIEW billing.totals AS SELECT customer_id, sum(amount) AS total FROM billing.invoices GROUP BY customer_id;
CREATE FUNCTION billing.invoice_count(c int) RETURNS bigint LANGUAGE sql AS $$ SELECT count(*) FROM billing.invoices WHERE customer_id = c $$;
Expand Down
2 changes: 2 additions & 0 deletions tests/integration/test_end_to_end.py
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,7 @@ def test_postgres_manifest():
objects = m["inventory"]["databases"]["app1"]["objects"]
assert objects["billing.invoices"] == {"kind": "table", "rows": 20000}
assert objects["billing.big_invoices"]["kind"] == "view"
assert set(m["inventory"]["databases"]["app1"]["extensions"]) == {"pg_trgm", "plpgsql"}
assert {"app_owner", "app_reader", "reporting"} <= set(m["inventory"]["users"])
assert m["warnings"] == []

Expand Down Expand Up @@ -220,6 +221,7 @@ def test_restore_postgres():
RESTORE_PG, "SET ROLE reporting; SELECT count(*) FROM billing.totals", db="app1", user="app_reader", password="reader-pw"
) == ["5000"]
assert pg(RESTORE_PG, "SHOW work_mem", db="app1") == ["8MB"]
assert pg(RESTORE_PG, "SELECT extname FROM pg_extension WHERE extname = 'pg_trgm'", db="app1") == ["pg_trgm"]
# ...and the restoring account's own password was left alone (pg() connects with it).
assert pg(RESTORE_PG, "SELECT 1") == ["1"]

Expand Down
26 changes: 25 additions & 1 deletion tests/test_engines.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
from dbbackup import proc
from dbbackup.config import parse_config
from dbbackup.engines import engine_for
from dbbackup.engines.base import safe_filename, version_tuple
from dbbackup.engines.base import EngineError, safe_filename, version_tuple
from dbbackup.engines.mariadb import option_file_value, statement_account, without_accounts
from dbbackup.engines.mongodb import _ns_escape
from dbbackup.engines.postgres import conninfo_dbname, without_roles
Expand Down Expand Up @@ -216,5 +216,29 @@ def test_select_databases(monkeypatch):
engine.select_databases()


def test_postgres_restore_refuses_a_server_without_the_extensions(monkeypatch, tmp_path):
rec = Recorder(monkeypatch)
engine = engine_for(target("postgres"))
monkeypatch.setattr(type(engine), "list_databases", lambda self: ["postgres"])
monkeypatch.setattr(type(engine), "query", lambda self, sql, db=None, **v: [["plpgsql"], ["pg_trgm"]])
manifest = {
"contents": {"globals": "globals.sql.zst", "databases": {"app": "app.pgdump"}},
"inventory": {"databases": {"app": {"extensions": {"plpgsql": "1.0", "vector": "0.8.1"}}}},
}
with pytest.raises(EngineError, match="doesn't have extension\\(s\\) the backup uses: vector"):
engine.restore(tmp_path, manifest, ["app"], force=False, include_globals=True)
assert rec.calls == [] # nothing restored


def test_postgres_restore_of_a_set_without_recorded_extensions(monkeypatch, tmp_path):
rec = Recorder(monkeypatch)
engine = engine_for(target("postgres"))
monkeypatch.setattr(type(engine), "list_databases", lambda self: ["postgres"])
monkeypatch.setattr(type(engine), "query", lambda self, sql, db=None, **v: []) # not even plpgsql
manifest = {"contents": {"globals": "globals.sql.zst", "databases": {"app": "app.pgdump"}}, "inventory": {"databases": {"app": {}}}}
engine.restore(tmp_path, manifest, ["app"], force=False, include_globals=False)
assert [cmd[0] for cmd, _env in rec.calls] == ["pg_restore", "vacuumdb"]


def test_ns_escape():
assert _ns_escape("a*b$c") == "a\\*b\\$c"
Loading
Loading