Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ jobs:
steps:
- uses: actions/checkout@v4
- run: make integration
- run: make restore-test-check
- if: failure()
run: docker compose logs --no-color
- if: always()
Expand Down
9 changes: 8 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ PYTHON ?= .venv/bin/python
IMAGE ?= ghcr.io/arkane-systems/db-backup
TAG ?= dev

.PHONY: venv lint format test integration stack-up stack-down image
.PHONY: venv lint format test integration restore-test-check stack-up stack-down image

venv:
python3 -m venv .venv
Expand All @@ -25,6 +25,13 @@ test:
integration: stack-up
docker compose --profile tool run --rm --build tests

# Phase-2 restore testing, end to end: back up the compose.yaml servers, then run
# scripts/restore-test.sh on the result (scratch servers, restore, inventory check).
restore-test-check: stack-up
rm -rf .backups && mkdir -p .backups
env UID=$$(id -u) GID=$$(id -g) docker compose --profile tool run --rm --build dbbackup backup pg maria mongo mongo-filtered
scripts/restore-test.sh --build .backups

stack-up:
mkdir -p .backups
docker compose up -d --wait
Expand Down
8 changes: 8 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ A containerized database backup solution: logical backups of any number of **Mar
- **Isolates failures.** One server's failure never stops the others. A target is pruned only after one of its own backups succeeds, and the newest good set is always kept.
- **Reports to MQTT,** with Home Assistant discovery. Failures, and backups that have *stopped happening*, can raise [Alert Redux](https://github.com/arkane-systems/ha-alert-redux) alerts; see [docs/home-assistant.md](docs/home-assistant.md).
- **Restores with one command,** `dbbackup restore`. Every set can also be restored with stock client tools; see [docs/restore.md](docs/restore.md).
- **Tests restores locally.** `scripts/restore-test.sh` restores sets into throwaway Docker servers of the matching version, and checks every database, table, collection, user and role against what the backup recorded.

Supported servers: MariaDB 11.4+, PostgreSQL 18+, MongoDB 8.3+ (as a replica set, for `--oplog`). Older versions may well work: the tool logs a warning and carries on. The image ships the PostgreSQL 18 client, the MariaDB 11.8 client and MongoDB Database Tools 100.19. It is built for linux/amd64 only, because MongoDB publishes its Debian tools packages only for x86-64.

Expand Down Expand Up @@ -64,6 +65,7 @@ Secrets never go in the file. Each one is named by an environment variable (`pas
| `uri_env`/`uri_file` | mongodb target | (none) | A full connection URI, instead of host/port/username. |
| `uri_options` | mongodb target | (none) | Extra URI options when connecting by host, e.g. `replicaSet: rs0`. |
| `dump_args` | target | (none) | Extra arguments for each per-database dump command. |
| `exact_counts` | `defaults` or target | `false` | Record exact `COUNT(*)` row counts in each set's inventory, for restore tests to check. This scans every table at backup time. Otherwise the cheap estimates are recorded; MariaDB's InnoDB estimates are too unreliable, so they're left out. |
| `mqtt` | top level | (none) | `host`, `port`, `username`, `password_env`/`password_file`, `tls`, `topic_prefix` (`dbbackup`), `ha_discovery` (`false`), `discovery_prefix` (`homeassistant`). |

For MongoDB, `include` or `exclude` changes how the backup is made. `mongodump` can't leave databases out of a whole-instance dump, so such a target gets one archive per database instead. Those dumps can't use `--oplog`, so they are consistent only per collection, and the manifest records a warning saying so. Leave both unset for point-in-time consistency.
Expand Down Expand Up @@ -129,8 +131,13 @@ dbbackup verify [TARGET ...] [--set S | --all] re-check checksums and dump in
dbbackup prune [TARGET ...] [--dry-run] apply retention without backing up
dbbackup restore TARGET [--set S] [-d DB ...] [--host H --port P --username U --password-env VAR | --uri-env VAR]
[--no-globals] [--force]
dbbackup inspect PATH [--set S] [--format text|env|json] describe a set (PATH: a set or target directory; no config needed)
dbbackup restore-test PATH [--set S] (--host H --username U --password-env VAR | --uri-env VAR) [--tolerance F]
restore a set into a scratch server and check it against the backup's inventory
```

To test restores routinely, without touching real servers or the cluster, run `scripts/restore-test.sh /path/to/backups` on any machine with Docker. It restores each target's latest set into a throwaway server of the matching version, checks the result, and cleans up. See [docs/restore.md](docs/restore.md#testing-restores).

Exit status: 0 on success, 1 if anything failed, 2 for configuration or usage errors.

## Development
Expand All @@ -139,6 +146,7 @@ Exit status: 0 on success, 1 if anything failed, 2 for configuration or usage er
make venv # .venv with the package, pytest and ruff
make lint test # ruff, and unit tests (no servers needed)
make integration # start compose.yaml's servers, then back up and restore them for real
make restore-test-check # back up those servers, then run scripts/restore-test.sh on the result
make stack-down # remove the test servers
```

Expand Down
92 changes: 92 additions & 0 deletions compose.restore-test.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# Scratch servers for restore testing; driven by scripts/restore-test.sh, which
# starts one scratch server per target in its own throwaway compose project,
# restores into it, checks the result, and removes everything (`down -v`).
#
# Set by the script:
# BACKUP_ROOT the backup share (mounted read-only)
# SCRATCH_IMAGE the scratch server's image, matching the backed-up server's version
# SCRATCH_STORAGE "volume" (default: an anonymous volume) or "tmpfs" (RAM: faster, but the
# restored data must fit in memory)
# DBBACKUP_IMAGE the db-backup image to test with
# RUN_AS uid:gid to read the backups as (the backup root's owner)
#
# Nothing is published on the host: the scratch servers are reachable only on
# the project's own network.

x-scratch-env: &scratch-env
# Throwaway credentials for throwaway servers.
SCRATCH_PASSWORD: restore-test
SCRATCH_MONGO_URI: mongodb://root:restore-test@scratch-mongodb:27017/?authSource=admin&directConnection=true

services:
scratch-postgres:
profiles: [postgres]
image: ${SCRATCH_IMAGE:-postgres:18}
environment:
POSTGRES_PASSWORD: restore-test
volumes:
- type: ${SCRATCH_STORAGE:-volume}
target: /var/lib/postgresql
healthcheck:
test: [CMD, pg_isready, -U, postgres]
interval: 2s
retries: 60

scratch-mariadb:
profiles: [mariadb]
image: ${SCRATCH_IMAGE:-mariadb:11.4}
environment:
MARIADB_ROOT_PASSWORD: restore-test
volumes:
- type: ${SCRATCH_STORAGE:-volume}
target: /var/lib/mysql
healthcheck:
test: [CMD, healthcheck.sh, --connect, --innodb_initialized]
interval: 2s
retries: 60

scratch-mongodb:
profiles: [mongodb]
image: ${SCRATCH_IMAGE:-mongo:8.3}
environment:
MONGO_INITDB_ROOT_USERNAME: root
MONGO_INITDB_ROOT_PASSWORD: restore-test
# A single-node replica set, so --oplogReplay restores behave as in production.
# Replica sets with auth need a keyfile; make a throwaway one.
entrypoint:
- bash
- -c
- |
openssl rand -base64 756 > /tmp/keyfile && chmod 400 /tmp/keyfile && chown mongodb:mongodb /tmp/keyfile
exec docker-entrypoint.sh "$$@"
- --
command: [mongod, --replSet, rs0, --keyFile, /tmp/keyfile, --bind_ip_all]
volumes:
- type: ${SCRATCH_STORAGE:-volume}
target: /data/db
- type: ${SCRATCH_STORAGE:-volume}
target: /data/configdb
healthcheck:
test:
- CMD
- mongosh
- --quiet
- -u
- root
- -p
- restore-test
- --eval
- "try { rs.status().myState === 1 ? 0 : quit(1) } catch (e) { rs.initiate({_id: 'rs0', members: [{_id: 0, host: 'scratch-mongodb:27017'}]}); quit(1) }"
interval: 3s
retries: 60

# The tool, with the backups mounted read-only; always run with --no-deps.
restore-test:
profiles: [tool]
image: ${DBBACKUP_IMAGE:-ghcr.io/arkane-systems/db-backup:latest}
user: ${RUN_AS:-1004:1004}
read_only: true
tmpfs: [/tmp]
environment: *scratch-env
volumes:
- ${BACKUP_ROOT:?BACKUP_ROOT must be set}:/backups:ro
3 changes: 3 additions & 0 deletions deploy/k8s/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,9 @@ defaults:
retry_delay_s: 60
stale_partial_hours: 24 # abandoned .partial sets older than this are removed
lock_stale_hours: 24 # a lock older than this is assumed abandoned
# Record exact row counts (a COUNT(*) of every table, each night) for restore
# tests to check, rather than cheap estimates. MariaDB/InnoDB has no usable estimate.
exact_counts: false

mqtt:
host: mqtt.example.lan # CHANGE ME
Expand Down
2 changes: 1 addition & 1 deletion deploy/k8s/kustomization.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ configMapGenerator:

images:
- name: ghcr.io/arkane-systems/db-backup
newTag: "0.1.1"
newTag: "0.2.0"

labels:
- pairs:
Expand Down
58 changes: 55 additions & 3 deletions docs/restore.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ This is the disaster-recovery runbook. It covers two cases:
- **With the tool**: `dbbackup restore`, the quick path.
- **With stock tools only**: `psql`, `pg_restore`, `mariadb`, `mongorestore` and `zstd`, for when the tool or its image isn't available. Nothing in a backup set needs this project to read it.

Test restores before you need them. A backup that has never been restored is only a hope.
Test restores before you need them. A backup that has never been restored is only a hope. [Testing restores](#testing-restores) shows how to do it routinely, on a workstation.

## Finding a backup set

Expand Down Expand Up @@ -68,7 +68,7 @@ spec:
securityContext: {runAsNonRoot: true, runAsUser: 1004, runAsGroup: 1004}
containers:
- name: restore
image: ghcr.io/arkane-systems/db-backup:0.1.1
image: ghcr.io/arkane-systems/db-backup:0.2.0
args: [restore, postgres-main, --host, new-postgres.example.lan, --username, postgres, --password-env, ADMIN_PASSWORD]
env:
- name: ADMIN_PASSWORD
Expand All @@ -85,7 +85,7 @@ spec:

```bash
docker run --rm -v /mnt/db-backups:/backups:ro -v $PWD/config.yaml:/etc/dbbackup/config.yaml:ro \
-e ADMIN_PASSWORD ghcr.io/arkane-systems/db-backup:0.1.1 \
-e ADMIN_PASSWORD ghcr.io/arkane-systems/db-backup:0.2.0 \
restore mariadb-main --host new-mariadb.example.lan --username root --password-env ADMIN_PASSWORD
```

Expand Down Expand Up @@ -169,3 +169,55 @@ zstd -dc admin.archive.zst | mongorestore --uri '...' --archive
- **Point applications at the new server,** and check they can log in. Restored application accounts keep their old passwords.
- **Check the restore against the backup's inventory.** The manifest's `inventory` lists every table and collection with an approximate row count, plus every user and role, as they were at backup time. It's a quick sanity check that nothing went missing.
- **Update the backup config** if the server's address changed, so tonight's backup covers the new server.

## Testing restores

`scripts/restore-test.sh` proves a backup can actually be restored, without touching any real server and without using the cluster. It runs on any machine with Docker and the backup share mounted (or a copy of it):

```bash
scripts/restore-test.sh /mnt/db-backups # the latest set of every target
scripts/restore-test.sh /mnt/db-backups postgres-main # just one target
scripts/restore-test.sh --set 20261009T021500Z /mnt/db-backups mariadb-main
```

For each target, the script:

1. Reads the set's manifest, using the db-backup image, and starts a throwaway scratch server matching the backed-up server's version: `postgres:<major>`, `mariadb:<major.minor>` or `mongo:<major.minor>`. MongoDB runs as a single-node replica set, so the oplog is replayed as it would be in production. Each scratch server has its own compose project and network, and nothing is published on the host.
2. Runs `dbbackup restore-test`, which:
- re-checks every file's checksum, and the dumps' integrity
- restores the set exactly as `dbbackup restore` would, users and roles included
- compares the restored server with the inventory recorded at backup time
3. Removes the scratch server and its data (unless you pass `--keep`), and goes on to the next target.

The comparison treats these as **errors**:

- a database missing from the restored server
- a missing table, view, materialized view or collection
- a missing user or role
- an object restored as a different kind (say, a view that came back as a table)

Row-count differences are only **warnings**. The restored server is always counted exactly; what it's compared with depends on the target's `exact_counts` setting:

- **`exact_counts: true`.** The set recorded a `COUNT(*)` of every table, just before the dump, and the counts must match exactly. Writes between the count and the dump's snapshot show up as small differences.
- **The default.** The set recorded the servers' cheap estimates (`pg_class.reltuples`, `estimatedDocumentCount`, and Aria/MyISAM row counts), and a difference of more than 25% (and more than 100 rows) is reported. InnoDB's estimate (`TABLE_ROWS`) can be off by orders of magnitude, for example straight after a bulk load, so MariaDB InnoDB tables have no recorded count: only their presence is checked, unless you set `exact_counts`. Sets made by 0.1.x recorded those InnoDB estimates anyway; their MariaDB counts are skipped, with a note saying why.

The run ends with a summary table, including how long each restore took, which is a rough figure for how long recovery would take. The script exits non-zero if any target fails.

| Option | Effect |
|---|---|
| `-s`, `--set SET` | Test this set instead of the latest. |
| `-i`, `--image IMAGE` | The db-backup image to use (default `$DBBACKUP_IMAGE`, else `ghcr.io/arkane-systems/db-backup:latest`). |
| `-b`, `--build` | Build the image from the checkout instead. |
| `-t`, `--tmpfs` | Keep the scratch servers' data in RAM. Faster, but the restored data must fit in memory. |
| `-k`, `--keep` | Leave each scratch server running afterwards, to look around in. The script prints how to remove it. |

To use a scratch image other than the one matching the recorded version, set `SCRATCH_IMAGE_POSTGRES`, `SCRATCH_IMAGE_MARIADB` or `SCRATCH_IMAGE_MONGODB`. For example, `SCRATCH_IMAGE_POSTGRES=postgres:19` rehearses an upgrade.

The backups are mounted read-only, and are read as the backup root's owner (`pg_dump`'s directory archives are private to their owner).

To restore-test one set by hand against a server you've started yourself:

```bash
dbbackup inspect /backups/postgres-main # what's in the latest set
dbbackup restore-test /backups/postgres-main --host scratch --username postgres --password-env SCRATCH_PASSWORD
```
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"

[project]
name = "dbbackup"
version = "0.1.1"
version = "0.2.0"
description = "Containerized logical backups of MariaDB, PostgreSQL and MongoDB servers, with GFS retention"
readme = "README.md"
license = "MIT"
Expand Down
Loading
Loading