Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ docker compose run --rm dev cargo test --workspace
- `pgbr-protocol` — JSON-line `Request` / `Response` message types + `read_message`/`write_message` codec
- `pgbr-postgres` — `crc32c_one`, `version` registry (PG 9.6 .. 18), `control` (`pg_control` header + per-version field offsets), `page` (`pg_checksum_page`)
- `pgbr-info` — on-disk info files: `InfoArchive`, `InfoBackup`, `Manifest`, shared INI+SHA-1 `format`
- `pgbr-command` — every command + the dispatch entry (`dispatch_multi` is the real router, taking all configured repositories; `dispatch` is the single-repo convenience wrapper that calls it with `[(1, repo)]`): backup (full/diff/incr), restore (+ delta + reference resolution), archive-push/get, expire (backup + WAL retention), verify, check, info, stanza-create/delete/upgrade, repo-ls/get/put/rm, annotate, manifest, start/stop, server/server-ping (TCP + TLS), help, version; plus the shared `pipeline::RepoTransform` (compress + encrypt)
- `pgbr-command` — every command + the dispatch entry (`dispatch_multi` is the real router, taking all configured repositories; `dispatch` is the single-repo convenience wrapper that calls it with `[(1, repo)]`): backup (full/diff/incr), restore (+ delta + reference resolution), archive-push/get, expire (backup + WAL retention), verify, check, info, stanza-create/delete/upgrade, repo-ls/get/put/rm, annotate, manifest, start/stop, server/server-ping (TCP + TLS), repo-sync (repo-to-repo mirroring of backups + WAL), help, version; plus the shared `pipeline::RepoTransform` (compress + encrypt)
- `pgbr-cli` — the `pgbackrust` binary: parse argv → load `config.yaml` + `pgbackrust.conf` → resolve → `pgbr_command::dispatch`

## Adding a configuration option
Expand Down Expand Up @@ -102,7 +102,7 @@ Beyond the in-crate `cargo test` suite, `integration/` drives the **release bina

- **Build the artifact first:** `./integration/build-binary.sh` compiles `pgbr-cli` in the **`build`** docker-compose service (Debian 12 bookworm, glibc 2.36 — *not* the trixie `dev`/`cargo` image) and drops a portable ELF at `integration/artifacts/pgbackrust`. glibc is backward- but not forward-compatible, so building on the lowest supported glibc yields **one** binary that runs on both the bookworm Vagrant VMs and the trixie Docker nodes. A trixie-built binary dies on the VMs with `GLIBC_2.39 not found`, so don't build the integration artifact with `dev`/`cargo`.
- **Docker topology** (`integration/docker/docker-compose.yml`) — a 3-node cluster (`principal` primary / `secondaire` standby / `depot` repo) plus a `minio` S3 endpoint, sharing an SSH key so the `postgres` user can hop between nodes. The binary is **bind-mounted** into each node (not COPYed — under Docker Desktop's WSL2 image store the COPYed layer is intermittently not materialised). This is what local/CI validation uses.
- **Scenarios** (`integration/scenarios/`) — 11 numbered scripts (`01-local-minimal` … `11-s3`: local, remote-pull SSH, PITR, encryption, standby, tablespaces, async queuing, multi-repo, TLS, bundling/block, S3) over shared helpers in `_lib.sh`. Run them with `./integration/scenarios/run-all.sh` (optionally filter: `run-all.sh 01 03`). Each scenario runs against a freshly `down -v`'d topology for isolation; a failing scenario is recorded and the loop continues rather than aborting.
- **Scenarios** (`integration/scenarios/`) — 12 numbered scripts (`01-local-minimal` … `12-repo-sync`: local, remote-pull SSH, PITR, encryption, standby, tablespaces, async queuing, multi-repo, TLS, bundling/block, S3, repo-sync) over shared helpers in `_lib.sh`. Run them with `./integration/scenarios/run-all.sh` (optionally filter: `run-all.sh 01 03`). Each scenario runs against a freshly `down -v`'d topology for isolation; a failing scenario is recorded and the loop continues rather than aborting.
- **Vagrant topology** (`integration/vagrant/`) — the same KB scenario matrix against VirtualBox VMs via `run-validation.sh` (run after `vagrant up`); an alternative to Docker that exercises real systemd/SSH. Provisioning scripts live in `integration/vagrant/provision/`.

## CI gating
Expand Down
28 changes: 28 additions & 0 deletions crates/pgbr-build/inputs/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,13 @@ command:
log-level-default: DEBUG
parameter-allowed: true

repo-sync:
command-role:
remote: {}
lock-remote-required: true
lock-required: true
lock-type: backup

restore:
command-role:
local: {}
Expand Down Expand Up @@ -403,6 +410,7 @@ option:
manifest:
default: latest
required: true
repo-sync: {}
restore:
default: latest
required: true
Expand Down Expand Up @@ -440,6 +448,7 @@ option:
required: false
repo-rm:
required: false
repo-sync: {}
restore: {}
stanza-create: {}
stanza-delete: {}
Expand Down Expand Up @@ -524,6 +533,12 @@ option:
- diff
- incr
default: full
repo-sync:
allow-list:
- all
- wal
- backup
default: all
info:
required: false
allow-list:
Expand Down Expand Up @@ -1202,6 +1217,15 @@ option:
command-role:
main: {}

repo-sync:
section: global
type: boolean
default: false
command:
backup: {}
command-role:
main: {}

start-fast:
section: global
type: boolean
Expand Down Expand Up @@ -1648,6 +1672,7 @@ option:
repo-ls: {}
repo-put: {}
repo-rm: {}
repo-sync: {}
restore: {}
server: {}
stanza-create:
Expand Down Expand Up @@ -1855,6 +1880,7 @@ option:
default: false
command:
backup: {}
repo-sync: {}
command-role:
main: {}

Expand All @@ -1866,6 +1892,7 @@ option:
allow-range: [1MiB, 1PiB]
command:
backup: {}
repo-sync: {}
command-role:
main: {}
depend:
Expand All @@ -1885,6 +1912,7 @@ option:
default: false
command:
backup: {}
repo-sync: {}
command-role:
main: {}
depend:
Expand Down
48 changes: 48 additions & 0 deletions crates/pgbr-build/inputs/help.xml
Original file line number Diff line number Diff line change
Expand Up @@ -1543,6 +1543,16 @@
<example>y</example>
</config-key>

<config-key id="repo-sync" name="Repo Sync">
<summary>Sync the just-completed backup to other repositories.</summary>

<text>
<p>When multiple repositories are configured and this option is enabled, the backup just written to the active repository is mirrored byte-for-byte to every other configured repository immediately after a successful backup. Because repositories in a set are identical mirrors the copy is a raw byte copy with no recompression or re-encryption. The standalone <cmd>repo-sync</cmd> command can also be run later to reconcile repositories.</p>
</text>

<example>y</example>
</config-key>

<config-key id="stop-auto" name="Stop Auto">
<summary>Stop prior failed backup on new backup.</summary>

Expand Down Expand Up @@ -2504,6 +2514,44 @@
</option-list>
</command>

<command id="repo-sync" name="Repository Sync">
<summary>Sync backups and WAL between repositories.</summary>

<text>
<p>Mirrors backups and archived WAL from the active repository (the highest priority repository, or the one given by <br-option>{[dash]}-repo</br-option>) to every other configured repository. Repositories in a set are byte-identical mirrors, so the sync is a raw byte copy of the stored objects with no recompression, re-encryption, or re-bundling.</p>

<p>The command is idempotent: objects already present in a destination repository are skipped, and the ancestors of a differential or incremental backup are synced first so dependency chains are never broken.</p>
</text>

<option-list>
<option id="type" name="Type">
<summary>What to sync.</summary>

<text>
<p>The following sync types are supported:</p>

<list>
<list-item><id>all</id> - sync both WAL and backups (the default).</list-item>
<list-item><id>wal</id> - sync archived WAL only.</list-item>
<list-item><id>backup</id> - sync backups only.</list-item>
</list>
</text>

<example>backup</example>
</option>

<option id="set" name="Set">
<summary>Backup set to sync.</summary>

<text>
<p>Sync only the specified backup set (the backup label and all of its dependent backups). When not specified, every backup in the source repository is synced.</p>
</text>

<example>20150131-153358F_20150131-153401I</example>
</option>
</option-list>
</command>

<command id="restore" name="Restore">
<summary>Restore a database cluster.</summary>

Expand Down
69 changes: 68 additions & 1 deletion crates/pgbr-cli/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -328,11 +328,40 @@ fn worker_loaded_config(resolved: &ResolvedCli) -> LoadedConfig {
const MULTI_REPO_COMMANDS: &[&str] = &[
"archive-push",
"archive-get",
"repo-sync",
"stanza-create",
"stanza-delete",
"stanza-upgrade",
];

/// Whether `loaded` must build one backend per configured repository.
///
/// The commands in [`MULTI_REPO_COMMANDS`] are unconditionally multi-repo.
/// `backup` is the one exception: it only needs every backend when the resolved
/// `repo-sync` boolean is on, because in that case a successful backup is
/// mirrored to the other repositories immediately afterwards (see
/// `pgbr_command::backup`'s `repo_sync_enabled`). With `repo-sync` off — the
/// default — building all backends is wasted and actively harmful: it would
/// connect every SFTP repo, spawn every `repoN-host` SSH worker and fail
/// construction on any partially configured cloud repo, so a plain `backup`
/// against a config with an extra unreachable repo would error before the
/// backup even starts. Constructing all backends is only justified when the
/// backup will mirror, so gate it on `repo-sync`.
fn needs_all_repo_storages(loaded: &LoadedConfig) -> bool {
MULTI_REPO_COMMANDS.contains(&loaded.command.as_str()) || (loaded.command == "backup" && repo_sync_enabled(loaded))
}

/// Whether the resolved `--repo-sync` boolean is enabled (default **false**).
///
/// Mirrors `pgbr_command::backup`'s reader so the CLI and the command agree on
/// when a backup fans out to every repository.
fn repo_sync_enabled(loaded: &LoadedConfig) -> bool {
matches!(
loaded.options.get(&("repo-sync".to_owned(), None)),
Some(OptionValue::Boolean(true))
)
}

/// Build storage (as needed) and dispatch `loaded` to its command, mapping the
/// outcome to a process exit code.
///
Expand Down Expand Up @@ -360,7 +389,7 @@ fn dispatch_loaded(loaded: &LoadedConfig) -> Result<i32, CliRunError> {
// `--repo`) is the one whose index matches `repo` — or the first configured
// when the active index has no backend in the set (it always does, since
// `configured_repo_indexes` includes the active index).
if MULTI_REPO_COMMANDS.contains(&loaded.command.as_str()) {
if needs_all_repo_storages(loaded) {
let all = storage_helper::build_all_repo_storages(loaded)?;
let active = storage_helper::active_repo_index(loaded);
let active_storage = all
Expand Down Expand Up @@ -1612,6 +1641,44 @@ option:
assert_eq!(log_level_from_id("nonsense"), None);
}

#[test]
fn backup_needs_all_repos_only_when_repo_sync_on() {
use super::needs_all_repo_storages;

// Plain backup with `repo-sync` unset takes the single-repo path: it
// must NOT build every configured backend, so a config with an extra
// unreachable repo does not fail before the backup starts.
let plain = loaded("backup", Some("demo"), &[]);
assert!(!needs_all_repo_storages(&plain));

// Backup with `repo-sync` explicitly off is likewise single-repo.
let off = loaded("backup", Some("demo"), &[("repo-sync", None, OptionValue::Boolean(false))]);
assert!(!needs_all_repo_storages(&off));

// Backup with `repo-sync` on must fan out to every repository, since it
// mirrors the completed backup to the others.
let on = loaded("backup", Some("demo"), &[("repo-sync", None, OptionValue::Boolean(true))]);
assert!(needs_all_repo_storages(&on));
}

#[test]
fn declared_multi_repo_commands_are_always_multi_repo() {
use super::{MULTI_REPO_COMMANDS, needs_all_repo_storages};

// Every command in the unconditional list fans out regardless of
// `repo-sync`. `repo-sync` (the command) in particular is always
// multi-repo.
for command in MULTI_REPO_COMMANDS {
let loaded_cfg = loaded(command, Some("demo"), &[]);
assert!(needs_all_repo_storages(&loaded_cfg), "{command} must be multi-repo");
}

// A non-multi command with `repo-sync` on stays single-repo: the gate is
// scoped to `backup` alone, not "any command with repo-sync set".
let restore = loaded("restore", Some("demo"), &[("repo-sync", None, OptionValue::Boolean(true))]);
assert!(!needs_all_repo_storages(&restore));
}

#[test]
fn resolve_log_level_uses_value_then_fallback() {
use pgbr_core::log::{LOG_LEVEL_DEBUG, LOG_LEVEL_INFO, LOG_LEVEL_WARN};
Expand Down
14 changes: 9 additions & 5 deletions crates/pgbr-command/src/archive.rs
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,11 @@ use crate::pipeline::{CompressType, RepoTransform};
/// once the plaintext form is found absent. Each maps to the compress codec
/// that produced it. `pgbackrust` names archived WAL with the codec's
/// extension, so a recovering client must try every suffix.
const COMPRESS_SUFFIXES: &[&str] = &[".gz", ".zst", ".bz2", ".lz4"];
///
/// Shared with the sync engine (`sync::archive`), which probes the same
/// suffix set when mirroring WAL between repositories; this is the single
/// source of truth so both paths stay in lockstep.
pub(crate) const COMPRESS_SUFFIXES: &[&str] = &[".gz", ".zst", ".bz2", ".lz4"];

/// Resolve the `compress-type` option to its file-name suffix. Returns
/// `""` for `none` (or when the option is unset), and `.gz`/`.bz2`/`.lz4`/
Expand Down Expand Up @@ -288,15 +292,15 @@ fn write_segment_to_dest_arg(bytes: &[u8], dest_arg: &Path) -> Result<(), Comman
/// `archive_id` is `<db-version>-<db-id>` from the stanza's `archive.info`
/// (see [`archive_id`]), matching pgBackRust's per-cluster archive directory
/// and the layout the `check` command polls.
fn repo_segment_path(stanza: &str, archive_id: &str, name: &str) -> PathBuf {
pub(crate) fn repo_segment_path(stanza: &str, archive_id: &str, name: &str) -> PathBuf {
PathBuf::from(format!("archive/{stanza}/{archive_id}/{name}"))
}

/// The archive-id directory name for a stanza's `archive.info`:
/// `<db-version>-<db-id>` (e.g. `"16-1"`). This is the per-cluster
/// subdirectory archived WAL is stored under, matching the C implementation
/// and the `check` command.
fn archive_id(info: &InfoArchive) -> String {
pub(crate) fn archive_id(info: &InfoArchive) -> String {
format!("{}-{}", info.db_version, info.db_id)
}

Expand Down Expand Up @@ -617,7 +621,7 @@ fn is_checkable_wal_segment(name: &str) -> bool {
/// [`CommandError::MissingOption`] when an encrypted repo has no
/// `repo-cipher-pass`; [`CommandError::Other`] when the info file cannot be
/// loaded / decrypted; [`CommandError::Storage`] on an underlying storage error.
fn load_archive_info(
pub(crate) fn load_archive_info(
config: &LoadedConfig,
repo_storages: &[&dyn Storage],
stanza: &str,
Expand Down Expand Up @@ -1590,7 +1594,7 @@ fn fetch_segment_with_retry(
/// Whether `repo` holds `segment` for `stanza` in any stored form (plaintext or
/// a compressed suffix). Used by [`get`] to pick the first repository that has
/// the segment.
fn repo_has_segment(repo: &dyn Storage, stanza: &str, archive_id: &str, segment: &str) -> Result<bool, CommandError> {
pub(crate) fn repo_has_segment(repo: &dyn Storage, stanza: &str, archive_id: &str, segment: &str) -> Result<bool, CommandError> {
if repo.exists(&repo_segment_path(stanza, archive_id, segment))? {
return Ok(true);
}
Expand Down
Loading