CRW-862: one definition of the state backup, its integrity gate and an operator command

docs/runtime-install.md

@@ -153,6 +153,7 @@ under the destination or in the host record is created before all three hold.
 `--backup-state-to <dir>` (on `install`, `update` and `rollback`) is the acknowledgement that the swap brings the additive DAG zone, or ordinary
 indexes on tables the store already holds, to a store that lacks them: the command copies the whole relay state directory to `<dir>` before
 promoting, and without it that swap refuses and names the flag ([the route](#why-the-schema-reading-compares-statements-and-not-versions)).
+The same artifact can be taken outside an install, by hand, with `crw install backup-state --to <dir>` ([the operator command](#the-state-backup-outside-an-install)).
 
 What follows is one run, in this order, and the result lists the steps it took:
 
@@ -419,22 +420,33 @@ it copies the whole state directory the gate read to `DIR`: copy only, byte for
 or deleted, here or on a failure. Directories and regular files are copied with their bytes and, once every byte is in place and verified, their permission bits, each synced after its mode is set so the mode and not only the bytes survives a power loss (the directory the backup is made in stays 0700, and a copy always keeps its owner able to open it: a source the installer could read only through its group gets the owner's read bit, and the manifest records both modes); a symbolic link to a regular file is copied as the
 file's bytes under the link's name (a link alone would back up nothing), and when `relay.sqlite3` is such a link the real file's `-wal`
 is copied beside it, so a restore opens with the commits only the log held; a socket, a FIFO or a device is listed as skipped. The store's two
-sidecars follow SQLite's WAL mode: `relay.sqlite3-shm` is never listed, copied or compared, because SQLite rebuilds that index from the log on
-open, and `relay.sqlite3-wal` is copied when it is there at its copy, while an empty one that goes or appears between the listing and the copy is
-not a refusal (a log listed and gone is dropped, one that appeared after the listing is not copied, and the manifest's `storeSidecars` records which
-of the four happened). An empty log is the case this route exists for: a read-only open of a store no connection holds leaves one. A log that was
+sidecars are recognised by the path SQLite resolves the database to, not by their top-level name: with `relay.sqlite3` linked to a file inside
+the state directory (or outside it), the sidecars are that file's `-wal` and `-shm`, wherever they lie. `-shm` is never listed, copied or compared,
+because SQLite rebuilds that index from the log on open, and `-wal` is copied when it is there at its copy, under the restore-compatible name
+`relay.sqlite3-wal`, while an empty one that goes or appears between the listing and the copy is not a refusal (a log listed and gone is dropped,
+one that appeared after the listing is not copied, and the manifest's `storeSidecars` records which of the four happened). An empty log is the case
+this route exists for: a read-only open of a store no connection holds leaves one. The log is compared by its identity, not its size: its kind, its
+link target and the file its bytes come from must be the same in both listings, so a log whose link target or resolved source moved under the copy
+refuses ("the store's write-ahead log changed"), while the log's size and its coming and going are left to the sidecar rules above. A log that was
 not copied and holds frames in the second listing refuses instead, because a commit that stays in the log does not touch `relay.sqlite3` until a
 checkpoint, so the digest check alone would not see it and the backup would claim success while the live store held a row the copy lacks.
 `relay.sqlite3` itself and every other file keep the rule above, so a store that was written under the copy still refuses, and a copied `-wal` that
 is still there at the verification must digest to what was copied: one that a checkpoint has taken away by then is not a refusal either, and
-`relay.sqlite3` is the consistency the verification keeps.
+`relay.sqlite3` is the consistency the verification keeps. A store's `-wal` or `-shm` that goes between a listing's directory read and the entry's own
+information is recorded as gone before its copy rather than refusing; every other file's disappearance there still refuses.
 Each file is
 hashed while it is read and synced; then the state directory is read again, and the listing, every size and every file's digest, and the digest of
 every file in the copy, must be what was copied. Any difference, in any file, refuses the swap ("the state directory changed under the copy"):
 the copy is of one moment or it is not made. The backup, its manifest and the directories made for them are synced in their parents after the manifest is written, so a power loss cannot keep the files and lose the names that reach them. `DIR` and its manifest must not exist, must not lie inside the state directory, and must not lie
 inside the runtime destination tree (a failed run removes its candidate runtime, and a backup there would go with it); the free space on its
-filesystem must cover the directory. The record is written last, beside the backup and not inside it (`DIR.manifest.json`: source, destination,
-time, issue, every entry with its size, mode and digest, what was skipped, an aggregate digest), and the command's result carries the same facts
+filesystem must cover the directory. Before the record is written the copy passes the integrity gate: a scratch duplicate of the copied store (its
+`relay.sqlite3` and, when copied, its `-wal`) is opened with SQLite and asked `PRAGMA integrity_check`. The duplicate is what SQLite touches, never
+the copy, so the backup’s bytes are unchanged by the check, and the duplicate is removed afterwards. The destination’s free-space preflight covers
+the backup and that duplicate together. The manifest records `integrityCheck` (SQLite’s
+answer) and `restoreCandidate` (true only when the check passed), and a copy that fails the check is not a restore candidate and refuses. The
+record is written last, beside the backup and not inside it (`DIR.manifest.json`: source, destination,
+time, issue, every entry with its size, mode and digest, what was skipped, an aggregate digest, `integrityCheck` and `restoreCandidate`), and the command's
+result carries the same facts
 as `swapGate.stateBackup`. A backup that fails part-way stays where it is and is reported with `partial: true`; a swap that fails after the backup keeps
 it too, and a rerun needs a new destination.
 
@@ -494,6 +506,25 @@ The warning against write commands stays: opening a store for writing with this
 relay command run by hand against a live state directory takes no backup), so the route makes the warning unnecessary only inside the install
 command. Run no write command of this build against a live state directory before the install has taken its backup.
 
+### The state backup outside an install
+
+`crw install backup-state --to <dir>` takes the same stopped byte copy the install route takes, outside any install, so an operator can
+take the default artifact on demand. It calls the same `backupState` routine, so the copy, its manifest and its `integrityCheck` and
+`restoreCandidate` fields are the same, and the destination rules (`<dir>` and its manifest must not exist, must not lie inside the state
+directory or the runtime destination tree, and the free space must cover the directory) are the same. It is routed before the generic install
+options, like `features`, `config` and `migrate-state`: it takes its own `--to` flag and prints its own JSON report (`command: "backup-state"`,
+`applied`, `stateBackup`), and the frozen `crw install` usage line still names only the commands it named before this one arrived.
+
+It refuses while the relay service runs, reading the relay the same way the swap gate does: the selected relay’s own `service status`. Stopping
+the service is the operator’s job. A reading that could not be taken refuses too, so a backup is never taken on the strength of an unasked
+service. While it copies, it holds the store’s write gate exclusively, so no relay writer reaches the store under the copy: the gate is the one
+the relay writers lock, beside the database as SQLite resolves it (`holdGate`), which is the state directory’s own `write-gate.lock` in the
+ordinary layout and the linked file’s directory otherwise. A writer that already holds it refuses the command rather than waiting, and the
+command takes no other lock.
+
+What this command does not do: it does not stop or start the service, it does not restore, and it does not schedule anything. The online
+`sqlite3_backup` snapshot and its schedule are a separate decision and are not built here.
+
 ### The order a swap commits in
 
 The selection in the host record and the pointer on disk are two truths, and both the order they

internal/runtime/install/backup_cmd.go

@@ -0,0 +1,161 @@
+package install
+
+// backup_cmd.go is `crw install backup-state --to <dir>`: the operator route to the stopped byte copy
+// outside an install (CRW-862, docs/port/decisions.md section 83 item 4 and slice S7). The install route reaches
+// backupState only when the swap actually introduces the additive DAG zone or an ordinary index
+// (swapgate.AdditiveArrivalOnly), so an operator cannot take the default artifact on demand; this command takes
+// the same artifact through the same routine. It is routed before the generic install options, like features,
+// config and migrate-state: it has its own flag and prints its own JSON report.
+
+import (
+	"context"
+	"errors"
+	"flag"
+	"fmt"
+	"io"
+	"io/fs"
+	"os"
+	"path/filepath"
+
+	"github.com/thisisjun786/codex-relay-workflow/internal/contract"
+	"github.com/thisisjun786/codex-relay-workflow/internal/relay/store"
+	"github.com/thisisjun786/codex-relay-workflow/internal/relay/store/ownership"
+	"github.com/thisisjun786/codex-relay-workflow/internal/runtime/definition"
+	"github.com/thisisjun786/codex-relay-workflow/internal/runtime/record"
+	"github.com/thisisjun786/codex-relay-workflow/internal/runtime/scope"
+	"github.com/thisisjun786/codex-relay-workflow/internal/runtime/swapgate"
+)
+
+// backupStateUsage is this command's own help. The legacy `crw install` usage line is a frozen contract and
+// does not name this command.
+const backupStateUsage = `usage: crw install backup-state --to <dir> [--state <dir>] [--socket <path>]
+
+Copy the whole relay state directory to <dir> outside an install: the same stopped byte copy the install
+route takes before a swap that brings the additive DAG zone or an ordinary index, with the same manifest
+beside it and the same integrity_check gate. The relay service must be stopped: this command refuses while
+the relay's own service reading reports it running, and it holds the store's write gate exclusively while
+it copies, so no relay write reaches the store under the copy. <dir> and its manifest must not exist and
+must not lie inside the state directory or the runtime destination tree, exactly as on the install route.
+
+The manifest records the copy's integrityCheck and whether it is a restoreCandidate (true only when
+PRAGMA integrity_check passed on a scratch duplicate of the copy, never on the copy itself).
+`
+
+// serviceReading is the relay's own service reading, the same signal the swap gate takes
+// (swapgate.DaemonCell over the relay's `service status`). It is a seam: a test fakes a running service
+// without starting a daemon, which the packet requires (the "service running" case is a faked reading).
+var serviceReading = func(ctx context.Context, o Options) Object {
+	rec, _ := record.Load(o.RecordPath, definition.Version).Value.(Object)
+	executable := filepath.Join(o.Dest, "current", "bin", definition.Relay)
+	if install := selectedInstall(rec, definition.Relay); install != nil {
+		if entry, ok := record.Get(install, "entryPoint").(string); ok && entry != "" {
+			executable = entry
+		}
+	}
+	return swapgate.DaemonCell(scope.Relay(ctx, []string{"service", "status"}, executable, o.Socket, o.State, o.Env, false, 0))
+}
+
+// runBackupState is `crw install backup-state`. It parses its own flags, resolves the same paths the other
+// commands read, and takes the backup.
+func runBackupState(ctx context.Context, args []string, env scope.Env, stdout, stderr io.Writer) int {
+	flags := flag.NewFlagSet("crw install backup-state", flag.ContinueOnError)
+	flags.SetOutput(stderr)
+	to := flags.String("to", "", "the directory the whole relay state directory is copied to (required)")
+	codexHome := flags.String("codex-home", "", "the Codex home (default $CODEX_HOME or ~/.codex)")
+	recordPath := flags.String("record", "", "the host record (default $XDG_STATE_HOME/codex-relay-workflow/host-record.json)")
+	issue := flags.String("issue", DefaultIssue, "the issue recorded as the evidence of this run")
+	socket := flags.String("socket", "", "the App Server socket the relay service reading is made against")
+	state := flags.String("state", "", "the relay state directory to copy")
+	for _, arg := range args {
+		if arg == "--help" || arg == "-h" || arg == "help" {
+			fmt.Fprint(stdout, backupStateUsage)
+			return OK
+		}
+	}
+	positional, err := parse(flags, args)
+	if err != nil {
+		if errors.Is(err, flag.ErrHelp) {
+			return OK
+		}
+		fmt.Fprintln(stderr, "crw install backup-state: error: "+err.Error())
+		return Usage
+	}
+	if len(positional) > 0 {
+		fmt.Fprintln(stderr, "crw install backup-state: error: unrecognized arguments: "+positional[0])
+		return Usage
+	}
+	if *to == "" {
+		fmt.Fprintln(stderr, "crw install backup-state: error: argument --to: the directory the state directory is copied to is required")
+		return Usage
+	}
+	if why := notUTF8([]namedValue{{"--to", *to}, {"--state", *state}, {"--socket", *socket}, {"--codex-home", *codexHome}, {"--record", *recordPath}}); why != "" {
+		fmt.Fprintln(stderr, "crw install backup-state: error: "+why)
+		return Usage
+	}
+	o, err := options(env, *codexHome, *recordPath, *issue, *socket, *state)
+	if err != nil {
+		fmt.Fprintln(stderr, "crw install backup-state: error: "+err.Error())
+		return Usage
+	}
+	if refused := foreignDestination(o, "backup-state"); refused != nil {
+		if err := contract.Emit(stdout, refused); err != nil {
+			fmt.Fprintln(stderr, "crw install backup-state: "+err.Error())
+		}
+		return Refused
+	}
+	dest, err := absolute(*to)
+	if err != nil {
+		fmt.Fprintln(stderr, "crw install backup-state: error: --to: "+err.Error())
+		return Usage
+	}
+	result, code := BackupState(ctx, o, dest)
+	if err := contract.Emit(stdout, result); err != nil {
+		fmt.Fprintln(stderr, "crw install backup-state: "+err.Error())
+		return Refused
+	}
+	return code
+}
+
+// BackupState takes the stopped byte copy outside an install. It refuses while the relay service runs, holds the
+// store's write gate exclusively for the whole copy, and takes the same artifact backupState makes.
+func BackupState(ctx context.Context, o Options, dest string) (Object, int) {
+	reading := serviceReading(ctx, o)
+	if record.Get(reading, "readable") != true {
+		return refusedResult("backup-state", "whether the relay service is running could not be established: "+record.Text(reading, "detail"),
+			"nothing was read and nothing was copied: a backup is taken only while the service is known stopped.")
+	}
+	if record.Get(reading, "answer") == scope.Running {
+		return refusedResult("backup-state", "the relay service is running ("+record.Text(reading, "detail")+")",
+			"nothing was read and nothing was copied: stop the relay service and run this command again.")
+	}
+	selection, err := store.ResolveStateDir(o.State, o.Socket)
+	if err != nil {
+		return refusedResult("backup-state", "the state directory could not be resolved: "+err.Error(), "nothing was read and nothing was copied.")
+	}
+	// The relay's write gate sits beside the database as SQLite resolves it (holdGate), which is the state
+	// directory's own relay.sqlite3 in the ordinary layout and the linked file's directory otherwise.
+	resolved, err := resolveStorePath(selection.DBPath())
+	if err != nil {
+		return refusedResult("backup-state", "the store's database path could not be resolved: "+err.Error(), "nothing was read and nothing was copied.")
+	}
+	gatePath := filepath.Join(filepath.Dir(resolved), "write-gate.lock")
+	if _, err := os.Lstat(gatePath); errors.Is(err, fs.ErrNotExist) {
+		return refusedResult("backup-state", "no relay store write gate exists at "+gatePath+", so the store cannot be held still for a copy",
+			"nothing was read and nothing was copied.")
+	} else if err != nil {
+		return refusedResult("backup-state", gatePath+" could not be examined: "+err.Error(), "nothing was read and nothing was copied.")
+	}
+	gate, err := ownership.Lock(gatePath, true, false)
+	if err != nil {
+		return refusedResult("backup-state", "the store's write gate could not be taken exclusively ("+err.Error()+"), so a relay writer may be reaching the store",
+			"nothing was read and nothing was copied: stop the relay service and run this command again.")
+	}
+	defer gate.Close()
+	backup, err := backupState(ctx, o, dest, backupForOperator)
+	if err != nil {
+		return Object{field("command", "backup-state"), field("applied", false), field("refused", err.Error()), field("stateBackup", backup),
+			field("note", "nothing but what is reported under stateBackup was written: a partial copy stays where it is and is never vouched for by a manifest it did not write.")}, Refused
+	}
+	return Object{field("command", "backup-state"), field("applied", true), field("stateBackup", backup),
+		field("note", "the state directory was copied byte for byte with the relay service stopped and the store's write gate held exclusively; the copy and its manifest stay where they are, and a rerun needs a new destination.")}, OK
+}