bb-app startup failures retry indefinitely and accumulate empty diagnostic files on disk failure

[MPIsaac-Per]

Summary

bb-app start retries unrecoverable managed-server startup failures indefinitely. During a disk-full incident, each failed start also attempted to create another diagnostic JSON file, leaving hundreds of empty files on the filesystem that had already run out of space. Repeated failures should back off, stop after a bounded number of attempts, and preserve stderr evidence when diagnostic writes fail.

Versions and environment

• bb-app 0.42.1, npm installation, running under a Linux systemd user unit with Restart=always.
• Ubuntu 24.04.4 LTS, Linux 6.8.0-138-generic, Node v22.23.2.
• The incident installation included an unrelated protocol-181 skill-discovery backport. The affected launcher and diagnostic code were unchanged from the release base.
• Source patch tested against 4ae0a7c893e9922721e0e367c72aa3a1a2990e7c and backported onto the deployed source lineage. No provider session is needed to trigger the failure.

Steps to reproduce

Use an unprivileged account and an isolated data directory; do not fill a real disk. An inaccessible SQLite path plus unwritable logs exercises the unrecoverable-startup path:

repro_dir=$(mktemp -d)
mkdir -p "$repro_dir/data/bb.db" "$repro_dir/data/logs"
chmod 500 "$repro_dir/data/logs"
BB_DATA_DIR="$repro_dir/data" \
BB_SERVER_PORT=49386 BB_HOST_DAEMON_PORT=49387 \
BB_SERVER_BIND_HOST=127.0.0.1 BB_TELEMETRY=false \
  bb-app start > "$repro_dir/stdout" 2> "$repro_dir/stderr"

bb.db is deliberately a directory, so SQLite cannot open it as a database. Choose unused ports. On an affected build, interrupt the continuing supervisor with Ctrl-C after observing repeated failures. Restore the fixture directory's permissions with chmod 700 "$repro_dir/data/logs" afterward.

Evidence scope: the original disk-full behavior below was recorded during the incident. The isolated commands were executed against the packaged fix in #3339; that run verifies the corrected failure path, not a new unpatched disk-full replay. A real full filesystem was not recreated.

Expected vs actual

The incident investigation recorded 78 restart attempts in five minutes and 980 total before the user unit was stopped. The interval stayed roughly 3.5 seconds. The resulting process-server-startupFailure-*.json files were all zero bytes.

The underlying error was available in the journal:

SqliteError: disk I/O error
    at Database.pragma (better-sqlite3/lib/methods/pragma.js:10:27)

Expected: capped exponential backoff, a terminal state after repeated failures, bounded diagnostic retention, atomic diagnostic publication, and the underlying error on stderr even if storage is unavailable. An external service manager must respect that terminal state instead of restarting the supervisor immediately.

Evidence

• Implementation and regression report: fix(bb-app): bound startup restart failures #3339.
• Launcher before the fix.
• Diagnostic writer before the fix.
• Supervisor regression.

The fixed package's isolated Linux run recorded delays of 1, 2, 4, 8, 16, 32, and 60 seconds, followed by:

Server failed 8 consecutive startup attempts; stopping bb

With Restart=always and RestartPreventExitStatus=75, systemd reported:

RestartPreventExitStatus=75
NRestarts=0
ExecMainStatus=75
ActiveState=failed

SQLite errors reached stderr and the unwritable logs directory contained zero diagnostic files. The source suite passed 83 launcher tests and 22 process-utils tests, with both typechecks. Separate tests cover the five-per-process/kind retention cap and cleanup after failed writes/renames. All executed PR CI checks passed, including Linux and macOS package smoke.

What was ruled out

• This is distinct from bb-app supervisor retries forever after another server takes its port #2993's competing healthy server/port-ownership case: SQLite startup failure occurs with no other server occupying the configured port, so detecting an existing healthy server does not resolve it.
• Disk cleanup restores immediate service but does not fix the unbounded retry or diagnostic-writing behavior.
• A systemd restart delay alone does not govern the child retry loop inside bb-app start.

Suggested priority

Reliability bug: an unrecoverable storage failure can produce sustained restart and diagnostic-write churn. The original service was restored; #3339 addresses the recurring failure behavior.

AGENT GENERATED

[bb-slop-cop]

🚨 SLOP COP 🚨 · new-issue-autopilot

Classification: Bug · High priority · Medium effort · host, cli (all previously missing; applied).

Reproduction report: PARTIALLY REPRODUCED · high root-cause confidence · partial-repro.

Two clean trusted-main checkouts produced 32 fixed-delay restart failures and 32 empty diagnostic files under injected write failure. The retry loop has no failure budget; diagnostic writes have neither atomic publication nor retention/cleanup. Initial startup exits separately, and server stderr fallback already works. No real disk-full/systemd replay; full build preparation could not complete.

No new PR: open #3339 already explicitly links to this issue.

AGENT GENERATED