Harden Factory against harness auth failures and false no-PR completion

[kjgbot]

Problem

Factory can currently turn recoverable infrastructure and harness failures into a false terminal success.

The incident on #180 exposed several coupled failure modes:

• the configured Claude reviewer could not start because its login had expired;
• the implementer left useful but uncommitted work in its retained worktree;
• Factory wrote factory:human-review and a durable complete lifecycle even though the lifecycle had no pull-request receipt and no PR existed;
• the single-implementer completion helper can return success without probing for a PR;
• Factory's process heartbeat remained fresh while Relayfile could no longer join the configured cloud workspace (workspace not found), making the process look healthy while discovery and event ingestion were stalled.

These conditions must be recoverable and observable, never reported as completed work.

Required behavior

Ordered harness fallback

• Support an ordered capability list per role rather than a single capability.
• Factory configuration must accept the full spawn-harness registry already supported by Pear/Relay: spawn:claude, spawn:codex, spawn:gemini, spawn:opencode, spawn:grok, spawn:aider, spawn:goose, spawn:cursor, and spawn:droid.
• Derive validation and CLI metadata from the shared/advertised harness capability registry rather than maintaining a smaller Factory-only enum. A newly registered spawn:* harness must not require another hard-coded Factory schema change.
• Allow each ordered entry to carry its harness-appropriate model and settings. Preserve the existing single-capability configuration as shorthand for a one-element list.
• Default the reviewer/babysitter fallback order to Claude, then Codex, then OpenCode unless explicitly configured otherwise; every supported harness must be selectable and orderable by the operator.
• Preflight each harness before dispatch and classify authentication/configuration failures separately from task failures.
• On a pre-start authentication failure, persist the failed attempt and move to the next configured harness without consuming the task retry budget.
• Preserve the deterministic worktree, branch, issue context, and durable lifecycle across fallback attempts. Use collision-safe agent/session identities and never reset or discard local work.
• Bound fallback attempts. If all configured harnesses are unavailable, leave the dispatch retryable/blocked with an actionable reason; do not complete or move the issue to human review.

PR-backed completion invariant

• A GitHub issue must not enter factory:human-review, writeback-applied, releasing, or complete unless every implementer route has a provider-confirmed PR receipt.
• Remove the single-implementer shortcut that returns success without checking for a PR.
• Confirm that each receipt belongs to the expected repository and deterministic head branch, that the remote branch is ahead of its base, and that the PR is open and non-draft before human-review writeback.
• Scope numeric GitHub issue references to the source/routed repository so #180 in another configured repository cannot satisfy Factory issue Add historian pr reviewer #180.
• Include the confirmed PR URL(s) in the issue writeback. Never emit “the pull request remains open” without a confirmed PR.
• If an agent exits with uncommitted/unpushed work, retain the worktree and move the lifecycle to a retryable/stalled state with an explicit recovery action.
• Add startup reconciliation for legacy false-terminal lifecycles whose PR receipt is absent.

Relayfile/workspace health

• Treat repeated token-resolution, join, and workspace not found failures as a degraded/unready Factory state.
• Do not report a healthy liveness status solely because the main loop heartbeat is advancing while issue discovery cannot read current provider state.
• Use bounded exponential backoff and rate-limited logs rather than continuously flooding the terminal.
• Recover automatically after workspace credentials/key selection becomes valid, or exit with a clear non-zero failure after the configured retry budget so a supervisor can restart it.
• Make SIGINT/SIGTERM abort pending mount operations and complete shutdown within a bounded deadline. A wedged Relayfile client must not force an operator or supervisor to use SIGKILL, and stopping Factory must not implicitly stop a shared Relay broker.

Observability and alerting

• Emit structured events/counters for harness auth failure, fallback selection/exhaustion, PR-completion guard rejection, retained dirty worktree, and Relayfile workspace unavailability.
• Surface degraded readiness in Factory status/heartbeat output.
• Ensure these signals reach the configured observability/PostHog path and are actionable by NightCTO alerting.

Verification

• Unit tests: Claude auth failure falls back to Codex; Codex failure falls back to OpenCode; Grok and every other Pear/Relay registry entry validate and dispatch; all harnesses unavailable remains non-terminal.
• Contract test: Factory's accepted built-in harness set cannot drift from the Pear/Relay spawn-harness registry, while advertised custom spawn:* capabilities remain configurable.
• Unit tests: a single implementer without a PR cannot complete; uncommitted or unpushed work cannot produce human-review writeback.
• Unit tests: same-number issue/PR references in another configured repository cannot satisfy completion.
• Recovery test: restart from a legacy complete lifecycle with no PR receipt restores a retryable state and preserves the worktree.
• Integration test: Relayfile workspace 404 makes readiness unhealthy, rate-limits retries, and recovers after a valid workspace is selected.
• Shutdown test: with a permanently blocked mount request, SIGTERM cancels the request, releases owned resources, leaves a shared broker running, and exits within the configured deadline.
• End-to-end test with fake harness executables proves fallback uses the same checkout and publishes exactly one PR.

Definition of done

• All tests above pass in CI.
• Existing Claude/Codex-only configurations remain backward compatible.
• Document ordered per-role fallback, per-entry model/settings, and examples using Claude, Codex, OpenCode, Grok, and another registry-backed harness.
• Factory cannot claim human review or completion without a durable, repository-scoped PR receipt.

Related incident: #180.