bug(ship): preserve PR binding after authorized head rewrite

[justinhelmer]

What happened

A one-unit Ship pipeline opened coreplanelabs/nominal#3768, completed an initial review/fix round, and then atomically rewrote the original owned ref to one coherent commit. The coding run resubmitted the current-head PR description and disposition, but the Ship parent failed before re-review with:

HTTP 409 {"ok":false,"error":"publication_facts_mismatch","message":"the open pull request was not durably bound"}

A later same-thread plain re-review request at the verified exact head was refused before execution:

Unit U1 does not retain the pull request's durable expected head, so continuation did not start. Nothing else ran.

Expected

The original unit should retain or safely refresh the durable PR/ref/head binding after its authorized --force-with-lease rewrite, then start one exact-head re-review. At minimum, a read-only same-thread re-review should be able to verify the open PR and exact head without requiring a new Ship/publication identity.

Actual

• Original Ship run: f3c12258-e487-4b31-bd66-12aa249187c7
• Original unit: plan-fix-issue-3743-scoped-to-524d5b:U1
• Original ref: plan/fix-issue-3743-scoped-to-524d5b/u1
• Coding fix run: b7c106c5-44e0-401c-9fa5-37f9c1d07491
• PR: https://github.com/coreplanelabs/nominal/pull/3768
• Verified remote head: 8733b9ac09c66468ab20548b0ecc0b70c3a9d230
• Slack thread: https://coreplanelabs.slack.com/archives/C0BRRHKFLCB/p1790272901878789
• Refusal reply: https://coreplanelabs.slack.com/archives/C0BRRHKFLCB/p1790276496416239
• Observed September 24, 2026 at 12:01 PM PDT.

The PR remained open/readable, one commit, mergeable, and green at the exact head. No alternate branch, PR, writer, or generic Ship retry was created.

[justinhelmer]

Recurrence — nominal#3784 — September 24, 2026, 4:17 PM PDT

The same failure recurred on a separate one-unit Ship pipeline after its findings writer successfully force-pushed the exact owned PR branch.

• Ship: 20d62f63-cf07-45b3-8f14-d5adcd581256
• Unit: plan-implement-issue-3764-as-6d6c70:U1
• Findings writer: c5816d9a-215a-478e-b71d-fcda02b2f95c
• PR: https://github.com/coreplanelabs/nominal/pull/3784
• Branch: plan/implement-issue-3764-as-6d6c70/u1
• Pre-rewrite review head: 807727d9f00199cbc0edeceecaff885614756daa
• Verified remote/PR head after rewrite: 7bdebfef49cb8ac22ddf67f5e6c07f7d31385f5e
• Thread: https://coreplanelabs.slack.com/archives/C0BRRHKFLCB/p1790289009959189

The findings writer reports a successful exact old-head lease push, but the parent immediately failed its U1/1/findings/pr-check with:

HTTP 409 {"ok":false,"error":"publication_facts_mismatch","message":"the open pull request was not durably bound"}

A same-thread continuation after independently reading back the exact GitHub head was then refused before execution:

Unit U1 does not retain the pull request's durable expected head, so continuation did not start. Nothing else ran.

This recurrence is higher impact than the earlier green case: the rewritten head exposed a deterministic typecheck failure, and the refusal prevents the owning unit from publishing its already-prepared typing correction or proceeding to exact-head re-review. No new unit/ref or blind retry was started.

[justinhelmer]

Post-fix recovery refusal — nominal#3784 — September 25, 2026, 8:09 AM PDT

Release v1.262.1 is live on the bot at exact commit b7f8f15f4bb351a30811697a93d6bac4d67a5db9, containing #2355's missing-lastPush recovery fix. After fresh readback proved PR #3784 and its original ref both unchanged at 7bdebfef49cb8ac22ddf67f5e6c07f7d31385f5e, zero in-flight runs, and no rival writer, the supported original-unit command was invoked once in the original thread:

agent:ship recover unit plan-implement-issue-3764-as-6d6c70:U1

It refused before any writer/model/tool execution with recovery_budget_unknown:

• Door run: 76ba371e-a521-42c1-8437-52051b995ba1
• Slack receipt: https://coreplanelabs.slack.com/archives/C0BRRHKFLCB/p1790348966819169?thread_ts=1790289009.959189&cid=C0BRRHKFLCB
• Run record: two events only (input, refusal), zero turns, $0

The deployed guard requires durable instance.caps, row.startedAt, and finite row.ending.at; this legacy instance's durable instance record has no caps, so the refusal is the designed fail-closed branch:

switchboard/src/channels/adminCoordinator.ts

Lines 2174 to 2179 in b7f8f15

	const caps = instance.caps;
	if (caps === undefined || row.startedAt === undefined || !Number.isFinite(row.ending.at))
	return json(409, { ok: false, error: "recovery_budget_unknown", at });
	// Prior spend is not yet durable on a terminal row. A cost-capped unit
	// cannot safely recover by resetting that total, so it stays parked.
	if (instance.grant?.costCapUsd !== undefined) return json(409, { ok: false, error: "recovery_budget_unknown", at });

No budget was inferred/reset, no retry was made, and no replacement unit/ref/writer was admitted. #2355 fixed the publication-binding shape but does not make this legacy unit's remaining budget recoverable.

[justinhelmer]

Self-hosting recurrence on the repair PR — September 25, 2026, 9:00:03 AM PDT

The active repair for this issue reproduced the same defect on its own PR after a valid findings rewrite.

• Original repair Ship: 62a52eb1-205a-4ed6-851b-29489e932a46
• Original unit/ref: plan-at-current-main-b7f8f15f-c90b61:U1 / plan/at-current-main-b7f8f15f-c90b61/u1
• PR: #2359
• Review head: ef9edd962993465c08117c588fb1cf1f8d2f7609
• Findings writer: 33eee9ff-1a43-46a1-bf88-aa11393e2d94
• Valid pushed head: 5663170a73c088ce8c4e65b36d671b4490d57436
• Slack refusal

The findings writer addressed F1 using the request's permitted serialized split, pushed the same owned ref, updated its disposition, and completed. Ship then failed at U1/1/findings/pr-check with HTTP 409 publication_facts_mismatch / “the open pull request was not durably bound.” No re-review started.

Independent GitHub readback at 9:01 AM PDT proves #2359 is open on the original ref at exact 5663170a…, one coherent commit, with all required/all CI green and no unresolved inline thread. It lacks exact-head approval; the body also fell back instead of retaining the submitted strict-ready map. No code is lost.

This is a production 1.262.1 recurrence of the precise post-findings head/binding failure, now blocking the fix itself. Parent is not retrying generic Ship or creating another writer/ref. The retained recovery shepherd is being routed to preserve #2359, repair its body through GitHub bookkeeping, and request one fresh independent read-only exact-head review; the one-time 90-minute extension remains deliberately serialized and unimplemented in this PR.

[justinhelmer]

Post-merge boundary — September 25, 2026, 9:35:49 AM PDT

#2359 merged at exact reviewed head 5663170a73c088ce8c4e65b36d671b4490d57436 as 21696a703d549c2655b5487ca0a738d408e306ac. It is in open release PR #2361, not in the live 1.262.1 bot build as of 9:38 AM PDT. No original-unit recovery retry is authorized by this merge alone; the once-only 90-minute extension and #2236 producer identity repair remain separate.

The additional exact-head review recorded three fail-closed follow-ups. F1 counts all finished /aN findings attempts, so a failed attempt followed by a valid pushing retry can still produce publication_facts_mismatch. F2 builds the ordinary prefix without /rN, excluding resumed units. F3 compares tag/binding JSON in key order. They were dispositioned as nits under the default-minor gate and are present in the merged code. They need a focused follow-up with regression proof before claiming those cases fixed; no duplicate writer, ref, unit, or blind product retry has been started. The recovery shepherd ses_f26dde85cffeuAOOqwB51Hw4Tz has completed; Codex owns new admission and will not reactivate that session. #2236 and extension work remain serialized under the existing plan.

[justinhelmer]

Containing build live — September 25, 2026, 10:16 AM PDT

The automatic v1.262.2 release workflow completed successfully after #2359 merged at exact reviewed head 5663170a73c088ce8c4e65b36d671b4490d57436 as 21696a703d549c2655b5487ca0a738d408e306ac and its containing #2361 merged as 6e30af1fca5413c37ae12556c8863f4fa0126476.

The deploy job reports memory Worker version 817c4e8d-0056-4456-9549-1fd033622c71, bot Worker version 52d7a590-f17d-4866-a59a-2392336061f3, and the bot container application advancing from image 1.262.1 to 1.262.2. An independent public /healthz read at 10:16:46 AM PDT returned ok:true, draining:false, connected Slack, and exact build commit 6e30af1fca5413c37ae12556c8863f4fa0126476.

This validates the containing build's live identity. It does not validate original-unit recovery behavior or resolve the follow-up F1/F2/F3 cases recorded above. The #2236 producer-identity repair and separate one-time 90-minute extension remain unimplemented and serialized in that order. No retained original unit was retried or replaced. Stability remains 0 certified / 24h; the latest conservative reset is the 9:45 AM PDT CI stall observation.

[justinhelmer]

Priority/ownership receipt, September 25, 2026, 10:55 AM PDT: labeled this existing issue P0 and placed it in Todo on the Switchboard v1.2 board. The 9:00:03 AM self-hosting recurrence and unproved original-unit recovery keep this on the factory recovery path. #2359 is merged and its containing v1.262.2 build is live, but its late-review retry/resume gaps and the original-unit functional receipt remain open.

Codex is the portfolio coordination owner. The next admitted repair remains the separate #2236 producer-identity prerequisite; the one-time 90-minute extension follows it. No duplicate issue, writer, ref, retained-unit retry, or source-overlapping parallel repair was admitted by this triage. Recheck at 11:05 AM PDT or on a supported capability/review event.

[justinhelmer]

Retained original-unit recurrence, September 25, 2026, 2:44 PM PDT

Three open Switchboard PRs remain stranded on their original units after the preventive #2359/#2363 source repairs shipped. Current runs.unit readback:

• #2345, plan-fix-p0-https-github-com-40cbdb:U1, ended at U1/1/findings with HTTP 409 publication_binding_missing. Exact retained head 5458e8b; three bot CI checks also fail.
• #2347, plan-at-current-main-9f4d0f4c-ab5030:U1, completed a findings push to exact 3505a6e8, then ended at U1/1/findings/pr-check with HTTP 409 publication_facts_mismatch. No exact-head re-review started; known getter-safety finding remains.
• #2357, plan-implement-the-screenshot-0b7e5e:U1, completed a findings push to exact f9d55793, then ended at the same publication_facts_mismatch boundary. No exact-head re-review started; CodeQL and two security threads remain.

The separate one-time 90-minute extension implementation unit plan-implement-the-separate-o-8dd912:U1 held before edits or PR. The current bare recovery handler only accepts a terminal review boundary, so that pre-PR held unit cannot be resumed through it. No original product unit was retried or replaced. This is not a CI flake: the parent failures are typed publication-state refusals, while the CI/security issues are independent work remaining after supported recovery. Keep this P0 open for a proven identity-preserving continuation through original PR/ref/head, with new-head CI/review and exact live acceptance; #2359's deployment alone did not close that path.

[justinhelmer]

P0 repair admission — September 25, 2026, 3:27 PM PDT

The remaining #2350 publication-state repair now has a distinct Switchboard Ship writer. This is a platform repair, not a retry or replacement of any stranded product unit.

• Original Ship thread
• Parent run: bc956777-9b4f-4730-bf59-3de318d96ee5 (working, round-0 coding at admission readback)
• Repair unit: plan-implement-p0-https-githu-7bdd8a:U1
• Repair branch: plan/implement-p0-https-githu-7bdd8a/u1

The request covers the recorded F1/F2/F3 gaps and the publication_binding_missing / publication_facts_mismatch continuation boundary. It requires same-unit, same-ref, existing-PR positive proof and stale/foreign refusal; current-head CI and review; and a person merge/release handoff. Source overlap with the live #2376 decision-record writer must be serialized before edits. The #2345/#2347/#2357 and polylanedotcom #405 original units, refs, heads and PRs remain untouched and terminal. The separately held one-time recovery-extension implementation is not folded into this repair. No original-unit recovery or product PR unblocking is yet claimed.

Release workflow v1.262.4 completed successfully; authenticated status.show serves exact 7e76aedb8ad5b5ed4a0a93daa300937a9b7272e8 / 1.262.4. This containing release does not include the new #2350 repair.

[coreplane-switchboard]

Repair scope reconciliation before edits: this child is on plan/implement-p0-https-githu-7bdd8a/u1, rebased to main at 7e76aedb8ad5b5ed4a0a93daa300937a9b7272e8. Proposed write set is src/channels/adminCoordinator.ts, its test file, and the publication/recovery proof rows in docs/reference/specs/agent-ship.md; any additional required path will be reconciled before editing.

Read #2376's latest admission receipt: its separate writer owns proposed decision record 0079, docs-only. Read #2370's terminal clean-tree hold: directory implementation is not admitted until that prerequisite merges. Remote readback places both original refs at 7e76aedb; the held extension ref remains at 8b5d4701. This repair will not edit decision records, identity/directory sources, extension logic, or any retained product ref/PR. The declared scopes do not overlap. No live runtime reservation MCP is connected to this child, so this is reconciliation against the published admission/hold receipts plus remote refs, not a claim of a live reservation-ledger read.

Next: regressions for F1/F2/F3 through the existing coordinator continuation boundary, preserving refusal of ambiguous, stale and foreign evidence. No original product recovery is being attempted.

[coreplane-switchboard]

Repair child pushed 0663d5efc18f4959fe77cc30aa494059dbe3da11 on the admitted original repair ref plan/implement-p0-https-githu-7bdd8a/u1 (unit plan-implement-p0-https-githu-7bdd8a:U1, parent bc956777-9b4f-4730-bf59-3de318d96ee5). Immediately before push it rebased onto main at 1ac78cc0, including the now-merged docs-only #2377 prerequisite; no source conflict occurred.

F1/F2/F3 regression fixes are implemented: distinguish a proven earlier failed/no-push attempt from competing publication, reconstruct the exact current resume namespace from durable wake answers, and compare binding fields independently of JSON key order. Also fixed terminal numeric /findings/aN/pr-check recovery attribution so the unique successful retry can proceed to read-only exact-head review. Added real coordinator-handler/recovery-driver tests for identity-preserving success and stale/foreign/ambiguous refusal.

Rebased-head proof: 278 coordinator tests passed; scoped TypeScript, ESLint, formatting, hygiene, spec bindings/coverage/test guard, docs consistency and PR-title checks exited 0. Strict description submitted. GitHub PR lookup for this ref currently returns no open PR: the child post-step still needs to publish it before parent Ship CI/review can complete. Not strict-ready or merge-ready yet. Full CI, independent exact-head review, person merge/release and any later live acceptance remain gates.

No deployment, protected product-unit recovery, product branch/PR update, or extension activation occurred. #2345/#2347/#2357, polylanedotcom #405 and plan-implement-the-separate-o-8dd912:U1 remain preserved. Unknown legacy budgets remain fail-closed. Keep this issue open; tests are not a live recovery receipt.

[coreplane-switchboard]

PR #2380 review fix round remains on the admitted repair ref plan/implement-p0-https-githu-7bdd8a/u1, with the same three-file write set: src/channels/adminCoordinator.ts, its tests, and docs/reference/specs/agent-ship.md. Rechecked scope receipts: #2376's docs-only prerequisite merged as #2377 (1ac78cc0); #2370 still records a no-edit hold and its remote ref remains at 7e76aedb. The held extension ref remains at 8b5d4701. No declared source overlap; no runtime reservation MCP is available, so this is published-receipt/remote-ref reconciliation, not a live ledger read. This round addresses review F1/F2: same-head invalid retry boundaries and competing pushes to different heads. No protected product unit/ref/PR or extension is being recovered or modified.

[justinhelmer]

Recurrence on 2026-09-25 PDT: Switchboard PR #2413, original Ship thread, unit plan-fix-issue-2250-so-coding-678140:U1.

After review F5, the authorized coding child rewrote the owned ref and pushed head 749f320b6b9870610ccdc3cb0ab402e9204a9dc0 (previous review head c4c8085d64211abaf0ce784e972fb2cf7c38db0c). Ship failed at U1/3/findings/pr-check before starting the final review with HTTP 409 publication_facts_mismatch: "the open pull request was not durably bound." A single same-thread plain re-review at the verified new head was refused because Switchboard expected the old c4c8085d head; no review ran. GitHub showed the PR open at 749f320b, and current-head Depot CI was green, including the memory Worker verify.

Recovery: one new top-level read-only review routed to review at the verified head. No second writer, branch, PR, retry, merge, deploy, or secret provisioning. The final verdict is pending.

This matches the binding and same-thread continuation failure documented above. The review workaround does not repair the Ship unit's durable PR binding.

[justinhelmer]

Further recurrence on Switchboard #2414, original #2412 Ship thread, unit plan-fix-p0-issue-2412-from-c-85c007:U1, September 25, 2026 around 9:26 PM PDT.

The first review requested F1/F2. Its single owned findings child b4b8acec-3176-4b76-9084-99b24f7ce802 completed and pushed the original ref to exact e8db530a578469166f7a7ab4516598371a4b1fd5, with fixed dispositions. GitHub has #2414 open at that head and current-head CI/CodeQL green. The original parent then ended failed at U1/1/findings/pr-check: HTTP 409 publication_facts_mismatch, “the open pull request was not durably bound.” It did not start re-review. The original branch and PR were not replaced; no alternate coding writer was admitted.

This is the same post-rewrite durable binding transition recorded here and on #2413. A separate explicit read-only exact-head review is being routed as a bounded review workaround; it cannot repair the original unit's publication record. Keep original-unit recovery and cumulative limits intact, and require a current-head review before any person merge.