Skip to content

Recover task lease streams without replacing ownership - #425

Draft
beinan wants to merge 4 commits into
mainfrom
codex/task-lease-stream-recovery
Draft

beinan wants to merge 4 commits into
mainfrom
codex/task-lease-stream-recovery

Conversation

@beinan

@beinan beinan commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

A keepalive stream error currently stops task lease renewal permanently, even if etcd connectivity recovers. This can discard in-progress work when its claim lease expires.

Reconnect the same existing lease with bounded RPCs and a fixed recovery window of one configured lease TTL after the stream fails. A missing acknowledgement is not proof that etcd failed to renew: a real-etcd regression verifies that requests can renew the server lease while delayed responses exhaust the client’s previous confirmation window. Let etcd decide whether the same lease is still renewable. Never grant a replacement lease or recreate ownership keys. Dropping the claim stops renewal, revoked leases cannot return, and a sustained outage terminates within the bounded retry window.

Renewal is not evidence of merge progress or permission to publish. Existing progress watchdogs and publication ownership checks remain independent. Add a recovery counter and failure/recovery logs.

Validation (Draft, final candidate checks still running):

  • Before this change, a corrected TCP-cut fixture proves the original implementation loses its claim after a short outage. The initial proxy fixture was invalid and its results are not used as short-outage evidence.
  • The earlier candidate passed three fault tests in CI stable and coverage, plus 12 concurrent local repetitions (36 test passes). One earlier local run also experienced unplanned etcd disk sync delays of 3.16 and 6.20 seconds with a six-second test lease; its exact renewal exit branch was not logged.
  • A new deterministic delayed-response test fails on the earlier candidate at unacknowledged renewals must be retried: the direct etcd observer has already verified that the original claim token, lease ID, and a positive server TTL remain intact. Logs confirm termination at the local confirmation deadline. This establishes the recovery-window defect without relying on the disk-delay hypothesis.
  • The delayed-response case passes individually on the updated product code. Its first four-way run exposed two test-timing defects during a 3.28-second etcd disk sync: the response gate was held closed while an observer RPC waited, and the client-stop deadline also included the server deletion/read delay. Both failures are retained. The fixture now restores responses before observer RPCs and independently asserts client termination within the original ten-second bound, then server-side deletion. Product retry limits are unchanged.
  • Four real-etcd fault cases are rerunning concurrently on the final test correction: short TCP interruption; delayed renewal acknowledgements; sustained outage; and preservation of a replacement owner after revocation. Results pending.
  • Formatting and diff checks pass. The previous stable CI failure was in the separate server demand-event test, not the lease tests; its failure is retained rather than hidden. No production deployment of this PR has occurred.

This is a separate follow-up to the qualified production image in services #9811. It does not retag or delay that release. Production claim-loss logs motivated the investigation but do not establish that this defect caused those particular failures.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant