Repository navigation
Conversation
added 4 commits
October 21, 2026 03:35
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A keepalive stream error currently stops task lease renewal permanently, even if etcd connectivity recovers. This can discard in-progress work when its claim lease expires.
Reconnect the same existing lease with bounded RPCs and a fixed recovery window of one configured lease TTL after the stream fails. A missing acknowledgement is not proof that etcd failed to renew: a real-etcd regression verifies that requests can renew the server lease while delayed responses exhaust the client’s previous confirmation window. Let etcd decide whether the same lease is still renewable. Never grant a replacement lease or recreate ownership keys. Dropping the claim stops renewal, revoked leases cannot return, and a sustained outage terminates within the bounded retry window.
Renewal is not evidence of merge progress or permission to publish. Existing progress watchdogs and publication ownership checks remain independent. Add a recovery counter and failure/recovery logs.
Validation (Draft, final candidate checks still running):
unacknowledged renewals must be retried: the direct etcd observer has already verified that the original claim token, lease ID, and a positive server TTL remain intact. Logs confirm termination at the local confirmation deadline. This establishes the recovery-window defect without relying on the disk-delay hypothesis.This is a separate follow-up to the qualified production image in services #9811. It does not retag or delay that release. Production claim-loss logs motivated the investigation but do not establish that this defect caused those particular failures.