Repository navigation
Conversation
A successor ended its watch once a lease and a half had passed on its own clock, so a listing whose answer stalled, or that it read late after a pause, counted silence storage never showed, and it took the lock from an owner that was still beating. The takeover then deleted the old lease directory, claim included, and the owner, which read only the claim, kept beating and serving beside the successor. A failed claim read also counted as no claim. Silence now counts only up to the send of a listing that found no newer beat. Before each beat the owner reads the claim and lock.json, the record whose if-match decides the owner and which never names a replaced lock id again, and fences when either says the lock is no longer its own; a read that gets no answer sends no beat and is retried until the deadline. A takeover keeps the previous owner's lease directory, so an owner whose clock stopped, or one that reads only the claim, stops at its next beat.
…ock bound Review of the lease fix: `take_over` no longer deletes the superseded owner's lease directory, so the mutable-key inventory's registration of that `delete_prefix` described the defect and failed the count. `state_lock::LOCK_RECORD_MAX_BYTES` is now the one bound on every read of `lock.json`, so the lease's ownership check cannot read with a smaller one and stop a live server at its deadline. The control-plane and storage-failure pages say the owner stops on a claim or a lost record at its next beat, at most a third of a lease later; only the lapse is a timer of its own.
The owner's fence ran in the lease's own task, so the deadline took effect only when that task ran. After a pause of the whole process any task may resume first: a writer could be admitted and send past the deadline, and an admitted request could send a retry or a part, before the lease task fenced the scope. A beat started after the deadline also counted from that start, so a resumed keeper could renew a lease that lapsed during the pause. `StorageIoScope::fence_at` gives the scope the deadline, which the keeper moves forward with each acknowledged beat. Admission, a multipart part and every poll of an admitted request check it before sending, and the first to find it passed fences the scope. A fenced scope's deadline no longer moves, and the keeper then reports the lapse. No beat starts at or after the deadline. `until_fenced` checks the fence before polling the request, so a request woken together with a fence sends nothing more.
|
Recommendation: ready. WhyThe owner's right to write ends at one monotonic deadline. On What
How the invariant holdsEvery scoped mutation checks the deadline and sends only while that deadline is still ahead, whichever task runs first. Heartbeats and the claim use the lease client, outside the scoped store. The deadline still covers the calls that can extend the lease.
A request already in the kernel or the SDK when the deadline passes can still land. The next poll sends nothing more. The RFC already treats that the same way as a crash: a fresh key, a conditional publication, or an audited mutable key. The clock is The scope is the right owner of the check. It is already the mutation send point. The keeper remains the owner of the instant, from the two-thirds rule. One ImpactA paused server stops storage writes at its deadline. That includes a writer, a retry, and a multipart part. A resumed keeper leaves a lapsed lease unrestored when the beat would start after the deadline. The server still exits 2 without releasing. Successor silence is unchanged. Each poll of an admitted mutation takes the scope mutex and reads the clock. Reads skip that check. There is no benchmark. The cost sits beside a storage call. GQT cannot express task order or a clock jump. BlockingNone. Optional
LiabilityThe fence and the send point are now the same object. One check on the scope covers that pause for every scoped caller. A further deadline is another The new surface is Open work does not pull this apart. #1217 is the parent protocol. #1176 changes conditional control-object re-issue and does not move this deadline. A future Lance write path that bypasses ValidationHead: Local isolated worktrees. Temporary probes were removed. Nothing was pushed. On
Failed. On
12 passed, including
Passed.
Passed. Before that restore, the same pause test with an added beat-listing equality also passed on CI on LimitsNo local measurement of the per-poll clock read. No Inline comment intended for The wait here is one full lease. A keeper that sent one beat after this jump, and counted that beat from the time after the jump, would set a new deadline two thirds of a lease ahead and then fence inside this wait. The test would still pass. Please compare the beat names from before the jump with the names after the fence, for both orders. I ran that comparison on this commit. The names were the same. On This is optional. The entry check in |
Stacked on #1217: review only the last commit. Until #1217 merges, this diff also shows its commits.
Closes the limit #1217 left open: after a pause of the whole process, a task other than the lease's could send a storage request past the owner's deadline.
What went wrong
The owner's fence ran in the lease's own task.
LeaseKeeperfenced the storage scope when its deadline timer fired, so the deadline took effect only when that task ran. After a pause of the whole process (a stop signal, a stalled host, a starved runtime) any task may resume first. A writer that resumed before the lease task was admitted and sent its request past the deadline. A request already admitted could send a retry or a multipart part the same way. A successor may act five sixths of a lease after that deadline, so a long enough pause let the old owner write beside it.The keeper had the same gap for its own beats. A beat it started after the deadline counted from that start, so with a store that answers at once, a resumed keeper could renew a lease that had lapsed during the pause.
The fix
StorageIoScope::fence_at(deadline)sets it, and the keeper moves it forward after each acknowledged beat, starting from the lock write's send. Admission, a multipart part, and every poll of an admitted request check it before sending. The first one to find it passed fences the scope: it refuses, cancels the admitted requests, and wakes them. Whichever task resumes first sends nothing past the deadline.fence_atreturns false once the scope is fenced, and the keeper then reportsLapsed, so the server exits instead of serving with a scope that refuses every write.until_fencednow checks the fence before each poll of the request, not after. Before, a request woken at the same time as a fence was polled first and could send one more attempt.Clean release is unaffected: it writes through a fresh scope that has no deadline, and its if-match decides the outcome at any time.
Docs: the scope paragraph (§4) and the owner and timing paragraphs (§7) of the storage-outcomes RFC, with a Decision-log entry naming the superseded sentence; the lease paragraph of
docs/dev/control-plane.md; anddocs/user/operations/storage-failures.md. New changelog notecluster-lease-deadline-fence.fixed.md.Tests
lease::tests::a_paused_owner_sends_nothing_after_its_deadline_issue_1067(new). It uses paused time and a holder adapter on the owner's scope. While the owner beats, the holder writes across three leases. Then the clock jumps a lease, and the next line runs before any other task. When the writer runs first, its write is refused before it is sent. When the lease task runs first, it starts no beat. Either way the owner fencesLapsedand the late file never exists. On the parent commit, both cases fail: the writer's write succeeds, and the keeper renews and never fences.scoped::tests::a_scope_fences_itself_at_its_deadline_issue_1067(new). A deadline moved forward before it passes keeps the scope open. A request admitted before the deadline, whose client would resend after it, sends one attempt and returns the ambiguous fenced error. Later parts, completion and new requests are refused as unsent. A fenced scope's deadline no longer moves.omnigraph-storagegainstokio'stest-utilas a dev-dependency for paused time, asomnigraph-clusteralready has.Limit that remains
An owner whose monotonic clock stopped (a suspended VM) resumes with its deadline still ahead. Only the reads before its next beat stop it, up to a third of a lease later, as #1217 and the RFC state.