fix(bucketindex): scope block identity to the tenure and fence commits on it

[tdakkota]

Part of #725: the first of three PRs. Blocks become term-scoped, and every clustered commit is fenced on the writer's tenure.

Changes

• Term-scoped block identity (bucketindex format v7). A block is now Block{Term, N}. The term is the etcd create revision of the claim that allocated it. Interval stays a canonical set over the (term, n) order. Held runs never cross terms, and a merge across tenures holds each tenure's blocks, with the rest of the lower term as an ordinary gap. NextBlock(term) allocates within the writer's term, so two tenures that never saw each other's index (a displaced owner and its successor, or diverged replicas) can't collide. A writer with no cluster (term 0) continues the highest term's sequence. AllocatedBlocks is now a Block.
• Writing term on entries and wants (Entry.Term, Want.Term). An identical block set at the same level resolves to the higher term through Lineage.Subsumes, and so through Supersedes, Satisfying, Revokes and Subsumed.
• Commit fence (bucketindex.CheckTenure, ErrSuperseded). Both engines check before every CAS attempt. A clustered writer refuses to commit when it holds no claim, when its tenure ended and restarted after a merge picked its inputs, or when a load or rebase shows the index was written by a later term. A rebase never adopts a higher term, so a displaced writer never commits under its successor's term (the Generation.Next path). A clustered engine without the claim now leaves LoadParts wants pending instead of committing them.
• Flush and Close under the fence. A flush checks the tenure before it writes anything, so a refused flush writes no part and its rows go back to the head. An engine's Close treats that refusal as nothing to flush: the head stays in the WAL. Storage.Close now flushes the engines it owns before it leaves the cluster, while it still holds the claims. Before this, the final flush ran after leaving and committed under a stale term. Now it would be refused, and every clean restart would re-flush heads the new owner had already flushed (the Windows CI failure on the first push, from part files still open under a refused commit).
• A tenure is established before it commits anything. The tenure check reads the claim locally, so a predecessor's CAS that passed it could still land after the claim moved. Before a tenure's first flush, merge or repair, the engine now reloads the index from scratch and commits it back under the new term (establishTenureLocked), and until that commit lands every commit of the term is refused.
  • Over a shared store, a predecessor's held CAS then either fails (the index version moved), rebases into the higher term and is refused, or it landed first. If it landed first, the fresh load makes it the new tenure's starting point, instead of a rebase carrying the inputs it consumed back in beside it.
  • Over private backends nothing orders the two owners' writes. There the protection is identity: term-scoped blocks, and the higher-term tie-break when the two indexes meet. Duplicates are the accepted cost.
  • Each flush and each merge makes one establishing attempt: one index load and one CAS, under the same locks a flush holds to publish. There is no retry loop, and no lock is held between attempts. While no attempt lands, the engine behaves like one whose flush keeps failing: nothing drains, and the head and WAL keep the rows and grow. The ingest bound is the optional per-tenant in-flight limit, and mandatory backpressure is feat(storage): opt-in blocking write backpressure when the head cannot drain #146 (not part of this PR).
  • It is observable: storage.index.establish_failures{signal, reason=conflict|unavailable} counts failed attempts, and SignalStats.TenureUnestablished marks an engine that holds the claim but has not established its tenure. Both are in ADMIN.md.
• Close reports a failed owned flush. flushOwned returns the first failure while still flushing every engine, and Storage.Close returns it.
• Leaving the cluster ends the tenure. After the lease was revoked on close, Ownership kept reporting the ended terms, so an engine's final flush committed under a term that no longer existed. Ownership.Forget now drops the held claims when the node leaves. An engine's Close also closes its WAL when the flush fails for any other reason; before this, the WAL handle leaked, which Windows CI caught.
• Shared planning and allocation. bucketindex.Plan, PlanMerge and Allocator replace the two identical blockid.go copies in engine and recordengine.
• Docs. backend/, engine/, recordengine/ and cluster/ARCH.md and ARCHITECTURE.md are updated: the new invariant, the v7 format and the v6 migration.

v1–v6 indexes still decode. A v6 index reads as term 0 throughout. FuzzDecode also found an older defect, fixed here. A v5 hull that names no set (inverted, or touching block 0) was kept verbatim, but it was written back as unset, so decode∘encode was not the identity. It now reads as unset, and the input is pinned in the corpus.

Measurements

Index encode/decode and the want trim, over a 100-entry index with 512 tombstones (backend/bucketindex/bench_test.go). Old is main's package, new is this PR. n=8, alternating runs:

               │   old.txt   │              new.txt               │
               │   sec/op    │   sec/op     vs base               │
IndexEncode-32   5.558µ ± 1%   7.093µ ± 2%  +27.63% (p=0.000 n=8)
IndexDecode-32   21.84µ ± 4%   25.01µ ± 3%  +14.49% (p=0.000 n=8)
TrimWants-32     179.9µ ± 1%   172.1µ ± 2%   -4.31% (p=0.000 n=8)

               │   allocs/op   │ allocs/op   vs base
TrimWants-32     1585.00 ± 0%   17.00 ± 0%  -98.93% (p=0.000 n=8)

Encode and decode cost 1.5µs and 3µs more per commit and load of a 100-part index, because every block now carries a term. Interval walks its runs without materializing them, and covered folds every entry's runs with one sort. Without that, the doubled Gap size made TrimWants 2.2× slower with 6× the allocations.

Tests

• Ungated bucketindex: part identity breaks under divergent replica lineage #725 reproducers: TestEngineSuite/{DisplacedWriterKeepsItsTerm, SplitBrainMergeSameInputs, SplitBrainMergeOverlappingInputs}, for both engines. The displaced merge and flush now fail with ErrSuperseded and commit nothing.
• New suite tests (both engines):
  • TenuresAllocateDisjointBlocks: the block-collision reproducer the issue asks for. Two tenures on diverged copies flush the next part.
  • StaleCommitCannotLandAfterTheSuccessorEstablishes: the predecessor's merge is held inside the index CAS, the claim moves, the successor commits, then the held CAS is released. It is refused, and its output never becomes live.
  • PredecessorCommitLandsBeforeTheSuccessorEstablishes: the held merge lands after the claim moved but before the successor's first write. The successor's establishing load takes it as its starting point, so the merged inputs don't come back (this test failed before the change).
  • UnestablishedTenureKeepsRowsInTheHead: establishment fails repeatedly, first on a lost CAS and then on a backend error. Each flush and merge makes exactly one CAS, and the counter rises by reason. No part is written, the rows stay readable from the head, and TenureUnestablished is set. The first cycle that lands publishes every held row and clears the bit.
  • LapsedClaimCommitsNothing: no claim means no commit, rows are kept, and a merge that outlives its tenure is refused.
  • WritingTermSurvivesTheNextTenure.
• bucketindex: v7 golden, v6 decode golden, v7 corruption cases, multi-term property tests for the set algebra, CheckTenure and Allocator tables, and multi-term seeds for FuzzRoundTrip and FuzzDecode.
• TestClusterHandoffUnsyncedFlushReadsOnce/logs is re-gated from bucketindex: part identity breaks under divergent replica lineage #725 to Deduplication: no signal dedups re-delivered records #572 (dedup), with its assertion unchanged. The old owner's unsynced flush, re-flushed by the new owner, is the duplicate the design accepts. No catch-up can see it, because the old owner is out of the ring.
• TestClusterCloseFlushesWhileTheClaimIsHeld: an owner's Close drains its head into a part, the node no longer claims the shard once it has left (this assertion failed before Forget), and a claimless node closes cleanly.
• TestClusterCloseReportsAFailedOwnedFlush: an injected index-commit failure during the owned flush is Close's error (this test failed before the change).
• TestClusterHandoffUnsyncedFlushReadsOnce now waits until the old owner has taken the shard back and merged, and the secondary has mirrored it, instead of running exactly three maintenance cycles. An establishing attempt that meets a concurrent index write waits for the next cycle, and on the Windows runner three cycles were sometimes not enough. The assertions are unchanged.
• TestRecoveredEngineRepairs* now restart the shard's compaction owner and take its claim before merging, since a claimless engine no longer commits.
• Gates:
  • go test -p 2 ./...
  • go test -race on the changed packages and ./internal/...
  • golangci-lint run --concurrency=2 ./... (exit 0)
  • OTELDB_STORAGE_REPRODUCE=1 go test -p 2 ./...: every failure is gated on another issue. They are TestClusterHandoffUnsyncedFlushReadsOnce/logs (Deduplication: no signal dedups re-delivered records #572), TestRebaseBlindToAdoptedPart (engine: rollup guard misses the layout of an unreadable adopted part #745, PR2), TestRerollLateOverwrite (engine: a late overwrite of an already-rolled sample cannot replace it #732), and TestRerollDuplicateSplitByMaxParts, TestRerollOracleDuplicates and FuzzRerollOracleDuplicates (engine: raw duplicates rolled in separate merges are double-counted by the rollup fold #739).
  • 30s runs of each fuzzer: FuzzRoundTrip, FuzzDecode, FuzzLivePart, FuzzPartOf, FuzzDecodeKeyList. The Codex round-1 fixes change no format and no fuzzed code, so these were not re-run for them.

Follow-ups: #746 (primary writes should require the claim). PR2 adds the lineage catalog, the rollup layout in the entry (#745) and the nested false hole. PR3 adds the bounded handoff catch-up.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KcLq1G8KpWBvM53JaQb1tS

[codecov]

Codecov Report

❌ Patch coverage is 96.87988% with 20 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
backend/bucketindex/block.go	95.23%	4 Missing ⚠️
engine/index.go	94.44%	4 Missing ⚠️
recordengine/index.go	94.44%	4 Missing ⚠️
engine/merge.go	76.92%	3 Missing ⚠️
recordengine/merge.go	76.92%	3 Missing ⚠️
backend/bucketindex/bucketindex.go	96.61%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!