fix(bucketindex): scope block identity to the tenure and fence commits on it

ADMIN.md

@@ -100,6 +100,7 @@ taking only a brief per-engine read lock to copy counters — safe to poll at da
 | `LostParts` | the shard's **monotone** data-loss count as its bucket index records it: every hole its writers have ever committed, revoked ones included. It never decreases and every owner reads the same number, so it is the durable answer to "did this shard ever lose data?" — unlike `Holes`, which is the current state. Alert on any increase |
 | `IndexFenced` | this engine's last bucket-index load failed — the index was unreadable, or a part it names could not be opened for a reason other than being gone (a corrupt manifest, a backend error, a format this binary cannot read) — so it **refuses every commit**: flush, merge, retention, repair and `CompactNow` fail with an error wrapping `bucketindex.ErrFenced`. A commit from an engine whose view is not the stored index could drop every entry it never opened. Without a cluster, reads keep answering from the last part set that loaded, and this flag with `IndexLoadError` is the health error to alert on; a cluster node instead disclaims every read of the shard (`rpc.shard_incomplete`), so reads fail over to another owner, and fail when every owner is fenced. Flushes keep their records in the head, which therefore grows until `tenant.Limits.MaxInFlightBytes` sheds writes (without bound when it is zero, the default); the WAL keeps them durable meanwhile. Every maintenance cycle retries the load, and the first that succeeds clears this and lets the held-back flush commit. A part found corrupt by three consecutive loads is handed to repair: the next load records it as a want (`WantedParts`, `corruption.detected{disposition="wanted"}`) and the fence lifts. Any other failure in between starts the count over. Two causes never clear on their own: a backend that keeps failing, and a part in a format newer than this binary (`index.fenced_loads{reason="unsupported_version"}`), which stays fenced until the node runs a release that reads it |
 | `IndexLoadError` | the error of the failed load behind `IndexFenced`, naming the index or part and why (`… corrupt metadata …`, `… unsupported version N …`, or the backend's error); empty while not fenced |
+| `TenureUnestablished` | this cluster node holds the shard's claim but its tenure's first index commit has not landed. A new owner reloads the index and commits it back under its term before anything else of the tenure may commit, so a displaced owner's held commit cannot land after it (`engine/ARCH.md`, *A clustered commit is fenced by tenure*). Until it lands the engine commits nothing — like `IndexFenced`, flushes keep their records in the head, which grows until `tenant.Limits.MaxInFlightBytes` sheds writes (without bound when it is zero, the default); the WAL keeps them durable — and every flush and merge makes one more attempt, counted in `index.establish_failures`. Set for a moment after every handoff; one that stays set is an index this node cannot load or commit to |
 | `HasReadGap` / `ReadGapAfterUnixNano` | this node holds the shard but came back without its unflushed head (a restart, or a rebalance handing it over), so it disclaims reads at or past that timestamp and lets another owner answer. Clears when a flush by the shard's compaction owner puts newer data in this node's parts. `math.MinInt64` means the shard had no parts at all — nothing here is known to be complete |
 
 Part *byte* sizes are intentionally omitted from `Inspect` (they would need backend stat calls) — use
@@ -254,6 +255,7 @@ Metric instruments (all prefixed `storage.`):
 | `wal.sync_failures` / `wal.sync_failing` | — | background fsyncs that failed, and a gauge: 1 while the latest sync pass had a failure (`StoreStats.WALSync`). A non-zero gauge means `WALSyncInterval`'s power-loss bound is not holding; the sync retries every tick, so it clears on its own once the disk recovers |
 | `corruption.detected` | `component`, `disposition` | on-disk artifacts that failed an integrity check. `disposition=fatal` is an operation that failed on one: a corrupt manifest or bucket index (`part`, `bucket_index`) fails the load. These count only errors carrying a corruption sentinel, never a backend that failed to answer. `disposition=wanted` is a part that stayed corrupt over three consecutive loads and was handed to repair as a want (`part`), one count when it is handed over. `disposition=tolerated` is corruption the engine fell back from and served on: a bloom sidecar (`bloom` — the part stops pruning on that column), a part's identity object (`part_identity` — its rows stay unresolvable until a merge rewrites the part), the paged series index or series-stats sidecar (`series_index`, `series_stats` — slower opens and aggregates), marks (`marks` — no granule pruning), an adopted part (`part` — left out of the read set), a record part whose rows are not timestamp-ordered within a stream, or whose stream column is not grouped in stream order (`stream_order` — merged without losing a row from a whole decode, once per merge that reads it, but its windowed reads are already unreliable), a damaged WAL region (`wal` — one count per region replay skipped at `Open`; its records are lost, a Warn log names the segment, offset and bytes, and the segment is copied aside as `{segment}.damaged`, the newest four kept). A tolerated artifact counts at every open or first read that meets it, so a steady rate is one damaged part read repeatedly, not new damage |
 | `index.fenced_loads` | `signal`, `reason` | bucket-index loads that failed and left the engine refusing commits (`SignalStats.IndexFenced`). `reason` is `corrupt` (a damaged index or part; a part that stays so is handed to repair after three loads), `unsupported_version` (a part written by a newer release; never clears until this node is upgraded) or `unavailable` (the backend failed to answer). A fenced engine retries once per maintenance cycle and each failure counts again, so a steady rate is one engine that stays fenced; a single count that stops is a transient failure the next retry cleared. A corrupt artifact behind it also counts in `corruption.detected` with `disposition=fatal` |
+| `index.establish_failures` | `signal`, `reason` | attempts to make a new tenure's first index commit that did not land (`SignalStats.TenureUnestablished`). `reason` is `conflict` (the commit lost the index CAS to another writer — a predecessor still committing, or another node that believes it owns the shard) or `unavailable` (the backend failed the load or the commit). Each flush and each merge makes one attempt, so a steady rate is one engine whose tenure never establishes and whose head is growing; a few counts right after a handoff are the predecessor's last commits |
 | `parts.orphans_swept` | `signal` | part objects no index entry names, deleted by the sweep at open: what failed flushes and merges of the previous process left behind. It moves only at `Open` (and a cluster owner's backfill reload) — a flush that fails at runtime leaves its objects until the next restart — so a flat line says nothing about orphans accumulating now |
 | `tenant.reserved_skipped` | `source` | a tenant whose id is reserved (`wal`, any case, or containing `\`) was found and left untouched, each also logged at Error with the tenant and prefix. `source` is `recovery` (a bucket index in the backend), `wal_replay` (a WAL directory), `bootstrap` (an etcd shard claim) or `partsync` (a peer's prefix). Only a release predating the reservation writes such a tenant; its data stays on disk, unserved, until an operator moves it |
 | `parts.orphans_deferred` | `signal` | unnamed part objects the same sweep **kept** because their part id is younger than `Options.OrphanGrace`: over a shared store such a part may be another node's flush or merge that has not committed yet. Counted by every load that defers them, so it is a rate of deferrals rather than a count of distinct objects. A steady rate on a store only this node writes means failed flushes whose leftovers wait for a load after they age; it only matters if it keeps growing, which is disk or inodes held by debris |

ARCHITECTURE.md

@@ -197,6 +197,14 @@ to fail over to: a read overlapping a want fails.
   stamping ownership terms, while still serving reads and still applying replicated writes. Gating
   on the lease deadline rather than on etcd reachability is what keeps a blip free and a partition
   safe (`cluster/ARCH.md`, *Lease fencing*).
+- **A clustered writer commits only as the shard's current tenure.** Every index commit first
+  re-checks that the writer still holds its claim, under the term the operation began with, and
+  that the index it builds on was not written by a later term (`bucketindex.CheckTenure`), and a
+  tenure's first commit re-establishes the index under its term, so over a shared store no displaced
+  writer lands a commit after its successor's first. Part identity is scoped the same way: a block is
+  a `(term, n)` pair, so two tenures that never saw each other's index — every handoff over private
+  backends — cannot allocate the same identity. What the fence leaves possible is duplicate rows,
+  never a collision (`backend/ARCH.md`, *Part identity*).
 - **The bucket index commits last, conditionally.** It is what makes a part durably visible — and in
   `recordengine` it also carries the flush watermark (the WAL replay floor, one slot per writer since
   the watermark is per node and the index is shared) — so its write is the
@@ -253,9 +261,9 @@ to fail over to: a read overlapping a want fails.
   reads as unknown, never raw) / marks (`OTMK`) / column object framing (the
   trailer-dictionary bytes column; the leading layout still read) and key layout, the attribute
   hash+binary encoding (the SeriesID pre-image), symbol table (`OTSY`), the bucket index (format
-  v6: entries with their block sets and split claims, tombstones, wants, per-writer flush
-  watermarks and the block high-water mark — readable back to v1, not writable by older
-  nodes), WAL record framing
+  v7: entries with their term-scoped block sets, split claims and writing term, tombstones, wants,
+  per-writer flush watermarks and the block high-water mark — readable back to v1, not writable by
+  older nodes), WAL record framing
   (additive record types), record-key footer (`OTKY`), the metric part column layout
   (`[series:int128, ts:int64, value:float64]` + optional `sf:float64`), and the profile
   symbol-store sidecar (`OTSP`, version 2: a compressed body with its raw length; version 1 still

backend/ARCH.md

@@ -165,8 +165,11 @@ a demonstration.
   to commit at all and wraps `ErrFenced`: a version is only safe to commit against together with
   the part set read with it (`engine/ARCH.md`).
 - **Part identity is a block *set* plus a level** (`bucketindex.Interval`, `Entry.Level`, format
-  v6). A flush writes `{n}` at level 0; a merge writes the union of what its inputs covered, above
-  them. `Entry.Supersedes` is then decidable from identity alone — no index diff, no bookkeeping,
+  v6). A flush writes `{b}` at level 0; a merge writes the union of what its inputs covered, above
+  them. **A block is a `(term, n)` pair** (`bucketindex.Block`, format v7): the ownership term of the
+  tenure that allocated it and a number within that term. Blocks order by term, then number, and a
+  merge across tenures holds each tenure's blocks with the rest of the lower term's number space as
+  an ordinary gap. `Entry.Supersedes` is then decidable from identity alone — no index diff, no bookkeeping,
   no data read — which is what lets a repair terminate: by the time a want is serviced the data may
   exist only inside a merged successor, and `Index.Satisfying` accepts that successor as discharging
   the want.
@@ -203,18 +206,52 @@ a demonstration.
   short of reading the rows says how. A part covering only some of a group's claim therefore
   overlaps every member while replacing none; no single-part rule resolves it, and repair refuses
   to publish one beside the other (`engine/ARCH.md`, "Repair").
-  **Allocation is the shard owner's alone**: `Index.NextBlock` is one above `AllocatedBlocks`, the
-  persisted high-water mark, and above every block the live entries, wants and group runs still
-  name. The mark is what keeps identity unique over a shard's *whole life* rather than over its
+  **Allocation is the shard owner's alone, and scoped to its tenure**: `Index.NextBlock(term)` is one
+  above `AllocatedBlocks`, the persisted high-water mark, and above every block the live entries,
+  wants and group runs still name — within `term`, starting at `(term, 1)` for a tenure the index has
+  not seen. Scoping is what makes identity unique across writers that never see each other's index:
+  a displaced owner and its successor, or two replicas whose lineages diverged, each allocate only
+  under their own claim's etcd create revision, which no other claim shares. The obvious alternative,
+  one number space with the loser of a CAS re-allocating above the winner, holds only while both
+  writers commit to one object; over diverged copies it hands two unrelated parts one identity, and
+  every identity relation then treats them as one part. A term below the index's highest one
+  continues that term's sequence instead of restarting its own; only a writer with no cluster (term
+  0, the prefix's only writer) reaches that, since a clustered one is fenced first (below). The mark is what keeps identity unique over a shard's *whole life* rather than over its
   current contents: the live set shrinks — retention can empty a shard outright, leaving tombstones
   that carry no blocks — and numbering derived from it would rewind and hand a new part the identity
   an expired one held, at which point a stale peer's old part satisfies a want for the new one and
   expired data is committed as a repair. Numbers are claimed by the same `CompareAndSwap`
   that adds the part; a writer rebasing on a rival's index takes the maximum of the two marks — see
   `engine/ARCH.md`, "Block identity is allocated by the commit that publishes the part". No etcd, no
   round trip on the flush path, and it works with the cluster layer absent. Two owners racing a
-  handoff resolve through that CAS: one commit lands, the loser wraps `ErrConflict`, re-reads, and
-  re-allocates above the winner.
+  handoff over one index resolve through that CAS: one commit lands, the loser wraps `ErrConflict`,
+  re-reads, and re-allocates above the winner.
+  **One identity written by two tenures resolves to the later one** (`Entry.Term`, format v7). Over
+  one shared index the fence below keeps it from arising; over private backends two owners can merge
+  the same inputs either side of a handoff, and the two outputs meet through `cluster/partsync` and
+  repair. Both cover the same blocks at the same level and hold the same rows, so `Lineage.Subsumes`
+  — and through it `Supersedes`, `Satisfying`, `Revokes` and `Subsumed` — lets the higher `Term`
+  replace the lower.
+- **A clustered writer commits only as the shard's current tenure** (`CheckTenure`, `ErrSuperseded`).
+  Before each CAS attempt the engine checks the term it holds now against the term its operation
+  began under and the term of the index it builds on. It refuses when it holds no claim (a lapsed or
+  fenced lease reads as term 0), when its tenure ended and restarted mid-operation, and when a
+  rebase shows a later tenure wrote the index — a rebase never adopts a higher term, so a displaced
+  writer never commits under its successor's. A refused commit leaves its output unreferenced, for
+  the orphan sweep. A writer with no cluster has no tenure and is not fenced.
+  The check reads the claim locally, so on its own it cannot stop a commit that passes it and then
+  lands after the claim moved. **A tenure's first commit is what closes that window**: the new
+  owner reloads the index and commits it back under its term before any flush, merge or repair of
+  the tenure may commit (`engine/ARCH.md`, "A clustered commit is fenced by tenure"). That moves the
+  index version, so a predecessor's held CAS either landed first — and is the new tenure's starting
+  point — or fails, rebases into the higher term and is refused. The CAS is the only serialization
+  point two writers share, so this is also why the guarantee is per index: **over a shared store**
+  a displaced writer never lands a commit after its successor's first one; **over private backends**
+  (`cluster.Config.PrivateBackend`) each node commits to its own index, nothing orders a displaced
+  owner's writes against its successor's, and the protection is identity alone — term-scoped blocks
+  cannot collide and an identical identity resolves to the later tenure. What that admits is
+  duplicate rows — an old owner's last flush, re-flushed by the new owner from its replica head, or
+  overlapping merges — never a collision, a stuck want or a false hole.
 - **`Entries → Removed | Wanted`** is the invariant the repair path enforces: a part leaves
   `Entries` only into a tombstone (`Removal`, a deliberate deletion) or into a `Want` (an
   obligation to fetch it back). Conflating the two would make "am I repaired?" unanswerable, which
@@ -262,13 +299,17 @@ a demonstration.
   is a fact, not a level: a per-node gauge that a restart or a successful repair clears erases the
   only record that a range of a shard was ever acknowledged as gone. It follows ClickHouse's
   `/lost_part_count`.
-- **Format v6 is a hard read break.** `Decode` rejects any version above the one it knows, so
-  reading is backward compatible (v1–v5 still decode, unset fields ordering below everything a
-  writer produces) but **writing is not**: a node on pre-v6 code fails on the first v6 index it
+- **Format v7 is a hard read break.** `Decode` rejects any version above the one it knows, so
+  reading is backward compatible (v1–v6 still decode, unset fields ordering below everything a
+  writer produces) but **writing is not**: a node on pre-v7 code fails on the first v7 index it
   reads. Every node that reads a given index must be upgraded together. The path differs by
   deployment and both matter — on a shared backend every node reads the same index object, and in
   `PrivateBackend` mode `cluster/partsync` reads its peers' indexes.
-  Two things migrate. A **v5 interval is a hull**, and what it actually covered is unknowable once
+  **A v6 index is term 0 throughout**: its blocks, its entries' and wants' writing terms and its
+  high-water mark. A clustered tenure then allocates from `(term, 1)` above all of it, and a writer
+  with no cluster continues term 0's numbering where v6 left it. An interval's lowest block number
+  leads its encoding, so a valid set never starts with the zero byte an unset one is written as.
+  Two things migrate from v5. A **v5 interval is a hull**, and what it actually covered is unknowable once
   decoded, so it is read as the contiguous set of its bounds — the meaning it had when it was
   written. Such an entry can still claim a block no part held, exactly as it did under v5; a merge
   rewriting it produces an exact set, so the defect drains as the shard compacts rather than being