fix(bucketindex): keep split lineage and rollup layouts in the index

[tdakkota]

Part of #725 (the second of three PRs). Fixes #745.

Changes

• Lineage catalog (bucketindex format v8). Index.Catalog records every split group's claim (RecordLineage, called on every commit). The record outlives the entries that carry the claim. That matters because a merge that consumes a whole group folds its claim away, and a group split again leaves the outer claim on no entry. The catalog is canonical: sorted by group, one entry per group, validated on decode.
  • MaxLineage (= MaxRemovals) is a target, not a hard cap. TrimCatalog ages the oldest groups out first, but never a claim that an outstanding want's or hole's ancestry passes through, or that a live entry carries. A catalog whose claims are all reachable stays above the target, and the engine warns once.
  • MergeLineage no longer trims; only a writer that has the whole index does.
• One lineage everywhere. Index.Lineage() returns the catalog plus the claims still on entries and wants. Satisfying, Discharging, TrimWants and TrimHoles (and through it Revokes) all relate identities by it, and so do joint coverage (Covered realizes any group the lineage records, transitively) and Missing. Missing now asks for the members of every group derived from a want's rows that no later group consumed, so a member that was split again is fetched through its own split.
  • The shared repair pass (internal/repair) carries the engine's catalog through every index it builds: planning, the member rounds, committable/groupsNeeded, Admit and prune. A successor that partsync found through a retired outer claim is admitted, not dropped. The catalog reaches the pass through Host.ObligationsLocked.
  • partsync FetchWants reads the local index and answers each want with SatisfyingWith over both sides' lineage. If the local index can't be read, every unanswered want in that pass is an error, not absence.
  • accountsFor uses both lineages. Installing a peer's index keeps the local catalog's groups, trims the merged catalog against the installed index, and installs it whenever its contents differ from the peer's (not its length).
  • Engines record the catalog only when a commit lands, because a lost CAS attempt's group run is handed out again. A rebase unions the rival's catalog.
• Rollup layout in the index entry (engine: rollup guard misses the layout of an unreadable adopted part #745). Entry.Rollup (nil means unknown) sits behind a new flag bit. Every commit writes its own parts' layouts from their manifests, and an adopted entry's layout is filled in once its part opens. The rollup guard reads an unopened adopted part's layout from its entry, and planning sees it through a stand-in part with no reader. An entry whose layout is still unknown and whose part can't be opened only exists in a pre-v8 index. It counts as raw and never aborts a commit; aborting would stall that merge for good.
• Groups only the catalog records (Codex round 3). Several answers looked for groups only on the entries, not in the whole lineage. They now use every group the lineage records.
  • Subsumed (and through it repair's prune) judges every group the parts complete, including an outer group whose members were all split again. It retires what the group claims below the lowest level among the parts derived from it. Before, a live divergent copy of the ancestry was never retired, the repaired unit was refused for overlapping it, and the want stayed stuck.
  • A joint answer (Satisfying, Discharging) names any part that descends through a recorded split, not only one that carries a claim. A merge that consumed a whole inner group has folded its claim away.
  • TrimHoles revokes a hole that a complete split group covers jointly. Run already counted such a hole as revoked on every pass, but it stayed in the index.
  • Repair's groupsNeeded now considers every group that a unit's rows derive through, including catalog-only ones, when the group's ancestry overlaps what the node holds. That also takes in the group's members that were split again. A lost outer member owed beside a divergent copy of its ancestry now pulls in the rest of the outer group through the members' own splits, and the commit retires the copy. Before, the inner fragments were refused for overlapping it on every pass.
• Catalog trimming keeps every live part's ancestry (Codex round 4). TrimCatalog used to protect only a live entry's own claim. A saturated catalog could then age out the outer claim a live nested member descends through. If that member was lost later, its want carried only the inner claim and the false hole came back. Every live entry now reaches its whole ancestry, as wants and holes already did. The walk shares one visited set across the whole part set (Relater.reachAncestry), so each group is followed once.
• Completion is not bounded by FetchesPerCycle: repair: split-group completion is not bounded by FetchesPerCycle #751. Split-group completion rounds have run outside the per-cycle cap since fix(repair): commit split groups whole and cap a cycle in units #720 (repair: a split group wider than repairFetchesPerCycle+1 is never repaired, and records duplicate its first fragment every cycle #717). This PR widens which groups they reach: catalog-derived Missing, and the ancestry groups in groupsNeeded. It does not change the loop, so the bound is tracked in repair: split-group completion is not bounded by FetchesPerCycle #751 (performance, area/cluster, area/engine, area/recordengine; no rows are dropped). There is a one-line note in engine/ARCH.md.
• Docs. Updated backend/, engine/ and cluster/ARCH.md and ARCHITECTURE.md: the catalog, the layout, the v8 format and the v7 migration.

v1–v7 indexes still decode. A v7 index has no layouts and an empty catalog, and the first v8 commit fills in both.

Measurements

Lineage closures (Codex round 2). Every identity question over a lineage now goes through one Relater, built per index or batch (Index.Relations). The Relater indexes the claims' block runs, follows each closure with a work queue that visits only the claims a newly gained run meets, and caches the result. So a commit's TrimWants, a repair pass, or partsync's answers to one peer compute each part's closure once. TestRelaterMatchesTheFixedPoints checks it against the naive fixed points on 3000 random nested lineages.

New scaling benchmarks (bench_lineage_test.go): a nested catalog near MaxLineage (500 chains, 8 levels deep, 4000 claims), 100 live parts, and a rising number of wants. Old is 35ea0fb, new is this head, n=6:

                                    │   35ea0fb    │                head                │
                                    │    sec/op    │   sec/op     vs base               │
IndexEncode-32                        7.070µ ± 1%   7.234µ ± 6%        ~ (p=0.065 n=6)
IndexDecode-32                        24.85µ ± 4%   25.60µ ± 5%        ~ (p=0.240 n=6)
IndexLoad-32                          37.56µ ± 1%   38.34µ ± 3%        ~ (p=0.240 n=6)
TrimWants-32                         143.16µ ± 1%   95.04µ ± 1%  -33.61% (p=0.002 n=6)
TrimWantsNestedLineage/wants=1-32    26.710m ± 1%   2.801m ± 4%  -89.51% (p=0.002 n=6)
TrimWantsNestedLineage/wants=16-32  422.159m ± 1%   2.964m ± 2%  -99.30% (p=0.002 n=6)
TrimWantsNestedLineage/wants=256-32 6739.778m ± 3%  8.564m ± 3%  -99.87% (p=0.002 n=6)
SatisfyingWithNestedLineage/1-32    113.286m ± 6%   3.250m ± 7%  -97.13% (p=0.002 n=6)
SatisfyingWithNestedLineage/16-32  1802.126m ± 2%   3.715m ± 4%  -99.79% (p=0.002 n=6)

(The 35ea0fb side of SatisfyingWithNestedLineage is partsync's old pattern, one SatisfyingWith per want. The head side builds relations once per peer, which is what partsync now does.)

Codex round 3 fix. Old is b02ed6f, new is this head. n=6, alternating runs; SubsumedNestedLineage is new and completes n nested chains over the 4000-claim catalog:

                                        │  b02ed6f   │               head                │
                                        │   sec/op   │   sec/op     vs base              │
TrimWantsNestedLineage/wants=1-32         2.686m ± 4%   2.801m ± 3%       ~ (p=0.065 n=6)
TrimWantsNestedLineage/wants=16-32        2.999m ± 6%   3.041m ± 6%       ~ (p=0.699 n=6)
TrimWantsNestedLineage/wants=256-32       8.702m ± 8%   8.670m ± 6%       ~ (p=0.394 n=6)
SatisfyingWithNestedLineage/wants=1-32    3.292m ± 7%   3.405m ± 6%       ~ (p=0.310 n=6)
SatisfyingWithNestedLineage/wants=16-32   3.578m ± 9%   3.760m ± 4%       ~ (p=0.132 n=6)
SubsumedNestedLineage/chains=1-32         1.497m ±11%   1.465m ± 2%       ~ (p=0.394 n=6)
SubsumedNestedLineage/chains=16-32        7.369m ± 3%   6.649m ± 3%  -9.77% (p=0.002 n=6)
SubsumedNestedLineage/chains=256-32       199.4m ± 2%   165.4m ± 3% -17.07% (p=0.002 n=6)
IndexLoad-32                              37.55µ ± 3%   36.93µ ± 2%  -1.65% (p=0.041 n=6)
TrimWants-32                              91.49µ ± 1%   97.54µ ± 0%  +6.61% (p=0.002 n=6)

TrimWants (100 entries, no claims, no catalog) runs the same code path on both sides: every want misses and returns at the coverage check before any changed code. The whole difference is flat time in Relations.satisfying, whose code is unchanged. Reverting this head's jointlySatisfying alone leaves the +6.3%, and so does a pointer-iteration variant (+10%). So this is code layout, not work.

Codex round 4: catalog trimming over the whole live set. RecordLineageSaturated (new) trims a catalog 200 claims over the target, with 4200 claims and 600 live parts: every chain's innermost member and the 100 successors. Old is e1da206, new is this head, n=8, alternating runs:

                          │   e1da206   │               head                │
                          │   sec/op    │   sec/op     vs base              │
RecordLineageSaturated-32   1.740m ± 3%   2.726m ± 8%  +56.63% (p=0.000 n=8)
TrimWants-32                97.69µ ± 0%   96.94µ ± 1%   -0.77% (p=0.015 n=8)

The +1ms is the fuller reachability: walking 600 live parts' ancestry, where the old code walked only the wants and holes (none here). One closure per part cost 6.6ms (19k allocations), and one closure over the union of all the parts' blocks cost 390ms, because growing an interval of thousands of runs rebuilds it on every step. The shared visited set costs one visit per group.

e2e profile. The engine package ran locally under the exact e2e command (go test -race -v -coverpkg=./... ./engine/):

tree	TestReseedHorizonRate	engine package
origin/main	164.6s	216.8s
35ea0fb	166.1s	217.3s
this head	164.5s	217.1s

The CI e2e timeouts on this PR come from the runner: main's own e2e already took 874.9s of the 900s budget. They are unrelated to this change, and #750 stays open for them.

Old is main (v7), new is this PR, on the same 100-entry index with 512 tombstones and no layouts or catalog. n=8, alternating runs:

               │   old.txt   │              new.txt              │
               │   sec/op    │   sec/op     vs base              │
IndexEncode-32   7.116µ ± 1%   7.130µ ± 1%       ~ (p=0.721 n=8)
IndexDecode-32   25.07µ ± 4%   24.87µ ± 2%       ~ (p=0.505 n=8)
IndexLoad-32     38.13µ ± 3%   37.59µ ± 3%       ~ (p=0.130 n=8)
TrimWants-32     173.9µ ± 1%   156.6µ ± 0%  -9.98% (p=0.000 n=8)

               │   B/op        │    B/op      vs base
IndexDecode-32   62.83Ki ± 0%   62.86Ki ± 0%  +0.05% (p=0.000 n=8)
IndexLoad-32     84.11Ki ± 0%   84.14Ki ± 0%  +0.04% (p=0.000 n=8)

The same index with a two-tier layout on every entry and a 256-group catalog (new benchmarks, n=6):

• encode: 17.6µs
• decode: 54.3µs

Decoding the catalog checks its order in one pass. The first version re-sorted it to compare, and that decode took 65.8µs.

Tests

• Codex round 1 regressions (each test was written first):
  • TestEngineSuite/NestedWantRepairedThroughTheCatalog (both engines, H1): the fetcher returns a successor that satisfies the want only through a retired outer claim, and repair admits it and discharges the want. The test fails with the catalog withheld from the pass.
  • TestJointCoverageThroughRetiredOuterClaim (H2): every outer member was split again into complete inner groups. The want for the original ancestor is answered jointly, a peer missing one inner member is asked for exactly that one, and a node that knows only the outer split asks for its members.
  • TestTrimCatalogKeepsReachedAncestry and TestTrimCatalogOutgrowsTheTargetOnlyByReachedClaims (M1): an old nested want plus more than MaxLineage newer groups keeps its ancestry, and the want is still satisfiable. When every claim is reachable, the catalog exceeds the target and reports it.
  • TestSyncKeepsTheLocalLineageBesideAFullCatalog (M2): a full peer catalog plus a newer claim only this node recorded; the merged result is installed.
• Codex round 3 regressions (each test was written first and failed before the fix):
  • TestAdmitRetiresAncestorOfCatalogOnlyGroup (repair): the reported case. Admit gets a catalog-only outer claim over {1,2}, complete inner groups, and a live divergent copy of block 1. The copy is retired, and the fragments commit.
  • TestRunMemberBesideDivergentAncestor (repair): Run plus Admit for a lost outer member beside the same copy. The pass fetches the whole outer group through member 11's split, and the commit retires the copy. It fails with the old groupsNeeded.
  • TestSubsumedByCatalogOnlyGroup, TestJointlySatisfyingByMergedFragments and TestTrimHolesRevokedByCompleteGroup (bucketindex).
  • TestRelaterMatchesTheFixedPoints also checks which groups realize completes against the naive fixed point.
• Codex round 4 regression: TestTrimCatalogKeepsLiveEntryAncestry (written first; it failed before the fix). A live nested member carries only the inner claim, the outer claim is only in the catalog, and there are more than MaxLineage newer groups. The outer claim is kept, and once the member is lost, a successor of the outer ancestry answers its want. TestRelaterMatchesTheFixedPoints checks reachAncestry against the naive ancestry, including accumulation across calls.
• Nested false hole: TestFetchWantNestedGroupFromOuterSuccessor (partsync) is the reproducer the issue asks for. Blocks {1,2} were split into {10..12}, and 12 with 7 into {20..22}. Member 20 is lost, and the only surviving copy is the peer's higher-level successor of {1,2,7}. The want is now answered instead of reported absent. It failed before the partsync change.
• Ungated: TestRebaseBlindToAdoptedPart (engine: rollup guard misses the layout of an unreadable adopted part #745).
• Layout tests:
  • TestEntryLayoutFilledFromTheManifest covers both fill paths, adopted on rebase and own at load.
  • TestUnknownLayoutOfAnUnreadablePartIsRaw covers the unreadable legacy path: the commit is not aborted.
• Catalog tests:
  • TestEngineSuite/SplitLineageOutlivesItsMembers (both engines): the claim stays in the catalog after retention drops every member.
  • bucketindex: catalog recording, the MaxLineage bound, SatisfyingWith, TrimWants/TrimHoles through the catalog, and Lineage.With aliasing.
  • partsync: accountsFor through either side's lineage, and install keeping the local catalog.
• Format: v8 golden, v7 decode golden, v8 corruption cases (unknown flag, layout, and a non-canonical catalog), multi-state layouts and catalogs in the property test and FuzzRoundTrip, and a v7 seed for FuzzDecode.
• Gates:
  • go test -p 2 ./...
  • go test -race on the changed packages and ./internal/...
  • golangci-lint run --concurrency=2 ./... (exit 0)
  • OTELDB_STORAGE_REPRODUCE=1 go test -p 2 ./...: every failure is gated on another issue. They are TestClusterHandoffUnsyncedFlushReadsOnce/logs (Deduplication: no signal dedups re-delivered records #572), TestRerollLateOverwrite (engine: a late overwrite of an already-rolled sample cannot replace it #732), and TestRerollDuplicateSplitByMaxParts, TestRerollOracleDuplicates and FuzzRerollOracleDuplicates (engine: raw duplicates rolled in separate merges are double-counted by the rollup fold #739).
  • 30s fuzz runs, all passing: FuzzRoundTrip, FuzzDecode, FuzzLivePart, FuzzPartOf, FuzzDecodeKeyList.

PR3 (bounded handoff catch-up) follows.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KcLq1G8KpWBvM53JaQb1tS

[codecov]

Codecov Report

❌ Patch coverage is 97.74306% with 13 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
cluster/partsync/repair.go	72.72%	3 Missing ⚠️
engine/index.go	75.00%	3 Missing ⚠️
recordengine/index.go	77.77%	2 Missing ⚠️
backend/bucketindex/block.go	95.23%	1 Missing ⚠️
backend/bucketindex/bucketindex.go	97.43%	1 Missing ⚠️
backend/bucketindex/relater.go	99.17%	1 Missing ⚠️
backend/bucketindex/relations.go	98.80%	1 Missing ⚠️
backend/bucketindex/rollup.go	96.42%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!