Skip to content

feat: configure WAL merge indexes with key-only ZoneMap deletes - #300

Merged
beinan merged 2 commits into
mainfrom
codex/verify-merge-index-cost
Oct 3, 2026
Merged

beinan merged 2 commits into
mainfrom
codex/verify-merge-index-cost

Conversation

@beinan

@beinan beinan commented Oct 3, 2026 •

Copy link
Copy Markdown
Collaborator

WAL merges currently require a BTree on the base key. Make that policy configurable with ROLLOUT_KEY_INDEX_TYPE=btree|zonemap, retaining BTree as the default and propagating the selected policy through workers, master indexing and native catch-up Jobs.

Selecting ZoneMap also changes deletion to bounded, typed ID predicates before the append. A direct index-type substitution is insufficient: Lance 9.0.0's non-exact-index merge_insert plan still scans target payload columns with an ID-only source. Predicates avoid that projection while preserving WAL drain ordering, retry behavior, byte budgets and existing commit fencing. Each expression is capped at 1,024 IDs; larger batches may require multiple delete commits.

Validation and measurements are documented in docs/benchmarks/merge-index-policy.md, with a reproducible Azure benchmark and JSON results. In fresh-process tests, ZoneMap was faster for 262k-key / 8-fragment layouts, similar in median time for 64 MiB payload batches, and slower with substantially more physical reads for 2,048 fragments. The results do not justify changing the fleet-wide default. One Azure manifest PUT timeout is recorded and excluded as an incomplete fixture; its fresh-prefix rerun passed.

All processes maintaining the same table must use the same policy to avoid repeatedly replacing id_idx. This PR changes no production configuration. Benchmarks used isolated synthetic datasets on real Azure storage; they do not measure end-to-end WAL throughput or prove behavior on full production-sized tables.

Local validation: workspace Clippy with -D warnings, formatting, and 37 merge-related core regression tests passed (one manual benchmark ignored). The new regression checks physical row counts and every payload across multiple predicate-delete chunks, repeated updates, unusual ID literals, scheduled index rebuilds and a subsequent switch back to BTree. GitHub CI covers the remaining workspace and platform checks.

@beinan
beinan merged commit 3c1ed02 into main Oct 3, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant