[beinan]

Native catch-up currently waits for a shard's complete read/index/delete/append/drain before reading the next shard, and repeats that commit sequence after at most eight generations even when the 64 MiB byte allowance is mostly unused.

This change overlaps the next shard's preparation with the current shard's commit using a one-entry queue. It reserves queue capacity before reading, retains at most two batches, and shares the existing memory budget. Commits stay in shard order inside the existing task-local maintenance fence; no ordinary worker fan-out or concurrent base commits are introduced. If a large prefetched generation cannot grow its memory reservation, the producer waits for the preceding commit and retries that shard once alone, without admitting later readers. Other read errors let the preceding commit finish; commit failure/cancellation drops the queued work. The generation cap becomes configurable and defaults to 64, with the byte and process memory bounds unchanged. Both controls are explicitly passed to new Jobs; setting CATCHUP_PIPELINE_ENABLED=false and CATCHUP_MERGE_MAX_GENERATIONS=8 restores the previous scheduling.

Validation covers overlap and bounded lookahead, commit order, a budget admitting only one batch, oversized-prefetch growth contention, read/commit failure, cancellation, Job configuration, and native multi-shard merges with continuous ingestion and overlapping keys. Includes an opt-in end-to-end comparison of serial/8, pipeline/8, and pipeline/64 on unique-key and repeated-update WAL. The expanded local catch-up suite and all-target master Clippy passed on final head b8631a5, including deterministic oversized-prefetch contention recovery and native multi-shard merges under three memory budgets. All 12 benchmark fixtures were rerun successfully on this head. All 10 CI checks passed on this exact head, including Rust/Python tests, etcd HA, coverage, three platform builds, Clippy and style. This does not activate catch-up or change production resources.

End-to-end measurements on local ARM64 storage with Lance 9.0.0 and local etcd (debug build, shared host; synthetic data, not production/Azure throughput): four shards × 32 generations × 8 rows × approximately 8 KiB payload, about 8 MiB sealed WAL per fixture. Each cell averages two fresh fixtures, with the second order reversed. Timing includes the native executor and ownership protocol; fixture creation and complete result verification are outside timing.

Workload	Serial / 8 generations	Pipeline / 8	Pipeline / 64
Unique IDs	3.563 s / 35.9 generations/s	3.084 s / 41.5 generations/s	1.573 s / 81.4 generations/s
Repeated updates	3.188 s / 40.1 generations/s	2.828 s / 45.3 generations/s	1.465 s / 87.4 generations/s

On the final head, pipeline alone improved throughput by 13–16%; combined with the larger count cap, 2.18–2.27×. Final base version was 49 with cap 8 versus 13 with cap 64, consistent with less repeated index/delete/append work. Every fixture reached zero pending and verified every final payload and the base live-row count (without WAL union deduplication). These small-WAL fixtures do not establish large-blob or cloud-storage speedup; the unchanged byte cap will often dominate those workloads.

Reproduce using CATCHUP_BENCH=1 CATCHUP_BENCH_WORKLOAD=unique ETCD_TEST_ENDPOINTS=http://127.0.0.1:2379 cargo test -p lance-context-master benchmark_native_catchup_pipeline -- --ignored --nocapture; repeat with workload updates. The benchmark is explicitly gated and does not run during the ignored CI suite without CATCHUP_BENCH=1.