Skip to content

perf: skip base ID checks for clean rollout WAL after worker restarts - #306

Merged
beinan merged 8 commits into
mainfrom
codex/rollout-append-replay-ranges
Oct 5, 2026
Merged

beinan merged 8 commits into
mainfrom
codex/rollout-append-replay-ranges

Conversation

@beinan

@beinan beinan commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

A writer restart currently expands rollout base-ID checks across every pending WAL generation. Record the original legacy prefix and each writer epoch’s first generation, so clean immutable generations skip base-table membership queries. Import expanded older cutoffs through bounded metadata history; missing history keeps conservative checks. Atomic fragment/watermark commits, fencing, and older coordinator compatibility remain intact.

CI also exposed Lance9’s existing batched WAL PK-sidecar error in TraceStore list/restart reads. Unfiltered GenericStore lists recover that exact error using native single-ID LSM queries on one consistent snapshot. Candidate keys are capped at1,024 IDs /1MiB /16,384 physical rows /10seconds; datasets are traversed one at a time without a backlog-sized path vector. Filtered and point reads retain the server’s existing cache refresh behavior. Box the recovery future to keep downstream master test types within rustc layout limits.

Validation:

  • Final source d2b9c52: local whole-workspace/all-targets test compilation, cargo check, and Clippy with warnings denied pass. GenericStore24/24, original TraceStore3/3, server GenericStore routes11/11, and both bounded fallback regressions pass. All runtime CI tests, including etcd-backed HA and instrumented coverage tests, also pass at this runtime revision. The final workflow-only commit upgrades the verified Codecov uploader to v5, saves the LCOV report as an artifact, and makes upload independently retryable after a v4 TLS handshake failure. All11 checks pass. LCOV is saved as a downloadable Actions artifact. The external Codecov TLS endpoint still fails; v5 respects the existing fail_ci_if_error:false policy rather than crashing the test job. Actual external upload did not succeed; TLS and GPG verification remain enabled.
  • Append/recovery regressions and populated-base ABBA benchmark pass; all1,056 IDs verified. Legacy staging0.758s average versus precise0.598s (~26.6% higher local staging throughput, not a production estimate).
  • Azure persisted-WAL upgrade/restart/steady/retry verification checks192 full unique records, zeroOOM, steady batches perform zero legacy ID lookups. This validates the unchanged merge implementation at3e871e9.

Limits: bounded list recovery does not fix the upstream PK-sidecar implementation; filtered or oversized failing scans still return errors. An immediate-crash test separately reproduced an existing Lance9 durability-acknowledgement bug with OLD writer+OLD merge (12/16 records). That ingestion issue remains unresolved and is not represented as fixed here.

@beinan
beinan marked this pull request as ready for review October 5, 2026 02:41
@beinan
beinan merged commit e57a94e into main Oct 5, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant