Repository navigation
perf: skip base ID checks for clean rollout WAL after worker restarts - #306
Merged
Merged
Conversation
beinan
marked this pull request as ready for review
October 5, 2026 02:41
added 6 commits
October 5, 2026 03:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A writer restart currently expands rollout base-ID checks across every pending WAL generation. Record the original legacy prefix and each writer epoch’s first generation, so clean immutable generations skip base-table membership queries. Import expanded older cutoffs through bounded metadata history; missing history keeps conservative checks. Atomic fragment/watermark commits, fencing, and older coordinator compatibility remain intact.
CI also exposed Lance9’s existing batched WAL PK-sidecar error in TraceStore list/restart reads. Unfiltered GenericStore lists recover that exact error using native single-ID LSM queries on one consistent snapshot. Candidate keys are capped at1,024 IDs /1MiB /16,384 physical rows /10seconds; datasets are traversed one at a time without a backlog-sized path vector. Filtered and point reads retain the server’s existing cache refresh behavior. Box the recovery future to keep downstream master test types within rustc layout limits.
Validation:
d2b9c52: local whole-workspace/all-targets test compilation, cargo check, and Clippy with warnings denied pass. GenericStore24/24, original TraceStore3/3, server GenericStore routes11/11, and both bounded fallback regressions pass. All runtime CI tests, including etcd-backed HA and instrumented coverage tests, also pass at this runtime revision. The final workflow-only commit upgrades the verified Codecov uploader to v5, saves the LCOV report as an artifact, and makes upload independently retryable after a v4 TLS handshake failure. All11 checks pass. LCOV is saved as a downloadable Actions artifact. The external Codecov TLS endpoint still fails; v5 respects the existingfail_ci_if_error:falsepolicy rather than crashing the test job. Actual external upload did not succeed; TLS and GPG verification remain enabled.Limits: bounded list recovery does not fix the upstream PK-sidecar implementation; filtered or oversized failing scans still return errors. An immediate-crash test separately reproduced an existing Lance9 durability-acknowledgement bug with OLD writer+OLD merge (12/16 records). That ingestion issue remains unresolved and is not represented as fixed here.