Repository navigation
Execute WAL catch-up inside resident masters - #326
Merged
Merged
Conversation
beinan
marked this pull request as ready for review
October 7, 2026 18:29
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WAL catch-up currently needs remote staging workers or a separate one-table executor. Add an explicit resident-master mode that reads and encodes WAL locally through the existing durable MergeWal queue and fenced publisher. Additional ordinary master replicas can share the queue without creating per-table Kubernetes Jobs.
ROLLOUT_APPEND_LOCALapplies only to opted-in owned append-only rollouts. Staging shares one process-wide Arrow budget and cache session; table concurrency uses the existing separate merge pool. Startup rejects missing shard identities, inadequate budgets, a shared general/merge pool, and conflicting Kubernetes catch-up configuration. Low-count tail scheduling also works without worker endpoints. Defaults retain the existing remote path.Existing old-writer guards, recovery fences and retry policy remain in force. Switching this flag does not adopt protected external publishers. Memory reservations are not a process RSS cap; oversized indivisible generations and other master work still require headroom.
Validation: all 174 master tests pass against local etcd, including two replicas sharing the queue without workers and a saturated resident merge budget while general indexing completes. All GitHub checks pass, including Rust workspace tests, coverage, Clippy and Python tests/builds. Production has not enabled resident local staging; the pinned release will be qualified on isolated staging before publisher handoff.