Repository navigation
Recover worker merges and route new-table catchup through masters - #422
Merged
Merged
Conversation
beinan
force-pushed
the
codex/durable-worker-merge-admission
branch
from
October 20, 2026 23:59
02891c9 to
89e89c8
Compare
added 7 commits
October 21, 2026 00:42
beinan
force-pushed
the
codex/durable-worker-merge-admission
branch
from
October 21, 2026 00:47
558604f to
72b2eb3
Compare
beinan
marked this pull request as ready for review
October 21, 2026 01:28
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Worker merges started by HTTP, generation thresholds or cleanup timers can outlive their caller without a durable owner, and new hot tables can fall outside the master's static native-merge list. This change records those executions before storage work, lets the master reconcile them using actual progress and storage fencing, and adds opt-in resident catchup for new rollout and generic tables.
Worker admission is coalesced and uses the existing bounded slots. Demand publication has a separate four-request metadata bound so occupied merge slots do not hide hot tables or block ingestion flushes. The master inventories executions after endpoint/configuration changes, preserves progressing writers and external helper ownership, and uses canonical claims, shared memory budgets and bounded continuation passes to keep catching up across masters. Persisted protocol mode prevents flag rollback from silently restoring untracked writes. Master maintenance is tracked; worker HTTP compaction is refused in fenced mode. No separate recovery pool or new crate is introduced, and worker fanout remains serial.
The combined workload exposed reads racing with WAL reclamation. Consistent reads now distinguish actual base-version progress from repeatedly missing data at the same version: up to 16 snapshot captures, three missing-file failures without progress, and a two-second budget for starting another attempt. Valid long scans are not cancelled and partial results are never accepted. Rollout point reads retain the cached-base fast path and WAL-only semantics; generic filtered reads recover actual missing-file races while preserving ordinary cache misses and pinned reads. Transitional append wildcards also keep new unowned targets on fenced worker fanout until automatic native routing is enabled.
Validation on the final
f37522druntime:Six worker admission/progress-isolation tests and six master routing/inventory tests also passed before the final read-only changes. All 11 exact-head CI checks passed, including Rust unit/integration, etcd HA, coverage and coverage upload. CI tested the merged tree with main
1e793de; its tree ID is642115cd00726663aeae2575f7404892b4ae66e9. Local fixtures use isolated etcd/storage and debug binaries, so their latency and memory observations are not production capacity claims.Production activation still requires the new release image to pass cloud-storage staging and a verified one-time transition covering old worker/master writers and external helpers.
MASTER_LOCAL_MERGE_AUTO=truewithMASTER_LOCAL_MERGE_TARGETS=*is enabled only after that transition; a protocol marker, Pod age or empty inventory is not proof that historical writes have drained. Seedocs/worker-merge-admission.mdfor configuration and rollback constraints. Existing production cleanup has not been interrupted or handed off by this PR's local validation.