Skip to content

Recover worker merges and route new-table catchup through masters - #422

Merged
beinan merged 10 commits into
mainfrom
codex/durable-worker-merge-admission
Oct 21, 2026
Merged

beinan merged 10 commits into
mainfrom
codex/durable-worker-merge-admission

Conversation

@beinan

@beinan beinan commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

Worker merges started by HTTP, generation thresholds or cleanup timers can outlive their caller without a durable owner, and new hot tables can fall outside the master's static native-merge list. This change records those executions before storage work, lets the master reconcile them using actual progress and storage fencing, and adds opt-in resident catchup for new rollout and generic tables.

Worker admission is coalesced and uses the existing bounded slots. Demand publication has a separate four-request metadata bound so occupied merge slots do not hide hot tables or block ingestion flushes. The master inventories executions after endpoint/configuration changes, preserves progressing writers and external helper ownership, and uses canonical claims, shared memory budgets and bounded continuation passes to keep catching up across masters. Persisted protocol mode prevents flag rollback from silently restoring untracked writes. Master maintenance is tracked; worker HTTP compaction is refused in fenced mode. No separate recovery pool or new crate is introduced, and worker fanout remains serial.

The combined workload exposed reads racing with WAL reclamation. Consistent reads now distinguish actual base-version progress from repeatedly missing data at the same version: up to 16 snapshot captures, three missing-file failures without progress, and a two-second budget for starting another attempt. Valid long scans are not cancelled and partial results are never accepted. Rollout point reads retain the cached-base fast path and WAL-only semantics; generic filtered reads recover actual missing-file races while preserving ordinary cache misses and pinned reads. Transitional append wildcards also keep new unowned targets on fenced worker fanout until automatic native routing is enabled.

Validation on the final f37522d runtime:

  • Five real merge/GC read regressions passed. The repeated-merge regression failed against the old implementation before the fix.
  • Four-reader HTTP workload: 2,312 calls without errors, four newly created tables drained to zero pending, exact content checked, and native merging continued after master restart with the initiation flag disabled.
  • The same workload with 1 MiB rollout blobs: 2,586 calls, all four tables at zero pending, full point/blob/content checks, maximum observed request duration 1.425 seconds and peak per-process RSS below 292 MB.
  • Actual paused-worker recovery: native masters reclaimed all 88 generations while the old process stayed stopped; after resumption its old merge was rejected, and both workers returned all 88 correct rows.
  • Transitional fenced-worker fanout: four generations merged across both workers with automatic native routing disabled.

Six worker admission/progress-isolation tests and six master routing/inventory tests also passed before the final read-only changes. All 11 exact-head CI checks passed, including Rust unit/integration, etcd HA, coverage and coverage upload. CI tested the merged tree with main 1e793de; its tree ID is 642115cd00726663aeae2575f7404892b4ae66e9. Local fixtures use isolated etcd/storage and debug binaries, so their latency and memory observations are not production capacity claims.

Production activation still requires the new release image to pass cloud-storage staging and a verified one-time transition covering old worker/master writers and external helpers. MASTER_LOCAL_MERGE_AUTO=true with MASTER_LOCAL_MERGE_TARGETS=* is enabled only after that transition; a protocol marker, Pod age or empty inventory is not proof that historical writes have drained. See docs/worker-merge-admission.md for configuration and rollback constraints. Existing production cleanup has not been interrupted or handed off by this PR's local validation.

@beinan
beinan force-pushed the codex/durable-worker-merge-admission branch from 02891c9 to 89e89c8 Compare October 20, 2026 23:59
@beinan beinan changed the title Integrate durable worker merge admission with master recovery Recover worker merges and route new-table catchup through masters Oct 21, 2026
@beinan
beinan force-pushed the codex/durable-worker-merge-admission branch from 558604f to 72b2eb3 Compare October 21, 2026 00:47
@beinan
beinan marked this pull request as ready for review October 21, 2026 01:28
@beinan
beinan merged commit b4c5e05 into main Oct 21, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant