Skip to content

Recover acknowledged own-shard WAL before cleanup after restart - #429

Merged
beinan merged 4 commits into
mainfrom
codex/recover-own-wal-before-cleanup
Oct 21, 2026
Merged

beinan merged 4 commits into
mainfrom
codex/recover-own-wal-before-cleanup

Conversation

@beinan

@beinan beinan commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

A restarted worker can acknowledge a cleanup merge without recovering its durable, unsealed WAL: ordinary flush is a no-op until a new append opens the resident writer. Add explicit ingestion-owner cleanup preparation that replays this worker's existing shard, seals it, and prepares the normal bounded merge. Worker rollout/generic HTTP cleanup and timed cleanup select this path, so recovery does not require a new application write.

The existing cleanup/prepare APIs retain their resident-only flush semantics. Native masters use those APIs to merge a live worker's sealed generations without claiming its ingestion epoch. The new recovery API requires exclusive ownership of the ingestion writer ID, just like append; it does not authorize takeover of a live legacy worker. Ordinary flush, count-only merge, pinned reads and absent shards remain cheap. Cold maintenance replay is serialized once per process and uses the normal writer configuration so subsequent ingestion retains normal generation sizes. Lance's memory ceilings are soft for indivisible entries.

Validation:

  • Crash/reopen regressions cover rollout 64 KiB payload recovery without new ingest, foreign-shard isolation, later ingestion/repeat cleanup, and generic last-write preservation.
  • Regression covers absent shards bypassing busy recovery admission. A further regression verifies cold maintenance preparation does not claim a live ingestion epoch.
  • Previous head cfb359e passed 9 cleanup and 2 ordinary-flush tests, but final etcd CI exposed two live-ingestion failures: generic native masters also use cleanup preparation. Fixed by keeping the original APIs unchanged and explicitly selecting recovery only at ingestion-owner server callsites. Final head 3e24444 passed 10 cleanup and 2 ordinary-flush tests, including the live-ingestion epoch regression. Both previously failed native master tests also passed against real local etcd (native executor with continued ingestion, and two generic resident replicas under live ingestion). The server merge HTTP test passed as well. All 11 CI checks passed, including Rust stable, coverage, real-etcd HA tests and Python.
  • Real Azure negative control: exact old production image 7255af2 to current release 4dc99ea, old process killed after acknowledged writes; both normal merge HTTP endpoints return but all 8 unsealed rollout records remain missing. A fixture-only append recovers them.
  • Three separate old9/new13 Azure delayed-write diagnostics passed: delayed base manifest, raw WAL, and shard manifest PUTs were rejected with 409 after proven old process exit and a successor barrier. They verify rows, 64 KiB blobs and generic last-write behavior; they are not patched-image automatic-recovery or complete production-retirement proof.
  • Patched-image Azure crash/normal-merge qualification remains pending. Cleanup must actually be scheduled or called; this change does not discover pending=0 unsealed WAL by itself.

@beinan
beinan marked this pull request as ready for review October 21, 2026 06:57
@beinan
beinan merged commit 85f4440 into main Oct 21, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant