Skip to content

fix: avoid same-store merge read-lock and memory-budget deadlock - #433

Merged
beinan merged 1 commit into
mainfrom
codex/serialize-worker-merge-preparation
Oct 21, 2026
Merged

beinan merged 1 commit into
mainfrom
codex/serialize-worker-merge-preparation

Conversation

@beinan

@beinan beinan commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Two concurrent merges on one resident store can deadlock: the first retains its merge-memory reservation and waits for the store write lock to commit; the second holds a store read lock while waiting for that same memory. A real local GenericStore regression running concurrent count-triggered and cleanup merges with a 1 MiB total budget times out on the unmodified server.

Add one merge-admission mutex per resident store. Every rollout/generic HTTP merge and both sweeper paths clone it under a short read lock, drop that read guard, then await admission and retain it through preparation and commit. Ingestion, reads, compaction and other tables do not acquire this mutex. Distributed ownership, writer epochs, commit validation and the process memory limit are unchanged. Independently opened store handles and SDK callers still require their existing coordination; this is not a new takeover mechanism.

Validation:

  • Before the fix, the real-store regression fails after 20 seconds with the read-lock/budget wait assertion (0 passed, 1 failed).
  • Added another real-store test requiring queued admission to release the store lock, concurrent ingestion to succeed, and a second table to finish merging while the first waits.
  • Formatting and diff checks pass. All nine normal local sweeper tests pass, including both new regressions and existing reader/ingest/slot behavior checks. All three isolated-etcd sweeper tests also pass, covering demand publication, a hung publisher, and owned count-trigger admission. Self-review covered all six split merge entry points and guard lifetime. Local Clippy (core and server, all targets) passes, and CI Clippy is also green. The remaining full Rust/coverage CI is still running.
  • The production worker waiting to drain has not been attributed to this cycle. No restart or fencing decision is justified solely by this local reproduction.

@beinan
beinan marked this pull request as ready for review October 21, 2026 11:49
@beinan
beinan merged commit 5ef9c44 into main Oct 21, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant