Skip to content

fix: keep blob misses under shared store locks - #413

Merged
beinan merged 1 commit into
mainfrom
codex/shared-blob-miss-read
Oct 20, 2026
Merged

beinan merged 1 commit into
mainfrom
codex/shared-blob-miss-read

Conversation

@beinan

@beinan beinan commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

Blob misses currently reacquire an exclusive store guard to refresh the resident dataset. A concurrent append/flush holds a shared guard, so one absent-ID lookup can wait behind the write and queue later blob hits behind itself.

Keep blob lookups under a shared guard and refresh cloned base snapshots around WAL fallback. After a complete WAL miss, check the latest base again so a row merged and drained between the initial base check and WAL discovery remains readable. Preserve resident and pinned versions, read caps, the base-hit fast path, and the bounded missing-file retries from #411.

Validation:

  • Before the fix, the server regression times out while an independent shared guard is held.
  • Before the fix, the deterministic core regression returns None for a row flushed before the read and merged during the base-to-WAL transition.
  • After the fix: 31 blob tests (21 core, 10 server) and 4 point-lookup tests (2 core, 2 server) pass, including both regressions. cargo fmt --all -- --check also passes. Full CI remains in progress.
  • Isolated Azure staging with a 291,593,422-byte same-table write, 53,248-byte blob reads, 32 pending generations, and concurrent hit/miss streams produced the maximum header latencies below. Each run used a fresh storage prefix; no OOM or container restart occurred. These are three sequential synthetic runs, not production p99 measurements.
Image/run Hit maximum Miss maximum
c9aa control 4.046 s 4.149 s
Patched run 1 0.0021 s 0.159 s
Patched run 2 0.0058 s 2.100 s

The second patched miss still took 2.1 s: this does not resolve every source of read latency. The runtime image IDs matched the immutable image built from this PR's exact head (768495e) in services build 2159160.

A separate isolated five-worker fixture with 100,586,357-byte payloads passed six status checks and four full payload comparisons during five concurrent writes. All three lookup types (base hit, WAL hit, absent ID) overlapped actual merges; the four writer shards reclaimed 32 generations. Maximum read header latency was 0.849 s, with no OOM/restarts. This is a bounded correctness run, not a soak test. All four fixture Pods were deleted and verified absent. Production has not been changed.

@beinan
beinan marked this pull request as ready for review October 20, 2026 20:07
@beinan
beinan merged commit f5e370e into main Oct 20, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant