Repository navigation
Keep rollout list results consistent when another worker drains MemWAL into base #245
Description
A long-lived reader scans its cached base dataset and the latest WAL manifests. When another worker merges a flushed generation and drains its manifest entry, that generation's rows exist only in a newer base version that the reader has not loaded.
Reproduction:
- Open a writer and a reader on the same dataset with distinct shard IDs.
- Append and flush one row through the writer; reader.list returns one row.
- Run writer.cleanup_own_shard.
- Call reader.list again without reopening.
Expected: the previously visible row remains visible.
Actual: list returns zero rows.
The REST list route also reuses the cached store without refresh/retry. Relevant code: RolloutStore::list_all_non_blob_records and server routes/rollouts.rs::list_rollouts.
Please capture a consistent base/WAL view or retry when concurrent merge changes the view. Refreshing only once before scanning also needs to handle a merge during the scan. Add both stale-reader and concurrent-merge regression coverage.
Related architectural work: #198 and #236. This issue is the concrete missing-row behavior. Confirmed locally.
Found while reviewing main at e292518; fixes will be based on refreshed main (currently 8fe27b2).
Activity
beinan commentedon Sep 10, 2026
Review of PR #251 at
948b970reproduced a missing retry when a merge has already committed its base append before a list read begins. If the reader captures the old WAL manifest and the merge then drains/deletes that generation, the scan fails with DatasetNotFound while the base version remains unchanged. The original implementation returned that error after one attempt even though the row was already in base.The follow-up retries missing-generation errors with fresh base and WAL snapshots independently of base-version changes, bounded to three total attempts. Persistent NotFound preserves the final error; pinned reads and unrelated errors do not retry. A deterministic regression pauses cleanup after base commit and verifies success on the second attempt.