Repository navigation
Exclude live MemWAL generations from explicitly checked-out snapshots #249
Description
StorageBase::checkout pins only the base dataset. Subsequent reads still discover the latest WAL manifests without consulting pinned_version, so a historical read can include later writes.
Reproduction (confirmed for ContextStore and RolloutStore):
- Add "old", merge it into base, and save that base version.
- Add "future" and make its WAL generation visible without merging.
- Check out the saved base version and list rows.
Expected: the pinned base snapshot returns only "old".
Actual: it returns ["old", "future"].
This reproduction uses a concrete already-merged base version; it does not assume that each rollout add advances the base version.
Relevant code: crates/lance-context-core/src/store_base.rs::checkout, wal_shard_snapshots, and LSM read construction.
Please define pinned reads as a stable historical view and apply that rule across list, point/blob reads, and observations. Background maintenance should not silently unpin an explicitly selected version. Add coverage for ContextStore and RolloutStore.
Found while reviewing main at e292518; fixes will be based on refreshed main (currently 8fe27b2).
Activity
beinan commentedon Sep 10, 2026
Review of PR #251 at
948b970reproduced a remaining write-side regression: aftercheckout(version), two sequentialContextStore::addcalls with different IDs and the same external_id both succeed because uniqueness validation reads only the pinned base snapshot. After refreshing, both rows are visible.The follow-up makes checked-out handles read-only until
refresh_latest(). The guard also covers upserts, updates, tombstones, external payload writes, schema migration, compaction, and index creation. Regression coverage checks that rejected operations preserve the pin and data, and that writes and uniqueness validation work again after refresh.