fix(rollout): flush sweeper only covers LRU-resident stores; evicted stores may never flush #185

Summary

The periodic flush sweeper snapshots the LRU of resident rollout stores. A store that is written to and then evicted from the LRU before the next flush tick has an unsealed active memtable that nobody is left to flush.

Evidence

crates/lance-context-server/src/state.rs:490-502 — the sweeper snapshots self.rollout_stores (the LRU) and iterates only over what is resident:

let resident: Vec<_> = {
    let cache = state.rollout_stores.lock().await;
    cache.iter().map(|(n, s)| (n.clone(), s.clone())).collect()
};

RolloutStore::close (crates/lance-context-core/src/rollout_store.rs:642-660) is the mechanism that drains the writer, and state.rs:565 calls it — but it needs confirming that every LRU eviction path reaches that call, including eviction triggered implicitly by LruCache::put when the cache is at capacity.

Note also that close() uses Arc::try_unwrap and silently gives up if another handle is live (rollout_store.rs:653-658):

Err(_shared) => {}

So even on a path that does call close(), a concurrent in-flight add holding a cloned Arc makes the drain a no-op.

Impact

Same failure mode as #184: rows are durable but potentially never become visible. Most likely to bite deployments with many rollout stores and a small rollout_store_cache_capacity, where eviction is routine rather than exceptional.

Proposed work

1. Confirm whether implicit LRU eviction (capacity overflow inside put) reaches close().
2. If not, hook eviction so the store is flushed/closed before the handle is dropped.
3. Consider flushing before eviction rather than relying on close(), since close()'s Arc::try_unwrap can no-op under concurrency.
4. Add a metric for evictions that skipped a drain, so this is observable rather than silent.

Related

• feat(rollout): concurrent writes with periodic flush #181, fix(rollout): rollout_flush_interval_secs=0 may leave rows durably written but permanently invisible #184