fix(rollout): rollout_flush_interval_secs=0 may leave rows durably written but permanently invisible #184

Summary

rollout_flush_interval_secs = 0 disables the periodic flush sweeper. The config doc claims rows then become visible via the cleanup/merge path instead, but reading that path suggests it cannot flush an active memtable that has never been sealed.

Evidence

crates/lance-context-server/src/config.rs:56-57:

Default 30; 0 disables periodic flush (rows then only become visible when the cleanup/merge path flushes them).

But the merge path short-circuits before it can do anything:

crates/lance-context-core/src/rollout_store.rs:703-706 (merge_own_shard_if_ready):

let pending = manifest.flushed_generations.len();
if pending == 0 || pending < threshold.max(1) {
    return Ok(0);
}

With no periodic flush, nothing ever seals the active memtable into a flushed generation, so flushed_generations stays empty, so merge_own_shard_if_ready returns 0 without ever reaching merge_own_shard — and therefore never reaches the self.close().await? at rollout_store.rs:782 that is the only other place a writer gets drained.

Impact

If the above reading is correct, with rollout_flush_interval_secs=0:

• rows are durable (put completed), so no data loss
• but they remain invisible to reads indefinitely — until the process restarts and WAL replay picks them up

That is a sharp edge for a value the config explicitly documents as supported.

Proposed work

1. Add a test: set flush interval to 0, add() rows, assert whether list() sees them. This confirms or refutes the analysis.
2. Depending on the result, either:
   • reject 0 at config validation with a clear message, or
   • keep 0 but rewrite the doc to warn that visibility then depends on process restart, or
   • make the cleanup sweeper force a seal before checking flushed_generations.

Related

• feat(rollout): concurrent writes with periodic flush #181 introduced the decoupled flush path