[beinan]

Summary

Adds a time-triggered per-shard WAL cleanup for the rollout store, complementing the existing count-triggered self-merge, and hardens the merge path against several concurrency corner cases discovered by reviewing the actual Lance 7.0.0 commit/OCC model.

Each instance cleans only the shard it owns and writes, reusing its own writer epoch, so it never fences a concurrent writer — the same safety invariant as the existing count-triggered merge.

Feature: periodic WAL cleanup

• RolloutStoreOptions gains cleanup_interval_secs (the time trigger) and cleanup_min_generations (the count gate a tick must meet before merging).
• cleanup_own_shard() runs a single pass; spawn_periodic_cleanup() runs a background timer that holds only a Weak reference, so it self-terminates when the store is dropped (e.g. on delete) — no leaked task, no store kept alive.
• Server exposes --rollout-cleanup-interval-secs / --rollout-cleanup-min-generations (env ROLLOUT_CLEANUP_INTERVAL_SECS / ROLLOUT_CLEANUP_MIN_GENERATIONS) and spawns the timer per rollout store. Embedded use leaves it disabled (no long-lived runtime to own the timer).

Concurrency hardening

• ensure_mem_wal: two concurrent first-writers both observe no index and both commit CreateIndex, which Lance treats as a hard, non-retried conflict — the loser previously failed the whole write. Now we reload and accept "index now present" as success. (This was caught by a new test.)
• Merge drain: the manifest drain now removes exactly the generations it merged instead of clearing the vec. A blanket clear would silently discard any generation flushed during the merge window without merging it (data loss). Removing only the merged ids preserves anything else for the next pass.

Corner cases analyzed (against Lance source) and confirmed safe

• Merge (append) racing base compaction — Append vs Rewrite are non-conflicting; OCC auto-rebases.
• Two shards merging into one base table concurrently — two Appends auto-retry; each drains only its own shard manifest.
• Merge racing an append on the same shard — serialized by &mut self + per-store RwLock, backstopped by MemWAL epoch fencing.
• Periodic timer racing the count-triggered merge — both take the write lock; the loser is a no-op.
• Crash between base-append and drain — rows dedup by id on read; next merge drains.
• Delete-store racing the timer — timer holds Weak, exits on next tick.

Test plan

• cleanup_own_shard_merges_only_at_min_generations — min-generation gating
• spawn_periodic_cleanup_reclaims_on_a_timer — timer drains the shard
• spawn_periodic_cleanup_disabled_returns_none — interval 0 spawns nothing
• concurrent_merges_from_two_shards_into_one_base_table — two shards merge into one base table, all rows conserved exactly once
• merge_drains_only_merged_generations — surgical drain conserves rows across two merge rounds
• Full lance-context-core suite, cargo clippy --workspace, and workspace build all green

🤖 Generated with Claude Code