[beinan]

Why

We want to partition rollouts by experiment_name — one physical Lance dataset per experiment. At the target scale (tens to hundreds of thousands of experiments) the current server runtime cannot cope:

1. rollout_stores is an unbounded HashMap of open handles → never evicted → OOM.
2. Each store spawns its own tokio cleanup timer (spawn_periodic_cleanup) → 100k+ background timers wedge the process.
3. list_rollout_stores only enumerates the in-memory cache, so with eviction it would be incomplete, and there is no durable directory to enumerate.

This PR decouples resident memory / background tasks from the number of stores.

What

• Durable registry (RolloutRegistry, new crates/lance-context-core/src/registry.rs): a single small Lance table (name, uri, created_at) that is the source of truth for which stores exist. create / list / delete / existence checks now go through it. Existence is no longer inferred from cache membership, so eviction cannot cause a false 404, and list returns every store.
• Bounded LRU replaces the unbounded HashMap; evicted stores are transparently reopened from storage on the next request (reusing open_existing_with_options from fix(server): lazily open stores on cache miss (multi-replica 404) #135).
• Single global sweeper (AppState::spawn_global_sweeper) replaces one-timer-per-store. It sweeps only the resident set each tick, with the same per-pass tokio::time::timeout hardening as fix(rollout): bound periodic WAL cleanup pass with a timeout #133. Cold (evicted) stores accrue no new generations and are swept again once re-opened.
• RolloutStoreInfo.version → Option<u64> (omitted in list to avoid opening every dataset; populated for single get/create). Client updated (unwrap_or(0)).
• New --rollout-cache-capacity / ROLLOUT_CACHE_CAPACITY (default 2000).

Tests

• registry.rs: upsert idempotency, contains/remove, list, survive-reopen.
• Server: evicted-store-transparently-reopened (capacity-1 LRU), list-enumerates-all-registered, delete-unregisters, sweeper gating. Existing multi-replica regression tests (fix(server): lazily open stores on cache miss (multi-replica 404) #135) still pass.
• cargo test --workspace, cargo clippy --workspace --all-targets, cargo fmt --check all clean.

Follow-ups (out of scope)

• Startup reconciliation (scan data_dir to backfill registry rows if a crash left a dataset without its row).
• Optional btree index on registry.name if the narrow scan becomes hot.

🤖 Generated with Claude Code