master: RSS grows ~1.5 GiB/h over long uptimes until OOM #260

Problem

lance-context-master resident memory climbs roughly linearly with uptime and the process is OOMKilled at the container limit (32 GiB) every 8–20 hours. Observed on a 3-replica deployment at commit 897a536 with COMPACTION_CONCURRENCY=2, TASK_CONCURRENCY=2, COMPACTION_THREADS=4, COMPACTION_MAX_SOURCE_FRAGMENTS=512, ROLLOUT_CACHE_BYTES=2 GiB, MALLOC_ARENA_MAX=2, STATS_SCAN_INTERVAL_SECS=300, ~7,900 experiments in the registry:

Two replicas were OOMKilled in the same window (after 7.5 h and 16 h). Only 2 compaction tasks were running at the time of each kill, so this is not the compaction fan-out that COMPACTION_CONCURRENCY bounds; it is a slow ratchet.

The mango deployment notes already record an earlier round of this ("compact RSS ratcheted up and never released") that was mitigated with MALLOC_ARENA_MAX=2; that helped but did not remove the growth.

What is known / suspects

• Growth is independent of COMPACTION_CONCURRENCY (2 vs 4 gives the same slope).
• Each stats scan touches every experiment (stats scan complete experiments=7917) and each auto-sweep queues hundreds (auto-sweep queued experiments queued=212, auto merge-wal sweep queued experiments queued=80). Candidates:
  • per-Dataset metadata retained across scans (manifest / fragment lists for thousands of datasets) outside the Session cache bounded by ROLLOUT_CACHE_BYTES;
  • Dataset handles kept alive by the scheduler's task state or the stats store between scans;
  • allocator retention after large Compaction task … Read progress N/40000 rewrites (the MALLOC_ARENA_MAX mitigation hints this is part of it).

Proposed next steps

1. Chart process_resident_memory_bytes from the master's /metrics over 24 h alongside master_task_* counters to see whether growth is smooth or steps at scan / compaction boundaries.
2. Add a gauge for the Session cache's actual size and the number of live Dataset handles, to rule the bounded cache in or out.
3. If it is retained Dataset state: drop/reopen per-experiment handles after each scan, or scope them to the scan.
4. If it is allocator retention: try jemalloc with background purging for the master binary, or a periodic malloc_trim(0) after compaction tasks.
5. Consider lowering COMPACTION_MAX_SOURCE_FRAGMENTS (512 → 256) as an interim to cut the per-compaction peak.

Independent of #256–#259 (worker-side WAL merge); filing separately so it is tracked.