master: a store whose merge/compact fails permanently is re-enqueued every sweep and starves the task queue #266

Problem

Five production rollout stores have a base-table manifest that references a data fragment that no longer exists in object storage:

task failed target=<store> error="Wrapped error: Not found: rocketkeep/<store>.rollout.lance/data/<frag>.lance"

Every MergeWal and Compact against them fails at the same point. All five were last written between 2026-09-09 and 2026-09-23, before #264 made the master the sole writer per base table — consistent with a compaction and a merge from two workers racing: one deletes a rewritten fragment, the other commits a manifest that still names it. One of them has 7,985 pending MemWAL generations (every read of it opens all of them) and a base-table version of 23,703.

The scheduler has no memory of failure. sweep_merge_wal_inner and sweep_candidates_inner read the stats table and enqueue every over-threshold store on every tick; enqueue only de-dupes against a queued/running task, so a task that just failed is re-enqueued on the next sweep. Each attempt fans out serially to 20 workers and burns a TASK_CONCURRENCY slot for minutes before failing. With five such stores and 6 masters × 2 slots, roughly half the fleet's merge capacity is spent failing the same five stores, and the queue for healthy stores climbs (observed 0 → 109 over an hour, 2026-09-26).

Proposed

1. Failure cooldown in the scheduler: track consecutive failures per (kind, target) in the task store (etcd, next to the dedupe key). After N consecutive failures (say 3), skip that target in sweeps for a backoff that doubles per failure (10 min → 20 → 40 → … cap 6 h). A success resets it. Manual enqueue via the API bypasses the cooldown. Expose the cooled-down set in GET /tasks or a new GET /scheduler/cooldowns so operators can see what the master has given up on.
2. Metric + warn: master_task_cooldowns_total{kind} and a warn! when a target enters cooldown, carrying the last error — so a permanently broken store pages someone instead of silently eating capacity.
3. Repair path (separate, larger): a POST /experiments/{name}/repair-manifest that drops fragments the manifest names but storage lacks, committing a new version and reporting rows lost. Today the only remedy is deleting the store.

Refs #264 (which prevents new instances of the corruption), #261 (the pending-generations warn that would have flagged the 7,985-generation store days earlier had it been alerted on).