[beinan]

High WAL backlogs currently require an operator to provision recovery Pods. Add an opt-in controller inside the master that admits native catch-up Jobs from scanned WAL pressure or POST /api/v1/catchup. Both entry points share table ownership checks, fleet capacity, deduplication, and durable retry budgets. GET /api/v1/catchup?target=... exposes Job identity, progress observations and recovery status.

Each dedicated process claims one table and merges sealed WAL serially across configured shard identities under the existing storage fence, preserving live ingestion epochs. Durable etcd reservations survive master restarts and ambiguous Kubernetes responses. Ordinary scheduler claims exclude the reserved table; unrelated jobs continue. Jobs have explicit CPU/memory limits, no preemption and no runtime deadline at creation.

Recovery follows completed work rather than total elapsed time:

• A soft admission slice stops starting another operation; useful work already in flight may finish beyond it. Owned worker merges and local maintenance also retain idle/cancel guards without a total runtime ceiling.
• Successful nonempty batch processing and manifest commits advance progress. Heartbeats, commit conflicts and errors do not.
• The master requires continuous lack of progress plus a confirmation window. Sampling gaps invalidate suspicion; new executions receive a fresh grace period.
• Before terminating a stalled Job, an etcd CAS checks the observed sequence and closes commit admission. Concurrent progress defeats cancellation. Kubernetes termination is guarded by Job UID/resource version, and its intent persists across API failures. Resource slots remain reserved until every Pod terminates; storage fencing remains required before replacement writes.

Defaults: disabled, four Jobs, 256 pending generations, 30-second observation interval, 1800-second soft slice/startup window, 600-second idle plus 60-second confirmation, 64 MiB/eight generations per batch, 1 GiB merge-buffer budget and 128 MiB shared cache. Failures back off across replacement and flag attention after three attempts. API callers cannot supply Pod specs, commands or force bypasses.

Activation requires namespace RBAC, a reviewed template with a pinned compatible image, and explicitly owned targets. All maintenance masters must understand the new catch-up discriminator. Older binaries can still enforce legacy runtime limits during rollout. Individual Lance index/storage calls lack internal progress checkpoints; large-table staging must validate idle thresholds and RSS. This does not migrate legacy writers, deploy production configuration, or evict ordinary workers.

Validation on 3065a9760f9787dfde3817e808b41458e9cb5ea8: 114 master tests (including isolated-etcd integration tests), 7 worker recovery tests, 5 manifest-scope tests and 15 coordinator tests passed. Clippy with --all-targets -D warnings passed for all four affected crates; formatting and diff checks passed. Coverage includes stale-progress/startup-phase CAS races, unchanged heartbeats, sampling gaps, survival past legacy runtime limits, ambiguous Job creation, UID protection, live ingestion and exact record conservation. All 10 GitHub checks passed, including Rust/Python tests, coverage, Clippy and builds on Linux x86_64/aarch64 and macOS arm64. No production mutations or real large-table staging run.