diff --git a/docs/operations/maintenance-runbook.md b/docs/operations/maintenance-runbook.md new file mode 100644 index 00000000..8a762b19 --- /dev/null +++ b/docs/operations/maintenance-runbook.md @@ -0,0 +1,187 @@ +# Maintenance scheduling runbook + +Symptom-driven guide for the master's maintenance layer: WAL merge, compaction, +key-index build, repair and cold retirement. It names the exact endpoint, +metric or etcd key to look at and what the answer means. Design background: +`docs/design/unified-maintenance-scheduler.md`; defect list and roadmap: +`docs/design/scheduler-improvement-backlog.md`; tracking: issue #333. + +`P` below is `ETCD_PREFIX` (default `/lance-context/master`). `` is the +lowercase hex of the target name's UTF-8 bytes; `generic:` targets +include the prefix in the hex. + +## 0. The one rule + +**Progress decides whether a merge, compaction or index build is stuck. +Nothing else does.** A live lease, a heartbeat, a Running task state, a retry +countdown, a Ready pod or long elapsed time are not evidence either way. The +only authoritative signal is the execution's completed-work sequence +(`P/merge-progress/`), and the only authoritative stall verdict +is the watchdog's: `MAINTENANCE_IDLE_TIMEOUT_SECS` (default 600 s) with no +completed step, confirmed again before cancellation. If you are about to +restart something because it "has been running a long time", stop and check +§3 first. + +## 1. Where to look, in order + +| Question | Look at | +|---|---| +| Is WAL piling up anywhere? | `master_wal_pending_generations_max`, `master_wal_pending_generations_total`, `master_stores_pending_over_read_cap`; per table `GET /api/v1/experiments/{name}` → `pending_wal_generations`, `fragment_count` | +| Are tasks waiting instead of running? | `master_task_schedule_to_bind_seconds{kind}` (enqueue → claim). Rising with `master_task_pool_in_use{pool}` at capacity = **capacity**. Rising with pools idle = **admission is blocking** (§4). | +| Is something queued at all? | `master_task_queue_depth`; `GET /api/v1/tasks/{id}` for a specific task | +| Is a running merge actually working? | `GET /api/v1/merge-progress?target=` (§3) | +| Is compaction/index getting its commit turn? | `master_commit_turn_requests_total{kind,result}`: `served` vs `expired`; `master_commit_turn_wait_seconds`; `master_merge_yield_to_compaction_total` (§5) | +| Is a table being skipped on purpose? | `GET /api/v1/scheduler/cooldowns`, `GET /api/v1/scheduler/repairs`, failure ledger `P/merge-failures//` (§4) | +| Did the last task fail, and how? | `GET /api/v1/tasks/{id}` → `error`; `master_tasks_total{kind,result}` | +| Is a catch-up Job stuck? | `master_catchup_*`; `P/catchup-records/`, `P/catchup-active/`; `kubectl get job -n $CATCHUP_NAMESPACE` | +| Is the stats view stale? | `master_scan_duration_seconds`, `master_stats_version`; `POST /api/v1/experiments/{name}/rescan` refreshes one table now | +| Is this replica draining? | `GET /api/v1/executor` | + +## 2. Backlog is growing on a table + +1. Confirm it is real: `GET /api/v1/experiments/{name}`. If `pending_wal_generations` + is stale (compare `master_stats_version` age with `STATS_SCAN_INTERVAL_SECS`), + `POST /api/v1/experiments/{name}/rescan` and re-read. +2. Is there a live task? `GET /api/v1/merge-progress?target={name}`: + * `no_active_record` → nothing is scheduled. Go to §4 (why not). + * `progress_unknown` → a legacy worker RPC holds the task with no telemetry. + Check the worker's own logs/metrics; the master cannot see inside it. + * `progress_details_unavailable` → an execution exists and has a progress + `sequence` but no work report (older executor). Sample twice ≥ 60 s apart: + a changing `sequence` is progress. + * `reported` → read `work_report` (§3). +3. If a task is live and progressing: this is **capacity**, not a fault. Check + `master_task_pool_in_use{pool="merge"}` and `{pool="resident_merge"}`; the + executor's memory budget (`ROLLOUT_APPEND_LOCAL_MEMORY_BYTES`, + `CATCHUP_MERGE_MEMORY_BYTES`); and whether the table is + oversized-generation bound (`work_report` shows waiting-for-memory phases). + Adding master replicas adds execution capacity only for tables in + resident/owned mode; legacy tables are bounded by their workers. +4. If `pending_wal_generations` ≥ 256 the read path warns; ≥ 4096 reads return + 503. There is no proportional write-side throttle today (backlog item C7a). + +## 3. Deciding whether a running execution is stuck + +Use `GET /api/v1/merge-progress?target=` and compare **two samples of the +same `execution.id`** taken ≥ 60 s apart. + +* `progress.sequence` increased → **working**. Leave it alone, however long it + has run. Compaction and index builds on large tables take many minutes with + coarse progress; index trainers in Lance 9 report progress only through + completed data/index file IO. +* `progress.sequence` unchanged for less than `MAINTENANCE_IDLE_TIMEOUT_SECS` + → **undetermined**. Wait. Heartbeats and phase changes in `work_report` do + not advance the sequence by design. +* unchanged for longer than the idle timeout → the watchdog will cancel the + preparation, drain admitted commits, fence the manifest version and release. + If it has not within ~2× the idle timeout, the executor process is likely + gone without releasing: see §6. + +What is **not** a stall signal: the task lease is alive; the pod is Ready; the +Job is Active; `needs_attention` is false; retry backoff is counting down; the +task has been Running for an hour. What is **not** a health signal: +`status: reported` (it means telemetry exists, not that work is advancing). + +## 4. A table has demand but nothing is scheduled + +Admission reasons, cheapest first: + +1. **Cooldown** (Compact/IndexId/Repair only; MergeWal is never cooled down): + `GET /api/v1/scheduler/cooldowns`. Produced after + `TASK_COOLDOWN_AFTER_FAILURES`; clears on success. +2. **Failure ledger**: `etcdctl get --prefix P/merge-failures//`. Each + record has `class`, `consecutive_attempts`, `next_retry_ms`, + `needs_attention`, `last_error`. Classes: `Retryable` (2 s ×3 then 30 s + doubling to 900 s), `OwnershipUnresolved` (30 s doubling to 300 s), + `DataOrConfiguration` and `Deadline` (3600 s, `needs_attention`). Errors + tagged `[LC_STALE_PREPARATION]` are retryable by code regardless of + wrapping. An hourly cooldown on a stale-preparation message from a binary + older than #338 is reinterpreted on read; nothing to do. +3. **Ownership**: a write-turn holder exists. `P/target-locks/`, + `P/merge-executions/`, `P/merge-claims/`, `P/catchup-active/`. + `master_task_admission_blocked_total` counts these skips. One holder per + table is correct; two tasks on one table is not. +4. **Fairness yield**: `P/compact-commit-ready/` equals + `P/compact-preparations/` → merge is deliberately yielding one turn to a + prepared compaction/index. `master_merge_yield_to_compaction_total`. +5. **Mode**: the table is in `MERGE_DRAIN_TARGETS` (maintenance refused), or not + in `MERGE_OWNED_TARGETS` while all MergeWal paths require ownership, or + resident-only pollers are configured but the table is not in + `ROLLOUT_APPEND_TARGETS`. There is no endpoint that prints the effective + mode yet (backlog A7); read the master's startup config log. +6. **Dependency**: `GET /api/v1/tasks/{id}` → `depends_on`; a Failed dependency + fails the dependent. +7. **Discovery lag**: nothing enqueued because no sweep has seen the demand. + `MERGE_WAL_INTERVAL_SECS` (600 s), `MERGE_WAL_TAIL_INTERVAL_SECS`, + `ROLLOUT_APPEND_RECONCILE_INTERVAL_SECS`, `CATCHUP_INTERVAL_SECS`, and the + stats scan all gate it. `POST /api/v1/tasks {"kind":"merge_wal","target":""}` + enqueues directly and is safe: dedupe prevents a duplicate. + +Manual enqueue is always safe. Deleting ownership keys by hand is **never** +safe while any process that might hold them is alive (§6). + +## 5. Compaction or index never commits on a hot table + +Symptom: `master_commit_turn_requests_total{result="expired"}` rising; +`master_index_phase_duration_seconds{phase="commit_wait"}` near +`INDEX_COMMIT_WAIT_SECS`/`COMPACTION_COMMIT_WAIT_SECS`; task errors +`maintenance commit ownership wait budget exhausted; reprepare`. + +* If `master_merge_yield_to_compaction_total` is **flat** while `expired` rises, + some contender for that table does not honour the commit-ready hint: a + master or catch-up executor older than #324, or a legacy worker self-merge. + Upgrade or stop that contender. The hint cannot interrupt a binary that + ignores it. +* If yield is rising and `expired` still rises, the merger wins the race after + yielding once. Raise the commit wait for that kind as a stopgap; the design + fix is `max_consecutive_turns` (design §4.2). +* A preparation that expires is discarded and rebuilt from scratch. Repeated + expiry on a large table is wasted IO, not a correctness problem. + +## 6. An executor died holding ownership + +Signs: `P/merge-executions/` with `phase` `Running` or `Uncertain`, its +progress sequence frozen past the idle timeout, no process with that +`instance` alive; or `P/target-locks/` = `merge-execution:` with no +live claim lease. + +Do not delete keys. Recovery is automatic and fenced: + +1. Any owning master's retry loop (15 s) sees the execution with + `maintenance` set and enqueues the matching kind for recovery; MergeWal + recovers worker executions. +2. The recovering task freezes the record, fences manifest versions (120 s), + finishes recovery, releases, and writes the failure ledger atomically. +3. `master_merge_storage_recoveries_total` increments. + +If a record stays `Uncertain` for more than 30 min, the fence step is failing: +check storage credentials and `master_tasks_total{kind,result="failed"}` with the +task error. Escalate before touching etcd. + +Leased keys (`claims/`, `merge-claims/`, `target-locks/` tokens, +`compact-preparations/`) expire with `ETCD_LEASE_TTL_SECS` when the process dies; +unleased keys (`merge-executions/`, `target-locks/` = `merge-execution:*`, +`catchup-*`) are deliberately durable and are what recovery reads. + +## 7. Draining a replica or stopping a table + +* Replica: `POST /api/v1/executor/drain` with the `executor_id` from + `GET /api/v1/executor`; it stops admitting and finishes in-flight claims. See + `docs/operations/master-drain.md`. +* Table: add it to `MERGE_DRAIN_TARGETS` on **every** replica and helper. Draining + resolves existing owned executions and refuses new mutations; it does not + cancel a legacy worker's in-flight self-merge. + +## 8. Known gaps (do not debug these as faults) + +* No endpoint prints a table's effective scheduling mode (A7). +* Queue order is time-ordered without priority: a 4000-generation backlog + enqueued after a 2-generation tail waits behind it (C6). +* Legacy worker RPCs can hold merge slots indefinitely; the resident pool + (`ROLLOUT_APPEND_LOCAL_TASK_CONCURRENCY`) is a reserved lane, not a fix (C3). +* Backpressure is a cliff at 4096 generations, not a slope (C7a). +* Five discovery loops with separate freshness windows; demand can be seen by + one and not another for minutes (A4). + +The unified scheduler work in #333 replaces these; until then this document is +the map.