From 8f9acebfee87b8519d6fa94a7d49f2cad61a8af2 Mon Sep 17 00:00:00 2001 From: Beinan Date: Thu, 8 Oct 2026 19:13:50 +0000 Subject: [PATCH] docs: operator runbook for maintenance scheduling Eleven topic documents describe individual mechanisms but none answers the operator's questions in order: is backlog real, is anything scheduled, is the running thing working, why is nothing scheduled, why does compaction never commit, what to do when an executor died holding ownership. This runbook does, naming the exact endpoint, metric or etcd key for each step and what the answer means. The leading rule is that progress alone decides whether an execution is stuck: leases, heartbeats, Running state, retry countdowns, Ready pods and elapsed time are not evidence either way. Recovery is automatic and fenced; the runbook says when to wait and when to escalate, and never to delete ownership keys by hand. Numbers (15 s retry loop, 15-failure attention threshold, 120 s fence, 30 s lease TTL, 3-failure cooldown, 256/4096 read thresholds) were checked against source. Known gaps are listed so they are not debugged as faults. Refs #333. Co-Authored-By: Claude Fable 5 --- docs/operations/maintenance-runbook.md | 187 +++++++++++++++++++++++++ 1 file changed, 187 insertions(+) create mode 100644 docs/operations/maintenance-runbook.md diff --git a/docs/operations/maintenance-runbook.md b/docs/operations/maintenance-runbook.md new file mode 100644 index 00000000..8a762b19 --- /dev/null +++ b/docs/operations/maintenance-runbook.md @@ -0,0 +1,187 @@ +# Maintenance scheduling runbook + +Symptom-driven guide for the master's maintenance layer: WAL merge, compaction, +key-index build, repair and cold retirement. It names the exact endpoint, +metric or etcd key to look at and what the answer means. Design background: +`docs/design/unified-maintenance-scheduler.md`; defect list and roadmap: +`docs/design/scheduler-improvement-backlog.md`; tracking: issue #333. + +`P` below is `ETCD_PREFIX` (default `/lance-context/master`). `` is the +lowercase hex of the target name's UTF-8 bytes; `generic:` targets +include the prefix in the hex. + +## 0. The one rule + +**Progress decides whether a merge, compaction or index build is stuck. +Nothing else does.** A live lease, a heartbeat, a Running task state, a retry +countdown, a Ready pod or long elapsed time are not evidence either way. The +only authoritative signal is the execution's completed-work sequence +(`P/merge-progress/`), and the only authoritative stall verdict +is the watchdog's: `MAINTENANCE_IDLE_TIMEOUT_SECS` (default 600 s) with no +completed step, confirmed again before cancellation. If you are about to +restart something because it "has been running a long time", stop and check +§3 first. + +## 1. Where to look, in order + +| Question | Look at | +|---|---| +| Is WAL piling up anywhere? | `master_wal_pending_generations_max`, `master_wal_pending_generations_total`, `master_stores_pending_over_read_cap`; per table `GET /api/v1/experiments/{name}` → `pending_wal_generations`, `fragment_count` | +| Are tasks waiting instead of running? | `master_task_schedule_to_bind_seconds{kind}` (enqueue → claim). Rising with `master_task_pool_in_use{pool}` at capacity = **capacity**. Rising with pools idle = **admission is blocking** (§4). | +| Is something queued at all? | `master_task_queue_depth`; `GET /api/v1/tasks/{id}` for a specific task | +| Is a running merge actually working? | `GET /api/v1/merge-progress?target=` (§3) | +| Is compaction/index getting its commit turn? | `master_commit_turn_requests_total{kind,result}`: `served` vs `expired`; `master_commit_turn_wait_seconds`; `master_merge_yield_to_compaction_total` (§5) | +| Is a table being skipped on purpose? | `GET /api/v1/scheduler/cooldowns`, `GET /api/v1/scheduler/repairs`, failure ledger `P/merge-failures//` (§4) | +| Did the last task fail, and how? | `GET /api/v1/tasks/{id}` → `error`; `master_tasks_total{kind,result}` | +| Is a catch-up Job stuck? | `master_catchup_*`; `P/catchup-records/`, `P/catchup-active/`; `kubectl get job -n $CATCHUP_NAMESPACE` | +| Is the stats view stale? | `master_scan_duration_seconds`, `master_stats_version`; `POST /api/v1/experiments/{name}/rescan` refreshes one table now | +| Is this replica draining? | `GET /api/v1/executor` | + +## 2. Backlog is growing on a table + +1. Confirm it is real: `GET /api/v1/experiments/{name}`. If `pending_wal_generations` + is stale (compare `master_stats_version` age with `STATS_SCAN_INTERVAL_SECS`), + `POST /api/v1/experiments/{name}/rescan` and re-read. +2. Is there a live task? `GET /api/v1/merge-progress?target={name}`: + * `no_active_record` → nothing is scheduled. Go to §4 (why not). + * `progress_unknown` → a legacy worker RPC holds the task with no telemetry. + Check the worker's own logs/metrics; the master cannot see inside it. + * `progress_details_unavailable` → an execution exists and has a progress + `sequence` but no work report (older executor). Sample twice ≥ 60 s apart: + a changing `sequence` is progress. + * `reported` → read `work_report` (§3). +3. If a task is live and progressing: this is **capacity**, not a fault. Check + `master_task_pool_in_use{pool="merge"}` and `{pool="resident_merge"}`; the + executor's memory budget (`ROLLOUT_APPEND_LOCAL_MEMORY_BYTES`, + `CATCHUP_MERGE_MEMORY_BYTES`); and whether the table is + oversized-generation bound (`work_report` shows waiting-for-memory phases). + Adding master replicas adds execution capacity only for tables in + resident/owned mode; legacy tables are bounded by their workers. +4. If `pending_wal_generations` ≥ 256 the read path warns; ≥ 4096 reads return + 503. There is no proportional write-side throttle today (backlog item C7a). + +## 3. Deciding whether a running execution is stuck + +Use `GET /api/v1/merge-progress?target=` and compare **two samples of the +same `execution.id`** taken ≥ 60 s apart. + +* `progress.sequence` increased → **working**. Leave it alone, however long it + has run. Compaction and index builds on large tables take many minutes with + coarse progress; index trainers in Lance 9 report progress only through + completed data/index file IO. +* `progress.sequence` unchanged for less than `MAINTENANCE_IDLE_TIMEOUT_SECS` + → **undetermined**. Wait. Heartbeats and phase changes in `work_report` do + not advance the sequence by design. +* unchanged for longer than the idle timeout → the watchdog will cancel the + preparation, drain admitted commits, fence the manifest version and release. + If it has not within ~2× the idle timeout, the executor process is likely + gone without releasing: see §6. + +What is **not** a stall signal: the task lease is alive; the pod is Ready; the +Job is Active; `needs_attention` is false; retry backoff is counting down; the +task has been Running for an hour. What is **not** a health signal: +`status: reported` (it means telemetry exists, not that work is advancing). + +## 4. A table has demand but nothing is scheduled + +Admission reasons, cheapest first: + +1. **Cooldown** (Compact/IndexId/Repair only; MergeWal is never cooled down): + `GET /api/v1/scheduler/cooldowns`. Produced after + `TASK_COOLDOWN_AFTER_FAILURES`; clears on success. +2. **Failure ledger**: `etcdctl get --prefix P/merge-failures//`. Each + record has `class`, `consecutive_attempts`, `next_retry_ms`, + `needs_attention`, `last_error`. Classes: `Retryable` (2 s ×3 then 30 s + doubling to 900 s), `OwnershipUnresolved` (30 s doubling to 300 s), + `DataOrConfiguration` and `Deadline` (3600 s, `needs_attention`). Errors + tagged `[LC_STALE_PREPARATION]` are retryable by code regardless of + wrapping. An hourly cooldown on a stale-preparation message from a binary + older than #338 is reinterpreted on read; nothing to do. +3. **Ownership**: a write-turn holder exists. `P/target-locks/`, + `P/merge-executions/`, `P/merge-claims/`, `P/catchup-active/`. + `master_task_admission_blocked_total` counts these skips. One holder per + table is correct; two tasks on one table is not. +4. **Fairness yield**: `P/compact-commit-ready/` equals + `P/compact-preparations/` → merge is deliberately yielding one turn to a + prepared compaction/index. `master_merge_yield_to_compaction_total`. +5. **Mode**: the table is in `MERGE_DRAIN_TARGETS` (maintenance refused), or not + in `MERGE_OWNED_TARGETS` while all MergeWal paths require ownership, or + resident-only pollers are configured but the table is not in + `ROLLOUT_APPEND_TARGETS`. There is no endpoint that prints the effective + mode yet (backlog A7); read the master's startup config log. +6. **Dependency**: `GET /api/v1/tasks/{id}` → `depends_on`; a Failed dependency + fails the dependent. +7. **Discovery lag**: nothing enqueued because no sweep has seen the demand. + `MERGE_WAL_INTERVAL_SECS` (600 s), `MERGE_WAL_TAIL_INTERVAL_SECS`, + `ROLLOUT_APPEND_RECONCILE_INTERVAL_SECS`, `CATCHUP_INTERVAL_SECS`, and the + stats scan all gate it. `POST /api/v1/tasks {"kind":"merge_wal","target":""}` + enqueues directly and is safe: dedupe prevents a duplicate. + +Manual enqueue is always safe. Deleting ownership keys by hand is **never** +safe while any process that might hold them is alive (§6). + +## 5. Compaction or index never commits on a hot table + +Symptom: `master_commit_turn_requests_total{result="expired"}` rising; +`master_index_phase_duration_seconds{phase="commit_wait"}` near +`INDEX_COMMIT_WAIT_SECS`/`COMPACTION_COMMIT_WAIT_SECS`; task errors +`maintenance commit ownership wait budget exhausted; reprepare`. + +* If `master_merge_yield_to_compaction_total` is **flat** while `expired` rises, + some contender for that table does not honour the commit-ready hint: a + master or catch-up executor older than #324, or a legacy worker self-merge. + Upgrade or stop that contender. The hint cannot interrupt a binary that + ignores it. +* If yield is rising and `expired` still rises, the merger wins the race after + yielding once. Raise the commit wait for that kind as a stopgap; the design + fix is `max_consecutive_turns` (design §4.2). +* A preparation that expires is discarded and rebuilt from scratch. Repeated + expiry on a large table is wasted IO, not a correctness problem. + +## 6. An executor died holding ownership + +Signs: `P/merge-executions/` with `phase` `Running` or `Uncertain`, its +progress sequence frozen past the idle timeout, no process with that +`instance` alive; or `P/target-locks/` = `merge-execution:` with no +live claim lease. + +Do not delete keys. Recovery is automatic and fenced: + +1. Any owning master's retry loop (15 s) sees the execution with + `maintenance` set and enqueues the matching kind for recovery; MergeWal + recovers worker executions. +2. The recovering task freezes the record, fences manifest versions (120 s), + finishes recovery, releases, and writes the failure ledger atomically. +3. `master_merge_storage_recoveries_total` increments. + +If a record stays `Uncertain` for more than 30 min, the fence step is failing: +check storage credentials and `master_tasks_total{kind,result="failed"}` with the +task error. Escalate before touching etcd. + +Leased keys (`claims/`, `merge-claims/`, `target-locks/` tokens, +`compact-preparations/`) expire with `ETCD_LEASE_TTL_SECS` when the process dies; +unleased keys (`merge-executions/`, `target-locks/` = `merge-execution:*`, +`catchup-*`) are deliberately durable and are what recovery reads. + +## 7. Draining a replica or stopping a table + +* Replica: `POST /api/v1/executor/drain` with the `executor_id` from + `GET /api/v1/executor`; it stops admitting and finishes in-flight claims. See + `docs/operations/master-drain.md`. +* Table: add it to `MERGE_DRAIN_TARGETS` on **every** replica and helper. Draining + resolves existing owned executions and refuses new mutations; it does not + cancel a legacy worker's in-flight self-merge. + +## 8. Known gaps (do not debug these as faults) + +* No endpoint prints a table's effective scheduling mode (A7). +* Queue order is time-ordered without priority: a 4000-generation backlog + enqueued after a 2-generation tail waits behind it (C6). +* Legacy worker RPCs can hold merge slots indefinitely; the resident pool + (`ROLLOUT_APPEND_LOCAL_TASK_CONCURRENCY`) is a reserved lane, not a fix (C3). +* Backpressure is a cliff at 4096 generations, not a slope (C7a). +* Five discovery loops with separate freshness windows; demand can be seen by + one and not another for minutes (A4). + +The unified scheduler work in #333 replaces these; until then this document is +the map.