Repository navigation
feat(master): expose scheduling backlog and fairness metrics - #339
Merged
Merged
Conversation
Three scheduling questions had no metric: how long tasks wait between enqueue and claim, whether prepared compactions/indexes actually get their commit turn or expire and rebuild, and how busy each dispatch pool is (the resident-only slot from lance-format#331 was invisible, so MERGE_WAL_CONCURRENCY=N looked like N while N+1 ran). - master_task_schedule_to_bind_seconds{kind}: enqueued_at -> started_at, the single backlog signal. Growth with idle pools means admission is blocking; growth with full pools means capacity. - master_commit_turn_wait_seconds{kind,result} and master_commit_turn_requests_total{kind,result} with result served or expired. The fairness hint from lance-format#324 is honored only by upgraded binaries; expired counts rising while master_merge_yield_to_compaction_total stays flat identifies a contender that ignores it. - master_task_pool_in_use{pool}: permits in use per dispatch pool (general, compact, merge, resident_merge), set each poll. No scheduling behavior changes. docs/compaction-commit-fairness.md explains how to read the new counters against the existing yield counter. Validation: clippy -D warnings, fmt, typos clean; master library builds. Refs lance-format#333. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
17 of 18 tasks
beinan
added a commit
that referenced
this pull request
Oct 8, 2026
Part of #333 (P0.6, backlog O2/O3). ## What `docs/operations/maintenance-runbook.md`: symptom-driven guide for WAL merge / compaction / index / repair scheduling. Sections: where to look (table of question → endpoint/metric/key), backlog growing, deciding whether a running execution is stuck, demand but nothing scheduled (seven admission reasons, cheapest first), compaction never commits on a hot table, executor died holding ownership, draining, known gaps. Leading rule: **progress decides whether something is stuck; leases, heartbeats, Running state, retry countdowns, Ready pods and elapsed time do not.** Recovery is fenced and automatic; never delete ownership keys by hand. Uses the metrics from #339 and the typed `/merge-progress` from #341. All numeric claims (15 s retry loop, 15-failure attention, 120 s fence, 30 s lease TTL, 3-failure cooldown, 256/4096 thresholds) verified against source. Docs only; `typos` clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
beinan
added a commit
that referenced
this pull request
Oct 8, 2026
…rom running (#348) Part of #333. Fixes a defect in #339 found in review (credit: production review). ## Problem `master_task_pool_in_use` was set once per poll iteration before any permit was taken. A claim blocked on etcd for minutes — the case the gauge is for — kept showing the pre-claim value, possibly zero. ## Fix Gauge updated at the permit boundaries, labelled by holder: `state="claiming"` during the etcd claim, `state="running"` from task spawn until permit drop. Runbook row updated: running at cap ⇒ capacity; claiming high ⇒ slow/blocked admission scan; both low ⇒ nothing eligible. ## Validation Clippy `-D warnings`, fmt, typos clean; `reserved_resident_merge_finishes_while_legacy_pool_is_saturated` passes locally against etcd (exercises the poller path). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #333 (P0.3, backlog C4 and A2).
Problem
Three scheduling questions had no metric: how long tasks wait from enqueue to claim; whether prepared compactions/indexes get their commit turn or expire and rebuild (#324's hint is honoured only by upgraded binaries and nothing records a miss); and how busy each dispatch pool is — the resident-only slot added in #331 was not covered by any gauge, so
MERGE_WAL_CONCURRENCY=Nread as N while N+1 could run.Approach
master_task_schedule_to_bind_secondskindenqueued_at → started_at. Growth with idle pools = admission blocking; with full pools = capacity.master_commit_turn_wait_secondskind, resultmaster_commit_turn_requests_totalkind, resultservedvsexpired(preparation discarded and rebuilt). Risingexpiredwith flatmaster_merge_yield_to_compaction_total⇒ a contender ignores the hint.master_task_pool_in_usepool∈ general/compact/merge/resident_mergeNo scheduling behaviour changes.
docs/compaction-commit-fairness.mdexplains how to read the new counters.Validation
cargo clippy --all-targets -D warnings,fmt,typosclean. Metrics are recorded at existing code points with no new control flow, so behaviour coverage is the existing scheduler suite (CI).🤖 Generated with Claude Code