Repository navigation
Report actual merge work, phases, and no-progress state - #330
Merged
Merged
Conversation
beinan
force-pushed
the
codex/merge-work-progress
branch
from
October 8, 2026 07:29
af5f482 to
cdc9d46
Compare
beinan
marked this pull request as ready for review
October 8, 2026 07:42
This was referenced Oct 8, 2026
beinan
added a commit
that referenced
this pull request
Oct 8, 2026
Part of #333. Companion to #334. ## What Adds `docs/design/scheduler-improvement-backlog.md`: the defects in the current maintenance scheduling layer, grouped by severity with file:line references, and the ordered P0–P4 plan to fix them. Highlights: - **Architectural**: no single owner of "what runs next"; capacity as semaphores without memory; 7-key etcd snapshot per candidate per claim; discovery duplicated five ways; dependencies as task DAG edges; stats scanner mutating tables directly; six allowlists. - **Correctness**: error classification by message text incl. source line numbers (#328); potential retry storm across #327/#330/#328 for slow-but-alive index builds; starvation of queued legacy tasks accepted as expected in #331's test; fairness hint (#324) without version or metric; `/merge-progress` pushing read consistency onto clients; random-id queue order; backpressure as a cliff at 4096 generations. - **Plan**: P0 is zero behaviour change (pure eligibility function, typed errors, metrics, e2e test, runbook); P1 shadow planner; P2–P4 per-table migration and deletion, each with mixed-version rules. ## Validation Docs only. `typos` clean locally. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This was referenced Oct 8, 2026
beinan
added a commit
that referenced
this pull request
Oct 8, 2026
) Part of #333 (P0.4, backlog C2). ## Problem `watch_maintenance_preparation` cancels a compaction/index preparation after `MAINTENANCE_IDLE_TIMEOUT_SECS` without a completed step. That boundary had no direct test. A regression to wall-clock cancellation, or a change that stopped counting real index IO as a step, would cancel every long index build, classify it retryable (#328) and rebuild it every few seconds — the #327 + #330 + #328 interaction flagged in the backlog. ## Approach One etcd-backed test, 3 s idle timeout, three cases: 1. **Slow but alive** — 8 s preparation completing one step per second finishes. 2. **Silent** — cancelled within `[IDLE, IDLE+4)` with the exact `"maintenance preparation made no progress"` error. 3. **Real index build** under the same short timeout completes, proving `data/` and `_indices/` IO is observed as progress through the preparation store wrapper. Mutation check: removing `checkpoint()` from case 1 fails the test with `alive preparation was cancelled` after ~4 s. ## Validation Test passes locally against real etcd (~12 s). Clippy `-D warnings`, fmt, typos clean. Test-only change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
beinan
added a commit
that referenced
this pull request
Oct 8, 2026
…sponse (#341) Part of #333 (P0.5, backlog C5). ## Problem `GET /api/v1/merge-progress` (#330) did four independent etcd reads — execution, progress, work report, execution again — and returned `progress_changed_retry` / `execution_changed_retry` when they disagreed, leaving the caller to retry. The body was `serde_json::json!` with no schema. ## Approach - `Coordinator::progress_snapshot(target)`: read the execution id, then one txn guarded on the execution bytes that fetches progress and work report together. All three fields come from a single revision, exposed as `revision`. Bounded to three attempts. A work report whose `sequence` trails the current progress sequence is dropped instead of being shown next to a newer heartbeat. - Handler returns `MergeProgressReport` (serde struct) with `status ∈ {reported, progress_details_unavailable, progress_unknown, no_active_record}`. The two `*_retry` statuses are removed. Absent fields are omitted, so consumers reading only `status`/`task_id` see no change. - `docs/merge-work-progress.md` updated with the schema and single-revision guarantee. ## Alternatives Reading all keys by target in one range: progress and work keys are per execution id, not per target, so a first read is unavoidable. Using `WithRevision` on separate gets: works, but the guarded txn also detects an execution swap for free. ## Validation - `lance-context-merge`: `detailed_work_keeps_legacy_progress_cas_…` extended — snapshot agrees with individual reads, stale report dropped after a heartbeat, absent target → `None`. - `lance-context-master`: `merge_progress_never_calls_a_live_legacy_task_healthy` asserts on the typed struct and its serialised form. - Both pass locally against real etcd; clippy `-D warnings`, fmt, typos clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A live merge claim and an opaque progress sequence do not tell operators whether a table is reading WAL, waiting for memory, encoding files, committing, or actually reclaiming generations. Expose an exact-target
/api/v1/merge-progressendpoint with execution/task/process identity, last actual progress time, active phase counts, and completed append-work counters.Reporting heartbeats and phase changes do not advance the progress sequence or extend its idle deadline. The existing watchdog still cancels preparation and drains/fences admitted commits before replacement; failure retry policy is unchanged. Legacy tasks with no execution telemetry explicitly report
progress_unknown.Detailed telemetry is published atomically alongside the original sequence in a companion key, preserving the old exact-byte cancellation CAS during mixed-version rollout. Regression coverage includes real WAL encoding versus committed reclamation, unchanged heartbeats, legacy unknown status, and late progress publication after revocation/release. Validation: five focused local master/merge regressions and the legacy-unknown endpoint test passed. All 11 CI checks passed on cdc9d46, including workspace unit/integration tests, real-etcd HA, coverage, Clippy, Python, and Linux/macOS wheels. See docs/merge-work-progress.md for counter semantics and limitations.