Repository navigation
feat(master): control-plane observability crate (stats scan + list API) - #137
Merged
Merged
Conversation
…t API)
Introduce `lance-context-master`, a standalone control-plane process for the
per-experiment rollout deployment. This first slice delivers the read-only
observability surface:
- core: `RolloutStore::observe() -> RolloutObservation` and a read-only
`pending_wal_generations()` — cheap metadata-only metrics (row/fragment
counts, manifest version + timestamp, pending MemWAL generations).
- api: `ExperimentSummary` / `ExperimentDetail` / `ExperimentListResponse` /
`CompactJobStatus` DTOs.
- master: `StatsStore` (`_stats.rollout.lance`, delete-then-append upsert,
LIKE search + pagination), a bounded-concurrency background `StatsScanner`
that reconciles against the registry, and `GET /api/v1/experiments` +
`GET /api/v1/experiments/{name}` (+ `POST /rescan`).
The master shares the data-plane's data_dir, reads the registry (data-plane
owned) and owns the stats table. Pure direct connection: it opens datasets
itself. CompactionScheduler + UI land in follow-up PRs.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
This was referenced Jul 14, 2026
beinan
added a commit
that referenced
this pull request
Jul 14, 2026
#138) ## Summary Second slice of `lance-context-master`. **Stacked on #137** — it contains #137's commit too; review/merge #137 first, then this diff is just the scheduler. Adds the compaction control loop: - **Single serial driver.** One background worker drains a single mpsc queue; both the periodic auto-sweep and manual API triggers push onto it, so compaction never runs two `Rewrite`s concurrently (OCC-safe). `Rewrite` vs the data-plane's `Append` is non-conflicting, so this is safe alongside live ingest. - **Auto-sweep** (gated on `compaction_interval_secs`): enqueues experiments whose `fragment_count >= min_fragments`, honoring quiet hours. - **Manual trigger**: `POST /api/v1/experiments/{name}/compact` → 202 + `CompactJobStatus`; `GET /api/v1/experiments/{name}/compact/status`. - After a successful compaction the worker re-observes the store and bumps `last_compaction` / `total_compactions` in the stats table. ## Test plan - [x] `manual_compaction_runs_and_updates_stats`: enqueue → worker compacts → job reaches `Done`, stats `total_compactions == 1` - [x] `enqueue_dedupes_queued_jobs`: duplicate enqueue produces a single queue message - [x] `cargo test` / `clippy` / `fmt` green 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4 <noreply@anthropic.com>
beinan
added a commit
that referenced
this pull request
Jul 14, 2026
…139) ## Summary Final slice of `lance-context-master`. **Stacked on #137 + #138** — this branch contains their commits too; review/merge #137 then #138 first, then this diff is just the `ui/` directory. React SPA served by the master's `ServeDir` when `--ui-dir` points at `ui/dist`: - **Stack**: Vite + React + TypeScript + Zustand (UI-local search/pagination state) + TanStack Query (data). - **Experiments table**: search by `experiment_name`, pagination, columns for row_count / fragment_count / pending WAL gens / last_updated / last_compaction, inline **Compact** button that polls `.../compact/status` until terminal. - **Detail panel**: metric cards with on-demand `?fresh=true` recompute + manual compact trigger. - `api.ts` mirrors the `lance-context-api` DTOs; `npm run build` emits `ui/dist` (git-ignored). ## Test plan - [x] `npm install` + `npm run build` (tsc -b && vite build) green - [ ] Manual: run `lance-context-master --data-dir <shared> --ui-dir crates/lance-context-master/ui/dist`, browse list/search, trigger a compaction, watch status flip to done 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First slice of the
lance-context-mastercontrol-plane process for the per-experiment rollout deployment (builds on #136's registry + LRU). Delivers the read-only observability surface; CompactionScheduler and the React UI follow in separate PRs.RolloutStore::observe() -> RolloutObservation+ read-onlypending_wal_generations()— cheap metadata-only metrics (row/fragment counts, manifest version + timestamp, pending MemWAL generations); no full scan.ExperimentSummary/ExperimentDetail/ExperimentListResponse/CompactJobStatusDTOs.StatsStore(_stats.rollout.lance, delete-then-append idempotent upsert,name LIKEsearch + pagination),StatsScannerthat enumerates the registry, observes each store read-only under a per-experiment timeout, and reconciles away deleted experiments,GET /api/v1/experiments,GET /api/v1/experiments/{name}(?fresh=truerecompute),POST /api/v1/rescan,--ui-dirServeDir wiring (SPA fallback) for the later UI PR.Design
Master shares the data-plane's
data_dir. It reads the registry (data-plane-owned) and owns the stats table — no write conflict. Pure direct connection: master opens datasets itself; it does not go through the data-plane.Test plan
observe_reports_cheap_metricscargo test/clippy/fmtgreen for core/api/master🤖 Generated with Claude Code