Repository navigation
feat(master): centralized serial compaction scheduler + compact routes - #138
Merged
Merged
Conversation
1 of 2 tasks
Add the compaction control loop to the master. A single background worker
drains one mpsc queue, so automatic sweeps and manual API triggers share the
same serial execution path — guaranteeing a single `Rewrite` driver and no
OCC conflict. `Rewrite` vs the data-plane's `Append` is non-conflicting, so
this runs safely alongside live ingest.
- state: `compact_tx`/`compact_rx` queue + per-experiment `jobs` status map.
- scheduler: `enqueue` (de-dupes queued/running), serial worker, optional
periodic auto-sweep gated on fragment_count >= min_fragments and quiet
hours; post-compaction it re-observes and bumps
last_compaction/total_compactions in the stats table.
- routes: `POST /experiments/{name}/compact` (202 + job status) and
`GET /experiments/{name}/compact/status`.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
beinan
force-pushed
the
master-compaction
branch
from
July 14, 2026 16:44
9b3a62c to
d5d8684
Compare
beinan
added a commit
that referenced
this pull request
Jul 14, 2026
…139) ## Summary Final slice of `lance-context-master`. **Stacked on #137 + #138** — this branch contains their commits too; review/merge #137 then #138 first, then this diff is just the `ui/` directory. React SPA served by the master's `ServeDir` when `--ui-dir` points at `ui/dist`: - **Stack**: Vite + React + TypeScript + Zustand (UI-local search/pagination state) + TanStack Query (data). - **Experiments table**: search by `experiment_name`, pagination, columns for row_count / fragment_count / pending WAL gens / last_updated / last_compaction, inline **Compact** button that polls `.../compact/status` until terminal. - **Detail panel**: metric cards with on-demand `?fresh=true` recompute + manual compact trigger. - `api.ts` mirrors the `lance-context-api` DTOs; `npm run build` emits `ui/dist` (git-ignored). ## Test plan - [x] `npm install` + `npm run build` (tsc -b && vite build) green - [ ] Manual: run `lance-context-master --data-dir <shared> --ui-dir crates/lance-context-master/ui/dist`, browse list/search, trigger a compaction, watch status flip to done 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Second slice of
lance-context-master. Stacked on #137 — it contains #137's commit too; review/merge #137 first, then this diff is just the scheduler.Adds the compaction control loop:
Rewrites concurrently (OCC-safe).Rewritevs the data-plane'sAppendis non-conflicting, so this is safe alongside live ingest.compaction_interval_secs): enqueues experiments whosefragment_count >= min_fragments, honoring quiet hours.POST /api/v1/experiments/{name}/compact→ 202 +CompactJobStatus;GET /api/v1/experiments/{name}/compact/status.last_compaction/total_compactionsin the stats table.Test plan
manual_compaction_runs_and_updates_stats: enqueue → worker compacts → job reachesDone, statstotal_compactions == 1enqueue_dedupes_queued_jobs: duplicate enqueue produces a single queue messagecargo test/clippy/fmtgreen🤖 Generated with Claude Code