Repository navigation
feat(rollout): base-table compaction (external-cron driven) - #132
Merged
Merged
Conversation
WAL merges append a fragment to the base table each time, so a long-lived
rollout store accumulates many small fragments. Add compaction to fold
them back together, mirroring ContextStore::compact.
- RolloutStore::compact() runs Lance compact_files on the base table,
reloads the handle, and tracks last_compaction / total_compactions /
last_error. should_compact(config) gates on min_fragments + quiet_hours;
compaction_stats() reports fragment count and history.
- Uses defer_index_remap: the rollout base table carries a fieldless
MemWAL index, and Lance's inline index remap panics on it ("An index
existed with no fields"). Deferring records a fragment-reuse index and
remaps lazily instead of touching the MemWAL index during the rewrite.
- Server: POST /api/v1/rollouts/{name}/compact and
GET .../compact/stats, reusing the existing Compact* DTOs.
Distributed model: compaction rewrites the SHARED base table, and two
concurrent Rewrite commits conflict in Lance, so this is meant to be
driven by a single external scheduler (cron / k8s CronJob), NOT a
per-worker background timer — unlike WAL merge, which each worker does
only on its own shard. It is safe to run alongside worker appends /
WAL merges: Append vs Rewrite is non-conflicting in Lance's matrix.
Tests: compact reduces fragment count while conserving rows and blobs;
should_compact respects min_fragments; compact composes with a
concurrent WAL merge from another shard without failing or losing rows.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds base-table compaction to the rollout store. Every WAL merge (
merge_own_shard) appends a fragment to the base table, so a long-lived rollout store accumulates many small fragments that slow scans. This folds them back together via Lance'scompact_files, mirroringContextStore::compact.Follow-up to #131 (WAL cleanup + concurrency hardening).
Core (
RolloutStore)compact(config)— runscompact_fileson the base table, reloads the handle, and trackslast_compaction/total_compactions/last_error.should_compact(config)— gates onmin_fragments+quiet_hoursso an external scheduler can pass the same config.compaction_stats()— fragment count + compaction history.defer_index_remap: true: the rollout base table carries a fieldless MemWAL index, and Lance's inline index remap panics on it ("An index existed with no fields"). Deferring records a fragment-reuse index and remaps lazily instead of touching the MemWAL index during the rewrite.Server
POST /api/v1/rollouts/{name}/compactandGET /api/v1/rollouts/{name}/compact/stats, reusing the existingCompact*DTOs.Distributed model (important)
Compaction rewrites the shared base table, and two concurrent
Rewritecommits conflict in Lance's commit matrix (one wins, the other must redo its work). So this is meant to be driven by a single external scheduler (cron / k8s CronJob), not a per-worker background timer — unlike WAL merge, where each worker only ever touches its own shard. It is safe to run alongside worker appends / WAL merges:AppendvsRewriteis non-conflicting in Lance's matrix, so a concurrent append simply rebases on the compaction (or vice-versa).Test plan
compact_reduces_fragments_and_preserves_reads— fragment count drops; all rows readable exactly once; inline artifact blobs still fetchable; stats updatedshould_compact_respects_min_fragmentscompact_composes_with_concurrent_wal_merge— a compaction (Rewrite) and a WAL merge (Append) from another shard run concurrently without failing or losing rowslance-context-core(114) andlance-context-server(28) suites,cargo fmt --all --check,cargo clippy --workspace --all-targetsall green🤖 Generated with Claude Code