Repository navigation
Conversation
Add the `maintain` and `reclaim` Cedar actions. `maintain` covers optimize, index builds, the full-text rebuild and the repair report and is judged on the branch maintained; `reclaim` covers cleanup and takes no branch scope. Maintenance has no actor-less public engine entry point any more: `optimize_as`, `cleanup_as`, `ensure_indices_as`, `ensure_indices_on_as`, `rebuild_full_text_indices_on_as` and the read-only `repair_report_as` each enforce their action before any effect. The full-text rebuild admits `maintain` or `change` through `PolicyChecker::check_any`. Repair loses its publishing mode, its options and its hard-coded actor strings. The CLI, DST, GQT and bench callers move to the `_as` forms; the direct CLI passes the actor it resolves for direct writes.
…cy-actions The index build from main checked `change`; it now checks `maintain` like the other index writers, and `ensure_indices_on_as` returns main's `IndexBuildResult`. The actor-less `ensure_indices` and `ensure_indices_on` stay removed. Main's unreleased index-build note is updated to match, and the maintenance page and notes are brought within their caps.
…utcomes optimize_as takes an OptimizeRequest (tables, start_at, max_gate_hold, planning and rewrite concurrency) and returns an OptimizeReport. A full-catalog run visits tables in type-key order from start_at and wraps; a tables request works in its own order and reports no resume position. Bad requests are refused before any effect. Plans run ahead in order and rewrites start in visit order. Once the gate-hold deadline passes, every table whose turn has not come is deferred, the finished tables publish once, and resume_at names the first deferred table. The configuration clear runs under the gates and reports its own outcome; the adjacency artifact is built after every writer gate is released. Neither fails a run whose commit is durable. The CLI gains --max-gate-hold and --tables. optimize --json reports one outcome per table, the input and published commits, the gate hold and both tail outcomes; the __manifest row becomes configuration_clear and the always-null published_dataset_version and lance_head_version go.
The shared publication helper takes an optional hook and calls it once, after the updates are prepared and before the coordinator lock and the publisher's first attempt, with the lineage intent's nonce and the captured head. Every attempt reads that head and carries that nonce, so a call per attempt would record nothing new. A hook failure publishes nothing and is labelled Unpublished whatever its own evidence says. The helper's other callers pass no hook. OptimizeRequest::with_before_publish passes the hook with the run's per-table outcomes. A run that stages no table never calls it.
A table's planning failure is labelled BeforeEffect and its staging failure Unpublished. Either reports the table failed, keeps the error on its stats, and leaves the other tables to publish in the run's one commit. A failure whose evidence stays uncertain still fails the run, as do failures outside one table: authority, catalog, gates and the publisher. The CLI reports a failed table with its error. The DST harnesses treat a failed table as the op's failure, so the convergence obligation still requires a rerun to finish every table.
OptimizeRequest::with_cancellation takes a shared handle. The run checks it before every table's plan and rewrite and before the publication, and awaits it meanwhile, so a table in progress is dropped where it stands; its detached effects stay unreachable. A cancelled run defers every table as cancelled, publishes nothing, leaves the configuration and the resume position as they were, and skips the artifact scan. A cancellation that arrives after the last check lets the run finish and is reported as having arrived late; only the artifact scan is skipped.
OptimizeLimits gains TableCeilings (source fragments, rows and bytes, managed Blob bytes, and the rows an index rebuild or build would scan) and run_source_bytes. Each set ceiling is measured from the table's compaction plan before its first write, fragments first so they bound the size requests, and counting stops once a value passes its ceiling. A table over one, or one whose size cannot be read, is deferred as resource_limit with the measure, and nothing is written for it. Source bytes are a data file's recorded size or a HEAD of the file, and a HEAD of each deletion file, which Lance 13 records no size for. Files resolve through the collector's base rule, now shared as data_file_location and deletion_file_location. Blob bytes come from the descriptor scan compaction sizing already runs, refactored into one visitor; external references count nothing. The run budget is also the default table byte ceiling, and a larger table ceiling is refused before any effect, so a first table within its ceilings always fits. The first table that would pass the remaining budget is deferred as byte_budget with every later one, and resume_at names it. Each table's measured bytes are reported on its stats.
OptimizeLimits::artifact_bytes bounds the whole object. Before the edge scan, a lower bound (the 20-byte prefix, every section prelude, one extra offset entry per array, the header at its shortest, and 8 bytes per live edge row) skips the artifact when it already passes the limit. During the scan, GraphIndex::build_counted reads each edge table one batch at a time and adds what the batch grows the object by, counted from the widest header: 4 bytes plus the UTF-8 length per new id, 4 bytes in every offset array its type sizes, and 8 bytes per edge. Once the count passes the limit it stops before keeping another batch or allocating the adjacency arrays. Either stop reports skipped(size_limit) and writes nothing. The unbounded build streams the same way instead of collecting every batch first. The artifact write runs in a spawned task the run awaits, so a dropped run cannot drop an issued write on the mutable key.
This was referenced Oct 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Rollout step 1 (Engine) of remote optimize through the serving owner. It is stacked on #1175 and includes its commits; review the six commits after it, one per slice. The server record and routes (step 2, #1205, closed until acceptance) and the CLI served path (step 3, #1206, closed until acceptance) stack on it.
The RFC is back under review as #1214 after #1203 reverted its early merge. This PR stays a draft until the RFC is accepted, and follows any change the review makes.
optimize_as(actor, OptimizeRequest)returns anOptimizeReport.OptimizeRequest::default()is today's full run.feat(engine)!).start_atand wraps. Atablesrequest works in its own order and neither reads nor reports a resume position. Bad requests are refused before any effect.deferred(gate_hold_limit). The finished tables publish once, andresume_atnames the first deferred table.--max-gate-holdand--tablesare added. JSON reports the outcomes and drops the__manifestrow and the always-null version fields (breaking note).Unpublished. Other callers pass no hook.BeforeEffect) or staging failure (Unpublished) reports itfailedwith its error, and the other tables still publish. An uncertain failure still fails the run. The DST harnesses treat a failed table as the op's failure, so their obligations keep their meaning.OptimizeRequest::with_cancellationis checked before every plan, before every rewrite, and before the publication. It is also awaited in between, so a table in progress is dropped where it stands.cancelledand publishes nothing. It skips the configuration clear and the artifact, and leaves the resume position where it was.arrived_late.TableCeilingscovers source fragments, rows and bytes, managed Blob bytes, and index rows. Each is measured from the table's compaction plan before its first write, fragments first, and only when set.data_file_locationanddeletion_file_location. Lance 13 records no deletion-file size, so deletion files always take a HEAD.deferred(resource_limit)with nothing written for it.run_source_bytesis also the default table byte ceiling, and a larger explicit ceiling is refused. The first table that would pass the budget, and every later one, isdeferred(byte_budget), andresume_atnames it.artifact_bytesis checked first against a lower bound: the fixed bytes at the shortest header, plus 8 bytes per live edge row.skipped(size_limit)and writes nothing.Tests
maintenance.rstests for:failpoints.rstests for:persist.rstest brackets the serialized size between the two bounds.maintenance::,traversal_adaptive::,traversal_indexed::andmerge_cleanup::;detached_commit_matrix,policy_engine_chassis,forbidden_apisand the seam guard;cargo fmt --check,check-docs.pyandtypos.