Repository navigation
Conversation
Add the `maintain` and `reclaim` Cedar actions. `maintain` covers optimize, index builds, the full-text rebuild and the repair report and is judged on the branch maintained; `reclaim` covers cleanup and takes no branch scope. Maintenance has no actor-less public engine entry point any more: `optimize_as`, `cleanup_as`, `ensure_indices_as`, `ensure_indices_on_as`, `rebuild_full_text_indices_on_as` and the read-only `repair_report_as` each enforce their action before any effect. The full-text rebuild admits `maintain` or `change` through `PolicyChecker::check_any`. Repair loses its publishing mode, its options and its hard-coded actor strings. The CLI, DST, GQT and bench callers move to the `_as` forms; the direct CLI passes the actor it resolves for direct writes.
…cy-actions The index build from main checked `change`; it now checks `maintain` like the other index writers, and `ensure_indices_on_as` returns main's `IndexBuildResult`. The actor-less `ensure_indices` and `ensure_indices_on` stay removed. Main's unreleased index-build note is updated to match, and the maintenance page and notes are brought within their caps.
…utcomes optimize_as takes an OptimizeRequest (tables, start_at, max_gate_hold, planning and rewrite concurrency) and returns an OptimizeReport. A full-catalog run visits tables in type-key order from start_at and wraps; a tables request works in its own order and reports no resume position. Bad requests are refused before any effect. Plans run ahead in order and rewrites start in visit order. Once the gate-hold deadline passes, every table whose turn has not come is deferred, the finished tables publish once, and resume_at names the first deferred table. The configuration clear runs under the gates and reports its own outcome; the adjacency artifact is built after every writer gate is released. Neither fails a run whose commit is durable. The CLI gains --max-gate-hold and --tables. optimize --json reports one outcome per table, the input and published commits, the gate hold and both tail outcomes; the __manifest row becomes configuration_clear and the always-null published_dataset_version and lance_head_version go.
The shared publication helper takes an optional hook and calls it once, after the updates are prepared and before the coordinator lock and the publisher's first attempt, with the lineage intent's nonce and the captured head. Every attempt reads that head and carries that nonce, so a call per attempt would record nothing new. A hook failure publishes nothing and is labelled Unpublished whatever its own evidence says. The helper's other callers pass no hook. OptimizeRequest::with_before_publish passes the hook with the run's per-table outcomes. A run that stages no table never calls it.
A table's planning failure is labelled BeforeEffect and its staging failure Unpublished. Either reports the table failed, keeps the error on its stats, and leaves the other tables to publish in the run's one commit. A failure whose evidence stays uncertain still fails the run, as do failures outside one table: authority, catalog, gates and the publisher. The CLI reports a failed table with its error. The DST harnesses treat a failed table as the op's failure, so the convergence obligation still requires a rerun to finish every table.
OptimizeRequest::with_cancellation takes a shared handle. The run checks it before every table's plan and rewrite and before the publication, and awaits it meanwhile, so a table in progress is dropped where it stands; its detached effects stay unreachable. A cancelled run defers every table as cancelled, publishes nothing, leaves the configuration and the resume position as they were, and skips the artifact scan. A cancellation that arrives after the last check lets the run finish and is reported as having arrived late; only the artifact scan is skipped.
OptimizeLimits gains TableCeilings (source fragments, rows and bytes, managed Blob bytes, and the rows an index rebuild or build would scan) and run_source_bytes. Each set ceiling is measured from the table's compaction plan before its first write, fragments first so they bound the size requests, and counting stops once a value passes its ceiling. A table over one, or one whose size cannot be read, is deferred as resource_limit with the measure, and nothing is written for it. Source bytes are a data file's recorded size or a HEAD of the file, and a HEAD of each deletion file, which Lance 13 records no size for. Files resolve through the collector's base rule, now shared as data_file_location and deletion_file_location. Blob bytes come from the descriptor scan compaction sizing already runs, refactored into one visitor; external references count nothing. The run budget is also the default table byte ceiling, and a larger table ceiling is refused before any effect, so a first table within its ceilings always fits. The first table that would pass the remaining budget is deferred as byte_budget with every later one, and resume_at names it. Each table's measured bytes are reported on its stats.
OptimizeLimits::artifact_bytes bounds the whole object. Before the edge scan, a lower bound (the 20-byte prefix, every section prelude, one extra offset entry per array, the header at its shortest, and 8 bytes per live edge row) skips the artifact when it already passes the limit. During the scan, GraphIndex::build_counted reads each edge table one batch at a time and adds what the batch grows the object by, counted from the widest header: 4 bytes plus the UTF-8 length per new id, 4 bytes in every offset array its type sizes, and 8 bytes per edge. Once the count passes the limit it stops before keeping another batch or allocating the adjacency arrays. Either stop reports skipped(size_limit) and writes nothing. The unbounded build streams the same way instead of collecting every batch first. The artifact write runs in a spawned task the run awaits, so a dropped run cannot drop an issued write on the mutable key.
…ecord
A cluster server can run optimize on a graph's main when
OMNIGRAPH_MAINTENANCE_ROUTES is true: POST
/graphs/{graph_id}/maintenance/optimize submits under a caller-chosen
ULID, GET /graphs/{graph_id}/maintenance/{maintenance_id} looks it up and
POST .../cancel stops it at its next safe boundary. Every route needs
maintain on main.
The record is a new cluster control object, __cluster/maintenance.json,
separate from state.json and written only by the lock holder, each write a
conditional put: at acceptance, when the run starts, in optimize's
before-publish hook (publishing, with the nonce and input head) and at the
end. It holds one active operation, at most four unresolved publications,
32 terminal operations and each graph's resume position, and it names the
cluster's ledger id; another ledger's record is refused, never merged.
A write with no definite answer stops the run's writes. The next admission
or lookup reads the record back: an accepted or running operation no live
run holds is interrupted, a publishing one waits on main's history, and a
lost terminal write is re-issued. Omnigraph::resolve_publication walks
main's parent chain from a fresh head to the input head's child and judges
it against the nonce.
The run is an owned server operation that holds its graph request; shutdown
cancels it, and a deployment that affects the graph cancels it before its
drain. A repeated submission of the same request answers the existing
record; the same id with another request, an outstanding deployment and a
fifth unresolved publication are 409, another active operation 429.
…e-routes # Conflicts: # docs/user/index.md
…s own code The same maintenance id submitted with another request is refused with a [maintenance_request_mismatch] prefix, so a client tells it apart from the other 409 refusals without reading prose.
The served optimize request names tables and source and index rows: the physical rewrite it bounds. The vocabulary inventory records those four OpenAPI occurrences as accurate physical terminology.
The maintenance cancel hook bound its graph list to `affected`, shadowing the preview's affected graphs. The serving snapshot read after the apply then left out every graph the deployment created, so a live apply that added a graph completed without activating it. The hook now reads the preview's list directly; a graph the deployment creates has no run to cancel.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Rollout step 2 (Server) of remote optimize through the serving owner. It is stacked on #1204, which is stacked on #1175; review the top commit.
The RFC is back under review: #1203 reverts its early merge, and its text reopens for continued review. This PR stays a draft until the RFC is accepted, and follows any change the review makes.
The routes are off by default (
OMNIGRAPH_MAINTENANCE_ROUTES). The RFC keeps them off in production until #1176 makes the record's conditional writes re-issue lost answers, and until an in-region measurement sets the gate-hold and artifact defaults.startOptimize:POST /graphs/{graph_id}/maintenance/optimize, which answers 202 while the operation is non-terminal and 200 once it is terminal.getMaintenanceOperation:GET /graphs/{graph_id}/maintenance/{maintenance_id}, which answers 200 with the record or withlookup: unknown.cancelMaintenanceOperation:POST /graphs/{graph_id}/maintenance/{maintenance_id}/cancel.maintainon main.outcome_unknownwith the id.omnigraph-cluster):__cluster/maintenance.json(MaintenanceLedger) holds one active operation, at most four unresolved publications, 32 terminal operations within 1 MiB each and 4 MiB in total, and each graph's resume position.maintenance_record_foreign.ClusterAdmission::{maintenance_context, read_maintenance, write_maintenance}check the lock before each write. Each write answersWritten,ConflictorInDoubt.publishingwrite happens in the engine's before-publish hook.Omnigraph::resolve_publicationwalks main'sparent_commit_idchain from a fresh head to the input head's child. It answersCommitted,NotCommitted,HeadUnchangedorUnproven.acceptedorrunningoperation becomesinterrupted, and a lost terminal write is re-issued first.maintenance_idadded to the allowed path parameters; regeneratedopenapi.json.Tests
omnigraph-clustermaintenance unit tests:maintenance::resolve_publication_judges_the_input_heads_child.maintenance_routes:succeededrecord with its commit, a repeat answering the same record, 409 on another request under the id,unknown, cancel of a terminal operation, and the record file's ledger id.maintenance::suite,forbidden_apis, the seam guard, Clippy in both feature graphs,cargo fmt --checkandcheck-docs.py.