From de23d980cd62c793cf72d17edafe729bb077f994 Mon Sep 17 00:00:00 2001 From: Ragnor Comerford Date: Mon, 26 Oct 2026 13:16:32 +0800 Subject: [PATCH 01/13] feat(policy)!: gate maintenance behind the maintain and reclaim actions Add the `maintain` and `reclaim` Cedar actions. `maintain` covers optimize, index builds, the full-text rebuild and the repair report and is judged on the branch maintained; `reclaim` covers cleanup and takes no branch scope. Maintenance has no actor-less public engine entry point any more: `optimize_as`, `cleanup_as`, `ensure_indices_as`, `ensure_indices_on_as`, `rebuild_full_text_indices_on_as` and the read-only `repair_report_as` each enforce their action before any effect. The full-text rebuild admits `maintain` or `change` through `PolicyChecker::check_any`. Repair loses its publishing mode, its options and its hard-coded actor strings. The CLI, DST, GQT and bench callers move to the `_as` forms; the direct CLI passes the actor it resolves for direct writes. --- ...ll-text-rebuild-change-grant.deprecated.md | 6 + ...nce-entry-points-take-an-actor.breaking.md | 12 + .../maintenance-policy-actions.added.md | 10 + crates/omnigraph-api-types/src/lib.rs | 3 + crates/omnigraph-bench/src/dataset_worker.rs | 5 +- .../src/legacy/branch_merge_oracle.rs | 2 +- .../src/commands/maintenance/mod.rs | 11 +- crates/omnigraph-cli/src/output.rs | 13 +- .../tests/integration/cli_data.rs | 5 +- crates/omnigraph-dst/src/concurrent.rs | 15 +- crates/omnigraph-dst/src/harness.rs | 17 +- crates/omnigraph-dst/tests/scenarios.rs | 54 +- crates/omnigraph-gqt-core/src/lib.rs | 2 +- crates/omnigraph-policy/src/lib.rs | 271 +++++++++- crates/omnigraph/src/db/graph_coordinator.rs | 7 +- crates/omnigraph/src/db/mod.rs | 2 +- crates/omnigraph/src/db/omnigraph.rs | 94 ++-- .../omnigraph/src/db/omnigraph/collector.rs | 18 +- crates/omnigraph/src/db/omnigraph/optimize.rs | 42 +- crates/omnigraph/src/db/omnigraph/repair.rs | 162 +----- .../omnigraph/src/db/omnigraph/table_ops.rs | 35 +- crates/omnigraph/src/db/upgrade/tests.rs | 6 +- crates/omnigraph/src/engine/plan_source.rs | 2 +- crates/omnigraph/src/runtime_cache.rs | 2 +- crates/omnigraph/tests/bound_storage.rs | 13 +- .../omnigraph/tests/detached_commit_matrix.rs | 47 +- crates/omnigraph/tests/failpoints.rs | 272 +++++----- crates/omnigraph/tests/forbidden_apis.rs | 118 +++- .../tests/helpers/expand_projection.rs | 2 +- crates/omnigraph/tests/helpers/mod.rs | 2 +- .../omnigraph/tests/integration/branching.rs | 79 ++- crates/omnigraph/tests/integration/changes.rs | 19 +- .../tests/integration/changes_cost.rs | 125 +---- .../tests/integration/composite_flow.rs | 13 +- .../tests/integration/consistency.rs | 4 +- .../omnigraph/tests/integration/end_to_end.rs | 6 +- .../omnigraph/tests/integration/engine_v2.rs | 4 +- .../tests/integration/engine_v2_memory.rs | 10 +- .../integration/engine_v2_plan_replay.rs | 6 +- .../tests/integration/literal_filters.rs | 2 +- .../tests/integration/maintenance.rs | 505 ++++++++++-------- .../tests/integration/merge_cleanup.rs | 4 +- .../tests/integration/point_in_time.rs | 5 +- .../omnigraph/tests/integration/recovery.rs | 8 +- .../tests/integration/repro_issue_563.rs | 2 +- .../tests/integration/rrf_prefilter_gate.rs | 14 +- .../tests/integration/scalar_indexes.rs | 4 +- .../tests/integration/traversal_adaptive.rs | 6 +- .../tests/integration/warm_read_cost.rs | 22 +- .../omnigraph/tests/integration/write_cost.rs | 16 +- .../omnigraph/tests/policy_engine_chassis.rs | 306 +++++++++-- crates/omnigraph/tests/schema_apply.rs | 27 +- crates/omnigraph/tests/search.rs | 35 +- .../omnigraph/tests/system_column_upgrade.rs | 6 +- docs/dev/control-plane.md | 1 + docs/dev/writes.md | 9 + docs/user/operations/maintenance.md | 15 +- docs/user/operations/policy.md | 18 +- skills/omnigraph/references/server-policy.md | 6 +- 59 files changed, 1566 insertions(+), 961 deletions(-) create mode 100644 changelog.d/full-text-rebuild-change-grant.deprecated.md create mode 100644 changelog.d/maintenance-entry-points-take-an-actor.breaking.md create mode 100644 changelog.d/maintenance-policy-actions.added.md diff --git a/changelog.d/full-text-rebuild-change-grant.deprecated.md b/changelog.d/full-text-rebuild-change-grant.deprecated.md new file mode 100644 index 000000000..b17d94953 --- /dev/null +++ b/changelog.d/full-text-rebuild-change-grant.deprecated.md @@ -0,0 +1,6 @@ +- The full-text rebuild accepts `maintain` or `change` on its branch in this + release; a later release accepts `maintain` only. Grant `maintain` to the + accounts that rebuild full-text indexes. See + [rebuild full-text indexes][full-text-rebuild-change-grant]. + +[full-text-rebuild-change-grant]: ../docs/user/operations/maintenance.md#rebuild-full-text-indexes diff --git a/changelog.d/maintenance-entry-points-take-an-actor.breaking.md b/changelog.d/maintenance-entry-points-take-an-actor.breaking.md new file mode 100644 index 000000000..d6f96b6fc --- /dev/null +++ b/changelog.d/maintenance-entry-points-take-an-actor.breaking.md @@ -0,0 +1,12 @@ +- Rust callers of the engine's maintenance methods move to their actor-taking + forms: `optimize_as(actor)`, `cleanup_as(options, actor)`, + `ensure_indices_as(actor)`, `ensure_indices_on_as(branch, actor)` and + `rebuild_full_text_indices_on_as(branch, actor)`; pass `None` where no policy + is installed. `repair` is now the read-only `repair_report_as(actor)`: + `RepairOptions`, the `healed` and `forced` actions and + `RepairStats::graph_manifest_version` are gone, and `repair --json` no longer + prints the always-null `graph_manifest_version`. An exhaustive match on + `PolicyAction` adds arms for `Maintain` and `Reclaim`. See + [repair][maintenance-entry-points-repair]. + +[maintenance-entry-points-repair]: ../docs/user/operations/maintenance.md#repair diff --git a/changelog.d/maintenance-policy-actions.added.md b/changelog.d/maintenance-policy-actions.added.md new file mode 100644 index 000000000..d7698ffce --- /dev/null +++ b/changelog.d/maintenance-policy-actions.added.md @@ -0,0 +1,10 @@ +- **Maintenance has its own Cedar actions.** `maintain` covers optimize, index + builds, the full-text rebuild and the repair report on the branch they + maintain, and `reclaim` covers cleanup with no branch scope; `change` implies + neither, so automation can compact a graph without the right to write data or + delete history. An embedded host that installs a policy checker enforces them + on every maintenance entry point, optimize and the index writers record the + requesting actor on their commit, and existing policies grant neither action. + See [policy actions][maintenance-policy-actions]. + +[maintenance-policy-actions]: ../docs/user/operations/policy.md#actions diff --git a/crates/omnigraph-api-types/src/lib.rs b/crates/omnigraph-api-types/src/lib.rs index f07f23307..ebe8b3a1c 100644 --- a/crates/omnigraph-api-types/src/lib.rs +++ b/crates/omnigraph-api-types/src/lib.rs @@ -2013,6 +2013,9 @@ fn access_slot(action: omnigraph_policy::PolicyAction) -> Option { P::InvokeQuery => Some(AccessSlot::GraphWide), // Server- or operator-scoped: no graph's access report names them. P::Admin | P::GraphList | P::ConfigManage => None, + // Maintenance runs only through the direct CLI and embedded callers; + // no served operation performs it, so the report does not name it. + P::Maintain | P::Reclaim => None, } } diff --git a/crates/omnigraph-bench/src/dataset_worker.rs b/crates/omnigraph-bench/src/dataset_worker.rs index f66ae24af..3999d6e98 100644 --- a/crates/omnigraph-bench/src/dataset_worker.rs +++ b/crates/omnigraph-bench/src/dataset_worker.rs @@ -803,7 +803,10 @@ pub(crate) async fn build_dataset( Engine::V2, )?; if plan.needs_indices { - session.ensure_indices().await.map_err(|e| e.to_string())?; + session + .ensure_indices_as(None) + .await + .map_err(|e| e.to_string())?; } (session, case, Some(exact)) } diff --git a/crates/omnigraph-bench/src/legacy/branch_merge_oracle.rs b/crates/omnigraph-bench/src/legacy/branch_merge_oracle.rs index b5151f265..78466e2be 100644 --- a/crates/omnigraph-bench/src/legacy/branch_merge_oracle.rs +++ b/crates/omnigraph-bench/src/legacy/branch_merge_oracle.rs @@ -28,7 +28,7 @@ pub async fn initialize_local_fixture( let optimized_user_tables = match plan.compaction_recency { CompactionRecency::Optimized => { let outcomes = db - .optimize() + .optimize_as(None) .await .map_err(|error| fixture_error(format!("optimize fixture main branch: {error}")))?; let intended_keys = (0..plan.node_tables()) diff --git a/crates/omnigraph-cli/src/commands/maintenance/mod.rs b/crates/omnigraph-cli/src/commands/maintenance/mod.rs index a6557054c..f39beb150 100644 --- a/crates/omnigraph-cli/src/commands/maintenance/mod.rs +++ b/crates/omnigraph-cli/src/commands/maintenance/mod.rs @@ -3,7 +3,7 @@ use std::io::Write; use color_eyre::eyre::{Result, bail}; -use omnigraph::db::{CleanupPolicyOptions, Omnigraph, RepairAction, RepairOptions}; +use omnigraph::db::{CleanupPolicyOptions, Omnigraph, RepairAction}; use crate::app::Completion; use crate::helpers::{confirm_destructive, parse_duration_arg, resolve_actor}; @@ -42,13 +42,14 @@ pub(crate) async fn run_optimize( let OptimizeArgs { json } = args; let uri = crate::scope::direct_graph_uri(scope.target, scope.graph).await?; let binding = crate::scope::capture_storage_binding(scope.target)?.for_uri(&uri)?; + let actor = resolve_actor(scope.operator, scope.actor, stderr)?; crate::admission::ensure_graph_with_binding(&binding, stderr).await?; let db = Omnigraph::open_with_storage( binding.uri(), omnigraph::storage::storage_for_binding(&binding)?, ) .await?; - let stats = db.optimize().await?; + let stats = db.optimize_as(actor.as_deref()).await?; write_optimize(&uri, &stats, json, stdout)?; Ok(Completion::success()) } @@ -93,13 +94,14 @@ pub(crate) async fn run_repair( let RepairArgs { json } = args; let uri = crate::scope::direct_graph_uri(scope.target, scope.graph).await?; let binding = crate::scope::capture_storage_binding(scope.target)?.for_uri(&uri)?; + let actor = resolve_actor(scope.operator, scope.actor, stderr)?; crate::admission::ensure_graph_with_binding(&binding, stderr).await?; let db = Omnigraph::open_with_storage( binding.uri(), omnigraph::storage::storage_for_binding(&binding)?, ) .await?; - let stats = db.repair(RepairOptions::default()).await?; + let stats = db.repair_report_as(actor.as_deref()).await?; let refused_count = stats .datasets .iter() @@ -159,13 +161,14 @@ pub(crate) async fn run_cleanup( keep_versions: keep, older_than: older_than_dur, }; + let actor = resolve_actor(scope.operator, scope.actor, stderr)?; crate::admission::ensure_graph_with_binding(&binding, stderr).await?; let db = Omnigraph::open_with_storage( binding.uri(), omnigraph::storage::storage_for_binding(&binding)?, ) .await?; - let stats = db.cleanup(options.clone()).await?; + let stats = db.cleanup_as(options.clone(), actor.as_deref()).await?; write_cleanup(&uri, &options, &policy_desc, &stats, json, stdout)?; Ok(Completion::success()) } diff --git a/crates/omnigraph-cli/src/output.rs b/crates/omnigraph-cli/src/output.rs index aa8280127..6455bd7d5 100644 --- a/crates/omnigraph-cli/src/output.rs +++ b/crates/omnigraph-cli/src/output.rs @@ -133,7 +133,6 @@ pub(crate) fn write_repair( if json { let value = serde_json::json!({ "uri": uri, - "graph_manifest_version": stats.graph_manifest_version, "datasets": stats.datasets.iter().map(|s| serde_json::json!({ "type_key": s.type_key, "published_dataset_version": s.published_dataset_version, @@ -1867,7 +1866,7 @@ mod tests { #[tokio::test] async fn maintenance_optimize_and_repair_reports_preserve_all_result_fields() { use omnigraph::db::{ - Omnigraph, PendingIndex, RepairAction, RepairClassification, RepairOptions, SkipReason, + Omnigraph, PendingIndex, RepairAction, RepairClassification, SkipReason, }; // These engine result types are non_exhaustive. Obtain one valid value // from a tiny graph, then set its public presentation fields; no engine @@ -1881,7 +1880,7 @@ mod tests { let db = Omnigraph::init(&uri, "node Person {\n slug: String @key\n}\n") .await .unwrap(); - let mut compacted = db.optimize().await.unwrap().pop().unwrap(); + let mut compacted = db.optimize_as(None).await.unwrap().pop().unwrap(); compacted.type_key = "node:Person".into(); compacted.fragments_removed = 3; compacted.fragments_added = 1; @@ -1933,7 +1932,7 @@ mod tests { |json, writer| super::write_optimize("graph", &[], json, writer), ); - let mut repair = db.repair(RepairOptions::default()).await.unwrap(); + let mut repair = db.repair_report_as(None).await.unwrap(); let mut clean = repair.datasets.pop().unwrap(); clean.type_key = "node:Person".into(); clean.published_dataset_version = 4; @@ -1950,7 +1949,6 @@ mod tests { refused.operations = vec!["Append".into(), "Delete".into()]; refused.error = Some("foreign write".into()); repair.datasets = vec![clean, refused]; - repair.graph_manifest_version = Some(9); assert_maintenance_report( concat!( "repair graph — 2 datasets\n", @@ -1958,7 +1956,7 @@ mod tests { " edge type 'Knows' refused foreign_drift 4 → 6 [Append, Delete] (foreign write)\n", ), serde_json::json!({ - "uri":"graph", "graph_manifest_version":9, + "uri":"graph", "datasets":[ {"type_key":"node:Person","published_dataset_version":4,"lance_head_version":4,"classification":"no_drift","action":"no_op","operations":[],"error":null}, {"type_key":"edge:Knows","published_dataset_version":4,"lance_head_version":6,"classification":"foreign_drift","action":"refused","operations":["Append","Delete"],"error":"foreign write"}, @@ -1967,10 +1965,9 @@ mod tests { |json, writer| super::write_repair("graph", &repair, json, writer), ); repair.datasets.clear(); - repair.graph_manifest_version = None; assert_maintenance_report( "repair graph — 0 datasets\n", - serde_json::json!({"uri":"graph","graph_manifest_version":null,"datasets":[]}), + serde_json::json!({"uri":"graph","datasets":[]}), |json, writer| super::write_repair("graph", &repair, json, writer), ); } diff --git a/crates/omnigraph-cli/tests/integration/cli_data.rs b/crates/omnigraph-cli/tests/integration/cli_data.rs index 939fe3de9..9d110089a 100644 --- a/crates/omnigraph-cli/tests/integration/cli_data.rs +++ b/crates/omnigraph-cli/tests/integration/cli_data.rs @@ -1211,7 +1211,8 @@ fn repair_json_reports_noop_on_clean_graph() { assert!(payload.get("confirm").is_none()); assert!(payload.get("force").is_none()); - assert_eq!(payload["graph_manifest_version"], Value::Null); + // The report publishes nothing, so it names no graph manifest version. + assert!(payload.get("graph_manifest_version").is_none()); assert!(payload.get("manifest_version").is_none()); assert!(payload.get("tables").is_none()); let datasets = payload["datasets"].as_array().unwrap(); @@ -1282,7 +1283,7 @@ fn repair_reports_foreign_drift_and_refuses_removed_flags() { .arg(&graph), ); let payload: Value = serde_json::from_slice(&output.stdout).unwrap(); - assert_eq!(payload["graph_manifest_version"], Value::Null); + assert!(payload.get("graph_manifest_version").is_none()); let person = payload["datasets"] .as_array() .unwrap() diff --git a/crates/omnigraph-dst/src/concurrent.rs b/crates/omnigraph-dst/src/concurrent.rs index a05a5274a..98fc5f188 100644 --- a/crates/omnigraph-dst/src/concurrent.rs +++ b/crates/omnigraph-dst/src/concurrent.rs @@ -1753,12 +1753,15 @@ fn maintenance_life( let mut occ_retries = 0usize; loop { let result = match kind { - 0 => Box::pin(db.ensure_indices()).await.map(|_| ()), - 1 => Box::pin(db.optimize()).await.map(|_| ()), - _ => Box::pin(db.cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - })) + 0 => Box::pin(db.ensure_indices_as(None)).await.map(|_| ()), + 1 => Box::pin(db.optimize_as(None)).await.map(|_| ()), + _ => Box::pin(db.cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + )) .await .map(|_| ()), }; diff --git a/crates/omnigraph-dst/src/harness.rs b/crates/omnigraph-dst/src/harness.rs index 109c1df1c..9f4e65730 100644 --- a/crates/omnigraph-dst/src/harness.rs +++ b/crates/omnigraph-dst/src/harness.rs @@ -1634,12 +1634,15 @@ async fn exec_op(db: &Session, branch: &str, op: &Op) -> OmniResult<()> { ) .await .map(|_| ()), - Op::Optimize => Box::pin(db.optimize()).await.map(|_| ()), + Op::Optimize => Box::pin(db.optimize_as(None)).await.map(|_| ()), Op::Cleanup => { - let stats = Box::pin(db.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - })) + let stats = Box::pin(db.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + )) .await?; let deferred: BTreeMap = stats .iter() @@ -1655,7 +1658,7 @@ async fn exec_op(db: &Session, branch: &str, op: &Op) -> OmniResult<()> { .insert(db.uri().to_string(), deferred); Ok(()) } - Op::EnsureIndices => Box::pin(db.ensure_indices()).await.map(|_| ()), + Op::EnsureIndices => Box::pin(db.ensure_indices_as(None)).await.map(|_| ()), Op::SchemaAddProperty { count } => Box::pin(db.apply_schema(&schema_with_extras(*count))) .await .map(|_| ()), @@ -6432,7 +6435,7 @@ impl UniverseScenario for Scenario { tokio::time::resume(); let lively = tokio::time::timeout( std::time::Duration::from_secs(120), - Box::pin(db.ensure_indices()), + Box::pin(db.ensure_indices_as(None)), ) .await; tokio::time::pause(); diff --git a/crates/omnigraph-dst/tests/scenarios.rs b/crates/omnigraph-dst/tests/scenarios.rs index 8266b52b3..c0cc84306 100644 --- a/crates/omnigraph-dst/tests/scenarios.rs +++ b/crates/omnigraph-dst/tests/scenarios.rs @@ -1045,7 +1045,7 @@ async fn dst_memory_graph_end_to_end() { db.load_jsonl(TEST_DATA, LoadMode::Overwrite) .await .expect("load test data"); - db.ensure_indices().await.expect("ensure indices"); + db.ensure_indices_as(None).await.expect("ensure indices"); let persons = count_rows(&db, "node:Person").await; assert!(persons > 0, "loaded graph must have Person rows"); @@ -3668,7 +3668,7 @@ fn reborn_branch_cache_poison_body() { .expect("op1"); db.branch_create("b0").await.expect("op2"); world_read(&db).await; // ARMING READ 1 (the universe's op-2 verify) - db.optimize().await.expect("op3"); + db.optimize_as(None).await.expect("op3"); m!( db, "b0", @@ -3682,7 +3682,7 @@ fn reborn_branch_cache_poison_body() { Box::pin(db.branch_merge("b0", "main")) .await .expect("op6 merge"); - db.ensure_indices().await.expect("op7"); + db.ensure_indices_as(None).await.expect("op7"); db.branch_delete("b0").await.expect("op8"); m!( db, @@ -3716,7 +3716,7 @@ fn reborn_branch_cache_poison_body() { &[] ) .expect("op12"); - db.ensure_indices().await.expect("op13"); + db.ensure_indices_as(None).await.expect("op13"); m!( db, "main", @@ -3757,10 +3757,13 @@ fn reborn_branch_cache_poison_body() { &[("$age", 20)] ) .expect("op20"); - db.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .expect("op21"); m!( @@ -3779,10 +3782,13 @@ fn reborn_branch_cache_poison_body() { &[("$age", 14), ("$ver", 8)] ) .expect("op23"); - db.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .expect("op24"); db.branch_create("b0").await.expect("op25"); @@ -5197,7 +5203,7 @@ fn dst_classa_ablation_matrix_body() { db.branch_create("b0").await.expect("bc b0 life1"); } if o { - db.optimize().await.expect("optimize"); + db.optimize_as(None).await.expect("optimize"); } if r { upsert(&db, "b0", "w6", 45, 2).await; @@ -5210,20 +5216,20 @@ fn dst_classa_ablation_matrix_body() { upsert(&db, "main", "w1", 19, 3).await; upsert(&db, "main", "w7", 7, 6).await; if c { - db.cleanup(omnigraph::db::CleanupPolicyOptions { + db.cleanup_as(omnigraph::db::CleanupPolicyOptions { keep_versions: Some(1), older_than: None, - }) + }, None) .await .expect("cleanup 1"); } upsert(&db, "main", "w7", 17, 7).await; upsert(&db, "main", "w5", 14, 8).await; if c { - db.cleanup(omnigraph::db::CleanupPolicyOptions { + db.cleanup_as(omnigraph::db::CleanupPolicyOptions { keep_versions: Some(1), older_than: None, - }) + }, None) .await .expect("cleanup 2"); } @@ -5284,26 +5290,26 @@ fn dst_classa_ablation_matrix_body() { m!(db, "main", "set_age_v", &[("$name", "w5")], &[("$age", 70), ("$ver", 1)]).expect("op0"); m!(db, "main", "insert_person", &[("$name", "w6")], &[("$age", 7)]).expect("op1"); db.branch_create("b0").await.expect("op2"); - db.optimize().await.expect("op3"); + db.optimize_as(None).await.expect("op3"); m!(db, "b0", "set_age_v", &[("$name", "w6")], &[("$age", 45), ("$ver", 2)]).expect("op4"); m!(db, "b0", "remove_friendships_from", &[("$from", "w6")], &[]).expect("op5"); Box::pin(db.branch_merge("b0", "main")).await.expect("op6 merge"); - db.ensure_indices().await.expect("op7"); + db.ensure_indices_as(None).await.expect("op7"); db.branch_delete("b0").await.expect("op8"); m!(db, "main", "remove_friendships_from", &[("$from", "w2")], &[]).expect("op9"); m!(db, "main", "insert_person_v", &[("$name", "w1")], &[("$age", 19), ("$ver", 3)]).expect("op10"); m!(db, "main", "set_age_v", &[("$name", "w2")], &[("$age", 3), ("$ver", 4)]).expect("op11"); m!(db, "main", "add_friend", &[("$from", "Charlie"), ("$to", "w6")], &[]).expect("op12"); - db.ensure_indices().await.expect("op13"); + db.ensure_indices_as(None).await.expect("op13"); m!(db, "main", "insert_person", &[("$name", "w1")], &[("$age", 52)]).expect("op15"); m!(db, "main", "remove_friendships_from", &[("$from", "w3")], &[]).expect("op16"); m!(db, "main", "set_age_v", &[("$name", "w3")], &[("$age", 24), ("$ver", 5)]).expect("op18"); m!(db, "main", "set_age_v", &[("$name", "w7")], &[("$age", 7), ("$ver", 6)]).expect("op19"); m!(db, "main", "insert_person", &[("$name", "w7")], &[("$age", 20)]).expect("op20"); - db.cleanup(omnigraph::db::CleanupPolicyOptions { keep_versions: Some(1), older_than: None }).await.expect("op21"); + db.cleanup_as(omnigraph::db::CleanupPolicyOptions { keep_versions: Some(1), older_than: None }, None).await.expect("op21"); m!(db, "main", "set_age_v", &[("$name", "w7")], &[("$age", 17), ("$ver", 7)]).expect("op22"); m!(db, "main", "set_age_v", &[("$name", "w5")], &[("$age", 14), ("$ver", 8)]).expect("op23"); - db.cleanup(omnigraph::db::CleanupPolicyOptions { keep_versions: Some(1), older_than: None }).await.expect("op24"); + db.cleanup_as(omnigraph::db::CleanupPolicyOptions { keep_versions: Some(1), older_than: None }, None).await.expect("op24"); db.branch_create("b0").await.expect("op25"); let empty_merge = Box::pin(db.branch_merge("b0", "main")).await; println!("faithful op26 empty merge: {:?}", empty_merge.map(|_| "ok")); @@ -5373,7 +5379,7 @@ fn dst_liveness_oracle_survives_cross_thread_work() { tokio::time::resume(); let lively = tokio::time::timeout( std::time::Duration::from_secs(120), - Box::pin(db.ensure_indices()), + Box::pin(db.ensure_indices_as(None)), ) .await; tokio::time::pause(); @@ -6338,7 +6344,7 @@ fn dst_optimize_races_branch_delete_minimal_two_thread_negative() { let _ = mutate_main(&db, MUTATION_QUERIES, "set_age", ¶ms).await; } - match Box::pin(db.optimize()).await { + match Box::pin(db.optimize_as(None)).await { Ok(_) => {} Err(err) => { let rendered = format!("{err:?}"); diff --git a/crates/omnigraph-gqt-core/src/lib.rs b/crates/omnigraph-gqt-core/src/lib.rs index 69f117dc1..02b8f3afc 100644 --- a/crates/omnigraph-gqt-core/src/lib.rs +++ b/crates/omnigraph-gqt-core/src/lib.rs @@ -4294,7 +4294,7 @@ pub async fn seed_case(session: &Session, seed: &Seed, needs_indices: bool) -> R seed.load(session).await?; if needs_indices { session - .ensure_indices() + .ensure_indices_as(None) .await .map_err(|e| format!("ensure_indices failed: {e}"))?; } diff --git a/crates/omnigraph-policy/src/lib.rs b/crates/omnigraph-policy/src/lib.rs index 3e1c18422..2b2383b7f 100644 --- a/crates/omnigraph-policy/src/lib.rs +++ b/crates/omnigraph-policy/src/lib.rs @@ -30,6 +30,16 @@ pub enum PolicyAction { /// Drop an unpublished graph commit (`commit drop ""`). Target-branch /// scoped: the branch the proposal would have landed on. CommitDrop, + /// Content-preserving maintenance: optimize, index builds, the full-text + /// rebuild and the repair report. Branch scoped like `change`: the scope + /// is the branch whose tables the operation maintains, `main` for + /// optimize and repair. It grants no right to change data or to delete + /// history. + Maintain, + /// Reclaim storage: cleanup, which permanently deletes table versions and + /// files no retained commit needs. A run covers every branch's history, + /// so the action is graph scoped and takes no branch scope. + Reclaim, /// Reserved action; no current engine or HTTP operation uses this grant. Admin, /// List the server's graph registry. This is server-scoped, with no branch @@ -57,6 +67,8 @@ impl PolicyAction { Self::BranchMerge => "branch_merge", Self::CommitPublish => "commit_publish", Self::CommitDrop => "commit_drop", + Self::Maintain => "maintain", + Self::Reclaim => "reclaim", Self::Admin => "admin", Self::GraphList => "graph_list", Self::InvokeQuery => "invoke_query", @@ -65,7 +77,10 @@ impl PolicyAction { } fn uses_branch_scope(self) -> bool { - matches!(self, Self::Read | Self::Export | Self::Change) + matches!( + self, + Self::Read | Self::Export | Self::Change | Self::Maintain + ) } fn uses_target_branch_scope(self) -> bool { @@ -98,6 +113,8 @@ impl PolicyAction { | Self::BranchMerge | Self::CommitPublish | Self::CommitDrop + | Self::Maintain + | Self::Reclaim | Self::Admin | Self::InvokeQuery => PolicyResourceKind::Graph, } @@ -160,6 +177,8 @@ impl FromStr for PolicyAction { "branch_merge" => Ok(Self::BranchMerge), "commit_publish" => Ok(Self::CommitPublish), "commit_drop" => Ok(Self::CommitDrop), + "maintain" => Ok(Self::Maintain), + "reclaim" => Ok(Self::Reclaim), "admin" => Ok(Self::Admin), "graph_list" => Ok(Self::GraphList), "invoke_query" => Ok(Self::InvokeQuery), @@ -942,6 +961,8 @@ namespace Omnigraph { action "branch_merge" appliesTo { principal: Actor, resource: Graph, context: RequestContext }; action "commit_publish" appliesTo { principal: Actor, resource: Graph, context: RequestContext }; action "commit_drop" appliesTo { principal: Actor, resource: Graph, context: RequestContext }; + action "maintain" appliesTo { principal: Actor, resource: Graph, context: RequestContext }; + action "reclaim" appliesTo { principal: Actor, resource: Graph, context: RequestContext }; action "admin" appliesTo { principal: Actor, resource: Graph, context: RequestContext }; action "invoke_query" appliesTo { principal: Actor, resource: Graph, context: RequestContext }; @@ -1095,6 +1116,35 @@ pub trait PolicyChecker: Send + Sync { scope: &ResourceScope, actor: &str, ) -> Result<(), PolicyError>; + + /// Gate for an operation that any one of `actions` admits on `scope`, + /// such as the full-text rebuild, which `maintain` and its older grant + /// `change` both admit. Each action is decided as [`Self::check`] decides + /// it, in order, and the first allowed one admits the operation. An + /// evaluation failure is returned at once rather than passed over for + /// the next action, so a broken rule never widens access. When every + /// action is denied, the denial names each one. + fn check_any( + &self, + actions: &[PolicyAction], + scope: &ResourceScope, + actor: &str, + ) -> Result<(), PolicyError> { + if actions.is_empty() { + return Err(PolicyError::Internal( + "no policy action to check".to_string(), + )); + } + let mut denials = Vec::with_capacity(actions.len()); + for &action in actions { + match self.check(action, scope, actor) { + Ok(()) => return Ok(()), + Err(PolicyError::Denied(message)) => denials.push(message), + Err(internal) => return Err(internal), + } + } + Err(PolicyError::Denied(denials.join("; "))) + } } impl PolicyChecker for PolicyEngine { @@ -1951,4 +2001,223 @@ rules: assert!(PolicyEngine::load_cluster_from_source(&mixed).is_err()); } } + + #[test] + fn every_action_round_trips_through_its_name() { + use clap::ValueEnum; + for &action in PolicyAction::value_variants() { + let name = action.as_str(); + assert_eq!(action.to_string(), name); + assert_eq!(name.parse::().unwrap(), action, "{name}"); + assert_eq!( + serde_json::to_value(action).unwrap(), + serde_json::Value::String(name.to_string()) + ); + assert_eq!( + serde_json::from_value::(serde_json::json!(name)).unwrap(), + action + ); + } + assert_eq!( + "maintain".parse::().unwrap(), + PolicyAction::Maintain + ); + assert_eq!( + "reclaim".parse::().unwrap(), + PolicyAction::Reclaim + ); + assert!("maintenance".parse::().is_err()); + } + + /// A rule granting any one action compiles under the Cedar schema and + /// allows that action, so no action can be added to the enum without its + /// declaration. + #[test] + fn every_action_is_declared_in_the_cedar_schema() { + use clap::ValueEnum; + for &action in PolicyAction::value_variants() { + let source = format!( + "version: 1\ngroups:\n g: [act]\nrules:\n - id: r\n allow:\n actors: {{ group: g }}\n actions: [{}]\n", + action.as_str() + ); + let engine = match action.resource_kind() { + PolicyResourceKind::Graph => PolicyEngine::load_graph_from_source(&source, "graph"), + PolicyResourceKind::Server => PolicyEngine::load_server_from_source(&source), + PolicyResourceKind::Cluster => PolicyEngine::load_cluster_from_source(&source), + } + .unwrap_or_else(|error| panic!("{action}: {error}")); + let decision = engine + .authorize( + "act", + &PolicyRequest { + action, + branch: None, + target_branch: None, + }, + ) + .unwrap(); + assert!(decision.allowed, "{action}: {}", decision.message); + } + } + + #[test] + fn maintain_is_branch_scoped_and_reclaim_takes_no_branch_scope() { + let policy: PolicyConfig = yaml_serde::from_str( + r#" +version: 1 +groups: + automation: [act-bot] + writers: [act-writer] +protected_branches: [main] +rules: + - id: automation-maintains-unprotected + allow: + actors: { group: automation } + actions: [maintain] + branch_scope: unprotected + - id: writers-change + allow: + actors: { group: writers } + actions: [change] + branch_scope: any +"#, + ) + .unwrap(); + let engine = PolicyCompiler::compile(&policy, "graph").unwrap(); + let decide = |actor: &str, action, branch: Option<&str>| { + engine + .authorize( + actor, + &PolicyRequest { + action, + branch: branch.map(str::to_string), + target_branch: None, + }, + ) + .unwrap() + .allowed + }; + assert!(decide("act-bot", PolicyAction::Maintain, Some("feature"))); + assert!(!decide("act-bot", PolicyAction::Maintain, Some("main"))); + // Maintenance grants neither data writes nor reclamation. + assert!(!decide("act-bot", PolicyAction::Change, Some("feature"))); + assert!(!decide("act-bot", PolicyAction::Reclaim, None)); + // And `change` grants no maintenance. + assert!(!decide( + "act-writer", + PolicyAction::Maintain, + Some("feature") + )); + assert!(!decide("act-writer", PolicyAction::Reclaim, None)); + + let reclaim = r#" +version: 1 +groups: + automation: [act-bot] +rules: + - id: automation-reclaims + allow: + actors: { group: automation } + actions: [reclaim] +"#; + let engine = PolicyEngine::load_graph_from_source(reclaim, "graph").unwrap(); + let allowed = engine + .authorize( + "act-bot", + &PolicyRequest { + action: PolicyAction::Reclaim, + branch: None, + target_branch: None, + }, + ) + .unwrap(); + assert!(allowed.allowed); + assert_eq!( + PolicyAction::Reclaim.resource_kind(), + PolicyResourceKind::Graph + ); + for scope in ["branch_scope", "target_branch_scope"] { + let scoped = reclaim.replace( + "actions: [reclaim]", + &format!("actions: [reclaim]\n {scope}: any"), + ); + let error = PolicyEngine::load_graph_from_source(&scoped, "graph").unwrap_err(); + assert!( + error.to_string().contains("unsupported action 'reclaim'"), + "{scope}: {error}" + ); + } + let target_scoped_maintain = reclaim.replace("[reclaim]", "[maintain]").replace( + "actions: [maintain]", + "actions: [maintain]\n target_branch_scope: any", + ); + assert!(PolicyEngine::load_graph_from_source(&target_scoped_maintain, "graph").is_err()); + } + + #[test] + fn check_any_admits_on_any_allowed_action_and_names_every_denial() { + use super::{PolicyChecker, PolicyError, ResourceScope}; + + let policy = PolicyEngine::load_graph_from_source( + r#" +version: 1 +groups: + maintainers: [act-maintainer] + writers: [act-writer] + readers: [act-reader] +rules: + - id: maintainers + allow: + actors: { group: maintainers } + actions: [maintain] + - id: writers + allow: + actors: { group: writers } + actions: [change] +"#, + "graph", + ) + .unwrap(); + let either = [PolicyAction::Maintain, PolicyAction::Change]; + let scope = ResourceScope::Branch("main".to_string()); + for actor in ["act-maintainer", "act-writer"] { + policy.check_any(&either, &scope, actor).unwrap(); + } + match policy.check_any(&either, &scope, "act-reader") { + Err(PolicyError::Denied(message)) => { + assert!( + message.contains("'maintain'") && message.contains("'change'"), + "{message}" + ); + } + other => panic!("expected a denial naming both actions, got {other:?}"), + } + assert!(matches!( + policy.check_any(&[], &scope, "act-maintainer"), + Err(PolicyError::Internal(_)) + )); + + /// Fails to evaluate `maintain` and allows everything else. + struct BrokenMaintain; + impl PolicyChecker for BrokenMaintain { + fn check( + &self, + action: PolicyAction, + _scope: &ResourceScope, + _actor: &str, + ) -> Result<(), PolicyError> { + match action { + PolicyAction::Maintain => Err(PolicyError::Internal("broken".to_string())), + _ => Ok(()), + } + } + } + assert!( + matches!( + BrokenMaintain.check_any(&either, &scope, "act"), + Err(PolicyError::Internal(message)) if message == "broken" + ), + "an evaluation failure must not fall through to the next action" + ); + } } diff --git a/crates/omnigraph/src/db/graph_coordinator.rs b/crates/omnigraph/src/db/graph_coordinator.rs index f2d9d1cfe..ae540613d 100644 --- a/crates/omnigraph/src/db/graph_coordinator.rs +++ b/crates/omnigraph/src/db/graph_coordinator.rs @@ -896,7 +896,10 @@ impl GraphCoordinator { /// each immutable table identity matches what the caller observed before /// writing; the diagnostic alias is checked as part of the expectation. /// Mismatches surface as `OmniError::Manifest` with - /// `ManifestConflictDetails::PublishedDatasetVersionMismatch`. + /// `ManifestConflictDetails::PublishedDatasetVersionMismatch`. It publishes + /// over whatever head the branch has (`PublishPrecondition::Any`), so no + /// production writer uses it; tests and failpoint fixtures do. + #[cfg(any(test, feature = "failpoints"))] pub(crate) async fn commit_updates_with_actor_with_expected( &mut self, updates: &[DatasetUpdate], @@ -915,6 +918,7 @@ impl GraphCoordinator { /// no separate write, no `commit_graph.refresh()` to pick a parent (the /// publisher resolves it under the CAS). The in-memory commit cache is then /// updated from the intent + the resolved parent without a re-read. + #[cfg(any(test, feature = "failpoints"))] async fn commit_changes_with_actor_with_expected( &mut self, changes: &[ManifestChange], @@ -1104,6 +1108,7 @@ pub(crate) fn commit_not_found(snapshot_id: &SnapshotId) -> OmniError { } /// Wrap each `DatasetUpdate` as a `ManifestChange::Update` for the publisher. +#[cfg(any(test, feature = "failpoints"))] fn updates_to_changes(updates: &[DatasetUpdate]) -> Vec { updates .iter() diff --git a/crates/omnigraph/src/db/mod.rs b/crates/omnigraph/src/db/mod.rs index 85f3d3415..2e19d14af 100644 --- a/crates/omnigraph/src/db/mod.rs +++ b/crates/omnigraph/src/db/mod.rs @@ -22,7 +22,7 @@ pub use omnigraph::{ DatasetRepairStats, EXPORT_CHUNK_MAX_BYTES, ExportCut, FullTextIndexRebuildResult, GraphCreateReconciliation, InitOptions, MergeOutcome, MergeResult, Omnigraph, OpenMode, PendingIndex, PreparedGraphCreate, PreparedSchemaApply, PreparedSchemaSettlement, ReadCheck, - RebuiltFullTextIndex, RepairAction, RepairClassification, RepairOptions, RepairStats, + RebuiltFullTextIndex, RepairAction, RepairClassification, RepairStats, RetainedManifestVersions, SYSTEM_COLUMNS_PREFLIGHT, SchemaApplyReconciliation, SchemaApplyResult, SchemaApplySettlement, SchemaContractDigest, SchemaNonPublicationProof, SkipReason, StagingVerdict, SystemColumnUpgradeFinding, SystemColumnUpgradeOptions, diff --git a/crates/omnigraph/src/db/omnigraph.rs b/crates/omnigraph/src/db/omnigraph.rs index 2a5b62ff7..db0e0e3c0 100644 --- a/crates/omnigraph/src/db/omnigraph.rs +++ b/crates/omnigraph/src/db/omnigraph.rs @@ -53,9 +53,7 @@ pub(crate) use export::{ pub use optimize::{CleanupPolicyOptions, DatasetCleanupStats, DatasetOptimizeStats, SkipReason}; use prepared_create::initial_schema_ir; pub use prepared_create::{GraphCreateReconciliation, PreparedGraphCreate}; -pub use repair::{ - DatasetRepairStats, RepairAction, RepairClassification, RepairOptions, RepairStats, -}; +pub use repair::{DatasetRepairStats, RepairAction, RepairClassification, RepairStats}; pub use schema_apply::{ PreparedSchemaApply, PreparedSchemaSettlement, SchemaApplyReconciliation, SchemaApplySettlement, SchemaContractDigest, SchemaNonPublicationProof, @@ -1130,6 +1128,19 @@ impl Omnigraph { action: omnigraph_policy::PolicyAction, scope: &omnigraph_policy::ResourceScope, actor: Option<&str>, + ) -> Result<()> { + self.enforce_any(&[action], scope, actor) + } + + /// [`Self::enforce`] for an operation that any one of `actions` admits on + /// `scope`, decided by [`omnigraph_policy::PolicyChecker::check_any`]: the + /// first allowed action admits it, an evaluation failure refuses it, and a + /// refusal names every denied action. + pub(crate) fn enforce_any( + &self, + actions: &[omnigraph_policy::PolicyAction], + scope: &omnigraph_policy::ResourceScope, + actor: Option<&str>, ) -> Result<()> { let (branch, target_branch) = scope.to_branch_pair(); for name in [branch, target_branch].into_iter().flatten() { @@ -1147,7 +1158,7 @@ impl Omnigraph { )); }; checker - .check(action, scope, actor) + .check_any(actions, scope, actor) .map_err(|err| OmniError::Policy(err.to_string())) } @@ -3528,14 +3539,23 @@ impl Omnigraph { /// under the final gates. /// Returns the declared indexes that could not be materialized on this /// pass (today: vector properties with no trainable vectors yet). They are - /// deferred, not errors; a later `ensure_indices`/`optimize` builds them - /// once the property is trainable. Reads stay correct (brute-force) meanwhile. - pub async fn ensure_indices(&self) -> Result> { - table_ops::ensure_indices(self).await + /// deferred, not errors; a later `ensure_indices_as`/`optimize_as` builds + /// them once the property is trainable. Reads stay correct (brute-force) + /// meanwhile. + /// + /// Works on the handle's current branch. Requires `maintain` on that + /// branch before any effect and records `actor` in the publication. + pub async fn ensure_indices_as(&self, actor: Option<&str>) -> Result> { + table_ops::ensure_indices_as(self, actor).await } - pub async fn ensure_indices_on(&self, branch: &str) -> Result> { - table_ops::ensure_indices_on(self, branch).await + /// [`Self::ensure_indices_as`] on `branch`, under `maintain` on `branch`. + pub async fn ensure_indices_on_as( + &self, + branch: &str, + actor: Option<&str>, + ) -> Result> { + table_ops::ensure_indices_on_as(self, branch, actor).await } /// Fully rebuild all declared or existing supported full-text indexes from @@ -3545,19 +3565,13 @@ impl Omnigraph { /// Rebuilt full-text indexes use the engine's default analyzer, including /// indexes originally created externally with custom tokenizer settings. /// - /// Unlike [`Self::ensure_indices_on`], this replaces existing FTS indexes - /// even when their row coverage is complete. It establishes the supported - /// analyzer generation after an upgrade; it does not rewrite old history. - /// With a policy installed, use the actor-aware counterpart. - pub async fn rebuild_full_text_indices_on( - &self, - branch: &str, - ) -> Result { - self.rebuild_full_text_indices_on_as(branch, None).await - } - - /// Actor-aware full-text rebuilding. Requires `Change` on the selected - /// branch before any effects and records the actor in the graph publication. + /// Unlike [`Self::ensure_indices_on_as`], this replaces existing FTS + /// indexes even when their row coverage is complete. It establishes the + /// supported analyzer generation after an upgrade; it does not rewrite old + /// history. + /// + /// Requires `maintain` or `change` on the selected branch before any + /// effect and records the actor in the graph publication. pub async fn rebuild_full_text_indices_on_as( &self, branch: &str, @@ -3584,26 +3598,34 @@ impl Omnigraph { } /// Compact small Lance fragments into fewer larger ones across every - /// backing dataset for a node or edge type on `main`. See [`optimize`] for details. - pub async fn optimize(&self) -> Result> { - optimize::optimize_all_datasets(self).await + /// backing dataset for a node or edge type on `main`. See [`optimize`] for + /// details. Requires `maintain` on `main` before any effect and records + /// `actor` in the publication. + pub async fn optimize_as( + &self, + actor: Option<&str>, + ) -> Result> { + optimize::optimize_all_datasets(self, actor).await } - /// Classify and explicitly repair uncovered graph-manifest/Lance-HEAD drift. See - /// [`repair`] for the distinction between safe maintenance drift and - /// suspicious/unverifiable drift. - pub async fn repair(&self, options: repair::RepairOptions) -> Result { - repair::repair_all_datasets(self, options).await + /// Report, per table on `main`, how its Lance linear history stands + /// against its registration. See [`repair`] for the classifications. The + /// report publishes nothing and adopts no drift. Requires `maintain` on + /// `main`. + pub async fn repair_report_as(&self, actor: Option<&str>) -> Result { + repair::repair_report(self, actor).await } /// Remove Lance manifests (and the fragments they uniquely own) per the /// given [`optimize::CleanupPolicyOptions`]. Destructive to version - /// history. See [`optimize`] for details. - pub async fn cleanup( + /// history. See [`optimize`] for details. Requires `reclaim` before any + /// effect; a run covers every branch, so the check names none. + pub async fn cleanup_as( &self, options: optimize::CleanupPolicyOptions, + actor: Option<&str>, ) -> Result> { - optimize::cleanup_all_datasets(self, options).await + optimize::cleanup_all_datasets(self, options, actor).await } /// The tracing collector's plan for `options`, deleting nothing and taking @@ -6185,7 +6207,7 @@ edge WorksAt: Person -> Company ); // The reconciler materializes it (Person has a row). - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let snapshot = db.snapshot().await; let ds = db .storage() @@ -6217,7 +6239,7 @@ edge WorksAt: Person -> Company db.apply_schema(&desired).await.unwrap(); // After the rewrite the reconciler restores index coverage. - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let snapshot = db.snapshot().await; let ds = db .storage() diff --git a/crates/omnigraph/src/db/omnigraph/collector.rs b/crates/omnigraph/src/db/omnigraph/collector.rs index f803fd81d..24c168196 100644 --- a/crates/omnigraph/src/db/omnigraph/collector.rs +++ b/crates/omnigraph/src/db/omnigraph/collector.rs @@ -2262,7 +2262,7 @@ mod borrowed_origin_tests { load_company(&db, "initial-company").await; db.branch_create("legacy").await.unwrap(); load_person(&db, "borrowed", 41).await; - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let source_snapshot = db.snapshot_of(ReadTarget::branch("main")).await.unwrap(); let entry = source_snapshot.dataset("node:Person").unwrap().clone(); let mut source = source_snapshot @@ -2376,7 +2376,7 @@ mod borrowed_origin_tests { assert!(!company_old.is_empty()); load_company(&db, "unrelated-current").await; load_person(&db, "main-current", 99).await; - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let options = super::super::optimize::CleanupPolicyOptions { keep_versions: Some(1), older_than: None, @@ -2391,7 +2391,7 @@ mod borrowed_origin_tests { source_plan.would_remove().contains(&source_version), "borrowed source manifest is reclaimable" ); - let rows = db.cleanup(options).await.unwrap(); + let rows = db.cleanup_as(options, None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); let source_store = source.object_store(None).await.unwrap(); for path in source_data.iter().chain(&source_indices) { @@ -2504,7 +2504,7 @@ mod borrowed_origin_tests { .any(|table| table.would_remove().contains(&version)), "old source pin must be reclaimable without the native tag's borrowed files" ); - let rows = db.cleanup(options).await.unwrap(); + let rows = db.cleanup_as(options, None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); drop(db); // forbidden-api-allow: test-only cold native read of a tagged fork absent from graph registrations @@ -2745,7 +2745,7 @@ mod overlay_retention_tests { .any(|object| object.path == overlay_path) ); assert!(expected.missing_paths().await.unwrap().is_empty()); - let rows = db.cleanup(options).await.unwrap(); + let rows = db.cleanup_as(options, None).await.unwrap(); let row = rows .iter() .find(|row| row.type_key == entry.type_key) @@ -2793,7 +2793,7 @@ mod overlay_retention_tests { .any(|object| object.path == overlay_path), "the independent pre-cleanup probe must include overlays" ); - let rows = db.cleanup(options.clone()).await.unwrap(); + let rows = db.cleanup_as(options.clone(), None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); let store = overlay.object_store(None).await.unwrap(); assert!(store.exists(&overlay_path).await.unwrap()); @@ -2842,7 +2842,7 @@ mod overlay_retention_tests { } else { load_person(&db, 99).await; } - let rows = db.cleanup(options).await.unwrap(); + let rows = db.cleanup_as(options, None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); assert!( !store.exists(&overlay_path).await.unwrap(), @@ -3020,7 +3020,7 @@ mod native_table_tag_tests { !plan.would_remove().contains(&version), "an exact native table tag must exclude its version from cleanup: {plan:?}", ); - let rows = db.cleanup(options.clone()).await.unwrap(); + let rows = db.cleanup_as(options.clone(), None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); assert!(store.exists(&manifest_path).await.unwrap()); // forbidden-api-allow: test-only cold native tag read after the graph collector ran @@ -3056,7 +3056,7 @@ mod native_table_tag_tests { .any(|plan| plan.location == location && plan.would_remove().contains(&version)), "removing the tag must make the old version reclaimable" ); - let rows = db.cleanup(options).await.unwrap(); + let rows = db.cleanup_as(options, None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); assert!( !store.exists(&manifest_path).await.unwrap(), diff --git a/crates/omnigraph/src/db/omnigraph/optimize.rs b/crates/omnigraph/src/db/omnigraph/optimize.rs index dd5517308..e1ff20c91 100644 --- a/crates/omnigraph/src/db/omnigraph/optimize.rs +++ b/crates/omnigraph/src/db/omnigraph/optimize.rs @@ -216,7 +216,15 @@ decide_seam! { /// with an exact pin CAS, so one public Optimize produces at most one graph /// commit. The publisher owns main's `__manifest` fragment layout; maintenance /// only removes obsolete automatic-cleanup configuration from that dataset. -pub async fn optimize_all_datasets(db: &Omnigraph) -> Result> { +pub async fn optimize_all_datasets( + db: &Omnigraph, + actor: Option<&str>, +) -> Result> { + db.enforce( + omnigraph_policy::PolicyAction::Maintain, + &omnigraph_policy::ResourceScope::Branch("main".to_string()), + actor, + )?; let _export_exclusion = db.reserve_export_destructive_control()?; db.ensure_schema_state_valid().await?; @@ -324,14 +332,14 @@ pub async fn optimize_all_datasets(db: &Omnigraph) -> Result, ) -> Result> { + db.enforce( + omnigraph_policy::PolicyAction::Reclaim, + &omnigraph_policy::ResourceScope::Graph, + actor, + )?; if options.keep_versions.is_none() && options.older_than.is_none() { return Err(OmniError::manifest( "cleanup requires at least one of keep_versions or older_than", @@ -1677,10 +1691,13 @@ mod tests { let bytes = store.inner.get(&path).await.unwrap().bytes().await.unwrap(); store.delete(&path).await.unwrap(); let rows = db - .cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + .cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); assert!( @@ -1694,10 +1711,13 @@ mod tests { ); store.inner.put(&path, bytes.into()).await.unwrap(); let rows = db - .cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + .cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); diff --git a/crates/omnigraph/src/db/omnigraph/repair.rs b/crates/omnigraph/src/db/omnigraph/repair.rs index ff0ac8ee8..d2d06fca2 100644 --- a/crates/omnigraph/src/db/omnigraph/repair.rs +++ b/crates/omnigraph/src/db/omnigraph/repair.rs @@ -1,27 +1,17 @@ -//! Explicit repair for uncovered published-dataset/Lance-HEAD drift. +//! The repair report: uncovered published-dataset/Lance-HEAD drift. //! //! Pending pins converge automatically: the next writer or `cleanup` promotes //! them. This module is for the different case: a dataset's Lance HEAD is //! ahead of the version recorded in `__manifest` and no published pin explains -//! the movement. `repair` classifies that uncovered drift from Lance -//! transactions and only auto-publishes maintenance-only drift when the -//! operator confirms. +//! the movement. The report classifies that drift from Lance transactions. It +//! publishes nothing: no read or write resolves a table's linear HEAD, so +//! drift never needs adopting. use lance::Dataset; use lance::dataset::transaction::Operation; use super::*; -/// Options for [`Omnigraph::repair`]. -#[derive(Debug, Clone, Copy, Default)] -pub struct RepairOptions { - /// Preview by default. With `confirm`, verified maintenance drift is - /// published to `__manifest`. - pub confirm: bool, - /// Also publish suspicious/unverifiable drift. Requires `confirm`. - pub force: bool, -} - /// Classification of a dataset's graph-manifest/Lance-HEAD state. #[derive(Debug, Clone, Copy, PartialEq, Eq)] #[non_exhaustive] @@ -65,20 +55,15 @@ impl std::fmt::Display for RepairClassification { } } -/// What repair did for a backing dataset. +/// What the repair report says about a backing dataset. #[derive(Debug, Clone, Copy, PartialEq, Eq)] #[non_exhaustive] pub enum RepairAction { - /// Nothing to do. + /// Nothing to report: no drift, or foreign drift no read or write resolves. NoOp, - /// Drift was reported but not published because this was a preview. + /// Drift the report classified and left in place. Preview, - /// Verified maintenance drift was published to `__manifest`. - Healed, - /// Suspicious/unverifiable drift was published because `force` was set. - Forced, - /// Drift was left untouched because it was not safe to publish without - /// `force`. + /// The table could not be judged; its `error` says why. Refused, } @@ -88,8 +73,6 @@ impl RepairAction { match self { Self::NoOp => "no_op", Self::Preview => "preview", - Self::Healed => "healed", - Self::Forced => "forced", Self::Refused => "refused", } } @@ -101,13 +84,13 @@ impl std::fmt::Display for RepairAction { } } -/// Per-dataset repair outcome. +/// One dataset's row of the repair report. #[derive(Debug, Clone)] #[non_exhaustive] pub struct DatasetRepairStats { /// Qualified graph type key, or `__manifest` for the system dataset. pub type_key: String, - /// Published dataset version before repair. + /// The published dataset version the registration pins. pub published_dataset_version: u64, pub lance_head_version: u64, pub classification: RepairClassification, @@ -116,13 +99,11 @@ pub struct DatasetRepairStats { pub error: Option, } -/// Whole-graph repair outcome. +/// The repair report for the whole graph. #[derive(Debug, Clone)] #[non_exhaustive] pub struct RepairStats { pub datasets: Vec, - /// New graph-manifest version if repair published any dataset pins. - pub graph_manifest_version: Option, } struct ClassificationResult { @@ -132,31 +113,27 @@ struct ClassificationResult { } struct RepairTableTask { - identity: crate::db::manifest::TableIdentity, table_key: String, full_path: String, pinned_data_version: u64, - pinned_native_ref: Option, entry: crate::db::manifest::DatasetEntry, } -pub async fn repair_all_datasets(db: &Omnigraph, options: RepairOptions) -> Result { - if options.force && !options.confirm { - return Err(OmniError::manifest("repair --force requires --confirm")); - } - +/// The repair report on `main`, under `maintain` on `main`. +pub(super) async fn repair_report(db: &Omnigraph, actor: Option<&str>) -> Result { + db.enforce( + omnigraph_policy::PolicyAction::Maintain, + &omnigraph_policy::ResourceScope::Branch("main".to_string()), + actor, + )?; db.ensure_schema_state_valid().await?; - // Repair may adopt physical HEADs into graph authority. Bind the entire - // attempt to one accepted view and revalidate after schema -> main -> table - // gates so concurrent graph movement refuses before publication. + // Bind the whole report to one accepted view, revalidated after the + // schema -> main -> table gates. The accepted catalog, the identity/path + // pairs and every raw Lance read stay under those gates, so a concurrent + // schema apply cannot pair an old dataset path with a replacement's alias, + // and a cleanup cannot remove a version the report is reading. let authority_txn = db.open_write_txn(None).await?; - - // Repair publishes manifest authority, so it joins the canonical writer - // envelope. The accepted catalog, identity/path pairs, raw Lance reads, and - // final publish all remain under schema -> main -> sorted-table gates. This - // prevents a concurrent drop/re-add from pairing the old dataset path with - // the replacement's same public alias and new identity. let _schema_permit = db.write_queue().acquire_schema_shared().await; db.refresh_coordinator_only().await?; let catalog = db.load_accepted_catalog_with_schema_gate_held().await?; @@ -175,41 +152,26 @@ pub async fn repair_all_datasets(db: &Omnigraph, options: RepairOptions) -> Resu .filter_map(|table_key| { let entry = snapshot.dataset(&table_key)?; Some(RepairTableTask { - identity: entry.identity, table_key, full_path: format!("{}/{}", db.root_uri, entry.dataset_path), pinned_data_version: entry.published_dataset_version, - pinned_native_ref: entry.native_dataset_branch.clone(), entry: entry.clone(), }) }) .collect::>(); - if table_tasks.is_empty() { - return Ok(RepairStats { - datasets: Vec::new(), - graph_manifest_version: None, - }); - } - let mut tables = Vec::with_capacity(table_tasks.len()); - let mut updates = Vec::new(); - let mut expected = crate::db::manifest::ExpectedTableVersions::new(); - let mut any_forced = false; - for task in table_tasks { let RepairTableTask { - identity, table_key, full_path, pinned_data_version, - pinned_native_ref, entry, } = task; // `classify_drift` inspects raw Lance transaction history // (`read_transaction_by_version`), a Lance-only maintenance read the // staged-write trait does not surface. The raw borrow is an enumerated, - // read-only escape; repair never takes ownership or moves Lance HEAD. + // read-only escape; the report never takes ownership or moves Lance HEAD. let handle = db.storage().open_dataset_head(&full_path, None).await?; if let Some(row) = judge_against_last_linear_version(&table_key, &entry, &handle)? { tables.push(row); @@ -262,90 +224,18 @@ pub async fn repair_all_datasets(db: &Omnigraph, options: RepairOptions) -> Resu } let classification = classify_drift(ds, pinned_data_version, lance_head_version).await; - let action = match ( - options.confirm, - options.force, - classification.classification, - ) { - (false, _, _) => RepairAction::Preview, - (true, _, RepairClassification::VerifiedMaintenance) => RepairAction::Healed, - (true, true, RepairClassification::Suspicious | RepairClassification::Unverifiable) => { - any_forced = true; - RepairAction::Forced - } - (true, _, RepairClassification::Suspicious | RepairClassification::Unverifiable) => { - RepairAction::Refused - } - (true, _, RepairClassification::NoDrift) => RepairAction::NoOp, - // Reported above, before drift is classified; never adopted. - (true, _, RepairClassification::BlockedPromotion) => RepairAction::Refused, - (true, _, RepairClassification::ForeignDrift) => RepairAction::NoOp, - }; - - if matches!(action, RepairAction::Healed | RepairAction::Forced) { - let state = db.storage().table_state(&full_path, &handle).await?; - updates.push(crate::db::DatasetUpdate { - identity, - type_key: table_key.clone(), - published_dataset_version: state.version, - native_dataset_branch: None, - entity_count: state.row_count, - version_metadata: state - .version_metadata - .with_last_linear_version(entry.version_metadata.last_linear_version()), - }); - expected.insert( - identity, - crate::db::manifest::TableVersionExpectation { - table_key: table_key.clone(), - table_version: pinned_data_version, - native_ref: crate::db::manifest::NativeRefPin::Exact(pinned_native_ref), - }, - ); - } - tables.push(DatasetRepairStats { type_key: table_key, published_dataset_version: pinned_data_version, lance_head_version, classification: classification.classification, - action, + action: RepairAction::Preview, operations: classification.operations, error: classification.error, }); } - let manifest_version = if updates.is_empty() { - None - } else { - let actor = if any_forced { - Some("omnigraph:repair:force") - } else { - Some("omnigraph:repair") - }; - let PublishedSnapshot { - graph_manifest_version: manifest_version, - .. - } = db - .coordinator - .write() - .await - .commit_updates_with_actor_with_expected(&updates, &expected, actor) - .await?; - db.runtime_cache.invalidate_all().await; - if updates - .iter() - .any(|update| update.type_key.starts_with("edge:")) - { - db.invalidate_graph_index().await; - } - Some(manifest_version) - }; - - Ok(RepairStats { - datasets: tables, - graph_manifest_version: manifest_version, - }) + Ok(RepairStats { datasets: tables }) } /// The detached-only rule: the recorded last linear version is the baseline diff --git a/crates/omnigraph/src/db/omnigraph/table_ops.rs b/crates/omnigraph/src/db/omnigraph/table_ops.rs index 68875383a..4e0619bba 100644 --- a/crates/omnigraph/src/db/omnigraph/table_ops.rs +++ b/crates/omnigraph/src/db/omnigraph/table_ops.rs @@ -35,19 +35,26 @@ pub(super) async fn graph_index_for_resolved( .await } -pub(super) async fn ensure_indices(db: &Omnigraph) -> Result> { +pub(super) async fn ensure_indices_as( + db: &Omnigraph, + actor: Option<&str>, +) -> Result> { let current_branch = db .coordinator .read() .await .current_branch() .map(str::to_string); - ensure_indices_for_branch(db, current_branch.as_deref()).await + ensure_indices_for_branch(db, current_branch.as_deref(), actor).await } -pub(super) async fn ensure_indices_on(db: &Omnigraph, branch: &str) -> Result> { +pub(super) async fn ensure_indices_on_as( + db: &Omnigraph, + branch: &str, + actor: Option<&str>, +) -> Result> { let branch = normalize_branch_name(branch)?; - ensure_indices_for_branch(db, branch.as_deref()).await + ensure_indices_for_branch(db, branch.as_deref(), actor).await } /// The indexes replaced by one successfully published full-text rebuild. @@ -86,8 +93,13 @@ pub(super) async fn rebuild_full_text_indices_on_as( ) -> Result { let branch = normalize_branch_name(branch)?; let public_branch = branch.as_deref().unwrap_or("main"); - db.enforce( - omnigraph_policy::PolicyAction::Change, + // `maintain` is the rebuild's grant; `change`, its grant before + // `maintain` existed, still admits it so existing policies keep working. + db.enforce_any( + &[ + omnigraph_policy::PolicyAction::Maintain, + omnigraph_policy::PolicyAction::Change, + ], &omnigraph_policy::ResourceScope::Branch(public_branch.to_string()), actor, )?; @@ -186,12 +198,19 @@ pub(super) async fn failpoint_publish_table_head_without_index_rebuild_for_test( .await } -pub(super) async fn ensure_indices_for_branch( +/// The index writer on `branch` (`None` is main), under `maintain` on it. +async fn ensure_indices_for_branch( db: &Omnigraph, branch: Option<&str>, + actor: Option<&str>, ) -> Result> { + db.enforce( + omnigraph_policy::PolicyAction::Maintain, + &omnigraph_policy::ResourceScope::Branch(branch.unwrap_or("main").to_string()), + actor, + )?; Ok( - maintain_indices_for_branch(db, branch, IndexMaintenanceMode::Ensure, None) + maintain_indices_for_branch(db, branch, IndexMaintenanceMode::Ensure, actor) .await? .pending, ) diff --git a/crates/omnigraph/src/db/upgrade/tests.rs b/crates/omnigraph/src/db/upgrade/tests.rs index 39ab4cb9d..30fa83bb8 100644 --- a/crates/omnigraph/src/db/upgrade/tests.rs +++ b/crates/omnigraph/src/db/upgrade/tests.rs @@ -2752,7 +2752,7 @@ async fn cleanup_after_upgrade_with_keep_four_and_older_than() { .collect(); assert_eq!(links.len(), 1, "{links:?}"); for (round, options) in [week, keep(4)].into_iter().enumerate() { - let stats = graph.cleanup(options).await.unwrap(); + let stats = graph.cleanup_as(options, None).await.unwrap(); let reclaimed = |table_key: &str| { if round == 0 && table_key == PERSON { links.len() as u64 @@ -2800,7 +2800,7 @@ async fn merge_with_legacy_base_pins_and_collector_keeps_it() { "the merge base of feature and main pins Person version {}: {roots:?}", source.a_root ); - let stats = graph.cleanup(keep(1)).await.unwrap(); + let stats = graph.cleanup_as(keep(1), None).await.unwrap(); assert!(stats.iter().all(|row| row.error.is_none()), "{stats:?}"); session(&graph) @@ -2897,7 +2897,7 @@ async fn leftover_merge_input_tag_on_bookkeeping_version_resolves() { roots.is_superset(&[source.b_root, source.c_root].into()), "{roots:?}" ); - let stats = graph.cleanup(keep(1)).await.unwrap(); + let stats = graph.cleanup_as(keep(1), None).await.unwrap(); assert!(stats.iter().all(|row| row.error.is_none()), "{stats:?}"); assert_eq!( open(root, None) diff --git a/crates/omnigraph/src/engine/plan_source.rs b/crates/omnigraph/src/engine/plan_source.rs index fcd88b8a7..f4bb53522 100644 --- a/crates/omnigraph/src/engine/plan_source.rs +++ b/crates/omnigraph/src/engine/plan_source.rs @@ -926,7 +926,7 @@ query people() { match { $p: Person } return { count($p) as n } } ) .await .unwrap(); - session.ensure_indices().await.unwrap(); + session.ensure_indices_as(None).await.unwrap(); let settings = SessionSettings::default(); let (view, catalog) = db .capture_read_view(ReadTarget::branch("main")) diff --git a/crates/omnigraph/src/runtime_cache.rs b/crates/omnigraph/src/runtime_cache.rs index be6267f34..3e3c00b6e 100644 --- a/crates/omnigraph/src/runtime_cache.rs +++ b/crates/omnigraph/src/runtime_cache.rs @@ -648,7 +648,7 @@ edge Likes: Person -> Person {} db.load_jsonl(DATA, crate::loader::LoadMode::Overwrite) .await .unwrap(); - db.optimize().await.unwrap(); + db.optimize_as(None).await.unwrap(); // Fresh handle: cold in-memory caches, the artifact stays on the store. let db = crate::db::Omnigraph::open(dir.path().to_str().unwrap()) .await diff --git a/crates/omnigraph/tests/bound_storage.rs b/crates/omnigraph/tests/bound_storage.rs index 4641a513d..a4f989498 100644 --- a/crates/omnigraph/tests/bound_storage.rs +++ b/crates/omnigraph/tests/bound_storage.rs @@ -303,11 +303,14 @@ async fn bound_handle_workload_child() { .contains("Feature") ); - db.optimize().await.unwrap(); - db.cleanup(CleanupPolicyOptions { - keep_versions: Some(2), - older_than: None, - }) + db.optimize_as(None).await.unwrap(); + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(2), + older_than: None, + }, + None, + ) .await .unwrap(); let reopened = session(Omnigraph::open_with_storage(&uri, storage()).await.unwrap()); diff --git a/crates/omnigraph/tests/detached_commit_matrix.rs b/crates/omnigraph/tests/detached_commit_matrix.rs index d0a5c2792..2f7e9dc31 100644 --- a/crates/omnigraph/tests/detached_commit_matrix.rs +++ b/crates/omnigraph/tests/detached_commit_matrix.rs @@ -414,15 +414,18 @@ fn rfc0067_matrix_child_process() { .block_on(async move { let db = helpers::session(Omnigraph::open(&uri).await.unwrap()); let outcome: omnigraph::error::Result<()> = match op.as_str() { - "cleanup" => db.cleanup(reclaim_everything()).await.map(|_| ()), + "cleanup" => db.cleanup_as(reclaim_everything(), None).await.map(|_| ()), "cleanup_keep" => db - .cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + .cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .map(|_| ()), - "ensure_indices" => db.ensure_indices().await.map(|_| ()), + "ensure_indices" => db.ensure_indices_as(None).await.map(|_| ()), "merge" => db .branch_merge(&format!("src_{name}"), "main") .await @@ -437,9 +440,12 @@ fn rfc0067_matrix_child_process() { .unwrap(); db.apply_prepared_schema_as(&intent, None).await.map(|_| ()) } - "optimize" => db.optimize().await.map(|_| ()), + "optimize" => db.optimize_as(None).await.map(|_| ()), "load" => load_two(&db, &name).await, - "fts_rebuild" => db.rebuild_full_text_indices_on("main").await.map(|_| ()), + "fts_rebuild" => db + .rebuild_full_text_indices_on_as("main", None) + .await + .map(|_| ()), "system_column_upgrade" => db .upgrade_system_columns(upgrade_execute()) .await @@ -696,7 +702,7 @@ async fn run_cell( ) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); } // A clear cell clears a photo a put stored first. if writer == Writer::BlobClear { @@ -736,8 +742,10 @@ async fn run_cell( let outcome: omnigraph::error::Result<()> = match writer { Writer::Insert => insert(&db, &write_name).await, Writer::MultiTable => insert_and_friend(&db, &write_name).await, - Writer::Cleanup => Box::pin(db.cleanup(reclaim_everything())).await.map(|_| ()), - Writer::EnsureIndices => db.ensure_indices().await.map(|_| ()), + Writer::Cleanup => Box::pin(db.cleanup_as(reclaim_everything(), None)) + .await + .map(|_| ()), + Writer::EnsureIndices => db.ensure_indices_as(None).await.map(|_| ()), Writer::Merge => db .branch_merge(&format!("src_{write_name}"), "main") .await @@ -747,9 +755,12 @@ async fn run_cell( .apply_prepared_schema_as(prepared.as_ref().unwrap(), None) .await .map(|_| ()), - Writer::Optimize => db.optimize().await.map(|_| ()), + Writer::Optimize => db.optimize_as(None).await.map(|_| ()), Writer::Load => load_two(&db, &write_name).await, - Writer::FtsRebuild => db.rebuild_full_text_indices_on("main").await.map(|_| ()), + Writer::FtsRebuild => db + .rebuild_full_text_indices_on_as("main", None) + .await + .map(|_| ()), Writer::SystemColumnUpgrade => db .upgrade_system_columns(upgrade_execute()) .await @@ -879,7 +890,7 @@ async fn run_cell( } Recovery::Cleanup => { let fresh = helpers::session(Omnigraph::open(&root).await.unwrap()); - Box::pin(fresh.cleanup(reclaim_everything())) + Box::pin(fresh.cleanup_as(reclaim_everything(), None)) .await .unwrap_or_else(|error| panic!("{cell}: recovery cleanup failed: {error}")); } @@ -974,7 +985,7 @@ async fn run_cell( if writer == Writer::Optimize && recovery != Recovery::ReadOnly { let fresh = helpers::session(Omnigraph::open(&root).await.unwrap()); for run in 1..=2 { - fresh.optimize().await.unwrap_or_else(|error| { + fresh.optimize_as(None).await.unwrap_or_else(|error| { panic!("{cell}: optimize run {run} after recovery failed: {error}") }); assert_pin_detached_and_head_unmoved(&cell, &fresh, &person_uri, head_before).await; @@ -984,12 +995,12 @@ async fn run_cell( if writer == Writer::EnsureIndices && recovery != Recovery::ReadOnly { let fresh = helpers::session(Omnigraph::open(&root).await.unwrap()); fresh - .ensure_indices() + .ensure_indices_as(None) .await .unwrap_or_else(|error| panic!("{cell}: index pass after recovery failed: {error}")); assert_pin_detached_and_head_unmoved(&cell, &fresh, &person_uri, head_before).await; fresh - .ensure_indices() + .ensure_indices_as(None) .await .unwrap_or_else(|error| panic!("{cell}: second index pass failed: {error}")); drop(fresh); @@ -998,7 +1009,7 @@ async fn run_cell( let fresh = helpers::session(Omnigraph::open(&root).await.unwrap()); for run in 1..=2 { let rebuilt = fresh - .rebuild_full_text_indices_on("main") + .rebuild_full_text_indices_on_as("main", None) .await .unwrap_or_else(|error| { panic!("{cell}: full-text rebuild run {run} after recovery failed: {error}") diff --git a/crates/omnigraph/tests/failpoints.rs b/crates/omnigraph/tests/failpoints.rs index d0e1a6822..77fd7d211 100644 --- a/crates/omnigraph/tests/failpoints.rs +++ b/crates/omnigraph/tests/failpoints.rs @@ -322,7 +322,7 @@ fn rfc023_external_writer_process() { } "collector_advance_merge_source" => { db.load("source", &payload, mode).await.unwrap(); - let rows = db.cleanup(keep_one()).await.unwrap(); + let rows = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); } "collector_retired" => { @@ -514,7 +514,7 @@ async fn branch_start_refuses_a_commit_an_interrupted_cleanup_emptied() { { let _fp = catalog::CLEANUP_SWEEP_PRE_MANIFEST_DELETE.fire_always(); - let _ = db.cleanup(keep_one()).await; + let _ = db.cleanup_as(keep_one(), None).await; } assert!( detached_versions(&person_uri).await.contains(&eve_person), @@ -546,7 +546,7 @@ async fn branch_start_refuses_a_commit_an_interrupted_cleanup_emptied() { assert_eq!(db.branch_list().await.unwrap(), vec!["main".to_string()]); assert_eq!(version_main(&db).await.unwrap(), main_version); - let stats = db.cleanup(keep_one()).await.unwrap(); + let stats = db.cleanup_as(keep_one(), None).await.unwrap(); let person = stats .iter() .find(|row| row.type_key == "node:Person") @@ -623,7 +623,7 @@ async fn branch_delete_cleanup_failure_converges_on_retry() { ); { let _fp = catalog::CLEANUP_TABLE_GC.fire_always(); - main.cleanup(keep_one()).await.unwrap(); + main.cleanup_as(keep_one(), None).await.unwrap(); } assert!( detached_versions(&person_uri) @@ -631,7 +631,7 @@ async fn branch_delete_cleanup_failure_converges_on_retry() { .contains(&former_staging), "failed cleanup preserves the deleted branch's staging" ); - main.cleanup(keep_one()).await.unwrap(); + main.cleanup_as(keep_one(), None).await.unwrap(); assert!( !detached_versions(&person_uri) .await @@ -723,7 +723,7 @@ async fn recreate_over_a_deleted_branch_writes_without_cleanup() { "writes do not collect the former staging" ); assert!(remaining.contains(&second_staging)); - main.cleanup(keep_one()).await.unwrap(); + main.cleanup_as(keep_one(), None).await.unwrap(); let remaining = detached_versions(&person_uri).await; assert!(!remaining.contains(&first_staging)); assert!( @@ -780,7 +780,7 @@ async fn branch_delete_acknowledges_with_staging_awaiting_cleanup() { .contains(&former_staging), "delete and reopen defer collection of the branch's staging" ); - main.cleanup(keep_one()).await.unwrap(); + main.cleanup_as(keep_one(), None).await.unwrap(); assert!( !detached_versions(&person_uri) .await @@ -850,7 +850,7 @@ async fn branch_recreate_completes_while_old_staging_awaits_cleanup() { let remaining = detached_versions(&person_uri).await; assert!(remaining.contains(&first_staging)); assert!(remaining.contains(&second_staging)); - main.cleanup(keep_one()).await.unwrap(); + main.cleanup_as(keep_one(), None).await.unwrap(); let remaining = detached_versions(&person_uri).await; assert!(!remaining.contains(&first_staging)); assert!(remaining.contains(&second_staging)); @@ -923,10 +923,13 @@ async fn branch_write_ignores_unavailable_cleanup_classifier() { helpers::count_rows_branch(&db, "feature", "node:Person").await, 5 ); - db.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); let branches = lance::Dataset::open(&person_uri) @@ -970,10 +973,13 @@ async fn cleanup_isolates_single_table_failure() { // One table's version GC fails once; the sweep must isolate it. let _fp = catalog::CLEANUP_TABLE_GC.fire_once_at(1); let stats = db - .cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + .cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .expect("a single table's GC failure must not abort cleanup"); @@ -1024,10 +1030,13 @@ async fn cleanup_isolates_reconcile_failure() { // not abort. { let _fp = catalog::CLEANUP_RECONCILE_FORK.fire_once_at(1); - db.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .expect("a reconcile force-delete failure must not abort cleanup"); } @@ -1044,10 +1053,13 @@ async fn cleanup_isolates_reconcile_failure() { ); } // A second cleanup with no injected failure converges. - db.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); { @@ -1096,10 +1108,13 @@ async fn reconcile_skips_fork_when_fresh_recheck_is_unavailable_then_converges() // branch is live but unreadable) → cleanup must SKIP it, not delete. { let _fp = catalog::CLASSIFY_FRESH_READ.fire_always(); - db.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); let ds = lance::Dataset::open(&person_uri).await.unwrap(); @@ -1114,10 +1129,13 @@ async fn reconcile_skips_fork_when_fresh_recheck_is_unavailable_then_converges() } // Read succeeds now → cleanup confirms the orphan and reclaims it (converges). - db.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); { @@ -2666,7 +2684,7 @@ async fn merge_target_delete_waits_and_source_delete_does_not_issue_643() { .unwrap() .unwrap_or_else(|error| panic!("deleting the target after the merge: {error}")); } else { - let rows = db.cleanup(keep_one()).await.unwrap(); + let rows = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); assert_eq!( helpers::count_rows_branch(&db, "t", "node:Person").await, @@ -3059,7 +3077,7 @@ async fn cancelled_merge_leaves_the_target_whole_issue_643() { ) .await .unwrap_or_else(|error| panic!("{seam}: a later write on the target: {error}")); - let rows = db.cleanup(keep_one()).await.unwrap(); + let rows = db.cleanup_as(keep_one(), None).await.unwrap(); assert!( rows.iter().all(|row| row.error.is_none()), "{seam}: {rows:?}" @@ -3705,7 +3723,7 @@ async fn ensure_indices_stage_btree_failure_leaves_existing_tables_writable() { &catalog::ENSURE_INDICES_POST_STAGE_PRE_COMMIT_BTREE, ); let writer_a_db = std::sync::Arc::clone(&db); - let writer_a = tokio::spawn(async move { writer_a_db.ensure_indices().await }); + let writer_a = tokio::spawn(async move { writer_a_db.ensure_indices_as(None).await }); rendezvous.wait_until_reached().await; db.load( @@ -4043,15 +4061,9 @@ async fn schema_apply_retry_preserves_the_original_empty_create() { assert_eq!(entry.published_dataset_version, if changed { 2 } else { 1 }); assert_eq!(entry.version_metadata.staged_version().is_some(), changed); assert_eq!(entry.version_metadata.last_linear_version(), Some(1)); - recovered - .repair(omnigraph::db::RepairOptions { - confirm: true, - force: false, - }) - .await - .unwrap(); - recovered.ensure_indices().await.unwrap(); - let cleaned = recovered.cleanup(keep_one()).await.unwrap(); + recovered.repair_report_as(None).await.unwrap(); + recovered.ensure_indices_as(None).await.unwrap(); + let cleaned = recovered.cleanup_as(keep_one(), None).await.unwrap(); assert!(cleaned.iter().all(|row| row.error.is_none()), "{cleaned:?}"); drop(recovered); let reopened = helpers::session(Omnigraph::open(&uri).await.unwrap()); @@ -4475,7 +4487,7 @@ async fn optimize_pre_publish_failure_leaves_no_residue() { .version; { let _failpoint = catalog::OPTIMIZE_POST_PHASE_B_PRE_MANIFEST_COMMIT.fire_always(); - let err = db.optimize().await.unwrap_err(); + let err = db.optimize_as(None).await.unwrap_err(); assert!( err.to_string().contains( "injected failpoint triggered: optimize.post_phase_b_pre_manifest_commit" @@ -4502,7 +4514,7 @@ async fn optimize_pre_publish_failure_leaves_no_residue() { assert_eq!(helpers::count_rows(&db, "node:Person").await, 4); assert_eq!(helpers::count_rows(&db, "node:Company").await, 4); let stats = db - .optimize() + .optimize_as(None) .await .expect("the retry re-plans from the pins"); assert!( @@ -4533,7 +4545,7 @@ async fn optimize_lost_publish_acknowledgement_leaves_published_detached_pins() let db = helpers::session(Omnigraph::open(&uri).await.unwrap()); let before = person_head_and_published(&db, "main").await; let _failpoint = catalog::GRAPH_PUBLISH_AFTER_MANIFEST_COMMIT.fire_once_at(1); - db.optimize() + db.optimize_as(None) .await .expect_err("a lost publish acknowledgement surfaces as an error"); before @@ -4549,7 +4561,7 @@ async fn optimize_lost_publish_acknowledgement_leaves_published_detached_pins() before.1 ); assert_person_pin_detached(&recovered, "main", before.0).await; - let stats = recovered.optimize().await.unwrap(); + let stats = recovered.optimize_as(None).await.unwrap(); assert!( stats .iter() @@ -4585,7 +4597,7 @@ node Embedding { ) .await .unwrap(); - let pending = db.ensure_indices().await.unwrap(); + let pending = db.ensure_indices_as(None).await.unwrap(); assert!( pending .iter() @@ -4598,7 +4610,7 @@ node Embedding { ) .await .unwrap(); - db.optimize().await.unwrap(); + db.optimize_as(None).await.unwrap(); let snapshot = db.snapshot_of(ReadTarget::branch("main")).await.unwrap(); let embedding = snapshot.open_dataset("node:Embedding").await.unwrap(); assert!(embedding.has_unindexed_fragments().await.unwrap()); @@ -4635,7 +4647,7 @@ node Embedding { .version; { let _failpoint = catalog::OPTIMIZE_POST_PHASE_B_PRE_MANIFEST_COMMIT.fire_once_at(1); - let error = db.optimize().await.unwrap_err(); + let error = db.optimize_as(None).await.unwrap_err(); assert!( !matches!(error, OmniError::RecoveryRequired { .. }), "{error}" @@ -4663,7 +4675,7 @@ node Embedding { ); drop(db); let db = helpers::session(Omnigraph::open(&uri).await.unwrap()); - let stats = db.optimize().await.unwrap(); + let stats = db.optimize_as(None).await.unwrap(); let work = stats .iter() .find(|stat| stat.type_key == "node:Work") @@ -4724,7 +4736,7 @@ async fn optimize_partial_table_effect_leaves_no_residue() { .version() .version; let _failpoint = catalog::OPTIMIZE_POST_TABLE_EFFECT.fire_once_at(1); - let error = db.optimize().await.unwrap_err(); + let error = db.optimize_as(None).await.unwrap_err(); assert!( !matches!(error, OmniError::RecoveryRequired { .. }), "a partial batch before publication needs no recovery: {error}" @@ -4768,7 +4780,7 @@ async fn optimize_partial_table_effect_leaves_no_residue() { recovered.list_commits(Some("main")).await.unwrap().len(), commits_before ); - let stats = recovered.optimize().await.unwrap(); + let stats = recovered.optimize_as(None).await.unwrap(); assert_eq!( stats .iter() @@ -4816,7 +4828,7 @@ async fn optimize_refuses_when_graph_authority_moves_before_its_gates() { &catalog::OPTIMIZE_POST_AUTHORITY_CAPTURE_PRE_GATES, ); let optimize_task_db = std::sync::Arc::clone(&optimize_db); - let optimize = tokio::spawn(async move { optimize_task_db.optimize().await }); + let optimize = tokio::spawn(async move { optimize_task_db.optimize_as(None).await }); rendezvous.wait_until_reached().await; // Advance the graph head underneath it with an ordinary committed write, so @@ -4912,7 +4924,7 @@ async fn optimize_holds_main_gate_through_disjoint_table_effects() { let uri_opt = uri.clone(); let optimize = tokio::spawn(async move { let db = helpers::session(Omnigraph::open(&uri_opt).await.unwrap()); - db.optimize().await + db.optimize_as(None).await }); rendezvous.wait_until_reached().await; @@ -4956,7 +4968,7 @@ async fn optimize_holds_main_gate_through_disjoint_table_effects() { 1, "queued table-disjoint Company insert must persist", ); - db.optimize() + db.optimize_as(None) .await .expect("graph must remain healthy / re-optimizable"); } @@ -4996,7 +5008,7 @@ async fn optimize_serializes_concurrent_delete_across_handles() { let uri_opt = uri.clone(); let optimize = tokio::spawn(async move { let db = helpers::session(Omnigraph::open(&uri_opt).await.unwrap()); - db.optimize().await + db.optimize_as(None).await }); rendezvous.wait_until_reached().await; @@ -5056,7 +5068,7 @@ async fn optimize_serializes_concurrent_delete_across_handles() { 3, "the concurrent delete must persist (alice removed)", ); - db.optimize() + db.optimize_as(None) .await .expect("graph must remain healthy / re-optimizable"); } @@ -5410,7 +5422,7 @@ async fn ensure_indices_phase_b_failure_does_not_leak_sidecar_when_no_work_neede ) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); } // Setup: trigger the failpoint. Steady-state ensure_indices @@ -5420,7 +5432,7 @@ async fn ensure_indices_phase_b_failure_does_not_leak_sidecar_when_no_work_neede { let db = helpers::session(Omnigraph::open(&uri).await.unwrap()); let _failpoint = catalog::ENSURE_INDICES_POST_PHASE_B_PRE_MANIFEST_COMMIT.fire_always(); - let err = db.ensure_indices().await.unwrap_err(); + let err = db.ensure_indices_as(None).await.unwrap_err(); assert!( err.to_string().contains( "injected failpoint triggered: ensure_indices.post_phase_b_pre_manifest_commit" @@ -7337,7 +7349,7 @@ async fn ensure_indices_failure_before_publish_leaves_no_residue() { let version = helpers::version_main(&db).await.unwrap(); { let _failpoint = catalog::ENSURE_INDICES_POST_TABLE_EFFECT.fire_always(); - let err = db.ensure_indices().await.unwrap_err(); + let err = db.ensure_indices_as(None).await.unwrap_err(); assert!( err.to_string() .contains("injected failpoint triggered: ensure_indices.post_table_effect"), @@ -7348,7 +7360,7 @@ async fn ensure_indices_failure_before_publish_leaves_no_residue() { assert_eq!(person_head_and_published(&db, "main").await, before); assert_eq!(helpers::version_main(&db).await.unwrap(), version); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); assert_person_pin_detached(&db, "main", before.0).await; assert_eq!( person_head_and_published(&db, "main").await.1, @@ -7409,7 +7421,7 @@ async fn full_text_rebuild_on_a_branch_stages_on_the_inherited_table() { db.load_jsonl(SEARCH_DATA, LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); db.branch_create("rebuild").await.unwrap(); let main_before = doc_head_and_published(&db, "main").await; let main_entry = helpers::snapshot_main(&db) @@ -7421,7 +7433,7 @@ async fn full_text_rebuild_on_a_branch_stages_on_the_inherited_table() { let main_pin = pinned_version(&db, "main", "node:Doc").await; let branch_version = helpers::version_branch(&db, "rebuild").await.unwrap(); - db.rebuild_full_text_indices_on("rebuild") + db.rebuild_full_text_indices_on_as("rebuild", None) .await .expect("a branch rebuild stages on the inherited table and never forks"); assert!(sidecar_operation_ids(dir.path()).is_empty()); @@ -7518,7 +7530,7 @@ async fn assert_optimize_detached_pin_survives_next_write(uri: &str) { { let _failpoint = catalog::GRAPH_PUBLISH_AFTER_MANIFEST_COMMIT.fire_once_at(1); let error = db - .optimize() + .optimize_as(None) .await .expect_err("compaction published before its acknowledgement was lost"); assert!( @@ -8604,10 +8616,13 @@ async fn rfc_0067_cleanup_reaps_aged_surplus_detached_manifests() { } } for _ in 0..2 { - db.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: Some(std::time::Duration::ZERO), - }) + db.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: Some(std::time::Duration::ZERO), + }, + None, + ) .await .unwrap(); } @@ -8672,10 +8687,13 @@ async fn rfc_0067_zero_age_cleanup_keeps_a_fresh_unpublished_detached_manifest() "the interrupted write must strand at least one detached manifest" ); - db.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: Some(std::time::Duration::ZERO), - }) + db.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: Some(std::time::Duration::ZERO), + }, + None, + ) .await .unwrap(); let after = detached_files(); @@ -8691,10 +8709,13 @@ async fn rfc_0067_zero_age_cleanup_keeps_a_fresh_unpublished_detached_manifest() .set_times(std::fs::FileTimes::new().set_modified(std::time::SystemTime::UNIX_EPOCH)) .unwrap(); } - db.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: Some(std::time::Duration::ZERO), - }) + db.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: Some(std::time::Duration::ZERO), + }, + None, + ) .await .unwrap(); let aged = detached_files(); @@ -8716,10 +8737,13 @@ fn rfc_0067_external_cleanup_process() { .unwrap() .block_on(async { let db = Omnigraph::open(&uri).await.unwrap(); - db.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: Some(std::time::Duration::ZERO), - }) + db.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: Some(std::time::Duration::ZERO), + }, + None, + ) .await .unwrap(); }); @@ -9330,20 +9354,20 @@ async fn live_handle_writes_after_every_write_family_seam_failure() { db.apply_schema(&render_schema(&schema_props, age_indexed)) .await .expect("ensure-row schema setup"); - db.ensure_indices() + db.ensure_indices_as(None) .await .map(|_| ()) .map_err(|error| error.to_string()) } Driver::Optimize => db - .optimize() + .optimize_as(None) .await .map(|_| ()) .map_err(|error| error.to_string()), - Driver::Cleanup => Box::pin(db.cleanup(omnigraph::db::CleanupPolicyOptions { + Driver::Cleanup => Box::pin(db.cleanup_as(omnigraph::db::CleanupPolicyOptions { keep_versions: Some(1), older_than: None, - })) + }, None)) .await .map(|_| ()) .map_err(|error| error.to_string()), @@ -9419,10 +9443,13 @@ async fn rfc_0067_merge_diff_keeps_every_chunk() { after_rows - before_rows ); db.branch_delete("feature").await.unwrap(); - db.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: Some(std::time::Duration::ZERO), - }) + db.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: Some(std::time::Duration::ZERO), + }, + None, + ) .await .unwrap(); let person_uri = node_table_uri(&db, "Person").await; @@ -9462,10 +9489,13 @@ async fn rfc_0067_cleanup_window_keeps_version_gc_on_a_written_table() { for pass in 0..2 { let stats = db - .cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: Some(std::time::Duration::from_secs(7 * 24 * 60 * 60)), - }) + .cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: Some(std::time::Duration::from_secs(7 * 24 * 60 * 60)), + }, + None, + ) .await .unwrap(); let person = stats @@ -9543,7 +9573,7 @@ async fn collector_keeps_live_staging_and_reclaims_it_once_the_head_moves() { ); assert!(staging.bytes > 0); } - let stats = db.cleanup(keep_one()).await.unwrap(); + let stats = db.cleanup_as(keep_one(), None).await.unwrap(); let row = stats .iter() .find(|row| row.type_key == "node:Person") @@ -9570,7 +9600,7 @@ async fn collector_keeps_live_staging_and_reclaims_it_once_the_head_moves() { staging.verdict ); } - let stats = db.cleanup(keep_one()).await.unwrap(); + let stats = db.cleanup_as(keep_one(), None).await.unwrap(); let row = stats .iter() .find(|row| row.type_key == "node:Person") @@ -9662,7 +9692,7 @@ async fn collector_keeps_pins_published_by_a_late_branch_in_another_process() { let expected_rows = count_rows(&db, "node:Person").await + 1; let cleaner = helpers::session(Omnigraph::open(db.uri()).await.unwrap()); let (guard, hold) = catalog::CLEANUP_COLLECTOR_POST_SNAPSHOT.hold(); - let cleanup = tokio::spawn(async move { cleaner.cleanup(keep_one()).await }); + let cleanup = tokio::spawn(async move { cleaner.cleanup_as(keep_one(), None).await }); let reached = hold.clone(); tokio::task::spawn_blocking(move || reached.wait_until_reached()) .await @@ -9705,7 +9735,7 @@ async fn collector_keeps_pins_published_by_a_late_branch_in_another_process() { path.display() ); } - let retry = reader.cleanup(keep_one()).await.unwrap(); + let retry = reader.cleanup_as(keep_one(), None).await.unwrap(); assert!(retry.iter().all(|row| row.error.is_none()), "{retry:?}"); assert_eq!(count_rows(&reader, "node:Person").await, expected_rows - 1); continue; @@ -9752,7 +9782,7 @@ async fn collector_refuses_mixed_branch_captures_before_deleting() { count_rows(&db, "node:Person").await + usize::from(action == "collector_mixed"); let cleaner = helpers::session(Omnigraph::open(db.uri()).await.unwrap()); let (guard, hold) = catalog::CLEANUP_COLLECTOR_MID_BRANCH_READ.hold(); - let cleanup = tokio::spawn(async move { cleaner.cleanup(keep_one()).await }); + let cleanup = tokio::spawn(async move { cleaner.cleanup_as(keep_one(), None).await }); let reached = hold.clone(); tokio::task::spawn_blocking(move || reached.wait_until_reached()) .await @@ -9790,7 +9820,7 @@ async fn collector_refuses_mixed_branch_captures_before_deleting() { let pin = pinned_version(&reader, "main", "node:Person").await; assert!(detached_versions(&person_uri).await.contains(&pin)); assert_eq!(count_rows(&reader, "node:Person").await, expected_rows); - let retry = reader.cleanup(keep_one()).await.unwrap(); + let retry = reader.cleanup_as(keep_one(), None).await.unwrap(); assert!(retry.iter().all(|row| row.error.is_none()), "{retry:?}"); assert!(detached_versions(&person_uri).await.contains(&pin)); } @@ -9808,7 +9838,7 @@ async fn collector_pass_interrupted_mid_sweep_counts_nothing_and_retries_whole() let stats = { let _fp = catalog::CLEANUP_SWEEP_BETWEEN_MANIFESTS.fail_once_at(1); - db.cleanup(keep_one()).await.unwrap() + db.cleanup_as(keep_one(), None).await.unwrap() }; let row = stats .iter() @@ -9824,7 +9854,7 @@ async fn collector_pass_interrupted_mid_sweep_counts_nothing_and_retries_whole() row.manifests_removed, 1, "the pass stopped after its first manifest: {row:?}" ); - let stats = db.cleanup(keep_one()).await.unwrap(); + let stats = db.cleanup_as(keep_one(), None).await.unwrap(); let row = stats .iter() .find(|row| row.type_key == "node:Person") @@ -9891,7 +9921,7 @@ async fn assert_collector_retry_treats_an_index_directory_an_interrupted_pass_re let stats = { let _fp = catalog::CLEANUP_SWEEP_PRE_FILE_DELETE.fire_always(); - db.cleanup(keep_one()).await.unwrap() + db.cleanup_as(keep_one(), None).await.unwrap() }; let row = stats .iter() @@ -9924,7 +9954,7 @@ async fn assert_collector_retry_treats_an_index_directory_an_interrupted_pass_re let stats = { let _fp = catalog::CLEANUP_SWEEP_PRE_MANIFEST_DELETE.fire_always(); - db.cleanup(keep_one()).await.unwrap() + db.cleanup_as(keep_one(), None).await.unwrap() }; let row = stats .iter() @@ -9945,7 +9975,7 @@ async fn assert_collector_retry_treats_an_index_directory_an_interrupted_pass_re ); } - let stats = db.cleanup(keep_one()).await.unwrap(); + let stats = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(stats.iter().all(|row| row.error.is_none()), "{stats:?}"); let row = stats .iter() @@ -10031,7 +10061,7 @@ async fn assert_collector_retry_after_a_pass_stopped_between_manifests_finishes_ let stats = { let _fp = catalog::CLEANUP_SWEEP_BETWEEN_MANIFESTS.fire_always(); - db.cleanup(keep_one()).await.unwrap() + db.cleanup_as(keep_one(), None).await.unwrap() }; let row = stats .iter() @@ -10079,7 +10109,7 @@ async fn assert_collector_retry_after_a_pass_stopped_between_manifests_finishes_ plan.sweep_paths ); - let stats = db.cleanup(keep_one()).await.unwrap(); + let stats = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(stats.iter().all(|row| row.error.is_none()), "{stats:?}"); let row = stats .iter() @@ -10134,7 +10164,7 @@ async fn assert_collector_retry_after_a_pass_stopped_before_the_dead_manifests_r let before = detached_versions(&person_uri).await; { let _fp = catalog::ENSURE_INDICES_POST_PHASE_B_PRE_MANIFEST_COMMIT.fire_always(); - let outcome = db.ensure_indices().await; + let outcome = db.ensure_indices_as(None).await; assert!( outcome.is_err(), "the seam fails the index build before publish" @@ -10172,7 +10202,7 @@ async fn assert_collector_retry_after_a_pass_stopped_before_the_dead_manifests_r let stats = { let _fp = catalog::CLEANUP_SWEEP_PRE_DEAD_MANIFEST_DELETE.fire_always(); - db.cleanup(keep_one()).await.unwrap() + db.cleanup_as(keep_one(), None).await.unwrap() }; let row = stats .iter() @@ -10213,7 +10243,7 @@ async fn assert_collector_retry_after_a_pass_stopped_before_the_dead_manifests_r plan.dead_paths ); - let stats = db.cleanup(keep_one()).await.unwrap(); + let stats = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(stats.iter().all(|row| row.error.is_none()), "{stats:?}"); let row = stats .iter() @@ -10358,7 +10388,7 @@ async fn branch_merge_in_doubt_keeps_input_tags_until_cleanup() { "the acknowledged source and target guards must survive an in-doubt fast-forward" ); - let rows = db.cleanup(keep_one()).await.unwrap(); + let rows = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); assert!( !native @@ -10892,7 +10922,7 @@ async fn unattached_live_chain_pins_are_rooted_and_survive_cleanup() { report.retained_unattached_tags ); - let rows = db.cleanup(keep_one_at_any_age()).await.unwrap(); + let rows = db.cleanup_as(keep_one_at_any_age(), None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); let remaining = detached_versions(&person_uri).await; assert!( @@ -10937,7 +10967,7 @@ async fn unattached_live_chain_pins_are_rooted_and_survive_cleanup() { let plan = main_plan(&report, "node:Person"); assert!(plan.roots.contains(&grandchild_pin)); assert!(plan.roots.contains(&root_pin)); - let rows = db.cleanup(keep_one_at_any_age()).await.unwrap(); + let rows = db.cleanup_as(keep_one_at_any_age(), None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); let remaining = detached_versions(&person_uri).await; assert!(remaining.contains(&grandchild_pin)); @@ -11013,7 +11043,9 @@ async fn cleanup_releases_expired_unattached_tags_descendants_first() { let interrupted = { let _fp = catalog::CLEANUP_POST_UNATTACHED_RELEASE.fail_once_at(1); - db.cleanup(keep_one_at_any_age()).await.unwrap_err() + db.cleanup_as(keep_one_at_any_age(), None) + .await + .unwrap_err() }; assert!( interrupted @@ -11027,7 +11059,7 @@ async fn cleanup_releases_expired_unattached_tags_descendants_first() { "the child's tag went first; the root's stays for the retry" ); - let rows = db.cleanup(keep_one_at_any_age()).await.unwrap(); + let rows = db.cleanup_as(keep_one_at_any_age(), None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); assert!(unattached_tags(&uri).await.is_empty()); let at = |id: &str| ReadTarget::snapshot(SnapshotId::new(id.to_string())); @@ -11045,7 +11077,7 @@ async fn cleanup_releases_expired_unattached_tags_descendants_first() { person_count_at(&db, ReadTarget::branch("main")).await, main_before + 1 ); - let rows = db.cleanup(keep_one_at_any_age()).await.unwrap(); + let rows = db.cleanup_as(keep_one_at_any_age(), None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); } @@ -11109,7 +11141,7 @@ async fn abandoned_merge_input_tags_expire_after_target_advances() { ) .await .unwrap(); - let rows = db.cleanup(keep_one()).await.unwrap(); + let rows = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); assert!( captured_path.is_file(), @@ -11122,7 +11154,7 @@ async fn abandoned_merge_input_tags_expire_after_target_advances() { ) .await .unwrap(); - let rows = db.cleanup(keep_one()).await.unwrap(); + let rows = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); assert!( !captured_path.exists(), @@ -11174,7 +11206,7 @@ async fn cleanup_keeps_a_late_retired_merge_base_provider() { db.branch_create("target").await.unwrap(); let cleaner = helpers::session(Omnigraph::open(&uri).await.unwrap()); let parked = helpers::failpoint::Rendezvous::park_first(&catalog::CLEANUP_TABLE_GC); - let cleanup = tokio::spawn(async move { cleaner.cleanup(keep_one()).await }); + let cleanup = tokio::spawn(async move { cleaner.cleanup_as(keep_one(), None).await }); parked.wait_until_reached().await; let child_uri = uri.clone(); let child = tokio::task::spawn_blocking(move || { @@ -11238,7 +11270,7 @@ async fn planning_source_failures_keep_their_error_class() { ) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let lookup = params(&[("$name", "Planned")]); for seam in [ &catalog::QUERY_INDEX_FACTS_PRE_LOAD, diff --git a/crates/omnigraph/tests/forbidden_apis.rs b/crates/omnigraph/tests/forbidden_apis.rs index 0b73be466..fff04ba6b 100644 --- a/crates/omnigraph/tests/forbidden_apis.rs +++ b/crates/omnigraph/tests/forbidden_apis.rs @@ -190,7 +190,6 @@ const SENTINEL: &str = "// forbidden-api-allow:"; enum WriteProtocol { Exact(&'static str), Composed(&'static str), - ManifestAdoption, NativeRefControl, PhysicalOnly, EphemeralScratch, @@ -205,7 +204,6 @@ impl WriteProtocol { match self { Self::Exact(name) => format!("exact adapter ({name})"), Self::Composed(name) => format!("composed protocol ({name})"), - Self::ManifestAdoption => "manifest adoption".into(), Self::NativeRefControl => "native ref control".into(), Self::PhysicalOnly => "physical-only maintenance".into(), Self::EphemeralScratch => "ephemeral scratch".into(), @@ -264,13 +262,11 @@ write_surfaces! { "db/omnigraph.rs" => WriteProtocol::Exact("unpublished commit publication: the proposal's recorded changes replayed through the manifest publisher under its captured authority, required to allocate the proposal's own id") => ["publish_commit_as"], "db/omnigraph.rs" => WriteProtocol::Composed("proposal tag removal confirmed by a readback, then merge-input tag release") => ["drop_commit_as"], "db/omnigraph.rs" => INDICES_V9 => [ - "ensure_indices", "ensure_indices_on", - "rebuild_full_text_indices_on", "rebuild_full_text_indices_on_as", + "ensure_indices_as", "ensure_indices_on_as", "rebuild_full_text_indices_on_as", ], "db/omnigraph.rs" => WriteProtocol::TestOnly => ["failpoint_publish_table_head_without_index_rebuild_for_test", "init_with_legacy_system_columns_for_tests"], - "db/omnigraph.rs" => OPTIMIZE_V9 => ["optimize"], - "db/omnigraph.rs" => WriteProtocol::ManifestAdoption => ["repair"], - "db/omnigraph.rs" => WriteProtocol::PhysicalOnly => ["cleanup"], + "db/omnigraph.rs" => OPTIMIZE_V9 => ["optimize_as"], + "db/omnigraph.rs" => WriteProtocol::PhysicalOnly => ["cleanup_as"], "db/omnigraph.rs" => WriteProtocol::NativeRefControl => ["branch_create", "branch_create_as", "branch_create_from", "branch_create_from_as", "branch_delete", "branch_delete_as"], } @@ -296,6 +292,9 @@ const READ_ONLY_SURFACES: &[(&str, &str)] = &[ ("db/omnigraph.rs", "open_read_only"), ("db/omnigraph.rs", "refresh"), ("db/omnigraph.rs", "cleanup_plan"), + // The repair report: it publishes nothing, but `maintain` gates it like + // the maintenance writers (`maintenance_entry_points_take_an_actor`). + ("db/omnigraph.rs", "repair_report_as"), ("db/omnigraph.rs", "cleanup_plan_missing_paths"), ("db/omnigraph.rs", "cleanup_plan_path_snapshot"), ("db/omnigraph.rs", "open_read_only_with_storage"), @@ -1091,7 +1090,6 @@ durable_calls! { ("db/omnigraph/table_ops.rs", ".commit_changes_with_intent_and_expected(", 1, WriteProtocol::Exact("shared publisher")), ("db/omnigraph/schema_apply.rs", ".commit_changes_with_intent_and_expected(", 1, SCHEMA_V9), ("db/omnigraph/schema_apply/settlement.rs", ".commit_changes_with_intent_and_expected(", 1, WriteProtocol::Exact("schema settlement: strict numeric neutral lineage publication")), - ("db/omnigraph/repair.rs", ".commit_updates_with_actor_with_expected(", 1, WriteProtocol::ManifestAdoption), ("db/graph_coordinator.rs", ".commit_changes_with_intent_and_expected(", 1, WriteProtocol::Exact("publisher gateway")), ("db/graph_coordinator.rs", ".commit_changes_with_lineage_and_precondition(", 1, WriteProtocol::Exact("lowest manifest publisher gateway")), ("omnigraph-catalog/lib.rs", ".publish_with_precondition(", 1, WriteProtocol::Exact("lowest manifest publisher gateway")), @@ -1141,7 +1139,8 @@ durable_calls! { ("db/omnigraph/collector.rs", ".dataset()", 17, WriteProtocol::ReadOnlyAccess), ("db/omnigraph/collector.rs", ".delete(", 1, WriteProtocol::Composed("detached-only sweep through the table's own Lance store: freed files, then the published manifests no retained `__manifest` version pins (links before tips) and the dead stagings; an interrupted pass is re-swept by the next")), ("db/omnigraph/schema_apply.rs", ".dataset()", 1, SCHEMA_V9), - ("db/omnigraph/repair.rs", ".dataset()", 1, WriteProtocol::ManifestAdoption), + // The repair report reads raw transaction history; it publishes nothing. + ("db/omnigraph/repair.rs", ".dataset()", 1, WriteProtocol::ReadOnlyAccess), // The sixth accessor reports deferred FTS coverage from an immutable // snapshot; it only reads index metadata and never stages or publishes. ("db/omnigraph/optimize.rs", ".dataset()", 10, WriteProtocol::Composed("Optimize planning + read-only coverage and native-fork inventory + physical cleanup, including the release of expired proposal tags and of expired unattached tags descendants first, each confirmed by a readback")), @@ -2136,6 +2135,107 @@ fn graph_write_surfaces_are_registered() { ); } +/// The public surfaces of maintenance: the graph writers registered under +/// these protocols, plus the read-only repair report the same gate covers. +fn is_maintenance_protocol(protocol: WriteProtocol) -> bool { + protocol == INDICES_V9 || protocol == OPTIMIZE_V9 || protocol == WriteProtocol::PhysicalOnly +} + +const READ_ONLY_MAINTENANCE_SURFACES: &[(&str, &str)] = &[("db/omnigraph.rs", "repair_report_as")]; + +/// Maintenance has no actor-less public form (invariant 10). Every public +/// optimize, cleanup, index or repair entry point is an `_as` method taking the +/// `actor: Option<&str>` its policy check judges, so neither an embedded nor a +/// direct caller reaches maintenance around the gate; with a policy installed, +/// a `None` actor is refused. `policy_engine_chassis.rs` owns which action and +/// scope each one checks. +#[test] +fn maintenance_entry_points_take_an_actor() { + let src = engine_src_root(); + let read_only = READ_ONLY_SURFACES.iter().copied().collect::>(); + for surface in READ_ONLY_MAINTENANCE_SURFACES { + assert!( + read_only.contains(surface), + "{surface:?} must stay a registered read-only surface" + ); + } + let surfaces = WRITE_SURFACES + .iter() + .filter(|surface| is_maintenance_protocol(surface.protocol)) + .map(|surface| (surface.file, surface.function)) + .chain(READ_ONLY_MAINTENANCE_SURFACES.iter().copied()) + .collect::>(); + assert_eq!( + surfaces, + BTreeSet::from([ + ("db/omnigraph.rs", "cleanup_as"), + ("db/omnigraph.rs", "ensure_indices_as"), + ("db/omnigraph.rs", "ensure_indices_on_as"), + ("db/omnigraph.rs", "optimize_as"), + ("db/omnigraph.rs", "rebuild_full_text_indices_on_as"), + ("db/omnigraph.rs", "repair_report_as"), + ]), + "a maintenance surface was added or removed; give it an actor and a policy case" + ); + for (file, function) in surfaces { + assert!( + function.ends_with("_as"), + "{file}::{function}: maintenance entry points are `_as` forms" + ); + let path = guarded_path(&src, file); + let contents = std::fs::read_to_string(&path) + .unwrap_or_else(|error| panic!("failed to read {}: {error}", path.display())); + let ast = parse_rust_source(&contents, file); + let signatures = function_signatures(&ast, function); + assert_eq!( + signatures.len(), + 1, + "{file}::{function} must resolve to exactly one function" + ); + let takes_actor = signatures[0].inputs.iter().any(|input| { + let syn::FnArg::Typed(argument) = input else { + return false; + }; + matches!(&*argument.pat, syn::Pat::Ident(name) if name.ident == "actor") + && type_final_ident(&argument.ty).is_some_and(|ident| ident == "Option") + }); + assert!( + takes_actor, + "{file}::{function} must take `actor: Option<&str>` for its policy check" + ); + } +} + +fn function_signatures(ast: &syn::File, function_name: &str) -> Vec { + struct Finder<'a> { + function_name: &'a str, + signatures: Vec, + } + + impl<'ast> Visit<'ast> for Finder<'_> { + fn visit_item_fn(&mut self, node: &'ast syn::ItemFn) { + if node.sig.ident == self.function_name { + self.signatures.push(node.sig.clone()); + } + visit::visit_item_fn(self, node); + } + + fn visit_impl_item_fn(&mut self, node: &'ast syn::ImplItemFn) { + if node.sig.ident == self.function_name { + self.signatures.push(node.sig.clone()); + } + visit::visit_impl_item_fn(self, node); + } + } + + let mut finder = Finder { + function_name, + signatures: Vec::new(), + }; + finder.visit_file(ast); + finder.signatures +} + #[test] fn low_level_coordinator_surfaces_are_registered() { let src = engine_src_root(); diff --git a/crates/omnigraph/tests/helpers/expand_projection.rs b/crates/omnigraph/tests/helpers/expand_projection.rs index 0fe1e877e..8f792dee2 100644 --- a/crates/omnigraph/tests/helpers/expand_projection.rs +++ b/crates/omnigraph/tests/helpers/expand_projection.rs @@ -63,7 +63,7 @@ pub async fn fixture(dir: &tempfile::TempDir, sources: usize, payload_bytes: usi .await .unwrap(); } - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); with_setting(&db, "engine", "v2") } diff --git a/crates/omnigraph/tests/helpers/mod.rs b/crates/omnigraph/tests/helpers/mod.rs index c39350972..af4c39238 100644 --- a/crates/omnigraph/tests/helpers/mod.rs +++ b/crates/omnigraph/tests/helpers/mod.rs @@ -347,7 +347,7 @@ pub async fn init_and_load_at(uri: &str, schema: &str) -> Session { db.load_jsonl(TEST_DATA, LoadMode::Overwrite).await.unwrap(); // Mutation/load publish only exact data effects; physical indexes are // reconciled separately as derived state. - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); db } diff --git a/crates/omnigraph/tests/integration/branching.rs b/crates/omnigraph/tests/integration/branching.rs index 388854f56..bb2379621 100644 --- a/crates/omnigraph/tests/integration/branching.rs +++ b/crates/omnigraph/tests/integration/branching.rs @@ -150,7 +150,7 @@ async fn init_search_db(dir: &tempfile::TempDir) -> Session { db.load_jsonl(SEARCH_DATA, LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); db } @@ -2261,10 +2261,13 @@ fn assert_native_version_case( ); let maintenance = Omnigraph::open(uri).await.unwrap(); maintenance - .cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(100), - older_than: None, - }) + .cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(100), + older_than: None, + }, + None, + ) .await .unwrap(); main.mutate( @@ -3623,7 +3626,10 @@ async fn ensure_indices_on_child_branch_keeps_inherited_table_when_no_work_is_ne "the child branch inherits feature's pin" ); - experiment.ensure_indices_on("experiment").await.unwrap(); + experiment + .ensure_indices_on_as("experiment", None) + .await + .unwrap(); let experiment_snap = snapshot_branch(&experiment, "experiment").await.unwrap(); assert_eq!( @@ -4210,10 +4216,13 @@ async fn branch_delete_defers_pin_collection_and_allows_recreate() { "recreated writes leave the former pin intact" ); assert!(detached.contains(&second_pin)); - main.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + main.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); let detached = helpers::collector::detached_versions(&person_uri).await; @@ -4409,10 +4418,13 @@ async fn branch_delete_retires_legacy_physical_path_parents() { "{refused}" ); let rows = db - .cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + .cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); @@ -4595,10 +4607,13 @@ async fn branch_delete_retires_native_parent_and_cleanup_preserves_live_child() "{invalid}" ); assert!( - main.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + main.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None + ) .await .is_err(), "{invalid}" @@ -4632,10 +4647,13 @@ async fn branch_delete_retires_native_parent_and_cleanup_preserves_live_child() .await .unwrap(); reopened - .cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + .cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); assert!(manifest.branches().get(&native).await.is_err()); @@ -4649,10 +4667,13 @@ async fn branch_delete_retires_native_parent_and_cleanup_preserves_live_child() ); reopened.branch_delete("experiment").await.unwrap(); reopened - .cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + .cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); assert!(manifest.branches().get(&native).await.is_err()); @@ -4681,7 +4702,7 @@ async fn merged_table_preserves_row_version_for_unchanged_rows() { let dir = tempfile::tempdir().unwrap(); let uri = dir.path().to_str().unwrap(); let main = init_and_load(&dir).await; - main.ensure_indices().await.unwrap(); + main.ensure_indices_as(None).await.unwrap(); main.branch_create("feature").await.unwrap(); let feature = helpers::session(Omnigraph::open(uri).await.unwrap()); @@ -4764,7 +4785,7 @@ async fn merged_table_preserves_row_version_for_unchanged_rows() { async fn edge_tables_have_id_btree_after_ensure_indices() { let dir = tempfile::tempdir().unwrap(); let db = init_and_load(&dir).await; - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let snap = snapshot_main(&db).await.unwrap(); let ds = snap.open_dataset("edge:Knows").await.unwrap(); @@ -4786,7 +4807,7 @@ async fn merge_delta_only_bumps_changed_rows() { let dir = tempfile::tempdir().unwrap(); let uri = dir.path().to_str().unwrap(); let main = init_and_load(&dir).await; - main.ensure_indices().await.unwrap(); + main.ensure_indices_as(None).await.unwrap(); main.branch_create("feature").await.unwrap(); let feature = helpers::session(Omnigraph::open(uri).await.unwrap()); diff --git a/crates/omnigraph/tests/integration/changes.rs b/crates/omnigraph/tests/integration/changes.rs index fc782044f..80e0815b6 100644 --- a/crates/omnigraph/tests/integration/changes.rs +++ b/crates/omnigraph/tests/integration/changes.rs @@ -17,10 +17,13 @@ use crate::helpers::*; /// name). async fn reclaim_person_history(db: &Session) { let stats = db - .cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + .cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); let person = stats @@ -550,7 +553,7 @@ async fn diff_after_merge_reports_actual_changes() { let dir = tempfile::tempdir().unwrap(); let uri = dir.path().to_str().unwrap(); let main = init_and_load(&dir).await; - main.ensure_indices().await.unwrap(); + main.ensure_indices_as(None).await.unwrap(); let v_before_branch = snapshot_id(&main, "main").await.unwrap(); main.branch_create("feature").await.unwrap(); @@ -2056,7 +2059,7 @@ node Document { // touching logical rows. The moved descriptors force the payload // tie-break, which must classify every row as unchanged: the maintenance // commit is an empty block, not a phantom full-table update. - let stats = db.optimize().await.unwrap(); + let stats = db.optimize_as(None).await.unwrap(); assert!( stats .iter() @@ -2158,7 +2161,7 @@ async fn physical_only_commit_over_a_large_fragment_reads_as_empty_issue_705() { ) .await .unwrap(); - let stats = main.optimize().await.unwrap(); + let stats = main.optimize_as(None).await.unwrap(); assert!( stats .iter() @@ -3186,7 +3189,7 @@ async fn change_feed_merge_commit_is_first_parent_relative_with_merged_parent_on let dir = tempfile::tempdir().unwrap(); let uri = dir.path().to_str().unwrap(); let main = init_and_load(&dir).await; - main.ensure_indices().await.unwrap(); + main.ensure_indices_as(None).await.unwrap(); let pre_merge_head = snapshot_id(&main, "main").await.unwrap(); main.branch_create("side").await.unwrap(); diff --git a/crates/omnigraph/tests/integration/changes_cost.rs b/crates/omnigraph/tests/integration/changes_cost.rs index c35e9a104..2f82e1716 100644 --- a/crates/omnigraph/tests/integration/changes_cost.rs +++ b/crates/omnigraph/tests/integration/changes_cost.rs @@ -26,14 +26,10 @@ use crate::helpers; -use std::sync::Arc; - use crate::helpers::cost::{IoCounts, assert_flat, assert_grows, cost_harness, measure}; -use lance::Dataset; -use lance::dataset::UpdateBuilder; use omnigraph::IndexCoverage; use omnigraph::changes::ChangeFeedScope; -use omnigraph::db::{Omnigraph, ReadTarget, RepairOptions}; +use omnigraph::db::{Omnigraph, ReadTarget}; use omnigraph::loader::LoadMode; use omnigraph_compiler::ir::ParamMap; @@ -93,7 +89,7 @@ node Company { // Build full coverage, then append a new parent fragment. // The measured update therefore runs with a normal partial // / stale index state. The false arm never builds an index. - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); db.load_with_receipt( "main", r#"{"type":"Person","data":{"name":"stale-index-tail","age":1}}"#, @@ -303,123 +299,6 @@ node Document { .await; } -/// Multi-version graph intervals are intentionally outside the candidate -/// contract. The fixture advances one physical dataset twice, then adopts both -/// logical updates in one forced repair graph commit. Two stateless -/// `max_changes=1` pages must each fall back before reading either transaction; -/// the page count cannot multiply a transaction-history walk. -#[tokio::test] -#[ignore = "the multi-version interval is built by `repair`, which promotes at head; phase 4 of RFC \"Detached-only tables\" restates it over a detached chain"] -async fn changes_page_size_one_skips_transaction_history_for_multi_version_intervals() { - cost_harness(async { - let dir = tempfile::tempdir().unwrap(); - let db = Omnigraph::init( - dir.path().to_str().unwrap(), - "node Person {\n name: String @key\n age: I32?\n}", - ) - .await - .unwrap(); - let db = helpers::session(db); - db.load_with_receipt( - "main", - concat!( - r#"{"type":"Person","data":{"name":"alice","age":1}}"#, - "\n", - r#"{"type":"Person","data":{"name":"bob","age":1}}"#, - ), - LoadMode::Merge, - ) - .await - .unwrap(); - let snapshot = db.snapshot_of(ReadTarget::branch("main")).await.unwrap(); - let entry = snapshot.dataset("node:Person").unwrap(); - let before = entry.published_dataset_version; - let dataset_uri = format!( - "{}/{}", - db.uri().trim_end_matches('/'), - entry.dataset_path.trim_start_matches('/') - ); - - let dataset = Dataset::open(&dataset_uri).await.unwrap(); - let dataset = UpdateBuilder::new(Arc::new(dataset)) - .update_where("name = 'alice'") - .unwrap() - .set("age", "2") - .unwrap() - .build() - .unwrap() - .execute() - .await - .unwrap() - .new_dataset; - let dataset = UpdateBuilder::new(dataset) - .update_where("name = 'bob'") - .unwrap() - .set("age", "3") - .unwrap() - .build() - .unwrap() - .execute() - .await - .unwrap() - .new_dataset; - assert_eq!(dataset.version().version, before + 2); - - let repair = db - .repair(RepairOptions { - confirm: true, - force: true, - }) - .await - .unwrap(); - assert!(repair.graph_manifest_version.is_some()); - let repair_commit = db - .list_commits(None) - .await - .unwrap() - .first() - .expect("repair publishes a graph lineage commit") - .graph_commit_id - .clone(); - - let (first, first_io) = measure(db.commit_changes_page( - &repair_commit, - &ChangeFeedScope::default(), - None, - Some(1), - None, - )) - .await; - let first = first.unwrap(); - assert_eq!(first.block.changes.len(), 1); - let token = first.next_page_token.expect("two updates need two pages"); - assert_eq!( - first_io.candidate_transaction_reads, 0, - "first multi-version page must fall back before transaction-history I/O" - ); - - let (second, second_io) = measure(db.commit_changes_page( - &repair_commit, - &ChangeFeedScope::default(), - Some(&token), - Some(1), - None, - )) - .await; - let second = second.unwrap(); - assert_eq!(second.block.changes.len(), 1); - assert!(second.next_page_token.is_none()); - assert_ne!(first.block.changes[0].id, second.block.changes[0].id); - for io in [first_io, second_io] { - assert_eq!( - io.candidate_transaction_reads, 0, - "every stateless multi-version page must skip transaction-history I/O" - ); - } - }) - .await; -} - /// A delete's commit path proves it from the recorded ids, so its page data /// reads stay flat as the table grows. #[tokio::test] diff --git a/crates/omnigraph/tests/integration/composite_flow.rs b/crates/omnigraph/tests/integration/composite_flow.rs index a9dfac8fd..863ab0271 100644 --- a/crates/omnigraph/tests/integration/composite_flow.rs +++ b/crates/omnigraph/tests/integration/composite_flow.rs @@ -309,7 +309,7 @@ async fn composite_flow_canonical_lifecycle() { // version, and this flow exercises exactly that previously-failing // write below. // ───────────────────────────────────────────────────────────────── - let optimize_stats = db.optimize().await.unwrap(); + let optimize_stats = db.optimize_as(None).await.unwrap(); assert!( !optimize_stats.is_empty(), "optimize must return per-table stats" @@ -362,10 +362,13 @@ async fn composite_flow_canonical_lifecycle() { use omnigraph::db::CleanupPolicyOptions; use std::time::Duration; let _cleanup_stats = db - .cleanup(CleanupPolicyOptions { - keep_versions: Some(10), - older_than: Some(Duration::from_secs(3600)), - }) + .cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(10), + older_than: Some(Duration::from_secs(3600)), + }, + None, + ) .await .unwrap(); diff --git a/crates/omnigraph/tests/integration/consistency.rs b/crates/omnigraph/tests/integration/consistency.rs index c7e3cd8fa..ab37588e4 100644 --- a/crates/omnigraph/tests/integration/consistency.rs +++ b/crates/omnigraph/tests/integration/consistency.rs @@ -1369,7 +1369,7 @@ node Thing { db.load_jsonl(&seed, LoadMode::Overwrite).await.unwrap(); // Build the key index explicitly before exercising the merge loads. - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let mut delta = String::new(); for i in (1..=25).chain(51..=55) { @@ -1382,7 +1382,7 @@ node Thing { db.load_jsonl(&delta, LoadMode::Merge) .await .expect("first merge after ensure_indices must succeed"); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); db.load_jsonl(&delta, LoadMode::Merge) .await .expect("second merge after ensure_indices must succeed"); diff --git a/crates/omnigraph/tests/integration/end_to_end.rs b/crates/omnigraph/tests/integration/end_to_end.rs index 02e8e5f45..8ed619ced 100644 --- a/crates/omnigraph/tests/integration/end_to_end.rs +++ b/crates/omnigraph/tests/integration/end_to_end.rs @@ -2989,9 +2989,9 @@ async fn ensure_indices_does_not_error_on_repeated_call() { // load commits now enforce required indices; repeated ensure_indices calls // should be a no-op at the manifest level. - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let version_after_first = version_main(&db).await.unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let version_after_second = version_main(&db).await.unwrap(); assert_eq!(version_after_first, version_after_load); @@ -3093,7 +3093,7 @@ async fn full_flow_optimize_then_query_update_and_reopen() { .unwrap(); } - let stats = db.optimize().await.unwrap(); + let stats = db.optimize_as(None).await.unwrap(); assert!( stats.iter().any(|s| s.committed), "a multi-fragment table should have compacted in this flow" diff --git a/crates/omnigraph/tests/integration/engine_v2.rs b/crates/omnigraph/tests/integration/engine_v2.rs index 078c32321..cf54ed2f0 100644 --- a/crates/omnigraph/tests/integration/engine_v2.rs +++ b/crates/omnigraph/tests/integration/engine_v2.rs @@ -413,7 +413,7 @@ async fn indexed_single_hop_switches_to_the_csr_between_input_batches() { db.load_jsonl(&lines.join("\n"), LoadMode::Overwrite) .await .unwrap(); - db.optimize().await.unwrap(); + db.optimize_as(None).await.unwrap(); let v2 = with_setting(&db, "engine", "v2"); let pinned = first_column_sorted( @@ -1187,7 +1187,7 @@ edge Likes: Person -> Doc ) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); for (source, name, params) in [ (ADULTS, "adults", int_params(&[("min", 18)])), (RANKED, "ranked", params(&[("q", "graph")])), diff --git a/crates/omnigraph/tests/integration/engine_v2_memory.rs b/crates/omnigraph/tests/integration/engine_v2_memory.rs index 02660da69..36b5a0120 100644 --- a/crates/omnigraph/tests/integration/engine_v2_memory.rs +++ b/crates/omnigraph/tests/integration/engine_v2_memory.rs @@ -520,7 +520,7 @@ node Doc { seed.push('\n'); } db.load_jsonl(&seed, LoadMode::Overwrite).await.unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let v2 = with_setting(&db, "engine", "v2"); let source = r#"query fused($q: String) { match { $d: Doc } @@ -608,7 +608,7 @@ edge Knows: Doc -> Doc db.load_jsonl(&lines.join("\n"), LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let v2 = with_setting( &with_setting(&db, "engine", "v2"), "rrf_plan", @@ -724,7 +724,7 @@ edge Knows: Person -> Person ) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let v2 = with_setting(&db, "engine", "v2"); let queries = r#" query fused($q: String) { @@ -846,7 +846,7 @@ async fn hash_join_output_is_the_refusing_reservation() { db.load_jsonl(&lines.join("\n"), LoadMode::Overwrite) .await .unwrap(); - db.optimize().await.unwrap(); + db.optimize_as(None).await.unwrap(); let v2 = with_setting(&db, "engine", "v2"); let limit = 8 * MIB; @@ -929,7 +929,7 @@ async fn underestimated_hash_build_retries_id_lookup() { db.load_jsonl(&lines.join("\n"), LoadMode::Overwrite) .await .unwrap(); - db.optimize().await.unwrap(); + db.optimize_as(None).await.unwrap(); let v2 = with_setting(&db, "engine", "v2"); let limit = 16 * MIB; let explain = with_query_memory_limit( diff --git a/crates/omnigraph/tests/integration/engine_v2_plan_replay.rs b/crates/omnigraph/tests/integration/engine_v2_plan_replay.rs index f866b91da..35e4fd92e 100644 --- a/crates/omnigraph/tests/integration/engine_v2_plan_replay.rs +++ b/crates/omnigraph/tests/integration/engine_v2_plan_replay.rs @@ -65,7 +65,7 @@ async fn incomplete_rank_fuse_row_tiebreak_refuses_replay_issue_659() { .collect::>() .join("\n"); db.load_jsonl(&edges, LoadMode::Append).await.unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let query = r#"query selected_page() { match { $p: Person $p $e:(knows | likes) $q } return { $e.@type as edge_type, $e.@id as edge_id, $q.name as target } @@ -305,7 +305,7 @@ async fn selected_cold_indexed_route_uses_persisted_members_without_building_csr ) .await .unwrap(); - db.optimize().await.unwrap(); + db.optimize_as(None).await.unwrap(); drop(db); let db = with_traversal( &session(Omnigraph::open(dir.path().to_str().unwrap()).await.unwrap()), @@ -2123,7 +2123,7 @@ async fn a_replay_rechecks_recorded_full_text_coverage() { ) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); db.load_jsonl( r#"{"type":"Doc","data":{"slug":"t1","text":"graph","year":2022}} {"type":"Doc","data":{"slug":"t2","text":"graph graph graph graph","year":1990}}"#, diff --git a/crates/omnigraph/tests/integration/literal_filters.rs b/crates/omnigraph/tests/integration/literal_filters.rs index e493037db..41588b1db 100644 --- a/crates/omnigraph/tests/integration/literal_filters.rs +++ b/crates/omnigraph/tests/integration/literal_filters.rs @@ -579,7 +579,7 @@ async fn standalone_string_predicate_is_hoisted_into_scan() { async fn string_predicates_compose_with_search_and_negation() { let dir = tempfile::tempdir().unwrap(); let mut db = metric_db(&dir).await; - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let q = r#" query prefix_and_search() { match { diff --git a/crates/omnigraph/tests/integration/maintenance.rs b/crates/omnigraph/tests/integration/maintenance.rs index 15afbc9e8..c41a001f0 100644 --- a/crates/omnigraph/tests/integration/maintenance.rs +++ b/crates/omnigraph/tests/integration/maintenance.rs @@ -17,7 +17,7 @@ use lance_core::datatypes::BlobHandling; use omnigraph::IndexCoverage; use omnigraph::db::{ CleanupPolicyOptions, MergeOutcome, Omnigraph, ReadTarget, RepairAction, RepairClassification, - RepairOptions, SnapshotId, + SnapshotId, }; use omnigraph::error::{BranchStartRefusal, OmniError}; use omnigraph::loader::LoadMode; @@ -130,7 +130,7 @@ async fn optimize_on_empty_graph_returns_stats_per_table_with_no_changes() { let uri = dir.path().to_str().unwrap(); let db = Omnigraph::init(uri, TEST_SCHEMA).await.unwrap(); - let stats = db.optimize().await.unwrap(); + let stats = db.optimize_as(None).await.unwrap(); assert_eq!(stats.len(), 5); for s in &stats { @@ -154,7 +154,7 @@ async fn optimize_after_load_then_again_is_idempotent() { let db = init_and_load(&dir).await; // First pass may compact (load wrote real fragments). - let _first = db.optimize().await.unwrap(); + let _first = db.optimize_as(None).await.unwrap(); let commits_before = db.list_commits(None).await.unwrap(); let head_before = commits_before @@ -165,7 +165,7 @@ async fn optimize_after_load_then_again_is_idempotent() { // Second pass should be a no-op: already-compacted graph produces no // fragments_removed / fragments_added. - let second = db.optimize().await.unwrap(); + let second = db.optimize_as(None).await.unwrap(); for s in &second { assert_eq!( s.fragments_removed, 0, @@ -229,7 +229,7 @@ async fn optimize_preserves_manifest_registry_fragments() { !cold_files.is_empty(), "the fixture must release more registry rows than the hot bound" ); - let stats = db.optimize().await.unwrap(); + let stats = db.optimize_as(None).await.unwrap(); let after = Dataset::open(&manifest_uri).await.unwrap(); let after_files: BTreeSet<_> = after .fragments() @@ -323,7 +323,7 @@ async fn optimize_clears_stale_auto_cleanup_and_preserves_versions() { .unwrap() .len(); - db.optimize().await.unwrap(); + db.optimize_as(None).await.unwrap(); let ds = Dataset::open(&manifest_uri).await.unwrap(); // (a) the stale auto_cleanup config was cleared (non-destructive by construction). @@ -363,10 +363,9 @@ async fn optimize_preserves_versions_under_stale_auto_cleanup_config_on_data_tab add_person_fragments(&db).await; // multiple fragments → will_compact // Simulate an upgraded graph: set an aggressive stored auto_cleanup config on - // the Person table. This is an out-of-band Lance commit (an `UpdateConfig` that - // advances HEAD past the manifest), so realign the manifest with a forced repair - // first — otherwise optimize skips the table as uncovered drift and never - // reaches the scrub. (Forced because UpdateConfig is not verified maintenance.) + // the Person table. This is an out-of-band Lance commit (an `UpdateConfig` + // above the table's last linear version); optimize stages on the pin, never + // the linear HEAD, so it still compacts the table. let (_, _, person_full) = person_pin_and_head(&db, &root).await; { let mut ds = Dataset::open(&person_full).await.unwrap(); @@ -377,12 +376,6 @@ async fn optimize_preserves_versions_under_stale_auto_cleanup_config_on_data_tab .await .unwrap(); } - db.repair(RepairOptions { - confirm: true, - force: true, - }) - .await - .unwrap(); let versions_before = Dataset::open(&person_full) .await @@ -393,7 +386,7 @@ async fn optimize_preserves_versions_under_stale_auto_cleanup_config_on_data_tab .len(); let rows_before = count_rows(&db, "node:Person").await; - db.optimize().await.unwrap(); + db.optimize_as(None).await.unwrap(); let ds = Dataset::open(&person_full).await.unwrap(); assert!( @@ -440,7 +433,7 @@ node Doc { ) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); // A second load with NEW keys appends a fragment the existing BTREEs do not // cover (the existence gate skips re-building an index that already exists). @@ -462,7 +455,7 @@ node Doc { ); } - let stats = db.optimize().await.unwrap(); + let stats = db.optimize_as(None).await.unwrap(); // Postcondition: optimize_indices folded the scalar tail in, but preserves // the existing FTS artifact instead of merging uncertified postings. @@ -487,7 +480,7 @@ node Doc { assert!(deferred[0].reason.contains("rebuild-full-text-indexes")); let commits_before = db.list_commits(None).await.unwrap().len(); for _ in 0..2 { - let repeated = db.optimize().await.unwrap(); + let repeated = db.optimize_as(None).await.unwrap(); let doc = repeated .iter() .find(|stat| stat.type_key == "node:Doc") @@ -506,7 +499,10 @@ node Doc { assert_eq!(recovery_sidecar_count(&dir), 0); } assert_eq!(db.list_commits(None).await.unwrap().len(), commits_before); - let rebuilt = db.rebuild_full_text_indices_on("main").await.unwrap(); + let rebuilt = db + .rebuild_full_text_indices_on_as("main", None) + .await + .unwrap(); assert_eq!(rebuilt.rebuilt_indexes.len(), 1); assert_eq!(rebuilt.rebuilt_indexes[0].property, "slug"); let snap = snapshot_main(&db).await.unwrap(); @@ -644,9 +640,10 @@ node Tag {\n slug: String @key\n}\n"; .clone(); let probes = omnigraph::instrumentation::MergeWriteProbes::default(); - let stats = omnigraph::instrumentation::with_merge_write_probes(probes.clone(), db.optimize()) - .await - .expect("optimize must not crash on a graph with a Blob table"); + let stats = + omnigraph::instrumentation::with_merge_write_probes(probes.clone(), db.optimize_as(None)) + .await + .expect("optimize must not crash on a graph with a Blob table"); assert_eq!( ( probes.compaction_blob_batch_calls(), @@ -778,9 +775,10 @@ async fn optimize_compacts_large_blob_rows_in_bounded_batches() { let commits_before = db.list_commits(None).await.unwrap().len(); let probes = omnigraph::instrumentation::MergeWriteProbes::default(); - let stats = omnigraph::instrumentation::with_merge_write_probes(probes.clone(), db.optimize()) - .await - .unwrap(); + let stats = + omnigraph::instrumentation::with_merge_write_probes(probes.clone(), db.optimize_as(None)) + .await + .unwrap(); assert_eq!( probes.compaction_blob_batch_calls(), 1, @@ -933,9 +931,10 @@ async fn optimize_sizes_a_blob_batch_by_the_sum_of_a_rows_blob_columns() { } let probes = omnigraph::instrumentation::MergeWriteProbes::default(); - let stats = omnigraph::instrumentation::with_merge_write_probes(probes.clone(), db.optimize()) - .await - .unwrap(); + let stats = + omnigraph::instrumentation::with_merge_write_probes(probes.clone(), db.optimize_as(None)) + .await + .unwrap(); assert_eq!(probes.compaction_blob_batch_calls(), 1); assert_eq!( probes.compaction_blob_batch_rows(), @@ -1022,7 +1021,7 @@ async fn optimize_compacts_blob_fragments_with_deleted_rows() { "test precondition: both fragments carry a deletion vector" ); - let stats = db.optimize().await.unwrap(); + let stats = db.optimize_as(None).await.unwrap(); let doc = stats .iter() .find(|stat| stat.type_key == "node:Doc") @@ -1076,14 +1075,15 @@ async fn optimize_sizes_each_compaction_task_from_its_own_fragments() { }; db.load_jsonl(&line, mode).await.unwrap(); if load == 1 { - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); } } let probes = omnigraph::instrumentation::MergeWriteProbes::default(); - let stats = omnigraph::instrumentation::with_merge_write_probes(probes.clone(), db.optimize()) - .await - .unwrap(); + let stats = + omnigraph::instrumentation::with_merge_write_probes(probes.clone(), db.optimize_as(None)) + .await + .unwrap(); assert_eq!( probes.compaction_blob_batch_calls(), 2, @@ -1159,9 +1159,10 @@ async fn optimize_does_not_size_a_blob_batch_by_external_references() { ); let probes = omnigraph::instrumentation::MergeWriteProbes::default(); - let stats = omnigraph::instrumentation::with_merge_write_probes(probes.clone(), db.optimize()) - .await - .unwrap(); + let stats = + omnigraph::instrumentation::with_merge_write_probes(probes.clone(), db.optimize_as(None)) + .await + .unwrap(); assert_eq!(probes.compaction_blob_batch_calls(), 1); assert_eq!( probes.compaction_blob_batch_rows(), @@ -1278,8 +1279,8 @@ async fn optimize_then_cleanup_erases_dropped_property_values() { .await .unwrap(); // Settle index work, then prove Lance's planner leaves both tables alone. - db.optimize().await.unwrap(); - for stat in db.optimize().await.unwrap() { + db.optimize_as(None).await.unwrap(); + for stat in db.optimize_as(None).await.unwrap() { assert!(!stat.committed, "test precondition: {stat:?}"); } assert_eq!( @@ -1325,9 +1326,10 @@ async fn optimize_then_cleanup_erases_dropped_property_values() { let commits_before = db.list_commits(None).await.unwrap().len(); let probes = omnigraph::instrumentation::MergeWriteProbes::default(); - let stats = omnigraph::instrumentation::with_merge_write_probes(probes.clone(), db.optimize()) - .await - .unwrap(); + let stats = + omnigraph::instrumentation::with_merge_write_probes(probes.clone(), db.optimize_as(None)) + .await + .unwrap(); let stat = |key: &str| stats.iter().find(|stat| stat.type_key == key).unwrap(); let doc = stat("node:Doc"); assert!( @@ -1375,7 +1377,7 @@ async fn optimize_then_cleanup_erases_dropped_property_values() { doc_blob_values(&db, &["content"]).await == expected, "the remaining properties survive the rewrite" ); - for stat in db.optimize().await.unwrap() { + for stat in db.optimize_as(None).await.unwrap() { assert!( !stat.committed, "a second optimize has nothing left: {stat:?}" @@ -1399,8 +1401,8 @@ async fn optimize_then_cleanup_erases_dropped_property_values() { db.branch_merge("retained", "main").await.unwrap().outcome, MergeOutcome::Merged ); - db.optimize().await.unwrap(); - let stats = db.cleanup(keep_one()).await.unwrap(); + db.optimize_as(None).await.unwrap(); + let stats = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(stats.iter().all(|row| row.error.is_none()), "{stats:?}"); for needle in &needles { assert!( @@ -1409,7 +1411,7 @@ async fn optimize_then_cleanup_erases_dropped_property_values() { ); } db.branch_delete("retained").await.unwrap(); - let stats = db.cleanup(keep_one()).await.unwrap(); + let stats = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(stats.iter().all(|row| row.error.is_none()), "{stats:?}"); for needle in &needles { let found = files_containing(dir.path(), needle); @@ -1463,7 +1465,7 @@ async fn optimize_publishes_compaction_to_manifest_so_schema_apply_succeeds() { .get_fragments() .len(); - let stats = db.optimize().await.unwrap(); + let stats = db.optimize_as(None).await.unwrap(); let person = stats .iter() .find(|s| s.type_key == "node:Person") @@ -1523,7 +1525,7 @@ async fn optimize_compacts_the_pin_under_foreign_head_drift() { let (pin_before, head_before, _) = forge_person_compaction_drift(&db, &root).await; let rows_before = count_rows(&db, "node:Person").await; - let stats = db.optimize().await.unwrap(); + let stats = db.optimize_as(None).await.unwrap(); let person = stats .iter() .find(|s| s.type_key == "node:Person") @@ -1541,10 +1543,11 @@ async fn optimize_compacts_the_pin_under_foreign_head_drift() { assert_eq!(count_rows(&db, "node:Person").await, rows_before); } -/// Repair preview names linear commits above the last linear version as -/// `foreign_drift` and publishes nothing. +/// The repair report names linear commits above the last linear version as +/// `foreign_drift` and publishes nothing, and writers stage on the pin +/// regardless: a strict schema apply still succeeds. #[tokio::test] -async fn repair_preview_reports_foreign_drift_without_adopting_it() { +async fn repair_report_names_foreign_drift_without_adopting_it() { let dir = tempfile::tempdir().unwrap(); let root = dir .path() @@ -1555,14 +1558,13 @@ async fn repair_preview_reports_foreign_drift_without_adopting_it() { let db = init_and_load(&dir).await; let (pin_before, head_before, _) = forge_person_compaction_drift(&db, &root).await; - let stats = db - .repair(RepairOptions { - confirm: false, - force: false, - }) - .await - .unwrap(); - assert_eq!(stats.graph_manifest_version, None); + let commits_before = db.list_commits(None).await.unwrap(); + let stats = db.repair_report_as(None).await.unwrap(); + assert_eq!( + db.list_commits(None).await.unwrap(), + commits_before, + "the repair report publishes nothing" + ); let person = stats .datasets .iter() @@ -1580,44 +1582,6 @@ async fn repair_preview_reports_foreign_drift_without_adopting_it() { let (pin_after, head_after, _) = person_pin_and_head(&db, &root).await; assert_eq!(pin_after, pin_before); assert_eq!(head_after, head_before); -} - -/// A confirmed repair never adopts foreign drift, and writers stage on the -/// pin regardless: the strict schema apply still succeeds. -#[tokio::test] -async fn repair_confirm_never_adopts_foreign_drift() { - let dir = tempfile::tempdir().unwrap(); - let root = dir - .path() - .to_str() - .unwrap() - .trim_end_matches('/') - .to_string(); - let db = init_and_load(&dir).await; - let (pin_before, head_before, _) = forge_person_compaction_drift(&db, &root).await; - - let stats = db - .repair(RepairOptions { - confirm: true, - force: false, - }) - .await - .unwrap(); - assert_eq!( - stats.graph_manifest_version, None, - "confirmed repair publishes nothing for foreign drift" - ); - let person = stats - .datasets - .iter() - .find(|s| s.type_key == "node:Person") - .expect("Person repair stat present"); - assert_eq!(person.classification, RepairClassification::ForeignDrift); - assert_eq!(person.action, RepairAction::NoOp); - - let (pin_after, head_after, _) = person_pin_and_head(&db, &root).await; - assert_eq!(pin_after, pin_before); - assert_eq!(head_after, head_before); let desired = TEST_SCHEMA.replace( " age: I32?\n}", @@ -1630,10 +1594,10 @@ async fn repair_confirm_never_adopts_foreign_drift() { assert!(result.applied); } -/// Force does not change the judgement: a foreign raw delete is reported as -/// `foreign_drift`, never adopted, and reads still resolve the pin. +/// A foreign raw delete is reported as `foreign_drift`, never adopted, and +/// reads still resolve the pin. #[tokio::test] -async fn repair_force_never_adopts_a_foreign_delete() { +async fn repair_report_never_adopts_a_foreign_delete() { let dir = tempfile::tempdir().unwrap(); let root = dir .path() @@ -1644,14 +1608,7 @@ async fn repair_force_never_adopts_a_foreign_delete() { let db = init_and_load(&dir).await; let (pin_before, head_before, _) = forge_person_delete_drift(&db, &root).await; - let stats = db - .repair(RepairOptions { - confirm: true, - force: true, - }) - .await - .unwrap(); - assert_eq!(stats.graph_manifest_version, None); + let stats = db.repair_report_as(None).await.unwrap(); let person = stats .datasets .iter() @@ -1717,17 +1674,7 @@ async fn write_on_a_never_written_table_succeeds_over_foreign_drift_and_repair_r "the write neither moves nor adopts the foreign HEAD" ); - let stats = db - .repair(RepairOptions { - confirm: true, - force: false, - }) - .await - .unwrap(); - assert_eq!( - stats.graph_manifest_version, None, - "repair publishes nothing for foreign drift" - ); + let stats = db.repair_report_as(None).await.unwrap(); let company = stats .datasets .iter() @@ -1775,7 +1722,7 @@ async fn full_text_rebuild_replaces_all_columns_and_segments_in_one_publication( db.load_jsonl(r#"{"type":"Person","data":{"name":"Alice","age":30,"biography":"organism university running"}}"#, LoadMode::Merge, ) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); helpers::forge_linear_head_from_pin(&db, "main", "node:Person", 0).await; let person_uri = node_table_uri(&db, "Person").await; @@ -1919,7 +1866,10 @@ async fn full_text_rebuild_replaces_all_columns_and_segments_in_one_publication( assert_eq!(recovery_sidecar_count(&dir), 0); // A force rebuild remains a real operation after coverage is complete. - let repeated = db.rebuild_full_text_indices_on("main").await.unwrap(); + let repeated = db + .rebuild_full_text_indices_on_as("main", None) + .await + .unwrap(); assert_eq!(repeated.rebuilt_indexes, expected); assert_ne!(repeated.graph_commit_id, result.graph_commit_id); assert_eq!( @@ -1936,7 +1886,7 @@ async fn full_text_rebuild_on_a_branch_stages_on_the_inherited_location_without_ let db = init_and_load(&dir).await; db.branch_create("feature").await.unwrap(); let inherited = db.snapshot_of(ReadTarget::branch("feature")).await.unwrap(); - db.ensure_indices_on("feature").await.unwrap(); + db.ensure_indices_on_as("feature", None).await.unwrap(); assert_same_dataset_entry( inherited.dataset("node:Person").unwrap(), db.snapshot_of(ReadTarget::branch("feature")) @@ -1950,7 +1900,10 @@ async fn full_text_rebuild_on_a_branch_stages_on_the_inherited_location_without_ let main_commits_before = db.list_commits(None).await.unwrap(); let feature_commits_before = db.list_commits(Some("feature")).await.unwrap(); - let result = db.rebuild_full_text_indices_on("feature").await.unwrap(); + let result = db + .rebuild_full_text_indices_on_as("feature", None) + .await + .unwrap(); assert_eq!(result.branch, "feature"); assert_eq!(result.rebuilt_indexes.len(), 2); let feature = db.snapshot_of(ReadTarget::branch("feature")).await.unwrap(); @@ -2036,7 +1989,10 @@ async fn full_text_rebuild_reports_empty_builds_but_not_no_work() { .await .unwrap(); let before = snapshot_main(&db).await.unwrap(); - let result = db.rebuild_full_text_indices_on("main").await.unwrap(); + let result = db + .rebuild_full_text_indices_on_as("main", None) + .await + .unwrap(); assert_eq!(result.branch, "main"); assert_eq!(result.graph_commit_id, None); assert!(result.rebuilt_indexes.is_empty()); @@ -2051,7 +2007,10 @@ async fn full_text_rebuild_reports_empty_builds_but_not_no_work() { .await .unwrap(); let before = snapshot_main(&db).await.unwrap(); - let result = db.rebuild_full_text_indices_on("main").await.unwrap(); + let result = db + .rebuild_full_text_indices_on_as("main", None) + .await + .unwrap(); assert_eq!(result.rebuilt_indexes.len(), 2); assert!(result.graph_commit_id.is_some()); let after = snapshot_main(&db).await.unwrap(); @@ -2138,7 +2097,10 @@ async fn full_text_rebuild_refuses_unsupported_physical_inventory_before_effects .version, ); } - let error = db.rebuild_full_text_indices_on("main").await.unwrap_err(); + let error = db + .rebuild_full_text_indices_on_as("main", None) + .await + .unwrap_err(); assert!( error.to_string().contains("unsupported_internal_id_fts"), "{error}" @@ -2181,7 +2143,7 @@ async fn cleanup_without_any_policy_option_errors() { let db = init_and_load(&dir).await; let err = db - .cleanup(CleanupPolicyOptions::default()) + .cleanup_as(CleanupPolicyOptions::default(), None) .await .expect_err("cleanup with no policy options must error"); @@ -2211,10 +2173,13 @@ async fn cleanup_keep_one_preserves_head_and_table_remains_readable() { ); let _stats = db - .cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + .cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); @@ -2247,7 +2212,7 @@ async fn cleanup_keep_exceeding_history_preserves_every_available_version() { ); let pins = pins_in_history(&db, "main", "node:Person").await; - db.cleanup(keep_ten).await.unwrap(); + db.cleanup_as(keep_ten, None).await.unwrap(); // Every version a commit pins stays; only a chain's links below its // pinned tip (the first load's effect beneath its full-text declaration) @@ -2269,10 +2234,13 @@ async fn cleanup_older_than_zero_preserves_head() { // Lance must still preserve the head manifest, so the table is openable // afterwards and a subsequent load still works. let _stats = db - .cleanup(CleanupPolicyOptions { - keep_versions: None, - older_than: Some(Duration::from_secs(0)), - }) + .cleanup_as( + CleanupPolicyOptions { + keep_versions: None, + older_than: Some(Duration::from_secs(0)), + }, + None, + ) .await .unwrap(); @@ -2311,10 +2279,13 @@ async fn cleanup_preserves_main_version_pinned_by_live_lazy_branch() { "precondition: lazy-branch pin must fall outside keep=1 retention" ); - db.cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); @@ -2328,10 +2299,13 @@ async fn cleanup_preserves_main_version_pinned_by_live_lazy_branch() { // timestamp and version predicates with AND, so injecting the branch pin // as `before_version` keeps the inherited version even when every old // manifest satisfies the timestamp cutoff. - db.cleanup(CleanupPolicyOptions { - keep_versions: None, - older_than: Some(Duration::from_secs(0)), - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: None, + older_than: Some(Duration::from_secs(0)), + }, + None, + ) .await .unwrap(); assert_eq!( @@ -2369,10 +2343,13 @@ async fn cleanup_retains_the_commit_a_branch_starts_at() { db.branch_create_from(at(&loaded), "at-load").await.unwrap(); let people_at_load = count_rows_branch(&db, "at-load", "node:Person").await; - db.cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); assert_eq!( @@ -2428,10 +2405,13 @@ async fn cleanup_uses_oldest_pin_across_multiple_live_lazy_branches() { .await .unwrap(); - db.cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); @@ -2471,10 +2451,13 @@ async fn cleanup_fails_closed_when_live_lazy_branch_pin_is_unopenable() { let person_before = detached_versions(&person_uri).await; let stats = db - .cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + .cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); let row = |type_key: &str| { @@ -2517,7 +2500,7 @@ async fn cleanup_reports_foreign_head_drift_and_keeps_it() { .expect("every registration records its last linear version"); let foreign: Vec = (last_linear + 1..=head_version).collect(); - let stats = db.cleanup(keep_one()).await.unwrap(); + let stats = db.cleanup_as(keep_one(), None).await.unwrap(); let person = stats .iter() .find(|row| row.type_key == "node:Person") @@ -2557,13 +2540,16 @@ async fn cleanup_then_optimize_preserves_rows_and_table_remains_writable() { "fixture should seed both Person and Company rows" ); - db.cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); - db.optimize().await.unwrap(); + db.optimize_as(None).await.unwrap(); // Head is preserved through both ops. assert_eq!(count_rows(&db, "node:Person").await, people_before); @@ -2643,10 +2629,13 @@ async fn cleanup_reconciles_orphaned_branch_forks() { ); } - db.cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); @@ -2695,10 +2684,13 @@ async fn cleanup_reconciles_orphaned_branch_forks() { ); // Idempotent: a second cleanup with the orphan already gone is a no-op. - db.cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); assert!( @@ -2774,10 +2766,13 @@ async fn cleanup_reconciles_live_branch_orphan_fork_and_keeps_the_branch_write() let main_people = count_rows(&db, "node:Person").await; let main_companies = count_rows(&db, "node:Company").await; - db.cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); @@ -2877,10 +2872,13 @@ async fn cleanup_age_window_preserves_a_recent_snapshot_across_a_pointer_switch( .await .unwrap(); assert_eq!(before.num_rows(), base_count + 1); - db.cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: Some(Duration::from_secs(30 * 24 * 60 * 60)), - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: Some(Duration::from_secs(30 * 24 * 60 * 60)), + }, + None, + ) .await .unwrap(); assert!( @@ -2898,10 +2896,13 @@ async fn cleanup_age_window_preserves_a_recent_snapshot_across_a_pointer_switch( .await .unwrap(); assert_eq!(after.num_rows(), base_count + 1); - db.cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: Some(Duration::ZERO), - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: Some(Duration::ZERO), + }, + None, + ) .await .unwrap(); assert!( @@ -2931,10 +2932,13 @@ async fn cleanup_age_window_preserves_recent_ref_absent_descendant() { .set_times(std::fs::FileTimes::new().set_modified(std::time::SystemTime::UNIX_EPOCH)) .unwrap(); std::fs::write(&recent_file, b"recent interrupted residue").unwrap(); - db.cleanup(CleanupPolicyOptions { - keep_versions: None, - older_than: Some(Duration::from_secs(30 * 24 * 60 * 60)), - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: None, + older_than: Some(Duration::from_secs(30 * 24 * 60 * 60)), + }, + None, + ) .await .unwrap(); assert!( @@ -2945,10 +2949,13 @@ async fn cleanup_age_window_preserves_recent_ref_absent_descendant() { recent_file.exists(), "a missing ref is not permission to discard a recent object" ); - db.cleanup(CleanupPolicyOptions { - keep_versions: None, - older_than: Some(Duration::ZERO), - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: None, + older_than: Some(Duration::ZERO), + }, + None, + ) .await .unwrap(); assert!( @@ -3018,10 +3025,13 @@ async fn cleanup_reclaims_retirement_archive_after_the_age_window() { > std::time::SystemTime::UNIX_EPOCH, "retirement writes a recent archive even when the tree is old" ); - db.cleanup(CleanupPolicyOptions { - keep_versions: None, - older_than: Some(Duration::from_secs(30 * 24 * 60 * 60)), - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: None, + older_than: Some(Duration::from_secs(30 * 24 * 60 * 60)), + }, + None, + ) .await .unwrap(); assert!( @@ -3032,10 +3042,13 @@ async fn cleanup_reclaims_retirement_archive_after_the_age_window() { tree.exists(), "the recent retirement metadata must protect the aged physical tree" ); - db.cleanup(CleanupPolicyOptions { - keep_versions: None, - older_than: Some(Duration::ZERO), - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: None, + older_than: Some(Duration::ZERO), + }, + None, + ) .await .unwrap(); assert!( @@ -3094,10 +3107,13 @@ async fn cleanup_age_window_preserves_recent_native_mutation_and_aged_parent() { let changed = child.delete("name = 'Alice'").await.unwrap(); assert_eq!(changed.num_deleted_rows, 1); let latest = changed.new_dataset.version().version; - db.cleanup(CleanupPolicyOptions { - keep_versions: None, - older_than: Some(Duration::from_secs(30 * 24 * 60 * 60)), - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: None, + older_than: Some(Duration::from_secs(30 * 24 * 60 * 60)), + }, + None, + ) .await .unwrap(); let branches = Dataset::open(&person_uri) @@ -3129,10 +3145,13 @@ async fn cleanup_age_window_preserves_recent_native_mutation_and_aged_parent() { retained.count_rows(None).await.unwrap(), count_rows(&db, "node:Person").await - 1 ); - db.cleanup(CleanupPolicyOptions { - keep_versions: None, - older_than: Some(Duration::ZERO), - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: None, + older_than: Some(Duration::ZERO), + }, + None, + ) .await .unwrap(); let branches = Dataset::open(&person_uri) @@ -3228,10 +3247,13 @@ async fn cleanup_preserves_the_pin_a_lazy_child_reads_after_its_parent_retires() .clone(); db.branch_delete("feature").await.unwrap(); db.branch_create("feature").await.unwrap(); - db.cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); @@ -3290,10 +3312,13 @@ async fn cleanup_reclaims_dead_incarnation_fork_of_live_branch() { ds.create_branch(&dead_native, base, None).await.unwrap(); } - db.cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); @@ -3359,10 +3384,13 @@ async fn cleanup_reclaims_dead_incarnation_fork_of_live_branch() { db.apply_schema(TEST_SCHEMA).await.unwrap(); let replacement_uri = node_table_uri(&db, "Company").await; assert_ne!(replacement_uri, company_uri); - db.cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); assert!( @@ -3421,7 +3449,7 @@ async fn index_build_tolerates_null_vector_rows() { // Must not abort: the untrainable vector column is deferred, while id, // slug (FTS), and n (BTREE) are built together in one table transaction. let pending = db - .ensure_indices() + .ensure_indices_as(None) .await .expect("ensure_indices must not abort when a vector column has no trainable vectors yet"); assert_eq!(pending.len(), 1, "only the null vector index is pending"); @@ -3488,7 +3516,7 @@ async fn optimize_materializes_index_declared_but_unbuilt() { ); } - db.optimize().await.unwrap(); + db.optimize_as(None).await.unwrap(); // Postcondition: optimize's reconciler materialized the declared index. let snap = snapshot_main(&db).await.unwrap(); @@ -3544,7 +3572,7 @@ async fn optimize_materializes_index_after_type_rename() { ); } - db.optimize().await.unwrap(); + db.optimize_as(None).await.unwrap(); let snap = snapshot_main(&db).await.unwrap(); let ds = snap.open_dataset("node:Item").await.unwrap(); @@ -3576,7 +3604,7 @@ async fn collector_retains_malformed_witnesses_after_retirement_history_is_gone( let identity = manifest.branches().get(&native).await.unwrap().identifier; assert_ne!(identity, BranchIdentifier::main()); db.branch_delete("retired").await.unwrap(); - let rows = db.cleanup(keep_one()).await.unwrap(); + let rows = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); assert!(!dir.path().join("__manifest/tree").join(&native).exists()); let identity = serde_json::to_string(&identity).unwrap(); @@ -3653,7 +3681,7 @@ async fn collector_retains_malformed_witnesses_after_retirement_history_is_gone( ); } } - let rows = db.cleanup(keep_one()).await.unwrap(); + let rows = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); let remaining = detached_versions(&table_uri).await; for (label, version) in versions { @@ -3707,7 +3735,7 @@ async fn collector_sweeps_the_pin_that_only_a_pruned_manifest_version_names() { ); assert!(plan.sweep_paths.is_disjoint(&plan.marked_paths)); - let stats = db.cleanup(keep_one()).await.unwrap(); + let stats = db.cleanup_as(keep_one(), None).await.unwrap(); let row = stats .iter() .find(|row| row.type_key == "node:Person") @@ -3921,7 +3949,7 @@ async fn collector_repeat_run_discovers_what_an_earlier_pass_left() { async fn collector_finds_nothing_to_sweep_after_todays_cleanup() { let dir = tempfile::tempdir().unwrap(); let db = init_and_load(&dir).await; - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); insert_person(&db, "main", "Eve").await; mutate_main( &db, @@ -3931,7 +3959,7 @@ async fn collector_finds_nothing_to_sweep_after_todays_cleanup() { ) .await .unwrap(); - db.optimize().await.unwrap(); + db.optimize_as(None).await.unwrap(); db.branch_create("feature").await.unwrap(); insert_person(&db, "feature", "Frank").await; db.branch_merge("feature", "main").await.unwrap(); @@ -3941,7 +3969,7 @@ async fn collector_finds_nothing_to_sweep_after_todays_cleanup() { ) .await .unwrap(); - let stats = db.cleanup(keep_one()).await.unwrap(); + let stats = db.cleanup_as(keep_one(), None).await.unwrap(); for row in &stats { assert!(row.error.is_none(), "every table's version GC ran: {row:?}"); } @@ -3990,10 +4018,13 @@ async fn cleanup_on_a_detached_only_graph_counts_manifest_versions_for_keep() { let third = staged_since(&person_uri, &after_second).await; let stats = db - .cleanup(CleanupPolicyOptions { - keep_versions: Some(2), - older_than: None, - }) + .cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(2), + older_than: None, + }, + None, + ) .await .unwrap(); let person = |stats: &[omnigraph::db::DatasetCleanupStats]| { @@ -4018,7 +4049,7 @@ async fn cleanup_on_a_detached_only_graph_counts_manifest_versions_for_keep() { assert!(!remaining.contains(&first), "{remaining:?}"); assert!(remaining.contains(&second) && remaining.contains(&third)); - let row = person(&db.cleanup(keep_one()).await.unwrap()); + let row = person(&db.cleanup_as(keep_one(), None).await.unwrap()); assert_eq!( row.manifests_removed, 1, "keep 1 prunes the publication that named the second pin: {row:?}" @@ -4027,7 +4058,7 @@ async fn cleanup_on_a_detached_only_graph_counts_manifest_versions_for_keep() { assert!(!remaining.contains(&second), "{remaining:?}"); assert!(remaining.contains(&third)); - let row = person(&db.cleanup(keep_one()).await.unwrap()); + let row = person(&db.cleanup_as(keep_one(), None).await.unwrap()); assert_eq!(row.manifests_removed, 0, "a rerun finds nothing: {row:?}"); let report = db.cleanup_plan(keep_one()).await.unwrap(); for plan in &report.tables { @@ -4222,7 +4253,7 @@ async fn collector_reclaims_aged_orphans_without_deleting_recent_blob_sidecars() .unwrap(); } let rows_before = count_rows(&db, "node:Person").await; - let stats = db.cleanup(keep_one()).await.unwrap(); + let stats = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(stats.iter().all(|row| row.error.is_none()), "{stats:?}"); for relative in recent { assert!( @@ -4290,7 +4321,7 @@ async fn collector_reclaims_blob_sidecars_after_the_last_retained_pin() { db.load_jsonl(&row(¤t_payload), LoadMode::Overwrite) .await .unwrap(); - let stats = db.cleanup(keep_one()).await.unwrap(); + let stats = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(stats.iter().all(|row| row.error.is_none()), "{stats:?}"); for path in &old_sidecars { assert!( @@ -4311,7 +4342,7 @@ async fn collector_reclaims_blob_sidecars_after_the_last_retained_pin() { let report = db.cleanup_plan(keep_one()).await.unwrap(); let plan = main_plan(&report, "node:Doc"); assert!(plan.errors.is_empty(), "{:?}", plan.errors); - let stats = db.cleanup(keep_one()).await.unwrap(); + let stats = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(stats.iter().all(|row| row.error.is_none()), "{stats:?}"); for path in &old_sidecars { assert!( diff --git a/crates/omnigraph/tests/integration/merge_cleanup.rs b/crates/omnigraph/tests/integration/merge_cleanup.rs index 471842586..bad2124f8 100644 --- a/crates/omnigraph/tests/integration/merge_cleanup.rs +++ b/crates/omnigraph/tests/integration/merge_cleanup.rs @@ -36,7 +36,7 @@ async fn repeated_merge_keeps_its_selected_base_through_cleanup() { for age in 1..=3 { set_age(&db, "source", "p", age).await; set_age(&db, "target", "q", age).await; - let rows = db.cleanup(keep_one()).await.unwrap(); + let rows = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); let reopened = helpers::session(Omnigraph::open(uri).await.unwrap()); assert_eq!( @@ -106,7 +106,7 @@ async fn imported_merge_base_survives_owner_retirement_and_recreation() { .starts_with("zz-unrelated.") }) .expect("unrelated retirement archive exists before cleanup"); - let rows = db.cleanup(keep_one()).await.unwrap(); + let rows = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); assert!( !unrelated.exists(), diff --git a/crates/omnigraph/tests/integration/point_in_time.rs b/crates/omnigraph/tests/integration/point_in_time.rs index 6f088bd5c..4cb87dc97 100644 --- a/crates/omnigraph/tests/integration/point_in_time.rs +++ b/crates/omnigraph/tests/integration/point_in_time.rs @@ -1184,7 +1184,10 @@ async fn historical_read_of_reclaimed_version_is_typed() { .await .unwrap(); - let stats = db.cleanup(helpers::collector::keep_one()).await.unwrap(); + let stats = db + .cleanup_as(helpers::collector::keep_one(), None) + .await + .unwrap(); let person = stats .iter() .find(|row| row.type_key == "node:Person") diff --git a/crates/omnigraph/tests/integration/recovery.rs b/crates/omnigraph/tests/integration/recovery.rs index 0a3608558..f0cd513b5 100644 --- a/crates/omnigraph/tests/integration/recovery.rs +++ b/crates/omnigraph/tests/integration/recovery.rs @@ -105,12 +105,12 @@ async fn recovery_ensure_indices_steady_state_no_sidecar() { {"type":"Company","data":{"name":"acme"}} "#; db.load_jsonl(test_data, LoadMode::Append).await.unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); drop(db); let db = Omnigraph::open(uri).await.unwrap(); let before = snapshot_main(&db).await.unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let after = snapshot_main(&db).await.unwrap(); assert_eq!( after.graph_manifest_version(), @@ -153,7 +153,7 @@ async fn recovery_ensure_indices_handles_empty_tables() { let db = Omnigraph::init(uri, TEST_SCHEMA).await.unwrap(); // Don't load any data — every table is empty. let before = snapshot_main(&db).await.unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let after = snapshot_main(&db).await.unwrap(); assert_eq!( after.graph_manifest_version(), @@ -173,7 +173,7 @@ async fn recovery_ensure_indices_handles_empty_tables() { drop(db); let db = Omnigraph::open(uri).await.unwrap(); let before = snapshot_main(&db).await.unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let after = snapshot_main(&db).await.unwrap(); assert_eq!( after.graph_manifest_version(), diff --git a/crates/omnigraph/tests/integration/repro_issue_563.rs b/crates/omnigraph/tests/integration/repro_issue_563.rs index 409ce09ac..733ca35c8 100644 --- a/crates/omnigraph/tests/integration/repro_issue_563.rs +++ b/crates/omnigraph/tests/integration/repro_issue_563.rs @@ -141,7 +141,7 @@ async fn ranked_read_with_join_returns_top_limit_issue_563() { db.load_jsonl(&edges, LoadMode::Append).await.unwrap(); } - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); // The ranked read without the join stays under the i32 offset ceiling. let no_join = query_main( diff --git a/crates/omnigraph/tests/integration/rrf_prefilter_gate.rs b/crates/omnigraph/tests/integration/rrf_prefilter_gate.rs index db0cdf997..121cef7f6 100644 --- a/crates/omnigraph/tests/integration/rrf_prefilter_gate.rs +++ b/crates/omnigraph/tests/integration/rrf_prefilter_gate.rs @@ -265,7 +265,7 @@ async fn init_gate_db(dir: &tempfile::TempDir) -> Session { db.load_jsonl(&gate_seed_data(), LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); db } @@ -457,7 +457,7 @@ async fn init_stats_db(dir: &tempfile::TempDir) -> Session { db.load_jsonl(&stats_seed_data(), LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); db } @@ -614,7 +614,7 @@ async fn natural_gate_prefilters_at_ratio_boundary() { db.load_jsonl(&rows.join("\n"), LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let db = db; let text_params = params(&[("$q1", "needle"), ("$q2", "sharp")]); @@ -767,7 +767,7 @@ async fn empty_eligible_set_falls_back() { ) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let db = db; let text_params = params(&[("$q1", "needle"), ("$q2", "sharp")]); @@ -807,7 +807,7 @@ async fn natural_gate_prefilters_selective_fixture() { db.load_jsonl(&rows.join("\n"), LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let db = db; let text_params = params(&[("$q1", "needle"), ("$q2", "sharp")]); @@ -880,7 +880,7 @@ async fn decoy_flood_winner_holds_under_both_plans() { db.load_jsonl(&rows.join("\n"), LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let db = db; let text_params = params(&[("$q1", "alpha"), ("$q2", "beta")]); @@ -926,7 +926,7 @@ async fn natural_gate_falls_back_on_broad_fixture() { db.load_jsonl(&rows.join("\n"), LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let db = db; let text_params = params(&[("$q1", "needle"), ("$q2", "sharp")]); diff --git a/crates/omnigraph/tests/integration/scalar_indexes.rs b/crates/omnigraph/tests/integration/scalar_indexes.rs index a87b5be4a..29443468b 100644 --- a/crates/omnigraph/tests/integration/scalar_indexes.rs +++ b/crates/omnigraph/tests/integration/scalar_indexes.rs @@ -42,7 +42,7 @@ async fn node_scalar_and_enum_index_columns_get_btree() { let uri = dir.path().to_str().unwrap(); let db = session(Omnigraph::init(uri, SCHEMA).await.unwrap()); db.load_jsonl(DATA, LoadMode::Overwrite).await.unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let snap = snapshot_main(&db).await.unwrap(); let ds = snap.open_dataset("node:Item").await.unwrap(); @@ -74,7 +74,7 @@ async fn node_scalar_and_enum_index_columns_get_btree() { // Idempotence: a second run finds every declared index present and // publishes nothing (dual indexes must not re-plan as missing forever). let version_before = ds.published_dataset_version(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let snap2 = snapshot_main(&db).await.unwrap(); let ds2 = snap2.open_dataset("node:Item").await.unwrap(); assert_eq!( diff --git a/crates/omnigraph/tests/integration/traversal_adaptive.rs b/crates/omnigraph/tests/integration/traversal_adaptive.rs index dc3f70ce3..e393372ff 100644 --- a/crates/omnigraph/tests/integration/traversal_adaptive.rs +++ b/crates/omnigraph/tests/integration/traversal_adaptive.rs @@ -83,7 +83,7 @@ async fn clique_db(dir: &tempfile::TempDir) -> Session { .unwrap(); // The endpoint BTREEs must exist, or dispatch prices a degraded scan and // takes CSR outright — the switch under test would never arm. - db.optimize().await.unwrap(); + db.optimize_as(None).await.unwrap(); db } @@ -189,7 +189,7 @@ query far($name: String) { jsonl.push('\n'); let db = session(Omnigraph::init(uri, TEST_SCHEMA).await.unwrap()); db.load_jsonl(&jsonl, LoadMode::Overwrite).await.unwrap(); - db.optimize().await.unwrap(); + db.optimize_as(None).await.unwrap(); let p = params(&[("$name", "p0")]); let csr_db = with_traversal(&db, Traversal::Csr); @@ -322,7 +322,7 @@ async fn corrupt_artifact_falls_back_to_the_scan_build() { async fn stale_artifact_is_rejected_after_an_edge_write() { let dir = tempfile::tempdir().unwrap(); let db = init_and_load(&dir).await; - db.optimize().await.unwrap(); + db.optimize_as(None).await.unwrap(); let artifact = dir.path().join("__graph_index/csr-current.bin"); assert!(artifact.is_file()); diff --git a/crates/omnigraph/tests/integration/warm_read_cost.rs b/crates/omnigraph/tests/integration/warm_read_cost.rs index d9e05e316..172c2d8b4 100644 --- a/crates/omnigraph/tests/integration/warm_read_cost.rs +++ b/crates/omnigraph/tests/integration/warm_read_cost.rs @@ -956,10 +956,13 @@ async fn assert_cached_borrower_survives_branch_delete( "{branch} payload changed" ); } - db.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); assert!( @@ -995,10 +998,13 @@ async fn assert_cached_borrower_survives_branch_delete( .unwrap(); let recreated_pin = helpers::pinned_version(writer, "feature", "node:Person").await; assert_ne!(recreated_pin, source_pin); - db.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); let pins = helpers::collector::detached_versions(&table_uri).await; diff --git a/crates/omnigraph/tests/integration/write_cost.rs b/crates/omnigraph/tests/integration/write_cost.rs index d33fd8053..7182c64e1 100644 --- a/crates/omnigraph/tests/integration/write_cost.rs +++ b/crates/omnigraph/tests/integration/write_cost.rs @@ -38,7 +38,7 @@ use crate::helpers::{ // `_graph_commits`/`_graph_commit_actors` tables are retired), so the manifest scan // also covers the lineage and actor rows the authenticated write path appends. // Without internal-table compaction these scans are O(fragments) and grow forever; -// step 2 brings the internal tables into `db.optimize()`, so after compaction the +// step 2 brings the internal tables into `db.optimize_as(None)`, so after compaction the // per-write scan is flat. The test runs the **authenticated (actorful) write path** // — every commit carries an actor — and compacts at each depth checkpoint before // measuring, pinning the production invariant "a periodically-compacted graph's @@ -62,7 +62,7 @@ async fn internal_table_scans_are_flat_in_history() { } // Step 2: compaction folds all three internal tables' O(depth) fragments back // to a small constant, so the following write's scan of them is flat. - db.optimize().await.unwrap(); + db.optimize_as(None).await.unwrap(); let io = measure_insert_as(&db, &format!("lock_{d}"), ACTOR).await; current += 1; // the measured write advanced depth by one eprintln!( @@ -87,7 +87,7 @@ async fn compacted_graph_at_depth(depth: u64) -> (u64, tempfile::TempDir, helper let dir = tempfile::tempdir().unwrap(); let db = local_graph(&dir).await; commit_many(&db, depth as usize).await; - db.optimize().await.unwrap(); + db.optimize_as(None).await.unwrap(); (depth, dir, db) } @@ -106,7 +106,7 @@ async fn ensure_indices_manifest_reads_are_flat_in_history() { for (depth, _dir, db) in [graphs.0, graphs.1] { let indexed_schema = helpers::TEST_SCHEMA.replace("age: I32?", "age: I32? @index"); db.apply_schema(&indexed_schema).await.unwrap(); - let (result, io) = measure(db.ensure_indices()).await; + let (result, io) = measure(db.ensure_indices_as(None)).await; result.unwrap(); eprintln!( "ensure_indices depth~{depth}: __manifest={} data={}", @@ -147,7 +147,7 @@ async fn ensure_indices_writes_no_control_object() { let before_write_text = counts.write_text(); let before_delete = counts.delete(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); assert_eq!( counts.write_text() - before_write_text, 0, @@ -274,7 +274,7 @@ async fn optimize_writes_no_control_object() { let before_write_text = counts.write_text(); let before_delete = counts.delete(); - let stats = db.optimize().await.unwrap(); + let stats = db.optimize_as(None).await.unwrap(); assert!( stats .iter() @@ -310,7 +310,7 @@ async fn optimize_manifest_reads_are_flat_in_history() { commit_many(&db, 3).await; let commits_before = db.list_commits(None).await.unwrap().len(); - let (result, io) = measure(db.optimize()).await; + let (result, io) = measure(db.optimize_as(None)).await; result.unwrap(); assert_eq!( db.list_commits(None).await.unwrap().len(), @@ -357,7 +357,7 @@ async fn internal_table_scans_are_flat_without_compaction() { commit_many_as(&db, (d - current) as usize, ACTOR).await; current = d; } - // NO `db.optimize()` here — that omission is the whole point. The flat gate + // NO `db.optimize_as(None)` here — that omission is the whole point. The flat gate // above compacts before measuring and so never exercises this served regime. let io = measure_insert_as(&db, &format!("served_{d}"), ACTOR).await; current += 1; // the measured write advanced depth by one diff --git a/crates/omnigraph/tests/policy_engine_chassis.rs b/crates/omnigraph/tests/policy_engine_chassis.rs index b0b806cc4..fa75a8553 100644 --- a/crates/omnigraph/tests/policy_engine_chassis.rs +++ b/crates/omnigraph/tests/policy_engine_chassis.rs @@ -8,7 +8,10 @@ //! PR #2 wired `apply_schema_as`. PR #3 fans the same `enforce()` call //! out to the remaining six writers — `mutate_as`, `load_as`, //! `ingest_as`, `branch_create_as` / `branch_create_from_as`, -//! `branch_delete_as`, `branch_merge_as`. Each writer pair below +//! `branch_delete_as`, `branch_merge_as`. The maintenance entry points — +//! `optimize_as`, `ensure_indices_as` / `ensure_indices_on_as`, +//! `rebuild_full_text_indices_on_as`, `repair_report_as` (`maintain`) and +//! `cleanup_as` (`reclaim`) — are covered the same way. Each writer pair below //! covers allow + deny via the engine-side gate; the allow case proves //! the enforce call is correctly scoped (i.e. doesn't reject a legit //! actor), the deny case proves it actually denies an unauthorized @@ -111,6 +114,31 @@ async fn init_with_policy_and_feature_branch( /// One JSONL record for `load_as` / `ingest_as` exercises. const ONE_PERSON_JSONL: &str = r#"{"type": "Person", "data": {"name": "Eve"}}"#; +/// Every directory and file under `root`, with file bytes: what a refused +/// call must leave as it found it. +fn physical_tree(root: &Path) -> std::collections::BTreeMap>> { + fn visit( + root: &Path, + path: &Path, + entries: &mut std::collections::BTreeMap>>, + ) { + for entry in fs::read_dir(path).unwrap() { + let entry = entry.unwrap(); + let path = entry.path(); + let relative = path.strip_prefix(root).unwrap().to_path_buf(); + if entry.file_type().unwrap().is_dir() { + entries.insert(relative, None); + visit(root, &path, entries); + } else { + entries.insert(relative, Some(fs::read(path).unwrap())); + } + } + } + let mut entries = std::collections::BTreeMap::new(); + visit(root, root, &mut entries); + entries +} + fn assert_denied(result: Result, what: &str) { match result { Err(OmniError::Policy(msg)) => { @@ -801,51 +829,44 @@ async fn non_canonical_branch_names_cannot_reach_a_protected_branch() { } #[tokio::test] -async fn full_text_rebuild_enforces_selected_branch_before_effects_and_records_actor() { - fn physical_tree( - root: &Path, - ) -> std::collections::BTreeMap>> { - fn visit( - root: &Path, - path: &Path, - entries: &mut std::collections::BTreeMap>>, - ) { - for entry in fs::read_dir(path).unwrap() { - let entry = entry.unwrap(); - let path = entry.path(); - let relative = path.strip_prefix(root).unwrap().to_path_buf(); - if entry.file_type().unwrap().is_dir() { - entries.insert(relative, None); - visit(root, &path, entries); - } else { - entries.insert(relative, Some(fs::read(path).unwrap())); - } - } - } - let mut entries = std::collections::BTreeMap::new(); - visit(root, root, &mut entries); - entries - } - +async fn full_text_rebuild_enforces_change_or_maintain_on_the_selected_branch_and_records_actor() { let dir = tempfile::tempdir().unwrap(); let db = init_and_load(&dir).await; db.branch_create("feature").await.unwrap(); drop(db); - // This actor can change unprotected feature, but not protected main. - // A hardcoded main scope or a different maintenance action must fail. - let policy = POLICY_YAML.replace(" branch_scope: any", " branch_scope: unprotected"); + // `act-allowed` can change unprotected feature, `act-maintainer` can + // maintain it, and neither reaches protected main. A hardcoded main scope + // or an action other than `change` or `maintain` must fail. + let policy = format!( + "{} - id: maintainers\n allow:\n actors: {{ group: maintainers }}\n actions: [maintain]\n branch_scope: unprotected\n", + POLICY_YAML + .replace(" branch_scope: any", " branch_scope: unprotected") + .replace( + " readers: [act-denied]\n", + " readers: [act-denied]\n maintainers: [act-maintainer]\n", + ) + ); let db = Omnigraph::open(dir.path().to_str().unwrap()).await.unwrap(); let (db, _engine) = install_policy_source(db, dir.path(), &policy); let main_before = db.list_commits(None).await.unwrap(); let feature_before = db.list_commits(Some("feature")).await.unwrap(); let files_before = physical_tree(dir.path()); - for (branch, actor) in [("feature", "act-denied"), ("main", "act-allowed")] { - assert_denied( - db.rebuild_full_text_indices_on_as(branch, Some(actor)) - .await, - "rebuild_full_text_indices_on_as", + for (branch, actor) in [ + ("feature", "act-denied"), + ("main", "act-allowed"), + ("main", "act-maintainer"), + ] { + let result = db + .rebuild_full_text_indices_on_as(branch, Some(actor)) + .await; + // Either grant would admit the rebuild, so the refusal names both. + assert!( + matches!(&result, Err(OmniError::Policy(message)) + if message.contains("'maintain'") && message.contains("'change'")), + "{actor} on {branch}: {result:?}" ); + assert_denied(result, "rebuild_full_text_indices_on_as"); assert_eq!( physical_tree(dir.path()), files_before, @@ -853,7 +874,7 @@ async fn full_text_rebuild_enforces_selected_branch_before_effects_and_records_a ); } let error = db - .rebuild_full_text_indices_on("feature") + .rebuild_full_text_indices_on_as("feature", None) .await .unwrap_err(); assert!( @@ -863,25 +884,36 @@ async fn full_text_rebuild_enforces_selected_branch_before_effects_and_records_a assert_eq!( physical_tree(dir.path()), files_before, - "the no-actor wrapper must not bypass policy" + "a missing actor must not bypass policy" ); - let result = db - .rebuild_full_text_indices_on_as("feature", Some("act-allowed")) - .await - .expect("Change on the selected unprotected branch must permit a real rebuild"); - assert_eq!(result.branch, "feature"); - assert_eq!(result.rebuilt_indexes.len(), 2); - let feature_after = db.list_commits(Some("feature")).await.unwrap(); - assert_eq!(feature_after.len(), feature_before.len() + 1); - assert_eq!( - result.graph_commit_id.as_deref(), - Some(feature_after[0].graph_commit_id.as_str()) - ); - assert_eq!(feature_after[0].actor_id.as_deref(), Some("act-allowed")); + let mut parent = feature_before[0].graph_commit_id.clone(); + for (actor, grant) in [("act-allowed", "change"), ("act-maintainer", "maintain")] { + let result = db + .rebuild_full_text_indices_on_as("feature", Some(actor)) + .await + .unwrap_or_else(|error| { + panic!( + "{grant} on the selected unprotected branch must permit a real rebuild: {error}" + ) + }); + assert_eq!(result.branch, "feature"); + assert_eq!(result.rebuilt_indexes.len(), 2); + let feature_after = db.list_commits(Some("feature")).await.unwrap(); + assert_eq!( + result.graph_commit_id.as_deref(), + Some(feature_after[0].graph_commit_id.as_str()) + ); + assert_eq!(feature_after[0].actor_id.as_deref(), Some(actor)); + assert_eq!( + feature_after[0].parent_commit_id.as_deref(), + Some(parent.as_str()) + ); + parent = feature_after[0].graph_commit_id.clone(); + } assert_eq!( - feature_after[0].parent_commit_id.as_deref(), - Some(feature_before[0].graph_commit_id.as_str()) + db.list_commits(Some("feature")).await.unwrap().len(), + feature_before.len() + 2 ); let main_after = db.list_commits(None).await.unwrap(); assert_eq!(main_after.len(), main_before.len()); @@ -891,6 +923,176 @@ async fn full_text_rebuild_enforces_selected_branch_before_effects_and_records_a ); } +/// Maintenance grants, apart from data writes: `act-maintainer` may maintain +/// any branch and `act-branch-maintainer` only unprotected ones; +/// `act-reclaimer` may reclaim; `act-writer` holds `change` everywhere and no +/// maintenance grant; `act-denied` holds nothing. +const MAINTENANCE_POLICY_YAML: &str = r#" +version: 1 +groups: + maintainers: [act-maintainer] + branch-maintainers: [act-branch-maintainer] + reclaimers: [act-reclaimer] + writers: [act-writer] + readers: [act-denied] +protected_branches: [main] +rules: + - id: maintainers + allow: + actors: { group: maintainers } + actions: [maintain] + branch_scope: any + - id: branch-maintainers + allow: + actors: { group: branch-maintainers } + actions: [maintain] + branch_scope: unprotected + - id: reclaimers + allow: + actors: { group: reclaimers } + actions: [reclaim] + - id: writers + allow: + actors: { group: writers } + actions: [change] + branch_scope: any +"#; + +/// A graph with a `feature` branch and two extra Person fragments on main, so +/// optimize has a compaction to publish, under [`MAINTENANCE_POLICY_YAML`]. +async fn init_with_maintenance_policy(dir: &tempfile::TempDir) -> Session { + let db = init_and_load(dir).await; + db.branch_create("feature").await.unwrap(); + for (name, age) in [("Eve", 40), ("Frank", 41)] { + let params = mixed_params(&[("$name", name)], &[("$age", age)]); + db.mutate("main", MUTATION_QUERIES, "insert_person", ¶ms) + .await + .unwrap(); + } + drop(db); + let db = Omnigraph::open(dir.path().to_str().unwrap()).await.unwrap(); + install_policy_source(db, dir.path(), MAINTENANCE_POLICY_YAML).0 +} + +fn assert_no_actor(result: Result, what: &str) { + match result { + Err(OmniError::Policy(message)) if message.contains("no actor") => {} + other => panic!("{what}: expected the no-actor refusal, got {other:?}"), + } +} + +/// Optimize, index builds and the repair report require `maintain` on the +/// branch they maintain, before any effect: `change` does not imply it, a +/// grant on unprotected branches does not reach main, and a missing actor is +/// refused. An allowed optimize records its actor on the commit it publishes. +#[tokio::test] +async fn maintenance_entry_points_enforce_maintain_on_their_branch_before_effects() { + let dir = tempfile::tempdir().unwrap(); + let db = init_with_maintenance_policy(&dir).await; + let main_before = db.list_commits(None).await.unwrap(); + let files_before = physical_tree(dir.path()); + + for actor in [ + "act-denied", + "act-writer", + "act-reclaimer", + "act-branch-maintainer", + ] { + let actor = Some(actor); + assert_denied(db.optimize_as(actor).await, "optimize_as"); + assert_denied(db.ensure_indices_as(actor).await, "ensure_indices_as"); + assert_denied( + db.ensure_indices_on_as("main", actor).await, + "ensure_indices_on_as(main)", + ); + assert_denied(db.repair_report_as(actor).await, "repair_report_as"); + } + for actor in ["act-denied", "act-writer", "act-reclaimer"] { + assert_denied( + db.ensure_indices_on_as("feature", Some(actor)).await, + "ensure_indices_on_as(feature)", + ); + } + assert_no_actor(db.optimize_as(None).await, "optimize_as"); + assert_no_actor(db.ensure_indices_as(None).await, "ensure_indices_as"); + assert_no_actor( + db.ensure_indices_on_as("feature", None).await, + "ensure_indices_on_as", + ); + assert_no_actor(db.repair_report_as(None).await, "repair_report_as"); + assert_eq!( + physical_tree(dir.path()), + files_before, + "a refused maintenance call must not write index artifacts, compactions or publications" + ); + assert_eq!(db.list_commits(None).await.unwrap(), main_before); + + db.ensure_indices_on_as("feature", Some("act-branch-maintainer")) + .await + .expect("maintain on an unprotected branch admits its index builds"); + let report = db + .repair_report_as(Some("act-maintainer")) + .await + .expect("maintain on main admits the repair report"); + assert!(!report.datasets.is_empty()); + db.ensure_indices_as(Some("act-maintainer")) + .await + .expect("maintain on main admits index builds"); + + let head_before_optimize = db.list_commits(None).await.unwrap()[0] + .graph_commit_id + .clone(); + let stats = db + .optimize_as(Some("act-maintainer")) + .await + .expect("maintain on main admits optimize"); + assert!( + stats + .iter() + .any(|stat| stat.type_key == "node:Person" && stat.committed), + "{stats:?}" + ); + let main_after = db.list_commits(None).await.unwrap(); + assert_eq!( + main_after[0].parent_commit_id.as_deref(), + Some(head_before_optimize.as_str()) + ); + assert_eq!(main_after[0].actor_id.as_deref(), Some("act-maintainer")); +} + +/// Cleanup requires `reclaim`, which no other grant implies, before any +/// effect. +#[tokio::test] +async fn cleanup_as_enforces_reclaim_before_effects() { + let dir = tempfile::tempdir().unwrap(); + let db = init_with_maintenance_policy(&dir).await; + let keep_one = || omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }; + let files_before = physical_tree(dir.path()); + for actor in ["act-denied", "act-writer", "act-maintainer"] { + assert_denied(db.cleanup_as(keep_one(), Some(actor)).await, "cleanup_as"); + } + assert_no_actor(db.cleanup_as(keep_one(), None).await, "cleanup_as"); + assert_eq!( + physical_tree(dir.path()), + files_before, + "a refused cleanup must not delete anything" + ); + + let rows = db + .cleanup_as(keep_one(), Some("act-reclaimer")) + .await + .expect("reclaim admits cleanup"); + assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); + assert_ne!( + physical_tree(dir.path()), + files_before, + "the admitted cleanup reclaims the history keep 1 does not retain" + ); +} + #[tokio::test] async fn schema_settlement_authorizes_current_executor_and_preserves_original_authorship() { use omnigraph::db::{SchemaApplySettlement, SchemaNonPublicationProof}; diff --git a/crates/omnigraph/tests/schema_apply.rs b/crates/omnigraph/tests/schema_apply.rs index 9ac4ee162..692b44ddd 100644 --- a/crates/omnigraph/tests/schema_apply.rs +++ b/crates/omnigraph/tests/schema_apply.rs @@ -308,7 +308,7 @@ query insert_project($name: String) { ); let detached_before_indices = helpers::collector::detached_versions(&project_uri).await; stale_handle - .ensure_indices_on("source") + .ensure_indices_on_as("source", None) .await .expect("index planning must use the same token-bound post-apply catalog"); let project_after_indices = project_published( @@ -372,7 +372,7 @@ async fn stale_handle_branch_delete_gates_tables_added_by_schema_apply() { let index_handle = Arc::clone(&index_reconciler); let mut index_task = - tokio::spawn(async move { index_handle.ensure_indices_on("target").await }); + tokio::spawn(async move { index_handle.ensure_indices_on_as("target", None).await }); let index_blocked = tokio::time::timeout(std::time::Duration::from_millis(250), &mut index_task) .await @@ -1821,7 +1821,7 @@ node Tag { .collect::>() .join("\n"); db.load_jsonl(&rows, LoadMode::Overwrite).await.unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let before = db.snapshot_of(ReadTarget::branch("main")).await.unwrap(); let before_snapshot = db.resolve_snapshot("main").await.unwrap(); @@ -1937,7 +1937,7 @@ node Tag { } // Reconciliation builds from the desired declarations, so nothing indexes // a dropped column again. - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); for (table, dropped_id) in dropped_ids { assert!( coverage(&db, table) @@ -2124,7 +2124,7 @@ query strong($min: I32) { "an update carries the untouched Blob of an evolved row" ); - db.optimize().await.unwrap(); + db.optimize_as(None).await.unwrap(); assert_eq!( read_managed_blob_bytes( &db, @@ -2479,7 +2479,7 @@ async fn apply_schema_pure_type_rename_preserves_identity_path_and_version() { ); let db = helpers::session(Omnigraph::init(uri, &initial).await.unwrap()); db.load_jsonl(&data, LoadMode::Overwrite).await.unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let before_snapshot_id = db.resolve_snapshot("main").await.unwrap(); let before_snapshot = db.snapshot_of(ReadTarget::branch("main")).await.unwrap(); let before_version = before_snapshot.graph_manifest_version(); @@ -2675,10 +2675,13 @@ edge WorksAt: Human -> Company /// prunes the `__manifest` version naming it. async fn reclaim_dropped_history(db: &Session) { let stats = db - .cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + .cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); for row in &stats { @@ -3110,7 +3113,7 @@ async fn apply_schema_defers_vector_index_on_empty_table() { db.load_jsonl(r#"{"type":"Doc","data":{"slug":"d1","body":"hello","embedding":[0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8]}}"#, LoadMode::Merge, ) .await .expect("loading a Doc with an embedding must succeed"); - db.ensure_indices() + db.ensure_indices_as(None) .await .expect("the deferred vector index must build once the table has a trainable vector"); } @@ -3261,7 +3264,7 @@ async fn full_text_analyzer_survives_schema_evolution_issue_904() { assert_eq!(search("summary").await, ["d3"]); // The reconciler builds the postings over the declarations. - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); assert_eq!(segments("text").await, [Some(2)]); assert_eq!(search("text").await, ["d1", "d2"]); assert_eq!(search("summary").await, ["d3"]); diff --git a/crates/omnigraph/tests/search.rs b/crates/omnigraph/tests/search.rs index fd79b55e4..2715c2b0c 100644 --- a/crates/omnigraph/tests/search.rs +++ b/crates/omnigraph/tests/search.rs @@ -207,7 +207,7 @@ async fn init_search_db(dir: &tempfile::TempDir) -> Session { db.load_jsonl(SEARCH_DATA, LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); db } @@ -226,7 +226,7 @@ async fn init_mock_embedding_search_db(dir: &tempfile::TempDir) -> Session { db.load_jsonl(&mock_embedding_seed_data(), LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); db } @@ -236,7 +236,7 @@ async fn init_model_recorded_search_db(dir: &tempfile::TempDir) -> Session { db.load_jsonl(&mock_embedding_seed_data(), LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); db } @@ -561,7 +561,7 @@ query delete_middle() { "#; let db = session(Omnigraph::init(uri, schema).await.unwrap()); db.load_jsonl(&seed, LoadMode::Overwrite).await.unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); helpers::forge_linear_head_from_pin(&db, "main", "node:Doc", 0).await; { use lance::index::DatasetIndexExt; @@ -1225,7 +1225,7 @@ query nearest_all($q: Vector(4)) { let uri = dir.path().to_str().unwrap(); let db = session(Omnigraph::init(uri, schema).await.unwrap()); db.load_jsonl(&rows, LoadMode::Overwrite).await.unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let db = with_setting(&db, "ann_nprobes", "100000"); use omnigraph::instrumentation::{QueryIoProbes, with_query_io_probes}; @@ -1306,7 +1306,7 @@ query nearest_all($q: Vector(4)) { let uri = dir.path().to_str().unwrap(); let db = session(Omnigraph::init(uri, schema).await.unwrap()); db.load_jsonl(&rows, LoadMode::Overwrite).await.unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let probes = QueryIoProbes::default(); let result = with_query_io_probes(probes.clone(), async { @@ -1413,7 +1413,7 @@ async fn issue_567_line_docs(uri: &str, rows: usize, edge_every: Option) db.load_jsonl(&lines.join("\n"), LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); db } @@ -2069,7 +2069,7 @@ async fn injected_embedding_config_is_used_instead_of_env() { db.load_jsonl(&mock_embedding_seed_data(), LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let result = query_main( &db, @@ -2160,7 +2160,7 @@ async fn bm25_ordered_aggregate_counts_all_matches_not_the_capped_scan() { db.load_jsonl(&underfill_seed_data(), LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let db = db; use omnigraph::instrumentation::{QueryIoProbes, with_query_io_probes}; @@ -2209,7 +2209,7 @@ async fn rrf_arms_scan_uncapped_in_one_pass() { db.load_jsonl(&underfill_seed_data(), LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); use omnigraph::instrumentation::{QueryIoProbes, with_query_io_probes}; // This test pins the POSTFILTER plan's invariants (uncapped one-pass @@ -2281,7 +2281,7 @@ query uncapped_all($q: String) { db.load_jsonl(&underfill_seed_data(), LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); // Appended AFTER the index build: their fragment is uncovered, and their // term frequency (40 > the seed's max 20) puts them at the top of any // correct ranking. @@ -2426,7 +2426,7 @@ async fn mutation_with_deferred_index_coverage_remains_searchable() { // Ordinary optimize must preserve certified postings, not silently replace // them with an uncertified incremental fold. Both old and tail rows remain // searchable after data compaction and unrelated index maintenance. - db.optimize().await.unwrap(); + db.optimize_as(None).await.unwrap(); for (term, slug) in [("Quasar", "quasar-notes"), ("Learning", "ml-intro")] { let result = query_main(&db, SEARCH_QUERIES, "text_search", ¶ms(&[("$q", term)])) .await @@ -2587,7 +2587,10 @@ async fn uncertified_full_text_refuses_all_search_routes_but_not_ordinary_reads( Err(OmniError::FullTextIndexRebuildRequired { .. }) )); - let rebuilt = db.rebuild_full_text_indices_on("main").await.unwrap(); + let rebuilt = db + .rebuild_full_text_indices_on_as("main", None) + .await + .unwrap(); assert!(!rebuilt.rebuilt_indexes.is_empty()); assert!( query_main( @@ -2690,7 +2693,7 @@ node Doc { 0, "load publishes exact data effects and leaves physical indexes pending" ); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let ds = snapshot_main(&db) .await @@ -2788,7 +2791,7 @@ async fn full_text_analyzer_holds_after_first_rows_and_overwrites_issue_904() { .unwrap(); assert_eq!(hits("main", "docs").await, ["d1", "d2"], "first rows"); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); db.load("main", &rows("Doc", "d"), LoadMode::Overwrite) .await .unwrap(); @@ -2880,7 +2883,7 @@ async fn a_full_text_index_with_no_segment_refuses_the_call_issue_747() { .unwrap_err(), ); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); assert_eq!( first_column_sorted( &query_branch(&db, "main", ANALYZER_QUERIES, "docs", &deep) diff --git a/crates/omnigraph/tests/system_column_upgrade.rs b/crates/omnigraph/tests/system_column_upgrade.rs index cfaf0e354..ca5a2dc84 100644 --- a/crates/omnigraph/tests/system_column_upgrade.rs +++ b/crates/omnigraph/tests/system_column_upgrade.rs @@ -71,7 +71,7 @@ async fn legacy_graph_with_data(dir: &tempfile::TempDir) -> Session { db.load_jsonl(LEGACY_DATA, LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); db } @@ -226,7 +226,7 @@ async fn assert_upgraded(db: &Session, dir: &tempfile::TempDir, expected_export: .await .expect_err("the respelled @unique(@src, @dst) still refuses a duplicate endpoint pair"); assert_eq!(count_rows(db, "edge:WorksAt").await, 3); - db.ensure_indices() + db.ensure_indices_as(None) .await .expect("index reconciliation after the rename is a no-op by field id"); } @@ -407,7 +407,7 @@ async fn system_column_upgrade_history_survives_unrelated_reclaimed_tables() { main.retained.contains(&company_moved) && main.would_prune.contains(&version_before), "the run keeps the Company write's `__manifest` version and drops the pre-write one: {main:?}" ); - let stats = db.cleanup(keep).await.unwrap(); + let stats = db.cleanup_as(keep, None).await.unwrap(); let row = stats.iter().find(|row| row.type_key == first).unwrap(); assert!(row.error.is_none(), "{row:?}"); assert!( diff --git a/docs/dev/control-plane.md b/docs/dev/control-plane.md index 72a1959cb..821427a0d 100644 --- a/docs/dev/control-plane.md +++ b/docs/dev/control-plane.md @@ -608,6 +608,7 @@ server. Public actor construction cannot grant signed-token permissions. on `AuthenticatedActor` exposes only verified claims. Policy embedders must handle `PolicyAction::ConfigManage`, +`PolicyAction::Maintain`, `PolicyAction::Reclaim`, `PolicyResourceKind::Cluster`, and `PolicyEngineKind::Cluster`. These public enums are exhaustive, so an external exhaustive match must add the relevant arm when upgrading. Preserving existing constructors and direct storage-holder diff --git a/docs/dev/writes.md b/docs/dev/writes.md index 120fff36a..367c85cff 100644 --- a/docs/dev/writes.md +++ b/docs/dev/writes.md @@ -582,6 +582,15 @@ callers must pass one when a policy checker is installed. Actor attribution travels with the pre-minted graph lineage and is published with the same manifest CAS. +Maintenance has no actor-less public form. `optimize_as`, `ensure_indices_as`, +`ensure_indices_on_as` and `rebuild_full_text_indices_on_as` check `maintain` +on the branch they maintain, `main` for optimize; the full-text rebuild also +admits `change`, its grant before `maintain` existed (`Omnigraph::enforce_any`, +`PolicyChecker::check_any`). `cleanup_as` checks `reclaim` with no branch, and +the read-only `repair_report_as` checks `maintain` on `main`. +`forbidden_apis.rs::maintenance_entry_points_take_an_actor` keeps every +registered maintenance surface an `_as` form with an `actor` parameter. + Bearer authentication (`401 unauthorized`), the HTTP contract check (`400 api_contract_mismatch`) and policy (`403 forbidden`) refuse a served write only before its first durable effect. The server runs diff --git a/docs/user/operations/maintenance.md b/docs/user/operations/maintenance.md index 7bf20669a..c89fe5c01 100644 --- a/docs/user/operations/maintenance.md +++ b/docs/user/operations/maintenance.md @@ -21,6 +21,14 @@ accepts local-file direct targets only; cloud-direct maintenance is not exposed by this CLI. Engine/server cloud support does not change that client boundary. See [operator configuration](../cli/targets.md#transport-and-capability-boundaries). +Each command acts as the operator configuration's `operator.actor`; +`rebuild-full-text-indexes` also accepts `--as` in its place. A commit the +command publishes records that actor. Direct CLI access, including `--target`, +does not load the server's Cedar policy; storage permissions are its trust +boundary. An embedded host that installs a policy checker enforces the +[policy actions](policy.md#actions) before any effect: `maintain` for optimize, +index builds, the full-text rebuild and repair, and `reclaim` for cleanup. + ## Optimize ```bash @@ -118,10 +126,9 @@ Success means the selected branch's planned rebuild was published, or an explici no-op with an empty index list, null commit, and empty `warnings` array. The default branch is `main`. -`--as` supplies actor attribution for this command. Direct CLI access, including -`--target`, does not load the server's Cedar policy; storage permissions are its -trust boundary. An embedded host that installs a policy checker also enforces -the Change permission before rebuilding. +`--as` supplies actor attribution for this command. An embedded host that +installs a policy checker admits the rebuild under `maintain` or `change` on the +selected branch; a later release admits it under `maintain` only. Other branches and historical snapshots are not rewritten. Rebuild every live branch that needs full-text search; restoring an older snapshot may require diff --git a/docs/user/operations/policy.md b/docs/user/operations/policy.md index 2187d6e8d..b3d122450 100644 --- a/docs/user/operations/policy.md +++ b/docs/user/operations/policy.md @@ -21,6 +21,8 @@ Graph-scoped actions: | `branch_merge` | Branch merge | | `commit_publish` | Publication of an unpublished commit onto its recorded target branch | | `commit_drop` | Drop of an unpublished commit | +| `maintain` | Content-preserving maintenance: optimize, index builds, the full-text rebuild and the repair report | +| `reclaim` | Cleanup, which permanently removes table versions and files no retained commit needs | | `invoke_query` | Entry to a stored query | | `admin` | Reserved graph administration | @@ -42,6 +44,17 @@ The reserved graph `admin` action does not grant cluster management. A stored mutation requires both `invoke_query` and `change`. A stored read requires `invoke_query` and `read`. +`maintain` and `reclaim` are separate from `change`, so an automation account +can compact a graph without the right to write data, and can hold `maintain` +without the right to delete history. `maintain` is judged on the branch the +operation maintains (`main` for optimize and the repair report) and takes +`branch_scope`; `reclaim` covers every branch's history and takes no branch +scope. The full-text rebuild accepts `change` or `maintain` on its branch; a +later release accepts `maintain` only, so grant `maintain` to the accounts that +run it. The engine checks these actions whenever a policy is installed on the +graph handle; the direct maintenance commands load no policy (see +[maintenance](maintenance.md)). + ## Bind a policy ```yaml @@ -124,7 +137,7 @@ empty, has leading or trailing whitespace, or contains a control character fails policy validation, and a request naming such a branch is refused as a bad request before any rule is evaluated (see [branch names](../branching/index.md#branch-names)). Server actions and graph-wide -`invoke_query` rules do not take branch scopes. `commit_publish` and +`invoke_query` and `reclaim` rules do not take branch scopes. `commit_publish` and `commit_drop` take `target_branch_scope`: the scope is the branch the unpublished commit recorded as its target. Writing an unpublished commit needs no new action: a mutation `as commit` is a `change` on its branch and a @@ -211,7 +224,8 @@ authentication allows every graph action and reports `actor_id: null`. The MCP server target ([what you may do at a target](../cli/targets.md#what-you-may-do-at-a-target)). It reports policy actions, not actor views, so a view can still refuse a -`change` the policy allows. +`change` the policy allows. It does not report `maintain` or `reclaim`, since +no server route runs maintenance. ## Actor identity diff --git a/skills/omnigraph/references/server-policy.md b/skills/omnigraph/references/server-policy.md index 5eccf49d8..9486333b6 100644 --- a/skills/omnigraph/references/server-policy.md +++ b/skills/omnigraph/references/server-policy.md @@ -120,6 +120,8 @@ Graph-scoped actions are: | `schema_apply` | Schema changes: `cluster apply --target` needs it on each existing graph whose schema changes | | `branch_create`, `branch_delete`, `branch_merge` | Corresponding branch operation | | `commit_publish`, `commit_drop` | Publish or drop an unpublished commit; drop an unattached commit | +| `maintain` | Optimize, index builds, the full-text rebuild (which also accepts `change` for now) and the repair report | +| `reclaim` | Cleanup, which permanently removes unretained history | | `invoke_query` | Entry to a stored query | | `admin` | Reserved; no current public operation | @@ -165,12 +167,12 @@ rules: actions: [invoke_query] ``` -`branch_scope` is valid only with `read`, `export`, and `change`; +`branch_scope` is valid only with `read`, `export`, `change`, and `maintain`; `target_branch_scope` only with `branch_create`, `branch_delete`, `branch_merge`, `schema_apply`, `commit_publish`, and `commit_drop`. Any other pairing, or both scopes on one rule, is refused. Values are `any`, `protected`, or `unprotected`, where protected branches are the bundle's top-level `protected_branches: [main, …]` -list. `invoke_query`, `graph_list` and `config_manage` take no branch scope. +list. `invoke_query`, `reclaim`, `graph_list` and `config_manage` take no branch scope. ```bash omnigraph policy validate --target . --graph knowledge From bc0c45f0e9db9764cc8a50c904a9021f6046ee3d Mon Sep 17 00:00:00 2001 From: Ragnor Comerford Date: Mon, 26 Oct 2026 13:31:35 +0800 Subject: [PATCH 02/13] test(policy): match the refused load instead of unwrapping it --- crates/omnigraph-policy/src/lib.rs | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/crates/omnigraph-policy/src/lib.rs b/crates/omnigraph-policy/src/lib.rs index 2b2383b7f..c0ea6200a 100644 --- a/crates/omnigraph-policy/src/lib.rs +++ b/crates/omnigraph-policy/src/lib.rs @@ -2141,7 +2141,9 @@ rules: "actions: [reclaim]", &format!("actions: [reclaim]\n {scope}: any"), ); - let error = PolicyEngine::load_graph_from_source(&scoped, "graph").unwrap_err(); + let Err(error) = PolicyEngine::load_graph_from_source(&scoped, "graph") else { + panic!("{scope} on reclaim must be refused"); + }; assert!( error.to_string().contains("unsupported action 'reclaim'"), "{scope}: {error}" From 9b45173c81defaab9002ef56e514f895d2e1585e Mon Sep 17 00:00:00 2001 From: Ragnor Comerford Date: Mon, 26 Oct 2026 13:43:09 +0800 Subject: [PATCH 03/13] fix(policy): gate the coordinator import the test-only publisher uses --- crates/omnigraph/src/db/graph_coordinator.rs | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/crates/omnigraph/src/db/graph_coordinator.rs b/crates/omnigraph/src/db/graph_coordinator.rs index ae540613d..e5a0b74ca 100644 --- a/crates/omnigraph/src/db/graph_coordinator.rs +++ b/crates/omnigraph/src/db/graph_coordinator.rs @@ -12,8 +12,10 @@ use super::commit_graph::{ CommitGraph, FirstParentPage, GraphCommit, HistoryCache, Lineage, graph_commit_from_manifest_row, }; +#[cfg(any(test, feature = "failpoints"))] +use super::manifest::DatasetUpdate; use super::manifest::{ - BranchRecords, CapturedManifestProbe, CommitBuffer, DatasetUpdate, ExpectedTableVersions, + BranchRecords, CapturedManifestProbe, CommitBuffer, ExpectedTableVersions, GenesisManifestAttempt, GraphHeadExpectation, HistoryRecord, HistoryReleaseBytes, LineageIntent, ManifestChange, ManifestCoordinator, ManifestIncarnation, ManifestInitError, PublishPrecondition, SchemaContractRow, UnattachedCommit, UnattachedIntent, From 85e79c28466d32fd066d1aea59c7fbe9ea865c2f Mon Sep 17 00:00:00 2001 From: Ragnor Comerford Date: Thu, 29 Oct 2026 09:59:01 -0700 Subject: [PATCH 04/13] feat(engine)!: bound optimize by its gate hold and report per-table outcomes optimize_as takes an OptimizeRequest (tables, start_at, max_gate_hold, planning and rewrite concurrency) and returns an OptimizeReport. A full-catalog run visits tables in type-key order from start_at and wraps; a tables request works in its own order and reports no resume position. Bad requests are refused before any effect. Plans run ahead in order and rewrites start in visit order. Once the gate-hold deadline passes, every table whose turn has not come is deferred, the finished tables publish once, and resume_at names the first deferred table. The configuration clear runs under the gates and reports its own outcome; the adjacency artifact is built after every writer gate is released. Neither fails a run whose commit is durable. The CLI gains --max-gate-hold and --tables. optimize --json reports one outcome per table, the input and published commits, the gate hold and both tail outcomes; the __manifest row becomes configuration_clear and the always-null published_dataset_version and lance_head_version go. --- changelog.d/optimize-gate-hold-bound.added.md | 3 + .../optimize-report-outcomes.changed.md | 3 + .../optimize-takes-a-request.breaking.md | 8 + .../src/legacy/branch_merge_oracle.rs | 11 +- .../src/commands/maintenance/args.rs | 9 + .../src/commands/maintenance/mod.rs | 19 +- crates/omnigraph-cli/src/output.rs | 170 +++- crates/omnigraph-dst/src/concurrent.rs | 4 +- crates/omnigraph-dst/src/harness.rs | 4 +- crates/omnigraph-dst/tests/scenarios.rs | 14 +- crates/omnigraph/src/db/mod.rs | 26 +- crates/omnigraph/src/db/omnigraph.rs | 19 +- crates/omnigraph/src/db/omnigraph/optimize.rs | 797 ++++++++++++++---- crates/omnigraph/src/runtime_cache.rs | 4 +- crates/omnigraph/src/seams/catalog.rs | 1 + crates/omnigraph/tests/bound_storage.rs | 4 +- .../omnigraph/tests/detached_commit_matrix.rs | 19 +- crates/omnigraph/tests/failpoints.rs | 121 ++- crates/omnigraph/tests/forbidden_apis.rs | 2 +- crates/omnigraph/tests/integration/changes.rs | 16 +- .../tests/integration/composite_flow.rs | 6 +- .../omnigraph/tests/integration/end_to_end.rs | 8 +- .../omnigraph/tests/integration/engine_v2.rs | 4 +- .../tests/integration/engine_v2_memory.rs | 8 +- .../integration/engine_v2_plan_replay.rs | 4 +- .../tests/integration/maintenance.rs | 430 ++++++++-- .../tests/integration/traversal_adaptive.rs | 12 +- .../omnigraph/tests/integration/write_cost.rs | 23 +- .../omnigraph/tests/policy_engine_chassis.rs | 22 +- crates/omnigraph/tests/schema_apply.rs | 4 +- crates/omnigraph/tests/search.rs | 4 +- docs/user/operations/maintenance.md | 47 +- 32 files changed, 1436 insertions(+), 390 deletions(-) create mode 100644 changelog.d/optimize-gate-hold-bound.added.md create mode 100644 changelog.d/optimize-report-outcomes.changed.md create mode 100644 changelog.d/optimize-takes-a-request.breaking.md diff --git a/changelog.d/optimize-gate-hold-bound.added.md b/changelog.d/optimize-gate-hold-bound.added.md new file mode 100644 index 000000000..edd40f7c8 --- /dev/null +++ b/changelog.d/optimize-gate-hold-bound.added.md @@ -0,0 +1,3 @@ +- `optimize --max-gate-hold` bounds how long a run holds main's writer gates and names the table the next run resumes at; `--tables` runs only the listed tables. See [bounding a run][optimize-bound]. + +[optimize-bound]: ../docs/user/operations/maintenance.md#bounding-a-run diff --git a/changelog.d/optimize-report-outcomes.changed.md b/changelog.d/optimize-report-outcomes.changed.md new file mode 100644 index 000000000..160cd393e --- /dev/null +++ b/changelog.d/optimize-report-outcomes.changed.md @@ -0,0 +1,3 @@ +- `optimize` reports one outcome per table, both graph commits and the gate hold; a failed configuration clear or adjacency artifact no longer fails a published run. See [bounding a run][optimize-report]. + +[optimize-report]: ../docs/user/operations/maintenance.md#bounding-a-run diff --git a/changelog.d/optimize-takes-a-request.breaking.md b/changelog.d/optimize-takes-a-request.breaking.md new file mode 100644 index 000000000..3b28f3cf7 --- /dev/null +++ b/changelog.d/optimize-takes-a-request.breaking.md @@ -0,0 +1,8 @@ +- Embedded hosts: `optimize_as` takes an `OptimizeRequest` (`OptimizeRequest::default()` + is a full run) and returns an `OptimizeReport` whose `tables` holds the per-table + stats; `committed` and `skipped` are methods. `optimize --json` reports the + `__manifest` row as `configuration_clear` and drops the always-null + `published_dataset_version` and `lance_head_version`. See + [bounding a run][optimize-request-upgrade]. + +[optimize-request-upgrade]: ../docs/user/operations/maintenance.md#bounding-a-run diff --git a/crates/omnigraph-bench/src/legacy/branch_merge_oracle.rs b/crates/omnigraph-bench/src/legacy/branch_merge_oracle.rs index 78466e2be..e33438f05 100644 --- a/crates/omnigraph-bench/src/legacy/branch_merge_oracle.rs +++ b/crates/omnigraph-bench/src/legacy/branch_merge_oracle.rs @@ -28,7 +28,7 @@ pub async fn initialize_local_fixture( let optimized_user_tables = match plan.compaction_recency { CompactionRecency::Optimized => { let outcomes = db - .optimize_as(None) + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) .await .map_err(|error| fixture_error(format!("optimize fixture main branch: {error}")))?; let intended_keys = (0..plan.node_tables()) @@ -36,6 +36,7 @@ pub async fn initialize_local_fixture( .chain((0..plan.edge_tables()).map(edge_table_key)); for key in intended_keys { let outcome = outcomes + .tables .iter() .find(|outcome| outcome.type_key == key) .ok_or_else(|| { @@ -43,17 +44,17 @@ pub async fn initialize_local_fixture( "optimized fixture returned no outcome for intended user table {key}" )) })?; - if outcome.skipped.is_some() - || !outcome.committed + if outcome.skipped().is_some() + || !outcome.committed() || outcome.fragments_removed == 0 || outcome.fragments_added == 0 { return Err(fixture_error(format!( "optimized fixture did not productively compact {key}: committed={}, fragments_removed={}, fragments_added={}, skipped={:?}", - outcome.committed, + outcome.committed(), outcome.fragments_removed, outcome.fragments_added, - outcome.skipped + outcome.skipped() ))); } } diff --git a/crates/omnigraph-cli/src/commands/maintenance/args.rs b/crates/omnigraph-cli/src/commands/maintenance/args.rs index fb4f3aebf..3cc6c7e51 100644 --- a/crates/omnigraph-cli/src/commands/maintenance/args.rs +++ b/crates/omnigraph-cli/src/commands/maintenance/args.rs @@ -60,6 +60,15 @@ pub(crate) struct UpgradeArgs { #[derive(Debug, Args)] pub(crate) struct OptimizeArgs { + /// Only these tables, in this order: `node:` or `edge:`, + /// comma-separated or repeated. Default: every table on main. + #[arg(long, value_delimiter = ',')] + pub(crate) tables: Vec, + /// Stop starting tables once the run has held main's writer gates this + /// long, and report the rest deferred. The first table always runs. + /// Accepts Go-style durations: `30s`, `5m`. + #[arg(long)] + pub(crate) max_gate_hold: Option, #[arg(long)] pub(crate) json: bool, } diff --git a/crates/omnigraph-cli/src/commands/maintenance/mod.rs b/crates/omnigraph-cli/src/commands/maintenance/mod.rs index 8db8fe121..6ea603145 100644 --- a/crates/omnigraph-cli/src/commands/maintenance/mod.rs +++ b/crates/omnigraph-cli/src/commands/maintenance/mod.rs @@ -42,7 +42,20 @@ pub(crate) async fn run_optimize( stdout: &mut dyn Write, stderr: &mut dyn Write, ) -> Result { - let OptimizeArgs { json } = args; + let OptimizeArgs { + tables, + max_gate_hold, + json, + } = args; + let mut request = omnigraph::db::OptimizeRequest::default(); + if !tables.is_empty() { + request = request.with_tables(tables); + } + if let Some(hold) = max_gate_hold { + request = request.with_limits( + omnigraph::db::OptimizeLimits::default().with_max_gate_hold(parse_duration_arg(&hold)?), + ); + } let uri = crate::scope::direct_graph_uri(scope.target, scope.graph).await?; let binding = crate::scope::capture_storage_binding(scope.target)?.for_uri(&uri)?; let actor = resolve_actor(scope.operator, scope.actor, stderr)?; @@ -52,8 +65,8 @@ pub(crate) async fn run_optimize( omnigraph::storage::storage_for_binding(&binding)?, ) .await?; - let stats = db.optimize_as(actor.as_deref()).await?; - write_optimize(&uri, &stats, json, stdout)?; + let report = db.optimize_as(actor.as_deref(), request).await?; + write_optimize(&uri, &report, json, stdout)?; Ok(Completion::success()) } diff --git a/crates/omnigraph-cli/src/output.rs b/crates/omnigraph-cli/src/output.rs index 52ade6c11..3d002047b 100644 --- a/crates/omnigraph-cli/src/output.rs +++ b/crates/omnigraph-cli/src/output.rs @@ -17,21 +17,51 @@ pub(crate) fn graph_type_subject(type_key: &str) -> String { pub(crate) fn write_optimize( uri: &str, - stats: &[omnigraph::db::DatasetOptimizeStats], + report: &omnigraph::db::OptimizeReport, json: bool, stdout: &mut dyn Write, ) -> Result<()> { + use omnigraph::db::{ArtifactRefresh, ConfigurationClear, TableOptimizeOutcome}; + let stats = &report.tables; + let reason = |outcome: &TableOptimizeOutcome| match outcome { + TableOptimizeOutcome::Deferred(reason) => Some(reason.as_str()), + TableOptimizeOutcome::Skipped(reason) => Some(reason.as_str()), + _ => None, + }; + let clear_error = match &report.configuration_clear { + ConfigurationClear::Failed(error) | ConfigurationClear::Unknown(error) => Some(error), + _ => None, + }; + let (artifact_reason, artifact_error) = match &report.artifact { + ArtifactRefresh::Skipped(reason) => (Some(reason.as_str()), None), + ArtifactRefresh::Failed(error) => (None, Some(error)), + _ => (None, None), + }; if json { let value = serde_json::json!({ "uri": uri, + "input_commit": report.input_commit, + "graph_commit_id": report.graph_commit_id, + "resume_at": report.resume_at, + "gate_hold_ms": report.gate_hold.as_millis() as u64, + "configuration_clear": { + "outcome": report.configuration_clear.as_str(), + "error": clear_error, + }, + "artifact": { + "outcome": report.artifact.as_str(), + "reason": artifact_reason, + "error": artifact_error, + }, "datasets": stats.iter().map(|s| serde_json::json!({ "type_key": s.type_key, + "outcome": s.outcome.as_str(), + "reason": reason(&s.outcome), "fragments_removed": s.fragments_removed, "fragments_added": s.fragments_added, - "committed": s.committed, - "skipped": s.skipped.map(|r| r.as_str()), - "published_dataset_version": s.published_dataset_version, - "lance_head_version": s.lance_head_version, + "committed": s.committed(), + "skipped": s.skipped().map(|r| r.as_str()), + "duration_ms": s.duration.as_millis() as u64, "pending_indexes": s.pending_indexes.iter().map(|p| serde_json::json!({ "property": p.property, "reason": p.reason, @@ -43,16 +73,21 @@ pub(crate) fn write_optimize( writeln!(stdout, "optimize {} — {} datasets", uri, stats.len())?; for s in stats { let subject = graph_type_subject(&s.type_key); - if let Some(reason) = s.skipped { - writeln!(stdout, " {subject:<40} skipped ({reason})")?; - } else if s.committed { - writeln!( - stdout, - " {:<40} frags {} → {} ✓", - subject, s.fragments_removed, s.fragments_added - )?; - } else { - writeln!(stdout, " {subject:<40} no-op")?; + match s.outcome { + TableOptimizeOutcome::Skipped(reason) => { + writeln!(stdout, " {subject:<40} skipped ({reason})")?; + } + TableOptimizeOutcome::Deferred(reason) => { + writeln!(stdout, " {subject:<40} deferred ({reason})")?; + } + TableOptimizeOutcome::Optimized => { + writeln!( + stdout, + " {:<40} frags {} → {} ✓", + subject, s.fragments_removed, s.fragments_added + )?; + } + _ => writeln!(stdout, " {subject:<40} no-op")?, } for p in &s.pending_indexes { writeln!( @@ -62,6 +97,28 @@ pub(crate) fn write_optimize( )?; } } + if let Some(commit) = &report.graph_commit_id { + writeln!(stdout, "graph commit: {commit}")?; + } + if let Some(key) = &report.resume_at { + writeln!(stdout, "resume at: {}", graph_type_subject(key))?; + } + match clear_error { + Some(error) => writeln!( + stdout, + "configuration clear: {} ({error})", + report.configuration_clear.as_str() + )?, + None if report.configuration_clear == ConfigurationClear::Cleared => { + writeln!(stdout, "configuration clear: cleared")? + } + None => {} + } + match (&report.artifact, artifact_error) { + (_, Some(error)) => writeln!(stdout, "adjacency artifact: failed ({error})")?, + (ArtifactRefresh::Refreshed, _) => writeln!(stdout, "adjacency artifact: refreshed")?, + _ => {} + } } Ok(()) } @@ -1932,7 +1989,8 @@ mod tests { #[tokio::test] async fn maintenance_optimize_and_repair_reports_preserve_all_result_fields() { use omnigraph::db::{ - Omnigraph, PendingIndex, RepairAction, RepairClassification, SkipReason, + ArtifactRefresh, ArtifactSkip, ConfigurationClear, DeferReason, Omnigraph, + PendingIndex, RepairAction, RepairClassification, SkipReason, TableOptimizeOutcome, }; // These engine result types are non_exhaustive. Obtain one valid value // from a tiny graph, then set its public presentation fields; no engine @@ -1946,14 +2004,16 @@ mod tests { let db = Omnigraph::init(&uri, "node Person {\n slug: String @key\n}\n") .await .unwrap(); - let mut compacted = db.optimize_as(None).await.unwrap().pop().unwrap(); + let mut report = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); + let mut compacted = report.tables.pop().unwrap(); compacted.type_key = "node:Person".into(); compacted.fragments_removed = 3; compacted.fragments_added = 1; - compacted.committed = true; - compacted.skipped = None; - compacted.published_dataset_version = None; - compacted.lance_head_version = None; + compacted.outcome = TableOptimizeOutcome::Optimized; + compacted.duration = std::time::Duration::from_millis(12); compacted.pending_indexes = vec![PendingIndex { type_key: "node:Person".into(), property: "embedding".into(), @@ -1963,39 +2023,73 @@ mod tests { skipped.type_key = "edge:Knows".into(); skipped.fragments_removed = 0; skipped.fragments_added = 0; - skipped.committed = false; - skipped.skipped = Some(SkipReason::DriftNeedsRepair); - skipped.published_dataset_version = Some(4); - skipped.lance_head_version = Some(5); + skipped.outcome = TableOptimizeOutcome::Skipped(SkipReason::DriftNeedsRepair); + skipped.duration = std::time::Duration::from_millis(3); skipped.pending_indexes.clear(); + let mut deferred = skipped.clone(); + deferred.type_key = "node:Zebra".into(); + deferred.outcome = TableOptimizeOutcome::Deferred(DeferReason::GateHoldLimit); + deferred.duration = std::time::Duration::ZERO; let mut noop = skipped.clone(); - noop.type_key = "__manifest".into(); - noop.skipped = None; - noop.published_dataset_version = None; - noop.lance_head_version = None; - let optimize = [compacted, skipped, noop]; + noop.type_key = "node:Yak".into(); + noop.outcome = TableOptimizeOutcome::Unchanged; + report.tables = vec![compacted, skipped, noop, deferred]; + report.input_commit = Some("hb1.parent".into()); + report.graph_commit_id = Some("hb1.child".into()); + report.resume_at = Some("node:Zebra".into()); + report.gate_hold = std::time::Duration::from_millis(40); + report.configuration_clear = ConfigurationClear::Unknown("timed out".into()); + report.artifact = ArtifactRefresh::Skipped(ArtifactSkip::NoEdgeTypes); assert_maintenance_report( concat!( - "optimize graph — 3 datasets\n", + "optimize graph — 4 datasets\n", " node type 'Person' frags 3 → 1 ✓\n", " ↳ index pending on property 'embedding': no trainable vectors\n", " edge type 'Knows' skipped (published dataset/Lance HEAD drift — run omnigraph repair)\n", - " dataset '__manifest' no-op\n", + " node type 'Yak' no-op\n", + " node type 'Zebra' deferred (the run reached its writer-gate hold limit)\n", + "graph commit: hb1.child\n", + "resume at: node type 'Zebra'\n", + "configuration clear: unknown (timed out)\n", ), serde_json::json!({ "uri":"graph", + "input_commit":"hb1.parent", + "graph_commit_id":"hb1.child", + "resume_at":"node:Zebra", + "gate_hold_ms":40, + "configuration_clear":{"outcome":"unknown","error":"timed out"}, + "artifact":{"outcome":"skipped","reason":"no_edge_types","error":null}, "datasets":[ - {"type_key":"node:Person","fragments_removed":3,"fragments_added":1,"committed":true,"skipped":null,"published_dataset_version":null,"lance_head_version":null,"pending_indexes":[{"property":"embedding","reason":"no trainable vectors"}]}, - {"type_key":"edge:Knows","fragments_removed":0,"fragments_added":0,"committed":false,"skipped":"drift_needs_repair","published_dataset_version":4,"lance_head_version":5,"pending_indexes":[]}, - {"type_key":"__manifest","fragments_removed":0,"fragments_added":0,"committed":false,"skipped":null,"published_dataset_version":null,"lance_head_version":null,"pending_indexes":[]}, + {"type_key":"node:Person","outcome":"optimized","reason":null,"fragments_removed":3,"fragments_added":1,"committed":true,"skipped":null,"duration_ms":12,"pending_indexes":[{"property":"embedding","reason":"no trainable vectors"}]}, + {"type_key":"edge:Knows","outcome":"skipped","reason":"drift_needs_repair","fragments_removed":0,"fragments_added":0,"committed":false,"skipped":"drift_needs_repair","duration_ms":3,"pending_indexes":[]}, + {"type_key":"node:Yak","outcome":"unchanged","reason":null,"fragments_removed":0,"fragments_added":0,"committed":false,"skipped":null,"duration_ms":3,"pending_indexes":[]}, + {"type_key":"node:Zebra","outcome":"deferred","reason":"gate_hold_limit","fragments_removed":0,"fragments_added":0,"committed":false,"skipped":null,"duration_ms":0,"pending_indexes":[]}, ], }), - |json, writer| super::write_optimize("graph", &optimize, json, writer), + |json, writer| super::write_optimize("graph", &report, json, writer), ); + let mut empty = report.clone(); + empty.tables.clear(); + empty.input_commit = None; + empty.graph_commit_id = None; + empty.resume_at = None; + empty.gate_hold = std::time::Duration::ZERO; + empty.configuration_clear = ConfigurationClear::NothingToClear; + empty.artifact = ArtifactRefresh::Unchanged; assert_maintenance_report( "optimize graph — 0 datasets\n", - serde_json::json!({"uri":"graph","datasets":[]}), - |json, writer| super::write_optimize("graph", &[], json, writer), + serde_json::json!({ + "uri":"graph", + "input_commit":null, + "graph_commit_id":null, + "resume_at":null, + "gate_hold_ms":0, + "configuration_clear":{"outcome":"nothing_to_clear","error":null}, + "artifact":{"outcome":"unchanged","reason":null,"error":null}, + "datasets":[], + }), + |json, writer| super::write_optimize("graph", &empty, json, writer), ); let mut repair = db.repair_report_as(None).await.unwrap(); diff --git a/crates/omnigraph-dst/src/concurrent.rs b/crates/omnigraph-dst/src/concurrent.rs index 98fc5f188..efc0a7d5a 100644 --- a/crates/omnigraph-dst/src/concurrent.rs +++ b/crates/omnigraph-dst/src/concurrent.rs @@ -1754,7 +1754,9 @@ fn maintenance_life( loop { let result = match kind { 0 => Box::pin(db.ensure_indices_as(None)).await.map(|_| ()), - 1 => Box::pin(db.optimize_as(None)).await.map(|_| ()), + 1 => Box::pin(db.optimize_as(None, omnigraph::db::OptimizeRequest::default())) + .await + .map(|_| ()), _ => Box::pin(db.cleanup_as( CleanupPolicyOptions { keep_versions: Some(1), diff --git a/crates/omnigraph-dst/src/harness.rs b/crates/omnigraph-dst/src/harness.rs index 7ff131f9b..70546e3e1 100644 --- a/crates/omnigraph-dst/src/harness.rs +++ b/crates/omnigraph-dst/src/harness.rs @@ -1634,7 +1634,9 @@ async fn exec_op(db: &Session, branch: &str, op: &Op) -> OmniResult<()> { ) .await .map(|_| ()), - Op::Optimize => Box::pin(db.optimize_as(None)).await.map(|_| ()), + Op::Optimize => Box::pin(db.optimize_as(None, omnigraph::db::OptimizeRequest::default())) + .await + .map(|_| ()), Op::Cleanup => { let stats = Box::pin(db.cleanup_as( omnigraph::db::CleanupPolicyOptions { diff --git a/crates/omnigraph-dst/tests/scenarios.rs b/crates/omnigraph-dst/tests/scenarios.rs index 69ddf6619..c79a5fb1a 100644 --- a/crates/omnigraph-dst/tests/scenarios.rs +++ b/crates/omnigraph-dst/tests/scenarios.rs @@ -3669,7 +3669,9 @@ fn reborn_branch_cache_poison_body() { .expect("op1"); db.branch_create("b0").await.expect("op2"); world_read(&db).await; // ARMING READ 1 (the universe's op-2 verify) - db.optimize_as(None).await.expect("op3"); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .expect("op3"); m!( db, "b0", @@ -5204,7 +5206,7 @@ fn dst_classa_ablation_matrix_body() { db.branch_create("b0").await.expect("bc b0 life1"); } if o { - db.optimize_as(None).await.expect("optimize"); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()).await.expect("optimize"); } if r { upsert(&db, "b0", "w6", 45, 2).await; @@ -5291,7 +5293,7 @@ fn dst_classa_ablation_matrix_body() { m!(db, "main", "set_age_v", &[("$name", "w5")], &[("$age", 70), ("$ver", 1)]).expect("op0"); m!(db, "main", "insert_person", &[("$name", "w6")], &[("$age", 7)]).expect("op1"); db.branch_create("b0").await.expect("op2"); - db.optimize_as(None).await.expect("op3"); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()).await.expect("op3"); m!(db, "b0", "set_age_v", &[("$name", "w6")], &[("$age", 45), ("$ver", 2)]).expect("op4"); m!(db, "b0", "remove_friendships_from", &[("$from", "w6")], &[]).expect("op5"); Box::pin(db.branch_merge("b0", "main")).await.expect("op6 merge"); @@ -6345,7 +6347,11 @@ fn dst_optimize_races_branch_delete_minimal_two_thread_negative() { let _ = mutate_main(&db, MUTATION_QUERIES, "set_age", ¶ms).await; } - match Box::pin(db.optimize_as(None)).await { + match Box::pin( + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()), + ) + .await + { Ok(_) => {} Err(err) => { let rendered = format!("{err:?}"); diff --git a/crates/omnigraph/src/db/mod.rs b/crates/omnigraph/src/db/mod.rs index db4cd7dd1..efcccdd67 100644 --- a/crates/omnigraph/src/db/mod.rs +++ b/crates/omnigraph/src/db/mod.rs @@ -16,18 +16,20 @@ pub use commit_graph::{GraphCommit, GraphCommitKind}; pub use graph_coordinator::{ReadTarget, ResolvedTarget, SnapshotId}; pub use manifest::{DatasetEntry, DatasetUpdate}; pub use omnigraph::{ - BranchCreated, BuiltIndex, BuiltIndexAction, BuiltIndexKind, COMMIT_LIST_DEFAULT_PAGE_COMMITS, - COMMIT_LIST_MAX_PAGE_COMMITS, CleanupPolicyOptions, CollectorCost, CollectorPathSnapshot, - CollectorReport, CollectorRowSummary, CommitListPage, DatasetCleanupStats, - DatasetOptimizeStats, DatasetRepairStats, EXPORT_CHUNK_MAX_BYTES, ExportCut, - FullTextIndexRebuildResult, GraphCreateReconciliation, IndexBuildResult, InitOptions, - MergeOutcome, MergeResult, Omnigraph, OpenMode, PendingIndex, PreparedGraphCreate, - PreparedSchemaApply, PreparedSchemaSettlement, ReadCheck, RebuiltFullTextIndex, RepairAction, - RepairClassification, RepairStats, RetainedManifestVersions, SYSTEM_COLUMNS_PREFLIGHT, - SchemaApplyReconciliation, SchemaApplyResult, SchemaApplySettlement, SchemaContractDigest, - SchemaNonPublicationProof, SkipReason, StagingVerdict, SystemColumnUpgradeFinding, - SystemColumnUpgradeOptions, SystemColumnUpgradeOutcome, SystemColumnUpgradeReport, - TableCollectionPlan, UnattachedTagRecord, UnpublishedManifest, unattached_release_order, + ArtifactRefresh, ArtifactSkip, BranchCreated, BuiltIndex, BuiltIndexAction, BuiltIndexKind, + COMMIT_LIST_DEFAULT_PAGE_COMMITS, COMMIT_LIST_MAX_PAGE_COMMITS, CleanupPolicyOptions, + CollectorCost, CollectorPathSnapshot, CollectorReport, CollectorRowSummary, CommitListPage, + ConfigurationClear, DatasetCleanupStats, DatasetOptimizeStats, DatasetRepairStats, DeferReason, + EXPORT_CHUNK_MAX_BYTES, ExportCut, FullTextIndexRebuildResult, GraphCreateReconciliation, + IndexBuildResult, InitOptions, MergeOutcome, MergeResult, Omnigraph, OpenMode, + OptimizeConcurrency, OptimizeLimits, OptimizeReport, OptimizeRequest, PendingIndex, + PreparedGraphCreate, PreparedSchemaApply, PreparedSchemaSettlement, ReadCheck, + RebuiltFullTextIndex, RepairAction, RepairClassification, RepairStats, + RetainedManifestVersions, SYSTEM_COLUMNS_PREFLIGHT, SchemaApplyReconciliation, + SchemaApplyResult, SchemaApplySettlement, SchemaContractDigest, SchemaNonPublicationProof, + SkipReason, StagingVerdict, SystemColumnUpgradeFinding, SystemColumnUpgradeOptions, + SystemColumnUpgradeOutcome, SystemColumnUpgradeReport, TableCollectionPlan, + TableOptimizeOutcome, UnattachedTagRecord, UnpublishedManifest, unattached_release_order, }; pub(crate) use omnigraph::{ LogicalBlobValue, RangedExternalBlobs, logical_row_image, row_blob_values, diff --git a/crates/omnigraph/src/db/omnigraph.rs b/crates/omnigraph/src/db/omnigraph.rs index 1df164f05..ce9007f10 100644 --- a/crates/omnigraph/src/db/omnigraph.rs +++ b/crates/omnigraph/src/db/omnigraph.rs @@ -50,7 +50,11 @@ pub use export::{EXPORT_CHUNK_MAX_BYTES, ExportCut}; pub(crate) use export::{ LogicalBlobValue, RangedExternalBlobs, logical_row_image, row_blob_values, }; -pub use optimize::{CleanupPolicyOptions, DatasetCleanupStats, DatasetOptimizeStats, SkipReason}; +pub use optimize::{ + ArtifactRefresh, ArtifactSkip, CleanupPolicyOptions, ConfigurationClear, DatasetCleanupStats, + DatasetOptimizeStats, DeferReason, OptimizeConcurrency, OptimizeLimits, OptimizeReport, + OptimizeRequest, SkipReason, TableOptimizeOutcome, +}; use prepared_create::initial_schema_ir; pub use prepared_create::{GraphCreateReconciliation, PreparedGraphCreate}; pub use repair::{DatasetRepairStats, RepairAction, RepairClassification, RepairStats}; @@ -3598,15 +3602,16 @@ impl Omnigraph { .await } - /// Compact small Lance fragments into fewer larger ones across every - /// backing dataset for a node or edge type on `main`. See [`optimize`] for - /// details. Requires `maintain` on `main` before any effect and records - /// `actor` in the publication. + /// Compact small Lance fragments into fewer larger ones across the + /// backing datasets `request` selects on `main`, and catch up their + /// indexes. See [`optimize`] for details. Requires `maintain` on `main` + /// before any effect and records `actor` in the publication. pub async fn optimize_as( &self, actor: Option<&str>, - ) -> Result> { - optimize::optimize_all_datasets(self, actor).await + request: optimize::OptimizeRequest, + ) -> Result { + optimize::optimize_all_datasets(self, actor, request).await } /// Report, per table on `main`, how its Lance linear history stands diff --git a/crates/omnigraph/src/db/omnigraph/optimize.rs b/crates/omnigraph/src/db/omnigraph/optimize.rs index 74393615c..32cb7c536 100644 --- a/crates/omnigraph/src/db/omnigraph/optimize.rs +++ b/crates/omnigraph/src/db/omnigraph/optimize.rs @@ -23,10 +23,12 @@ //! to unreferenced version history — callers should gate this behind an //! explicit confirm flag at the CLI layer. +use std::collections::VecDeque; use std::time::Duration; -use futures::stream::StreamExt; +use futures::stream::{FuturesOrdered, FuturesUnordered, StreamExt}; use lance::dataset::optimize::{CompactionMetrics, CompactionOptions}; +use tokio::time::Instant; use super::*; use crate::seams::{decide_seam, fail}; @@ -92,50 +94,358 @@ impl std::fmt::Display for SkipReason { } } -/// Per-dataset outcome of `optimize_all_datasets`. This is a returned result type, +/// What one optimize run did to one table. A run reports exactly one outcome +/// for every table it considered. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[non_exhaustive] +pub enum TableOptimizeOutcome { + /// A rewrite or index link was staged and is in the run's publication. + Optimized, + /// Planning found nothing to do. + Unchanged, + /// Not started: nothing was written for the table. + Deferred(DeferReason), + /// Deliberately left alone. + Skipped(SkipReason), +} + +impl TableOptimizeOutcome { + /// Stable machine-readable token for serialized output. Once emitted this + /// is part of the output contract — keep it stable. + pub fn as_str(&self) -> &'static str { + match self { + TableOptimizeOutcome::Optimized => "optimized", + TableOptimizeOutcome::Unchanged => "unchanged", + TableOptimizeOutcome::Deferred(_) => "deferred", + TableOptimizeOutcome::Skipped(_) => "skipped", + } + } +} + +/// Why a run did not start a table. Typed so callers branch on the reason. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[non_exhaustive] +pub enum DeferReason { + /// The run had held main's writer gates for `max_gate_hold` when the + /// table's turn came. This table and every later one wait for a later + /// run, which [`OptimizeReport::resume_at`] starts at. + GateHoldLimit, +} + +impl DeferReason { + /// Stable machine-readable token for serialized output. Once emitted this + /// is part of the output contract — keep it stable. + pub fn as_str(&self) -> &'static str { + match self { + DeferReason::GateHoldLimit => "gate_hold_limit", + } + } +} + +impl std::fmt::Display for DeferReason { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str(match self { + DeferReason::GateHoldLimit => "the run reached its writer-gate hold limit", + }) + } +} + +/// Per-table outcome of `optimize_all_datasets`. This is a returned result type, /// not built by callers, so it is `#[non_exhaustive]`: future fields stay /// non-breaking and downstream code reads fields rather than constructing it. #[derive(Debug, Clone)] #[non_exhaustive] pub struct DatasetOptimizeStats { - /// Qualified graph type key, or `__manifest` for the system dataset. + /// Qualified graph type key. pub type_key: String, + /// What the run did to this table. + pub outcome: TableOptimizeOutcome, /// Number of source fragments that were rewritten by Lance. pub fragments_removed: usize, /// Number of new, larger fragments Lance produced. pub fragments_added: usize, - /// Whether this dataset advanced to a new Lance version that this run - /// published through the graph manifest. This may result from compaction, - /// index maintenance or materialization, or stale `auto_cleanup` removal. - pub committed: bool, - /// `Some(reason)` if this dataset was deliberately not compacted. When set, - /// `fragments_removed == 0`, `fragments_added == 0`, and `!committed`. - pub skipped: Option, - /// Published dataset version observed by optimize for drift skips. `None` for - /// normal compaction/no-op outcomes. - pub published_dataset_version: Option, - /// Lance HEAD version observed by optimize for drift skips. `None` for - /// normal compaction/no-op outcomes. - pub lance_head_version: Option, /// Index work deferred this run, with the reason and remedy: a vector /// property without trainable vectors, or full-text coverage requiring an /// explicit rebuild. Deferred work alone does not stage or publish. pub pending_indexes: Vec, + /// Time from the start of the table's planning to the end of its staging; + /// zero for a table the run did not start. + pub duration: Duration, } impl DatasetOptimizeStats { - /// Stat for a dataset that Lance actually compacted. - fn compacted(type_key: String, metrics: &CompactionMetrics, committed: bool) -> Self { + fn planned(type_key: String, metrics: &CompactionMetrics, committed: bool) -> Self { Self { type_key, + outcome: if committed { + TableOptimizeOutcome::Optimized + } else { + TableOptimizeOutcome::Unchanged + }, fragments_removed: metrics.fragments_removed, fragments_added: metrics.fragments_added, - committed, - skipped: None, - published_dataset_version: None, - lance_head_version: None, pending_indexes: Vec::new(), + duration: Duration::ZERO, + } + } + + fn deferred(type_key: String, reason: DeferReason) -> Self { + Self { + type_key, + outcome: TableOptimizeOutcome::Deferred(reason), + fragments_removed: 0, + fragments_added: 0, + pending_indexes: Vec::new(), + duration: Duration::ZERO, + } + } + + /// Whether this table advanced to a new Lance version that this run + /// published through the graph manifest: compaction, index maintenance or + /// materialization. + pub fn committed(&self) -> bool { + self.outcome == TableOptimizeOutcome::Optimized + } + + /// `Some(reason)` if this table was deliberately not compacted. + pub fn skipped(&self) -> Option { + match self.outcome { + TableOptimizeOutcome::Skipped(reason) => Some(reason), + _ => None, + } + } +} + +/// The configuration clear that follows a run's publication under the writer +/// gates: stale `lance.auto_cleanup.*` keys on `__manifest` are removed with +/// one config-only commit. It never fails the run; the next run clears keys +/// that are still present. +#[derive(Debug, Clone, PartialEq, Eq)] +#[non_exhaustive] +pub enum ConfigurationClear { + /// The keys were removed. + Cleared, + /// No key was present. + NothingToClear, + /// The keys are still present. + Failed(String), + /// Whether the removal committed has no definite answer. + Unknown(String), +} + +impl ConfigurationClear { + /// Stable machine-readable token for serialized output. Once emitted this + /// is part of the output contract — keep it stable. + pub fn as_str(&self) -> &'static str { + match self { + ConfigurationClear::Cleared => "cleared", + ConfigurationClear::NothingToClear => "nothing_to_clear", + ConfigurationClear::Failed(_) => "failed", + ConfigurationClear::Unknown(_) => "unknown", + } + } +} + +/// The adjacency artifact refresh, after the writer gates are released. The +/// artifact is derived state: a stale or missing one is rejected on load and +/// rebuilt in memory, so no outcome here changes an answer or fails the run. +#[derive(Debug, Clone, PartialEq, Eq)] +#[non_exhaustive] +pub enum ArtifactRefresh { + /// A new artifact was written. + Refreshed, + /// No edge table advanced and an artifact exists. + Unchanged, + /// Deliberately not built. + Skipped(ArtifactSkip), + /// The build or the write failed; traversals keep building in memory. + Failed(String), +} + +impl ArtifactRefresh { + /// Stable machine-readable token for serialized output. Once emitted this + /// is part of the output contract — keep it stable. + pub fn as_str(&self) -> &'static str { + match self { + ArtifactRefresh::Refreshed => "refreshed", + ArtifactRefresh::Unchanged => "unchanged", + ArtifactRefresh::Skipped(_) => "skipped", + ArtifactRefresh::Failed(_) => "failed", + } + } +} + +/// Why a run did not build the adjacency artifact. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[non_exhaustive] +pub enum ArtifactSkip { + /// The catalog declares no edge type. + NoEdgeTypes, + /// A declared edge type has no table yet, so the artifact has no complete + /// identity to stamp. + IncompleteEdgeTables, +} + +impl ArtifactSkip { + /// Stable machine-readable token for serialized output. Once emitted this + /// is part of the output contract — keep it stable. + pub fn as_str(&self) -> &'static str { + match self { + ArtifactSkip::NoEdgeTypes => "no_edge_types", + ArtifactSkip::IncompleteEdgeTables => "incomplete_edge_tables", + } + } +} + +/// What one optimize run did. A returned result type, so `#[non_exhaustive]`. +#[derive(Debug, Clone)] +#[non_exhaustive] +pub struct OptimizeReport { + /// main's graph head as the run captured it under the writer gates: the + /// parent of the commit the run publishes. + pub input_commit: Option, + /// The `__manifest` version of that head. + pub input_manifest_version: u64, + /// The graph commit the run published; `None` when it staged no table. + pub graph_commit_id: Option, + /// One entry per table the run considered, in the order it visited them. + pub tables: Vec, + /// The first table a full-catalog run deferred because of a run-wide + /// bound. A full-catalog run that passes it as `start_at` visits it + /// first. `None` when the run stopped on no bound, and for a `tables` + /// request, which neither reads nor moves the resume position. + pub resume_at: Option, + /// How long the run held main's writer gates, from acquisition to release. + pub gate_hold: Duration, + /// The configuration clear, under the gates after the publication. + pub configuration_clear: ConfigurationClear, + /// The adjacency artifact refresh, after the gates are released. + pub artifact: ArtifactRefresh, +} + +/// How many tables a run plans ahead of the ones it rewrites, and how many it +/// rewrites at once. Planning only reads. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct OptimizeConcurrency { + pub planning: usize, + pub rewrite: usize, +} + +impl OptimizeConcurrency { + pub fn new(planning: usize, rewrite: usize) -> Self { + Self { planning, rewrite } + } + + /// The direct path's setting: both from `OMNIGRAPH_MAINTENANCE_CONCURRENCY` + /// (default 8). + pub fn from_env() -> Self { + let concurrency = maint_concurrency(); + Self::new(concurrency, concurrency) + } +} + +/// Bounds on one run. An absent limit does not bound it. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +#[non_exhaustive] +pub struct OptimizeLimits { + /// Stop starting tables once the run has held main's writer gates this + /// long. The first table always starts, so every run makes progress; the + /// writer wait is at most this, plus that table, the publication and the + /// configuration clear. + pub max_gate_hold: Option, +} + +impl OptimizeLimits { + pub fn with_max_gate_hold(mut self, hold: Duration) -> Self { + self.max_gate_hold = Some(hold); + self + } +} + +/// What one optimize run is asked to do. The default is the direct run as it +/// has always been: every table on main, no bound, and both concurrencies from +/// `OMNIGRAPH_MAINTENANCE_CONCURRENCY`. +#[derive(Debug, Clone, Default)] +#[non_exhaustive] +pub struct OptimizeRequest { + /// Only these table keys (`node:`, `edge:`), in this order. + /// A key that names no declared type is refused before any effect. + pub tables: Option>, + /// For a full-catalog run: start at this table key, or at the first key + /// after it when no table has it, and wrap around, so the run visits every + /// table at most once. + pub start_at: Option, + pub limits: OptimizeLimits, + /// `None` reads [`OptimizeConcurrency::from_env`]. + pub concurrency: Option, +} + +impl OptimizeRequest { + pub fn with_tables(mut self, tables: I) -> Self + where + I: IntoIterator, + S: Into, + { + self.tables = Some(tables.into_iter().map(Into::into).collect()); + self + } + + pub fn with_start_at(mut self, table_key: impl Into) -> Self { + self.start_at = Some(table_key.into()); + self + } + + pub fn with_limits(mut self, limits: OptimizeLimits) -> Self { + self.limits = limits; + self + } + + pub fn with_concurrency(mut self, concurrency: OptimizeConcurrency) -> Self { + self.concurrency = Some(concurrency); + self + } + + /// The tables this request visits, in order, out of the catalog's sorted + /// keys. Refuses a malformed request before any effect. + fn visit_order(&self, table_keys: &[String]) -> Result> { + if let Some(concurrency) = self.concurrency + && (concurrency.planning == 0 || concurrency.rewrite == 0) + { + return Err(OmniError::manifest( + "optimize concurrency must be at least 1 for planning and rewriting", + )); } + let Some(tables) = &self.tables else { + let start = self + .start_at + .as_ref() + .map_or(0, |start| table_keys.partition_point(|key| key < start)); + let mut order = table_keys[start..].to_vec(); + order.extend_from_slice(&table_keys[..start]); + return Ok(order); + }; + if self.start_at.is_some() { + return Err(OmniError::manifest( + "optimize takes a start position only for a full-catalog run, not with a table list", + )); + } + if tables.is_empty() { + return Err(OmniError::manifest("optimize table list is empty")); + } + let mut seen = std::collections::BTreeSet::new(); + for table in tables { + if table_keys.binary_search(table).is_err() { + return Err(OmniError::manifest(format!( + "optimize table '{table}' names no node or edge type" + ))); + } + if !seen.insert(table.as_str()) { + return Err(OmniError::manifest(format!( + "optimize table '{table}' is listed twice" + ))); + } + } + Ok(tables.clone()) } } @@ -208,21 +518,30 @@ decide_seam! { pub static OPTIMIZE_POST_AUTHORITY_CAPTURE_PRE_GATES = ("optimize.post_authority_capture_pre_gates", Unreachable, [Fail]); } -/// Run Lance maintenance across every node + edge dataset on `main` under one -/// graph visibility envelope. Physical dataset work remains bounded-parallel, -/// but every productive dataset stages detached and the batch publishes once -/// with an exact pin CAS, so one public Optimize produces at most one graph -/// commit. The publisher owns main's `__manifest` fragment layout; maintenance -/// only removes obsolete automatic-cleanup configuration from that dataset. +/// Run Lance maintenance across the node + edge datasets on `main` that +/// `request` selects, under one graph visibility envelope. Tables are visited +/// in order: up to `concurrency.planning` are planned ahead of the ones being +/// rewritten, rewrites start in visit order, and every productive dataset +/// stages detached. The batch publishes once with an exact pin CAS, so one +/// public Optimize produces at most one graph commit. Once the run has held +/// the writer gates for `max_gate_hold`, it starts no further table and +/// reports the rest deferred. The publisher owns main's `__manifest` fragment +/// layout; maintenance only removes obsolete automatic-cleanup configuration +/// from that dataset, under the gates, and refreshes the adjacency artifact +/// after releasing them. pub async fn optimize_all_datasets( db: &Omnigraph, actor: Option<&str>, -) -> Result> { + request: OptimizeRequest, +) -> Result { db.enforce( omnigraph_policy::PolicyAction::Maintain, &omnigraph_policy::ResourceScope::Branch("main".to_string()), actor, )?; + let concurrency = request + .concurrency + .unwrap_or_else(OptimizeConcurrency::from_env); let _export_exclusion = db.reserve_export_destructive_control()?; db.ensure_schema_state_valid().await?; @@ -239,26 +558,28 @@ pub async fn optimize_all_datasets( let schema_permit = db.write_queue().acquire_schema_shared().await; db.refresh_coordinator_only().await?; let catalog = db.load_accepted_catalog_with_schema_gate_held().await?; + let table_keys = all_table_keys(&catalog); + let visit_order = request + .visit_order(&table_keys) + .map_err(OmniError::before_effect)?; + let full_catalog = request.tables.is_none(); // Optimize's one visibility point advances main's graph head, so its // authority is branch-wide even though physical effects are table-local. - let _main_branch_guard = db.write_queue().acquire_branch(None).await; - - let table_keys = all_table_keys(&catalog); + let main_branch_guard = db.write_queue().acquire_branch(None).await; let queue_keys = table_keys .iter() .map(|table_key| (table_key.clone(), None)) .collect::>(); let table_guards = db.write_queue().acquire_many(&queue_keys).await; + let gates_acquired = Instant::now(); let snapshot = db.revalidate_write_txn(&authority_txn).await?; - - // Whether this run advanced any edge table — consumed by the graph-index - // artifact gate at the tail (a no-work optimize skips the rebuild+PUT). - let mut edge_tables_committed = false; + let input_commit = authority_txn.authority.graph_head.clone(); + let input_manifest_version = authority_txn.base.graph_manifest_version(); let witness = authority_txn.authority.staging_witness()?; - let table_tasks = table_keys + let table_tasks = visit_order .into_iter() .filter_map(|table_key| { let entry = snapshot.dataset(&table_key)?; @@ -273,103 +594,220 @@ pub async fn optimize_all_datasets( }) .collect::>(); - let concurrency = maint_concurrency().min(table_tasks.len()).max(1); - - let preparations: Vec> = futures::stream::iter(table_tasks) - .map(|task| { - let catalog = std::sync::Arc::clone(&catalog); - async move { prepare_optimize_table(db, catalog.as_ref(), task).await } - }) - .buffer_unordered(concurrency) - .collect() - .await; - - let mut prepared = Vec::new(); - let mut stats = Vec::new(); - for preparation in preparations { - match preparation? { - OptimizePreparation::Work(work) => prepared.push(work), - OptimizePreparation::Stat(stat) => stats.push(stat), + let deadline = request + .limits + .max_gate_hold + .map(|hold| gates_acquired + hold); + let visited = run_table_pipeline(db, &catalog, table_tasks, concurrency, deadline).await?; + + let mut updates = Vec::new(); + let mut expected_versions = crate::db::manifest::ExpectedTableVersions::new(); + let mut tables = Vec::with_capacity(visited.tables.len()); + for (stat, effect) in visited.tables { + if let Some(effect) = effect { + expected_versions.insert( + effect.update.identity, + crate::db::manifest::TableVersionExpectation { + table_key: effect.update.type_key.clone(), + table_version: effect.expected_version, + native_ref: crate::db::manifest::NativeRefPin::Exact(None), + }, + ); + updates.push(effect.update); } + tables.push(stat); } - prepared.sort_by(|left, right| left.table_key.cmp(&right.table_key)); - - if !prepared.is_empty() { - let effect_concurrency = maint_concurrency().min(prepared.len()).max(1); - let effect_results: Vec> = futures::stream::iter(prepared) - .map(|work| { - let catalog = std::sync::Arc::clone(&catalog); - async move { apply_optimize_table_effects(db, catalog.as_ref(), work).await } - }) - .buffer_unordered(effect_concurrency) - .collect() - .await; - let mut outcomes = Vec::new(); - for result in effect_results { - outcomes.push(result?); - } - fail(&OPTIMIZE_POST_PHASE_B_PRE_MANIFEST_COMMIT)?; - let mut updates = Vec::new(); - let mut expected_versions = crate::db::manifest::ExpectedTableVersions::new(); - for outcome in &mut outcomes { - if let Some(effect) = outcome.effect.take() { - expected_versions.insert( - effect.update.identity, - crate::db::manifest::TableVersionExpectation { - table_key: effect.update.type_key.clone(), - table_version: effect.expected_version, - native_ref: crate::db::manifest::NativeRefPin::Exact(None), - }, - ); - updates.push(effect.update); - } - } - let any_committed = !updates.is_empty(); - let edge_committed = updates - .iter() - .any(|update| update.type_key.starts_with("edge:")); - if any_committed { - let lineage = db - .new_lineage_intent_for_branch(None, actor, HistoryReleaseBytes::PRODUCTION) - .await?; - super::table_ops::commit_updates_on_branch_with_expected( - db, - None, - &updates, - &expected_versions, - actor, - &authority_txn, - lineage, - super::table_ops::Publication::Publish, - ) - .await?; - } - edge_tables_committed = edge_committed; - stats.extend(outcomes.into_iter().map(|outcome| outcome.stat)); - if any_committed { - db.runtime_cache.invalidate_all().await; - if edge_committed { - db.invalidate_graph_index().await; - } + let edge_tables_committed = updates + .iter() + .any(|update| update.type_key.starts_with("edge:")); + let mut graph_commit_id = None; + if !updates.is_empty() { + fail(&OPTIMIZE_POST_PHASE_B_PRE_MANIFEST_COMMIT).map_err(OmniError::unpublished)?; + let lineage = db + .new_lineage_intent_for_branch(None, actor, HistoryReleaseBytes::PRODUCTION) + .await + .map_err(OmniError::unpublished)?; + let commit = super::table_ops::commit_updates_on_branch_with_expected( + db, + None, + &updates, + &expected_versions, + actor, + &authority_txn, + lineage, + super::table_ops::Publication::Publish, + ) + .await?; + graph_commit_id = Some(commit.graph_commit_id); + db.runtime_cache.invalidate_all().await; + if edge_tables_committed { + db.invalidate_graph_index().await; } } - stats.sort_by(|left, right| left.type_key.cmp(&right.type_key)); + // The graph commit is durable from here: nothing below fails the run. + let configuration_clear = clear_manifest_auto_cleanup(db).await; + let gate_hold = gates_acquired.elapsed(); drop(table_guards); + drop(main_branch_guard); drop(schema_permit); - let mut all = stats.into_iter().map(Ok).collect::>>(); - all.push(clear_manifest_auto_cleanup(db).await); + let artifact = match persist_graph_index_artifact(db, &catalog, edge_tables_committed).await { + Ok(artifact) => artifact, + Err(error) => { + tracing::warn!( + target: "omnigraph::optimize", + error = %error, + "graph index artifact persist failed; traversals keep building in memory" + ); + ArtifactRefresh::Failed(error.to_string()) + } + }; + + Ok(OptimizeReport { + input_commit, + input_manifest_version, + graph_commit_id, + tables, + resume_at: visited.resume_at.filter(|_| full_catalog), + gate_hold, + configuration_clear, + artifact, + }) +} - if let Err(error) = persist_graph_index_artifact(db, &catalog, edge_tables_committed).await { - tracing::warn!( - target: "omnigraph::optimize", - error = %error, - "graph index artifact persist failed; traversals keep building in memory" - ); +/// The tables a run visited, in visit order, each with its staged effect if +/// it has one, and the first table a run-wide bound deferred. +struct VisitedTables { + tables: Vec<(DatasetOptimizeStats, Option)>, + resume_at: Option, +} + +type PlannedTable = (usize, Instant, Result); + +/// Plan and stage `tasks` in order under the held gates. Plans run up to +/// `concurrency.planning` tables ahead; a table's rewrite starts only after +/// every earlier table's has, with at most `concurrency.rewrite` at once. +/// When a table's turn comes after `deadline`, that table and every later +/// one are deferred and their plans, which only read, are dropped; the first +/// table always starts. A failure fails the run before its publication: +/// `BeforeEffect` until a table's staging starts, `Unpublished` after. +async fn run_table_pipeline( + db: &Omnigraph, + catalog: &std::sync::Arc, + tasks: Vec, + concurrency: OptimizeConcurrency, + deadline: Option, +) -> Result { + let keys = tasks + .iter() + .map(|task| task.table_key.clone()) + .collect::>(); + let mut visited: Vec)>> = + (0..keys.len()).map(|_| None).collect(); + let plan = |index: usize, task: OptimizeTableTask| { + let catalog = std::sync::Arc::clone(catalog); + async move { + let started = Instant::now(); + let result = prepare_optimize_table(db, catalog.as_ref(), task).await; + (index, started, result) + } + }; + let stage = |index: usize, started: Instant, work: PreparedOptimizeTable| { + let catalog = std::sync::Arc::clone(catalog); + async move { + let result = apply_optimize_table_effects(db, catalog.as_ref(), work).await; + (index, started, result) + } + }; + + let mut unplanned = tasks.into_iter().enumerate(); + let mut planning = FuturesOrdered::new(); + let mut ready: VecDeque = VecDeque::new(); + let mut staging = FuturesUnordered::new(); + let mut turns = 0usize; + let mut stopped_at = None; + let mut staged_any = false; + let label = |error: OmniError, staged_any: bool| { + if staged_any { + error.unpublished() + } else { + error.before_effect() + } + }; + + loop { + if stopped_at.is_none() { + while planning.len() + ready.len() < concurrency.planning { + let Some((index, task)) = unplanned.next() else { + break; + }; + planning.push_back(plan(index, task)); + } + while staging.len() < concurrency.rewrite { + let Some((index, started, result)) = ready.pop_front() else { + break; + }; + if turns > 0 && deadline.is_some_and(|deadline| Instant::now() >= deadline) { + stopped_at = Some(index); + break; + } + turns += 1; + match result.map_err(|error| label(error, staged_any))? { + OptimizePreparation::Stat(mut stat) => { + stat.duration = started.elapsed(); + visited[index] = Some((stat, None)); + } + OptimizePreparation::Work(work) => { + staged_any = true; + staging.push(stage(index, started, work)); + } + } + } + if stopped_at.is_some() { + planning.clear(); + ready.clear(); + } + } + // Filling planning above leaves it non-empty while any table is + // unplanned, so an empty pipeline has visited every table or stopped. + if planning.is_empty() && ready.is_empty() && staging.is_empty() { + break; + } + // The deadline stops only a table whose turn has not come. + let timer = deadline.filter(|_| turns > 0 && turns < keys.len() && stopped_at.is_none()); + tokio::select! { + biased; + Some((index, started, result)) = staging.next(), if !staging.is_empty() => { + let mut outcome = result.map_err(OmniError::unpublished)?; + outcome.stat.duration = started.elapsed(); + visited[index] = Some((outcome.stat, outcome.effect)); + } + Some(planned) = planning.next(), if !planning.is_empty() && stopped_at.is_none() => { + ready.push_back(planned); + } + () = tokio::time::sleep_until(timer.unwrap_or_else(Instant::now)), if timer.is_some() => { + stopped_at = Some(turns); + planning.clear(); + ready.clear(); + } + } } - all.into_iter().collect() + let resume_at = stopped_at.map(|index| keys[index].clone()); + let tables = visited + .into_iter() + .zip(keys) + .map(|(entry, key)| { + entry.unwrap_or_else(|| { + ( + DatasetOptimizeStats::deferred(key, DeferReason::GateHoldLimit), + None, + ) + }) + }) + .collect(); + Ok(VisitedTables { tables, resume_at }) } /// Build the full-catalog graph index from a fresh main snapshot and write it @@ -387,14 +825,15 @@ pub async fn optimize_all_datasets( /// with no edge writes is likewise rewritten only by the next edge-committing /// optimize — the gate cannot cheaply see inside the object (a bounded read /// of the whole body is the only primitive; a ranged header read is the -/// planned follow-up). +/// planned follow-up). The caller holds no writer gate: the build reads one +/// fresh snapshot, and the artifact's stamps name the table versions it read. async fn persist_graph_index_artifact( db: &Omnigraph, catalog: &omnigraph_compiler::catalog::Catalog, edge_tables_committed: bool, -) -> Result<()> { +) -> Result { if catalog.edge_types.is_empty() { - return Ok(()); + return Ok(ArtifactRefresh::Skipped(ArtifactSkip::NoEdgeTypes)); } let uri = crate::graph_index::persist::artifact_uri(db.root_uri()); if !edge_tables_committed && db.storage_adapter().exists(&uri).await? { @@ -402,7 +841,7 @@ async fn persist_graph_index_artifact( target: "omnigraph::optimize", "graph index artifact refresh skipped: no edge table advanced and an artifact exists" ); - return Ok(()); + return Ok(ArtifactRefresh::Unchanged); } let edge_types: std::collections::HashMap = catalog .edge_types @@ -420,7 +859,7 @@ async fn persist_graph_index_artifact( target: "omnigraph::optimize", "graph index artifact skipped: not every declared edge table is materialized yet" ); - return Ok(()); + return Ok(ArtifactRefresh::Skipped(ArtifactSkip::IncompleteEdgeTables)); } let index = crate::graph_index::GraphIndex::build(&snapshot, &edge_types, db.catalog().system_columns) @@ -428,19 +867,20 @@ async fn persist_graph_index_artifact( let written = crate::graph_index::persist::save(&snapshot, db.storage_adapter(), &edge_types, &index) .await?; - if let Some(uri) = written { - // A fresh artifact may serve scope keys whose earlier load attempt - // failed; those negative verdicts are stamp-keyed, and when this - // optimize advanced no table version the stamps (hence keys) are - // unchanged — drop them so the new object gets loaded. - db.runtime_cache.note_artifact_replaced().await; - tracing::debug!( - target: "omnigraph::optimize", - uri = %uri, - "graph index artifact persisted" - ); - } - Ok(()) + let Some(uri) = written else { + return Ok(ArtifactRefresh::Skipped(ArtifactSkip::IncompleteEdgeTables)); + }; + // A fresh artifact may serve scope keys whose earlier load attempt + // failed; those negative verdicts are stamp-keyed, and when this + // optimize advanced no table version the stamps (hence keys) are + // unchanged — drop them so the new object gets loaded. + db.runtime_cache.note_artifact_replaced().await; + tracing::debug!( + target: "omnigraph::optimize", + uri = %uri, + "graph index artifact persisted" + ); + Ok(ArtifactRefresh::Refreshed) } /// Pure planning: classify drift/no-work/productive state without advancing @@ -472,7 +912,7 @@ async fn prepare_optimize_table( .await?; if !will_compact && !needs_reindex && !index_work.needs_commit { let mut stat = - DatasetOptimizeStats::compacted(task.table_key, &CompactionMetrics::default(), false); + DatasetOptimizeStats::planned(task.table_key, &CompactionMetrics::default(), false); stat.pending_indexes = index_work.pending; super::table_ops::append_deferred_full_text_indexes( &snapshot, @@ -591,7 +1031,7 @@ async fn apply_optimize_table_effects( fail(&OPTIMIZE_POST_TABLE_EFFECT)?; } let Some(identity) = tip_identity else { - let mut stat = DatasetOptimizeStats::compacted(table_key, &metrics, false); + let mut stat = DatasetOptimizeStats::planned(table_key, &metrics, false); stat.pending_indexes = index_work.pending; super::table_ops::append_deferred_full_text_indexes( &tip, @@ -603,7 +1043,7 @@ async fn apply_optimize_table_effects( .await?; return Ok(OptimizeEffectOutcome { stat, effect: None }); }; - let mut stat = DatasetOptimizeStats::compacted(table_key.clone(), &metrics, true); + let mut stat = DatasetOptimizeStats::planned(table_key.clone(), &metrics, true); stat.pending_indexes = index_work.pending; super::table_ops::append_deferred_full_text_indexes( &tip, @@ -668,35 +1108,72 @@ async fn clear_stale_auto_cleanup_config( Ok(true) } -/// Remove stored automatic GC configuration without changing manifest fragments. -async fn clear_manifest_auto_cleanup(db: &Omnigraph) -> Result { +decide_seam! { + /// After the run's publication, before the configuration clear reads + /// `__manifest`. A failure here must leave the published run reported, + /// with the clear `failed`. + pub static OPTIMIZE_CONFIGURATION_CLEAR = ("optimize.configuration_clear", Unreachable, [Fail]); +} + +/// Remove stored automatic GC configuration without changing manifest +/// fragments, and report what happened. A failed removal reads `__manifest` +/// again: keys still present are `Failed`, keys gone are `Cleared`, and a +/// read that fails too is `Unknown`. Nothing here fails the run. +async fn clear_manifest_auto_cleanup(db: &Omnigraph) -> ConfigurationClear { let uri = crate::db::manifest::manifest_uri(db.root_uri()); + let mut last_error = None; for attempt in 0..INTERNAL_CONFIG_RETRY_BUDGET { - let handle = db.storage().open_dataset_head(&uri, None).await?; - let mut ds = handle.into_dataset(); + if let Err(error) = fail(&OPTIMIZE_CONFIGURATION_CLEAR) { + return ConfigurationClear::Failed(error.to_string()); + } + let mut ds = match db.storage().open_dataset_head(&uri, None).await { + Ok(handle) => handle.into_dataset(), + Err(error) => return ConfigurationClear::Failed(error.to_string()), + }; match clear_stale_auto_cleanup_config(&mut ds).await { - Ok(cleared) => { - if cleared { - db.coordinator.write().await.refresh().await?; + Ok(false) => return ConfigurationClear::NothingToClear, + Ok(true) => { + if let Err(error) = db.coordinator.write().await.refresh().await { + tracing::warn!( + target: "omnigraph::optimize", + error = %error, + "coordinator refresh after the configuration clear failed" + ); } - return Ok(DatasetOptimizeStats::compacted( - "__manifest".to_string(), - &CompactionMetrics::default(), - false, - )); + return ConfigurationClear::Cleared; } Err(error) if attempt + 1 < INTERNAL_CONFIG_RETRY_BUDGET && is_retryable_lance_conflict(&error) => { + last_error = Some(error); continue; } - Err(error) => return Err(OmniError::storage(error)), + Err(error) => { + last_error = Some(error); + break; + } + } + } + let message = last_error.map_or_else( + || format!("removal exhausted {INTERNAL_CONFIG_RETRY_BUDGET} retries"), + |error| error.to_string(), + ); + match db.storage().open_dataset_head(&uri, None).await { + Ok(handle) => { + let present = handle + .dataset() + .config() + .keys() + .any(|key| key.starts_with("lance.auto_cleanup.")); + if present { + ConfigurationClear::Failed(message) + } else { + ConfigurationClear::Cleared + } } + Err(error) => ConfigurationClear::Unknown(format!("{message}; read-back failed: {error}")), } - Err(OmniError::manifest_conflict(format!( - "manifest auto-cleanup configuration removal exhausted {INTERNAL_CONFIG_RETRY_BUDGET} retries" - ))) } decide_seam! { diff --git a/crates/omnigraph/src/runtime_cache.rs b/crates/omnigraph/src/runtime_cache.rs index 3e3c00b6e..b17466183 100644 --- a/crates/omnigraph/src/runtime_cache.rs +++ b/crates/omnigraph/src/runtime_cache.rs @@ -648,7 +648,9 @@ edge Likes: Person -> Person {} db.load_jsonl(DATA, crate::loader::LoadMode::Overwrite) .await .unwrap(); - db.optimize_as(None).await.unwrap(); + db.optimize_as(None, crate::db::OptimizeRequest::default()) + .await + .unwrap(); // Fresh handle: cold in-memory caches, the artifact stays on the store. let db = crate::db::Omnigraph::open(dir.path().to_str().unwrap()) .await diff --git a/crates/omnigraph/src/seams/catalog.rs b/crates/omnigraph/src/seams/catalog.rs index df637aa8f..50a621b12 100644 --- a/crates/omnigraph/src/seams/catalog.rs +++ b/crates/omnigraph/src/seams/catalog.rs @@ -50,6 +50,7 @@ omnigraph_seams::catalog! { crate::db::omnigraph::optimize::CLEANUP_POST_UNATTACHED_RELEASE, crate::db::omnigraph::optimize::CLEANUP_TABLE_GC, crate::db::omnigraph::optimize::OPTIMIZE_BEFORE_COMPACT, + crate::db::omnigraph::optimize::OPTIMIZE_CONFIGURATION_CLEAR, crate::db::omnigraph::optimize::OPTIMIZE_POST_AUTHORITY_CAPTURE_PRE_GATES, crate::db::omnigraph::optimize::OPTIMIZE_POST_TABLE_EFFECT, crate::db::omnigraph::optimize::OPTIMIZE_POST_PHASE_B_PRE_MANIFEST_COMMIT, diff --git a/crates/omnigraph/tests/bound_storage.rs b/crates/omnigraph/tests/bound_storage.rs index a4f989498..7b2bdefe3 100644 --- a/crates/omnigraph/tests/bound_storage.rs +++ b/crates/omnigraph/tests/bound_storage.rs @@ -303,7 +303,9 @@ async fn bound_handle_workload_child() { .contains("Feature") ); - db.optimize_as(None).await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); db.cleanup_as( CleanupPolicyOptions { keep_versions: Some(2), diff --git a/crates/omnigraph/tests/detached_commit_matrix.rs b/crates/omnigraph/tests/detached_commit_matrix.rs index 2f7e9dc31..daca0a1a7 100644 --- a/crates/omnigraph/tests/detached_commit_matrix.rs +++ b/crates/omnigraph/tests/detached_commit_matrix.rs @@ -440,7 +440,10 @@ fn rfc0067_matrix_child_process() { .unwrap(); db.apply_prepared_schema_as(&intent, None).await.map(|_| ()) } - "optimize" => db.optimize_as(None).await.map(|_| ()), + "optimize" => db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .map(|_| ()), "load" => load_two(&db, &name).await, "fts_rebuild" => db .rebuild_full_text_indices_on_as("main", None) @@ -755,7 +758,10 @@ async fn run_cell( .apply_prepared_schema_as(prepared.as_ref().unwrap(), None) .await .map(|_| ()), - Writer::Optimize => db.optimize_as(None).await.map(|_| ()), + Writer::Optimize => db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .map(|_| ()), Writer::Load => load_two(&db, &write_name).await, Writer::FtsRebuild => db .rebuild_full_text_indices_on_as("main", None) @@ -985,9 +991,12 @@ async fn run_cell( if writer == Writer::Optimize && recovery != Recovery::ReadOnly { let fresh = helpers::session(Omnigraph::open(&root).await.unwrap()); for run in 1..=2 { - fresh.optimize_as(None).await.unwrap_or_else(|error| { - panic!("{cell}: optimize run {run} after recovery failed: {error}") - }); + fresh + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap_or_else(|error| { + panic!("{cell}: optimize run {run} after recovery failed: {error}") + }); assert_pin_detached_and_head_unmoved(&cell, &fresh, &person_uri, head_before).await; } drop(fresh); diff --git a/crates/omnigraph/tests/failpoints.rs b/crates/omnigraph/tests/failpoints.rs index 66da0aea0..87f2e61ca 100644 --- a/crates/omnigraph/tests/failpoints.rs +++ b/crates/omnigraph/tests/failpoints.rs @@ -4487,7 +4487,10 @@ async fn optimize_pre_publish_failure_leaves_no_residue() { .version; { let _failpoint = catalog::OPTIMIZE_POST_PHASE_B_PRE_MANIFEST_COMMIT.fire_always(); - let err = db.optimize_as(None).await.unwrap_err(); + let err = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap_err(); assert!( err.to_string().contains( "injected failpoint triggered: optimize.post_phase_b_pre_manifest_commit" @@ -4514,11 +4517,12 @@ async fn optimize_pre_publish_failure_leaves_no_residue() { assert_eq!(helpers::count_rows(&db, "node:Person").await, 4); assert_eq!(helpers::count_rows(&db, "node:Company").await, 4); let stats = db - .optimize_as(None) + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) .await - .expect("the retry re-plans from the pins"); + .expect("the retry re-plans from the pins") + .tables; assert!( - stats.iter().any(|stat| stat.committed), + stats.iter().any(|stat| stat.committed()), "the retry compacts what the failed attempt staged again" ); assert_person_pin_detached(&db, "main", person_head).await; @@ -4531,6 +4535,50 @@ async fn optimize_pre_publish_failure_leaves_no_residue() { .expect("schema apply after optimize must succeed"); } +/// Once Optimize's graph commit is durable, a failed configuration clear is +/// reported beside it and fails nothing: the run returns its commit, and +/// the writer gates are released for the next writer. +#[tokio::test] +#[serial(optimize)] +async fn optimize_reports_a_failed_configuration_clear_after_its_publication() { + let _scenario = FailScenario::setup(); + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap().to_string(); + seed_two_productive_optimize_tables(&uri).await; + let db = helpers::session(Omnigraph::open(&uri).await.unwrap()); + let report = { + let _failpoint = catalog::OPTIMIZE_CONFIGURATION_CLEAR.fire_always(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .expect("a post-commit clear failure does not fail the run") + }; + let published = report + .graph_commit_id + .clone() + .expect("the productive run published"); + assert!( + matches!( + report.configuration_clear, + omnigraph::db::ConfigurationClear::Failed(_) + ), + "{:?}", + report.configuration_clear + ); + assert_eq!( + db.list_commits(None).await.unwrap()[0].graph_commit_id, + published, + "the reported commit is main's head" + ); + db.mutate( + "main", + OCC_DISJOINT_MUTATIONS, + "insert_company", + ¶ms(&[("$name", "after-optimize")]), + ) + .await + .expect("the run released its writer gates"); +} + /// RFC 0067: a lost acknowledgement of Optimize's manifest commit reports /// an error, but the detached pins are published: reads serve the compacted /// rows through them and nothing is left to compact. @@ -4545,7 +4593,7 @@ async fn optimize_lost_publish_acknowledgement_leaves_published_detached_pins() let db = helpers::session(Omnigraph::open(&uri).await.unwrap()); let before = person_head_and_published(&db, "main").await; let _failpoint = catalog::GRAPH_PUBLISH_AFTER_MANIFEST_COMMIT.fire_once_at(1); - db.optimize_as(None) + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) .await .expect_err("a lost publish acknowledgement surfaces as an error"); before @@ -4561,11 +4609,15 @@ async fn optimize_lost_publish_acknowledgement_leaves_published_detached_pins() before.1 ); assert_person_pin_detached(&recovered, "main", before.0).await; - let stats = recovered.optimize_as(None).await.unwrap(); + let stats = recovered + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap() + .tables; assert!( stats .iter() - .all(|stat| stat.type_key == "__manifest" || !stat.committed), + .all(|stat| stat.type_key == "__manifest" || !stat.committed()), "nothing is left to compact after the published compaction: {stats:?}" ); } @@ -4610,7 +4662,9 @@ node Embedding { ) .await .unwrap(); - db.optimize_as(None).await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); let snapshot = db.snapshot_of(ReadTarget::branch("main")).await.unwrap(); let embedding = snapshot.open_dataset("node:Embedding").await.unwrap(); assert!(embedding.has_unindexed_fragments().await.unwrap()); @@ -4647,7 +4701,10 @@ node Embedding { .version; { let _failpoint = catalog::OPTIMIZE_POST_PHASE_B_PRE_MANIFEST_COMMIT.fire_once_at(1); - let error = db.optimize_as(None).await.unwrap_err(); + let error = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap_err(); assert!( !matches!(error, OmniError::RecoveryRequired { .. }), "{error}" @@ -4675,17 +4732,24 @@ node Embedding { ); drop(db); let db = helpers::session(Omnigraph::open(&uri).await.unwrap()); - let stats = db.optimize_as(None).await.unwrap(); + let stats = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap() + .tables; let work = stats .iter() .find(|stat| stat.type_key == "node:Work") .expect("Work stat present"); - assert!(work.committed, "the retry compacts Work"); + assert!(work.committed(), "the retry compacts Work"); let embedding = stats .iter() .find(|stat| stat.type_key == "node:Embedding") .expect("Embedding stat present"); - assert!(!embedding.committed, "pending-only table must stay a no-op"); + assert!( + !embedding.committed(), + "pending-only table must stay a no-op" + ); assert!( embedding .pending_indexes @@ -4739,7 +4803,10 @@ async fn optimize_partial_table_effect_leaves_no_residue() { .version() .version; let _failpoint = catalog::OPTIMIZE_POST_TABLE_EFFECT.fire_once_at(1); - let error = db.optimize_as(None).await.unwrap_err(); + let error = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap_err(); assert!( !matches!(error, OmniError::RecoveryRequired { .. }), "a partial batch before publication needs no recovery: {error}" @@ -4783,11 +4850,15 @@ async fn optimize_partial_table_effect_leaves_no_residue() { recovered.list_commits(Some("main")).await.unwrap().len(), commits_before ); - let stats = recovered.optimize_as(None).await.unwrap(); + let stats = recovered + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap() + .tables; assert_eq!( stats .iter() - .filter(|stat| stat.type_key != "__manifest" && stat.committed) + .filter(|stat| stat.type_key != "__manifest" && stat.committed()) .count(), 2, "the retry compacts both tables: {stats:?}" @@ -4831,7 +4902,11 @@ async fn optimize_refuses_when_graph_authority_moves_before_its_gates() { &catalog::OPTIMIZE_POST_AUTHORITY_CAPTURE_PRE_GATES, ); let optimize_task_db = std::sync::Arc::clone(&optimize_db); - let optimize = tokio::spawn(async move { optimize_task_db.optimize_as(None).await }); + let optimize = tokio::spawn(async move { + optimize_task_db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + }); rendezvous.wait_until_reached().await; // Advance the graph head underneath it with an ordinary committed write, so @@ -4927,7 +5002,8 @@ async fn optimize_holds_main_gate_through_disjoint_table_effects() { let uri_opt = uri.clone(); let optimize = tokio::spawn(async move { let db = helpers::session(Omnigraph::open(&uri_opt).await.unwrap()); - db.optimize_as(None).await + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await }); rendezvous.wait_until_reached().await; @@ -4971,7 +5047,7 @@ async fn optimize_holds_main_gate_through_disjoint_table_effects() { 1, "queued table-disjoint Company insert must persist", ); - db.optimize_as(None) + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) .await .expect("graph must remain healthy / re-optimizable"); } @@ -5011,7 +5087,8 @@ async fn optimize_serializes_concurrent_delete_across_handles() { let uri_opt = uri.clone(); let optimize = tokio::spawn(async move { let db = helpers::session(Omnigraph::open(&uri_opt).await.unwrap()); - db.optimize_as(None).await + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await }); rendezvous.wait_until_reached().await; @@ -5071,7 +5148,7 @@ async fn optimize_serializes_concurrent_delete_across_handles() { 3, "the concurrent delete must persist (alice removed)", ); - db.optimize_as(None) + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) .await .expect("graph must remain healthy / re-optimizable"); } @@ -7533,7 +7610,7 @@ async fn assert_optimize_detached_pin_survives_next_write(uri: &str) { { let _failpoint = catalog::GRAPH_PUBLISH_AFTER_MANIFEST_COMMIT.fire_once_at(1); let error = db - .optimize_as(None) + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) .await .expect_err("compaction published before its acknowledgement was lost"); assert!( @@ -9363,7 +9440,7 @@ async fn live_handle_writes_after_every_write_family_seam_failure() { .map_err(|error| error.to_string()) } Driver::Optimize => db - .optimize_as(None) + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) .await .map(|_| ()) .map_err(|error| error.to_string()), diff --git a/crates/omnigraph/tests/forbidden_apis.rs b/crates/omnigraph/tests/forbidden_apis.rs index 9d84bcd2e..0e20efc90 100644 --- a/crates/omnigraph/tests/forbidden_apis.rs +++ b/crates/omnigraph/tests/forbidden_apis.rs @@ -1146,7 +1146,7 @@ durable_calls! { ("db/omnigraph/repair.rs", ".dataset()", 1, WriteProtocol::ReadOnlyAccess), // The sixth accessor reports deferred FTS coverage from an immutable // snapshot; it only reads index metadata and never stages or publishes. - ("db/omnigraph/optimize.rs", ".dataset()", 9, WriteProtocol::Composed("Optimize planning + read-only coverage and native-fork inventory + physical cleanup, including the release of expired proposal tags and of expired unattached tags descendants first, each confirmed by a readback")), + ("db/omnigraph/optimize.rs", ".dataset()", 10, WriteProtocol::Composed("Optimize planning + read-only coverage and native-fork inventory + the configuration clear's read-back after a failed removal + physical cleanup, including the release of expired proposal tags and of expired unattached tags descendants first, each confirmed by a readback")), ("db/omnigraph/optimize.rs", ".into_dataset()", 1, WriteProtocol::PhysicalOnly), ("exec/merge.rs", "SnapshotHandle::new(", 4, MERGE_V9), } diff --git a/crates/omnigraph/tests/integration/changes.rs b/crates/omnigraph/tests/integration/changes.rs index 80e0815b6..23964392a 100644 --- a/crates/omnigraph/tests/integration/changes.rs +++ b/crates/omnigraph/tests/integration/changes.rs @@ -2059,11 +2059,15 @@ node Document { // touching logical rows. The moved descriptors force the payload // tie-break, which must classify every row as unchanged: the maintenance // commit is an empty block, not a phantom full-table update. - let stats = db.optimize_as(None).await.unwrap(); + let stats = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap() + .tables; assert!( stats .iter() - .any(|table| table.fragments_removed > 0 && table.committed), + .any(|table| table.fragments_removed > 0 && table.committed()), "the fixture must actually compact so descriptors move" ); let optimize_commit = head_commit_id(uri, None).await; @@ -2161,11 +2165,15 @@ async fn physical_only_commit_over_a_large_fragment_reads_as_empty_issue_705() { ) .await .unwrap(); - let stats = main.optimize_as(None).await.unwrap(); + let stats = main + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap() + .tables; assert!( stats .iter() - .any(|table| table.fragments_removed > 0 && table.committed), + .any(|table| table.fragments_removed > 0 && table.committed()), "the fixture must actually compact: {stats:?}" ); let optimize_commit = head_commit_id(dir.path().to_str().unwrap(), None).await; diff --git a/crates/omnigraph/tests/integration/composite_flow.rs b/crates/omnigraph/tests/integration/composite_flow.rs index 863ab0271..cefa9c820 100644 --- a/crates/omnigraph/tests/integration/composite_flow.rs +++ b/crates/omnigraph/tests/integration/composite_flow.rs @@ -309,7 +309,11 @@ async fn composite_flow_canonical_lifecycle() { // version, and this flow exercises exactly that previously-failing // write below. // ───────────────────────────────────────────────────────────────── - let optimize_stats = db.optimize_as(None).await.unwrap(); + let optimize_stats = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap() + .tables; assert!( !optimize_stats.is_empty(), "optimize must return per-table stats" diff --git a/crates/omnigraph/tests/integration/end_to_end.rs b/crates/omnigraph/tests/integration/end_to_end.rs index 8ed619ced..41a72159d 100644 --- a/crates/omnigraph/tests/integration/end_to_end.rs +++ b/crates/omnigraph/tests/integration/end_to_end.rs @@ -3093,9 +3093,13 @@ async fn full_flow_optimize_then_query_update_and_reopen() { .unwrap(); } - let stats = db.optimize_as(None).await.unwrap(); + let stats = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap() + .tables; assert!( - stats.iter().any(|s| s.committed), + stats.iter().any(|s| s.committed()), "a multi-fragment table should have compacted in this flow" ); diff --git a/crates/omnigraph/tests/integration/engine_v2.rs b/crates/omnigraph/tests/integration/engine_v2.rs index cf54ed2f0..1f1ab045a 100644 --- a/crates/omnigraph/tests/integration/engine_v2.rs +++ b/crates/omnigraph/tests/integration/engine_v2.rs @@ -413,7 +413,9 @@ async fn indexed_single_hop_switches_to_the_csr_between_input_batches() { db.load_jsonl(&lines.join("\n"), LoadMode::Overwrite) .await .unwrap(); - db.optimize_as(None).await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); let v2 = with_setting(&db, "engine", "v2"); let pinned = first_column_sorted( diff --git a/crates/omnigraph/tests/integration/engine_v2_memory.rs b/crates/omnigraph/tests/integration/engine_v2_memory.rs index 36b5a0120..a9c4347b0 100644 --- a/crates/omnigraph/tests/integration/engine_v2_memory.rs +++ b/crates/omnigraph/tests/integration/engine_v2_memory.rs @@ -846,7 +846,9 @@ async fn hash_join_output_is_the_refusing_reservation() { db.load_jsonl(&lines.join("\n"), LoadMode::Overwrite) .await .unwrap(); - db.optimize_as(None).await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); let v2 = with_setting(&db, "engine", "v2"); let limit = 8 * MIB; @@ -929,7 +931,9 @@ async fn underestimated_hash_build_retries_id_lookup() { db.load_jsonl(&lines.join("\n"), LoadMode::Overwrite) .await .unwrap(); - db.optimize_as(None).await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); let v2 = with_setting(&db, "engine", "v2"); let limit = 16 * MIB; let explain = with_query_memory_limit( diff --git a/crates/omnigraph/tests/integration/engine_v2_plan_replay.rs b/crates/omnigraph/tests/integration/engine_v2_plan_replay.rs index 35e4fd92e..ccfd40170 100644 --- a/crates/omnigraph/tests/integration/engine_v2_plan_replay.rs +++ b/crates/omnigraph/tests/integration/engine_v2_plan_replay.rs @@ -305,7 +305,9 @@ async fn selected_cold_indexed_route_uses_persisted_members_without_building_csr ) .await .unwrap(); - db.optimize_as(None).await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); drop(db); let db = with_traversal( &session(Omnigraph::open(dir.path().to_str().unwrap()).await.unwrap()), diff --git a/crates/omnigraph/tests/integration/maintenance.rs b/crates/omnigraph/tests/integration/maintenance.rs index da034cce9..1c582bfa6 100644 --- a/crates/omnigraph/tests/integration/maintenance.rs +++ b/crates/omnigraph/tests/integration/maintenance.rs @@ -15,11 +15,14 @@ use lance::Dataset; use lance::dataset::optimize::{CompactionOptions, compact_files}; use lance_core::datatypes::BlobHandling; use omnigraph::IndexCoverage; +use omnigraph::db::{ + ArtifactRefresh, ConfigurationClear, DeferReason, OptimizeLimits, TableOptimizeOutcome, +}; use omnigraph::db::{ CleanupPolicyOptions, MergeOutcome, Omnigraph, ReadTarget, RepairAction, RepairClassification, SnapshotId, }; -use omnigraph::error::{BranchStartRefusal, OmniError}; +use omnigraph::error::{BranchStartRefusal, CompletionEvidence, OmniError}; use omnigraph::loader::LoadMode; use omnigraph::{ExternalBlobBase, ExternalBlobExecutionScope, ExternalBlobPolicy}; @@ -130,21 +133,192 @@ async fn optimize_on_empty_graph_returns_stats_per_table_with_no_changes() { let uri = dir.path().to_str().unwrap(); let db = Omnigraph::init(uri, TEST_SCHEMA).await.unwrap(); - let stats = db.optimize_as(None).await.unwrap(); + let head = db.list_commits(None).await.unwrap()[0] + .graph_commit_id + .clone(); + let report = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); - assert_eq!(stats.len(), 5); - for s in &stats { + // One outcome per node and edge table, in type-key order; `__manifest` + // is not a table the run optimizes. + let keys = report + .tables + .iter() + .map(|s| s.type_key.as_str()) + .collect::>(); + let mut sorted = keys.clone(); + sorted.sort(); + assert_eq!( + keys, sorted, + "a full-catalog run visits tables in key order" + ); + assert_eq!(keys.len(), 4); + for s in &report.tables { + assert_eq!(s.outcome, TableOptimizeOutcome::Unchanged, "{s:?}"); assert_eq!(s.fragments_removed, 0, "{} should not remove", s.type_key); assert_eq!(s.fragments_added, 0, "{} should not add", s.type_key); } - // `__manifest` is present and reported as a no-op on an empty graph. - let s = stats + assert_eq!(report.input_commit.as_deref(), Some(head.as_str())); + assert_eq!(report.graph_commit_id, None, "nothing to publish"); + assert_eq!(report.resume_at, None, "no bound stopped the run"); + assert_eq!( + report.configuration_clear, + ConfigurationClear::NothingToClear + ); + assert!( + !matches!(report.artifact, ArtifactRefresh::Failed(_)), + "{:?}", + report.artifact + ); +} + +/// A run whose gate hold is used up when its second table's turn comes runs +/// only the first, defers the rest, publishes once, and names the first +/// deferred table; a full-catalog run that starts there visits it first and +/// wraps, so successive runs reach every table. +#[tokio::test] +async fn optimize_stops_at_the_gate_hold_and_resumes_at_the_first_deferred_table() { + let dir = tempfile::tempdir().unwrap(); + let db = init_and_load(&dir).await; + let bounded = || { + omnigraph::db::OptimizeRequest::default() + .with_limits(OptimizeLimits::default().with_max_gate_hold(Duration::ZERO)) + }; + + let first = db.optimize_as(None, bounded()).await.unwrap(); + let keys = first + .tables .iter() - .find(|s| s.type_key == "__manifest") - .expect("optimize stats missing internal table __manifest"); + .map(|s| s.type_key.clone()) + .collect::>(); + assert_eq!(keys.len(), 4, "every table is reported: {first:?}"); assert!( - !s.committed, - "__manifest should be a no-op on an empty graph" + !matches!(first.tables[0].outcome, TableOptimizeOutcome::Deferred(_)), + "the first table always runs: {first:?}" + ); + for stat in &first.tables[1..] { + assert_eq!( + stat.outcome, + TableOptimizeOutcome::Deferred(DeferReason::GateHoldLimit), + "{stat:?}" + ); + assert_eq!( + stat.duration, + Duration::ZERO, + "a deferred table never started" + ); + } + assert_eq!(first.resume_at.as_deref(), Some(keys[1].as_str())); + + let second = db + .optimize_as(None, bounded().with_start_at(keys[1].clone())) + .await + .unwrap(); + let visited = second + .tables + .iter() + .map(|s| s.type_key.as_str()) + .collect::>(); + assert_eq!( + visited, + [&keys[1], &keys[2], &keys[3], &keys[0]].map(String::as_str), + "the run starts at the resume position and wraps" + ); + assert_eq!(second.resume_at.as_deref(), Some(keys[2].as_str())); + assert_eq!( + second.input_commit, + first.graph_commit_id.clone().or(first.input_commit.clone()), + "each run builds on the head the previous one left" + ); + + // A position after every key wraps to the first, and an unbounded run + // finishes the catalog and clears the position. + let wrapped = db + .optimize_as( + None, + omnigraph::db::OptimizeRequest::default().with_start_at("~"), + ) + .await + .unwrap(); + assert_eq!(wrapped.tables[0].type_key, keys[0]); + assert_eq!(wrapped.resume_at, None); +} + +/// A table list runs only those tables, in its order, and neither reads nor +/// sets a resume position; a malformed request is refused before any effect. +#[tokio::test] +async fn optimize_table_list_runs_only_those_tables_and_refuses_bad_requests() { + let dir = tempfile::tempdir().unwrap(); + let db = init_and_load(&dir).await; + let all = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); + let keys = all + .tables + .iter() + .map(|s| s.type_key.clone()) + .collect::>(); + + let listed = db + .optimize_as( + None, + omnigraph::db::OptimizeRequest::default().with_tables([&keys[2], &keys[0]]), + ) + .await + .unwrap(); + let visited = listed + .tables + .iter() + .map(|s| s.type_key.as_str()) + .collect::>(); + assert_eq!(visited, [keys[2].as_str(), keys[0].as_str()]); + assert_eq!(listed.resume_at, None); + + // A bounded list defers its tail but reports no resume position: a list + // neither reads nor moves it. + let bounded = db + .optimize_as( + None, + omnigraph::db::OptimizeRequest::default() + .with_tables([&keys[2], &keys[0]]) + .with_limits(OptimizeLimits::default().with_max_gate_hold(Duration::ZERO)), + ) + .await + .unwrap(); + assert_eq!( + bounded.tables[1].outcome, + TableOptimizeOutcome::Deferred(DeferReason::GateHoldLimit), + "{bounded:?}" + ); + assert_eq!(bounded.resume_at, None); + + let head = db.list_commits(None).await.unwrap()[0] + .graph_commit_id + .clone(); + for request in [ + omnigraph::db::OptimizeRequest::default().with_tables(["node:Missing"]), + omnigraph::db::OptimizeRequest::default().with_tables([&keys[0], &keys[0]]), + omnigraph::db::OptimizeRequest::default().with_tables(Vec::::new()), + omnigraph::db::OptimizeRequest::default() + .with_tables([&keys[0]]) + .with_start_at(keys[1].clone()), + omnigraph::db::OptimizeRequest::default() + .with_concurrency(omnigraph::db::OptimizeConcurrency::new(0, 1)), + ] { + let error = db.optimize_as(None, request.clone()).await.unwrap_err(); + assert_eq!( + error.completion_evidence(), + Some(CompletionEvidence::BeforeEffect), + "{request:?}: {error}" + ); + } + assert_eq!( + db.list_commits(None).await.unwrap()[0].graph_commit_id, + head, + "a refused request publishes nothing" ); } @@ -154,7 +328,10 @@ async fn optimize_after_load_then_again_is_idempotent() { let db = init_and_load(&dir).await; // First pass may compact (load wrote real fragments). - let _first = db.optimize_as(None).await.unwrap(); + let _first = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); let commits_before = db.list_commits(None).await.unwrap(); let head_before = commits_before @@ -165,7 +342,11 @@ async fn optimize_after_load_then_again_is_idempotent() { // Second pass should be a no-op: already-compacted graph produces no // fragments_removed / fragments_added. - let second = db.optimize_as(None).await.unwrap(); + let second = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap() + .tables; for s in &second { assert_eq!( s.fragments_removed, 0, @@ -178,7 +359,7 @@ async fn optimize_after_load_then_again_is_idempotent() { s.type_key ); assert!( - !s.committed, + !s.committed(), "{} re-optimize should not commit a new version", s.type_key ); @@ -229,7 +410,11 @@ async fn optimize_preserves_manifest_registry_fragments() { !cold_files.is_empty(), "the fixture must release more registry rows than the hot bound" ); - let stats = db.optimize_as(None).await.unwrap(); + let report = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); + let stats = &report.tables; let after = Dataset::open(&manifest_uri).await.unwrap(); let after_files: BTreeSet<_> = after .fragments() @@ -241,13 +426,14 @@ async fn optimize_preserves_manifest_registry_fragments() { "optimize rewrote cold registry files" ); - let manifest_stats = stats - .iter() - .find(|s| s.type_key == "__manifest") - .expect("optimize stats missing internal table __manifest"); assert!( - !manifest_stats.committed && manifest_stats.fragments_removed == 0, - "the manifest publisher owns hot and cold fragments, so optimize must leave them intact: {manifest_stats:?}" + stats.iter().all(|s| s.type_key != "__manifest"), + "the manifest publisher owns hot and cold fragments, so optimize must not compact them" + ); + assert_eq!( + report.configuration_clear, + ConfigurationClear::NothingToClear, + "a graph with no stale auto-cleanup keys has nothing to clear" ); assert!( @@ -323,7 +509,9 @@ async fn optimize_clears_stale_auto_cleanup_and_preserves_versions() { .unwrap() .len(); - db.optimize_as(None).await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); let ds = Dataset::open(&manifest_uri).await.unwrap(); // (a) the stale auto_cleanup config was cleared (non-destructive by construction). @@ -386,7 +574,9 @@ async fn optimize_preserves_versions_under_stale_auto_cleanup_config_on_data_tab .len(); let rows_before = count_rows(&db, "node:Person").await; - db.optimize_as(None).await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); let ds = Dataset::open(&person_full).await.unwrap(); assert!( @@ -455,7 +645,11 @@ node Doc { ); } - let stats = db.optimize_as(None).await.unwrap(); + let stats = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap() + .tables; // Postcondition: optimize_indices folded the scalar tail in, but preserves // the existing FTS artifact instead of merging uncertified postings. @@ -484,12 +678,16 @@ node Doc { ); let commits_before = db.list_commits(None).await.unwrap().len(); for _ in 0..2 { - let repeated = db.optimize_as(None).await.unwrap(); + let repeated = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap() + .tables; let doc = repeated .iter() .find(|stat| stat.type_key == "node:Doc") .unwrap(); - assert!(!doc.committed, "deferred FTS alone is not optimize work"); + assert!(!doc.committed(), "deferred FTS alone is not optimize work"); assert_eq!(doc.pending_indexes.len(), 1); assert_eq!(doc.pending_indexes[0].property, "slug"); assert_same_dataset_entry( @@ -644,10 +842,12 @@ node Tag {\n slug: String @key\n}\n"; .clone(); let probes = omnigraph::instrumentation::MergeWriteProbes::default(); - let stats = - omnigraph::instrumentation::with_merge_write_probes(probes.clone(), db.optimize_as(None)) - .await - .expect("optimize must not crash on a graph with a Blob table"); + let stats = omnigraph::instrumentation::with_merge_write_probes( + probes.clone(), + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()), + ) + .await + .expect("optimize must not crash on a graph with a Blob table"); assert_eq!( ( probes.compaction_blob_batch_calls(), @@ -658,16 +858,18 @@ node Tag {\n slug: String @key\n}\n"; ); let doc = stats + .tables .iter() .find(|s| s.type_key == "node:Doc") .expect("Doc stat present"); let tag = stats + .tables .iter() .find(|s| s.type_key == "node:Tag") .expect("Tag stat present"); // Lance 10's null/empty-safe blob-v2 compaction uses the ordinary path. - assert_eq!(doc.skipped, None, "blob table must no longer be skipped"); - assert!(doc.committed, "blob table compaction must be published"); + assert_eq!(doc.skipped(), None, "blob table must no longer be skipped"); + assert!(doc.committed(), "blob table compaction must be published"); assert!( doc.fragments_removed >= 2 && doc.fragments_added >= 1, "expected a real rewrite of the multi-fragment blob table, got \ @@ -675,8 +877,8 @@ node Tag {\n slug: String @key\n}\n"; doc.fragments_removed, doc.fragments_added ); - assert_eq!(tag.skipped, None, "non-blob table must not be skipped"); - assert!(tag.committed, "plain table compaction must be published"); + assert_eq!(tag.skipped(), None, "non-blob table must not be skipped"); + assert!(tag.committed(), "plain table compaction must be published"); // Both productive tables share one graph visibility point. Physical // __manifest compaction below may add Lance versions, so lineage is the @@ -779,10 +981,12 @@ async fn optimize_compacts_large_blob_rows_in_bounded_batches() { let commits_before = db.list_commits(None).await.unwrap().len(); let probes = omnigraph::instrumentation::MergeWriteProbes::default(); - let stats = - omnigraph::instrumentation::with_merge_write_probes(probes.clone(), db.optimize_as(None)) - .await - .unwrap(); + let stats = omnigraph::instrumentation::with_merge_write_probes( + probes.clone(), + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()), + ) + .await + .unwrap(); assert_eq!( probes.compaction_blob_batch_calls(), 1, @@ -794,11 +998,15 @@ async fn optimize_compacts_large_blob_rows_in_bounded_batches() { "the task runs with the batch derived from its largest row" ); let doc = stats + .tables .iter() .find(|stat| stat.type_key == "node:Doc") .expect("Doc stat present"); - assert_eq!(doc.skipped, None); - assert!(doc.committed, "the Blob table compaction must be published"); + assert_eq!(doc.skipped(), None); + assert!( + doc.committed(), + "the Blob table compaction must be published" + ); assert_eq!(doc.fragments_removed, fragments_before); assert_eq!( db.list_commits(None).await.unwrap().len(), @@ -935,10 +1143,12 @@ async fn optimize_sizes_a_blob_batch_by_the_sum_of_a_rows_blob_columns() { } let probes = omnigraph::instrumentation::MergeWriteProbes::default(); - let stats = - omnigraph::instrumentation::with_merge_write_probes(probes.clone(), db.optimize_as(None)) - .await - .unwrap(); + let stats = omnigraph::instrumentation::with_merge_write_probes( + probes.clone(), + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()), + ) + .await + .unwrap(); assert_eq!(probes.compaction_blob_batch_calls(), 1); assert_eq!( probes.compaction_blob_batch_rows(), @@ -946,10 +1156,14 @@ async fn optimize_sizes_a_blob_batch_by_the_sum_of_a_rows_blob_columns() { "the batch is derived from the row's 2 MiB, not from its largest 1 MiB column" ); let doc = stats + .tables .iter() .find(|stat| stat.type_key == "node:Doc") .expect("Doc stat present"); - assert!(doc.committed, "the Blob table compaction must be published"); + assert!( + doc.committed(), + "the Blob table compaction must be published" + ); assert_eq!(doc.fragments_removed, 2); let expected = rows .into_iter() @@ -1025,12 +1239,19 @@ async fn optimize_compacts_blob_fragments_with_deleted_rows() { "test precondition: both fragments carry a deletion vector" ); - let stats = db.optimize_as(None).await.unwrap(); + let stats = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap() + .tables; let doc = stats .iter() .find(|stat| stat.type_key == "node:Doc") .expect("Doc stat present"); - assert!(doc.committed, "the Blob table compaction must be published"); + assert!( + doc.committed(), + "the Blob table compaction must be published" + ); assert_eq!(doc.fragments_removed, 2); let compacted = helpers::open_pinned_dataset_for_test(&db, "main", "node:Doc").await; let fragments = compacted.get_fragments(); @@ -1084,10 +1305,12 @@ async fn optimize_sizes_each_compaction_task_from_its_own_fragments() { } let probes = omnigraph::instrumentation::MergeWriteProbes::default(); - let stats = - omnigraph::instrumentation::with_merge_write_probes(probes.clone(), db.optimize_as(None)) - .await - .unwrap(); + let stats = omnigraph::instrumentation::with_merge_write_probes( + probes.clone(), + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()), + ) + .await + .unwrap(); assert_eq!( probes.compaction_blob_batch_calls(), 2, @@ -1099,10 +1322,14 @@ async fn optimize_sizes_each_compaction_task_from_its_own_fragments() { "the second task is sized by its own 64 KiB row, not the first task's 1 MiB row" ); let doc = stats + .tables .iter() .find(|stat| stat.type_key == "node:Doc") .expect("Doc stat present"); - assert!(doc.committed, "the Blob table compaction must be published"); + assert!( + doc.committed(), + "the Blob table compaction must be published" + ); let expected = rows .into_iter() .map(|(slug, content)| (slug.to_string(), vec![Some(content)])) @@ -1163,10 +1390,12 @@ async fn optimize_does_not_size_a_blob_batch_by_external_references() { ); let probes = omnigraph::instrumentation::MergeWriteProbes::default(); - let stats = - omnigraph::instrumentation::with_merge_write_probes(probes.clone(), db.optimize_as(None)) - .await - .unwrap(); + let stats = omnigraph::instrumentation::with_merge_write_probes( + probes.clone(), + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()), + ) + .await + .unwrap(); assert_eq!(probes.compaction_blob_batch_calls(), 1); assert_eq!( probes.compaction_blob_batch_rows(), @@ -1174,10 +1403,14 @@ async fn optimize_does_not_size_a_blob_batch_by_external_references() { "a 1 MiB external reference must not lower the batch to 32 rows" ); let doc = stats + .tables .iter() .find(|stat| stat.type_key == "node:Doc") .expect("Doc stat present"); - assert!(doc.committed, "the Blob table compaction must be published"); + assert!( + doc.committed(), + "the Blob table compaction must be published" + ); assert_eq!(doc.fragments_removed, 2); } @@ -1283,9 +1516,16 @@ async fn optimize_then_cleanup_erases_dropped_property_values() { .await .unwrap(); // Settle index work, then prove Lance's planner leaves both tables alone. - db.optimize_as(None).await.unwrap(); - for stat in db.optimize_as(None).await.unwrap() { - assert!(!stat.committed, "test precondition: {stat:?}"); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); + for stat in db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap() + .tables + { + assert!(!stat.committed(), "test precondition: {stat:?}"); } assert_eq!( helpers::open_pinned_dataset_for_test(&db, "main", "node:Doc") @@ -1330,18 +1570,26 @@ async fn optimize_then_cleanup_erases_dropped_property_values() { let commits_before = db.list_commits(None).await.unwrap().len(); let probes = omnigraph::instrumentation::MergeWriteProbes::default(); - let stats = - omnigraph::instrumentation::with_merge_write_probes(probes.clone(), db.optimize_as(None)) - .await - .unwrap(); - let stat = |key: &str| stats.iter().find(|stat| stat.type_key == key).unwrap(); + let stats = omnigraph::instrumentation::with_merge_write_probes( + probes.clone(), + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()), + ) + .await + .unwrap(); + let stat = |key: &str| { + stats + .tables + .iter() + .find(|stat| stat.type_key == key) + .unwrap() + }; let doc = stat("node:Doc"); assert!( - doc.committed, + doc.committed(), "the fragment holding dropped values is rewritten" ); assert_eq!((doc.fragments_removed, doc.fragments_added), (1, 1)); - assert!(!stat("node:Tag").committed, "Tag has nothing to rewrite"); + assert!(!stat("node:Tag").committed(), "Tag has nothing to rewrite"); assert_eq!( helpers::pinned_version(&db, "main", "node:Tag").await, tag_pin @@ -1381,9 +1629,14 @@ async fn optimize_then_cleanup_erases_dropped_property_values() { doc_blob_values(&db, &["content"]).await == expected, "the remaining properties survive the rewrite" ); - for stat in db.optimize_as(None).await.unwrap() { + for stat in db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap() + .tables + { assert!( - !stat.committed, + !stat.committed(), "a second optimize has nothing left: {stat:?}" ); } @@ -1405,7 +1658,9 @@ async fn optimize_then_cleanup_erases_dropped_property_values() { db.branch_merge("retained", "main").await.unwrap().outcome, MergeOutcome::Merged ); - db.optimize_as(None).await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); let stats = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(stats.iter().all(|row| row.error.is_none()), "{stats:?}"); for needle in &needles { @@ -1469,13 +1724,17 @@ async fn optimize_publishes_compaction_to_manifest_so_schema_apply_succeeds() { .get_fragments() .len(); - let stats = db.optimize_as(None).await.unwrap(); + let stats = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap() + .tables; let person = stats .iter() .find(|s| s.type_key == "node:Person") .expect("Person stat present"); assert!( - person.committed, + person.committed(), "Person is multi-fragment, so optimize must have compacted it" ); @@ -1529,13 +1788,20 @@ async fn optimize_compacts_the_pin_under_foreign_head_drift() { let (pin_before, head_before, _) = forge_person_compaction_drift(&db, &root).await; let rows_before = count_rows(&db, "node:Person").await; - let stats = db.optimize_as(None).await.unwrap(); + let stats = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap() + .tables; let person = stats .iter() .find(|s| s.type_key == "node:Person") .expect("Person stat present"); - assert_eq!(person.skipped, None); - assert!(person.committed, "optimize compacts the multi-fragment pin"); + assert_eq!(person.skipped(), None); + assert!( + person.committed(), + "optimize compacts the multi-fragment pin" + ); let (pin_after, head_after, _) = person_pin_and_head(&db, &root).await; assert_ne!(pin_after, pin_before, "optimize publishes a new pin"); @@ -2705,7 +2971,9 @@ async fn cleanup_then_optimize_preserves_rows_and_table_remains_writable() { ) .await .unwrap(); - db.optimize_as(None).await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); // Head is preserved through both ops. assert_eq!(count_rows(&db, "node:Person").await, people_before); @@ -3672,7 +3940,9 @@ async fn optimize_materializes_index_declared_but_unbuilt() { ); } - db.optimize_as(None).await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); // Postcondition: optimize's reconciler materialized the declared index. let snap = snapshot_main(&db).await.unwrap(); @@ -3728,7 +3998,9 @@ async fn optimize_materializes_index_after_type_rename() { ); } - db.optimize_as(None).await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); let snap = snapshot_main(&db).await.unwrap(); let ds = snap.open_dataset("node:Item").await.unwrap(); @@ -4115,7 +4387,9 @@ async fn collector_finds_nothing_to_sweep_after_todays_cleanup() { ) .await .unwrap(); - db.optimize_as(None).await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); db.branch_create("feature").await.unwrap(); insert_person(&db, "feature", "Frank").await; db.branch_merge("feature", "main").await.unwrap(); diff --git a/crates/omnigraph/tests/integration/traversal_adaptive.rs b/crates/omnigraph/tests/integration/traversal_adaptive.rs index e393372ff..5d1764d07 100644 --- a/crates/omnigraph/tests/integration/traversal_adaptive.rs +++ b/crates/omnigraph/tests/integration/traversal_adaptive.rs @@ -83,7 +83,9 @@ async fn clique_db(dir: &tempfile::TempDir) -> Session { .unwrap(); // The endpoint BTREEs must exist, or dispatch prices a degraded scan and // takes CSR outright — the switch under test would never arm. - db.optimize_as(None).await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); db } @@ -189,7 +191,9 @@ query far($name: String) { jsonl.push('\n'); let db = session(Omnigraph::init(uri, TEST_SCHEMA).await.unwrap()); db.load_jsonl(&jsonl, LoadMode::Overwrite).await.unwrap(); - db.optimize_as(None).await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); let p = params(&[("$name", "p0")]); let csr_db = with_traversal(&db, Traversal::Csr); @@ -322,7 +326,9 @@ async fn corrupt_artifact_falls_back_to_the_scan_build() { async fn stale_artifact_is_rejected_after_an_edge_write() { let dir = tempfile::tempdir().unwrap(); let db = init_and_load(&dir).await; - db.optimize_as(None).await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); let artifact = dir.path().join("__graph_index/csr-current.bin"); assert!(artifact.is_file()); diff --git a/crates/omnigraph/tests/integration/write_cost.rs b/crates/omnigraph/tests/integration/write_cost.rs index 7182c64e1..6b155a215 100644 --- a/crates/omnigraph/tests/integration/write_cost.rs +++ b/crates/omnigraph/tests/integration/write_cost.rs @@ -38,7 +38,7 @@ use crate::helpers::{ // `_graph_commits`/`_graph_commit_actors` tables are retired), so the manifest scan // also covers the lineage and actor rows the authenticated write path appends. // Without internal-table compaction these scans are O(fragments) and grow forever; -// step 2 brings the internal tables into `db.optimize_as(None)`, so after compaction the +// step 2 brings the internal tables into `db.optimize_as(None, omnigraph::db::OptimizeRequest::default())`, so after compaction the // per-write scan is flat. The test runs the **authenticated (actorful) write path** // — every commit carries an actor — and compacts at each depth checkpoint before // measuring, pinning the production invariant "a periodically-compacted graph's @@ -62,7 +62,9 @@ async fn internal_table_scans_are_flat_in_history() { } // Step 2: compaction folds all three internal tables' O(depth) fragments back // to a small constant, so the following write's scan of them is flat. - db.optimize_as(None).await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); let io = measure_insert_as(&db, &format!("lock_{d}"), ACTOR).await; current += 1; // the measured write advanced depth by one eprintln!( @@ -87,7 +89,9 @@ async fn compacted_graph_at_depth(depth: u64) -> (u64, tempfile::TempDir, helper let dir = tempfile::tempdir().unwrap(); let db = local_graph(&dir).await; commit_many(&db, depth as usize).await; - db.optimize_as(None).await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); (depth, dir, db) } @@ -274,11 +278,15 @@ async fn optimize_writes_no_control_object() { let before_write_text = counts.write_text(); let before_delete = counts.delete(); - let stats = db.optimize_as(None).await.unwrap(); + let stats = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap() + .tables; assert!( stats .iter() - .any(|stat| stat.type_key == "node:Person" && stat.committed), + .any(|stat| stat.type_key == "node:Person" && stat.committed()), "Person must compact: {stats:?}" ); assert_eq!( @@ -310,7 +318,8 @@ async fn optimize_manifest_reads_are_flat_in_history() { commit_many(&db, 3).await; let commits_before = db.list_commits(None).await.unwrap().len(); - let (result, io) = measure(db.optimize_as(None)).await; + let (result, io) = + measure(db.optimize_as(None, omnigraph::db::OptimizeRequest::default())).await; result.unwrap(); assert_eq!( db.list_commits(None).await.unwrap().len(), @@ -357,7 +366,7 @@ async fn internal_table_scans_are_flat_without_compaction() { commit_many_as(&db, (d - current) as usize, ACTOR).await; current = d; } - // NO `db.optimize_as(None)` here — that omission is the whole point. The flat gate + // NO `db.optimize_as(None, omnigraph::db::OptimizeRequest::default())` here — that omission is the whole point. The flat gate // above compacts before measuring and so never exercises this served regime. let io = measure_insert_as(&db, &format!("served_{d}"), ACTOR).await; current += 1; // the measured write advanced depth by one diff --git a/crates/omnigraph/tests/policy_engine_chassis.rs b/crates/omnigraph/tests/policy_engine_chassis.rs index dc6b9dae0..dd3f2017e 100644 --- a/crates/omnigraph/tests/policy_engine_chassis.rs +++ b/crates/omnigraph/tests/policy_engine_chassis.rs @@ -1005,7 +1005,11 @@ async fn maintenance_entry_points_enforce_maintain_on_their_branch_before_effect "act-branch-maintainer", ] { let actor = Some(actor); - assert_denied(db.optimize_as(actor).await, "optimize_as"); + assert_denied( + db.optimize_as(actor, omnigraph::db::OptimizeRequest::default()) + .await, + "optimize_as", + ); assert_denied(db.ensure_indices_as(actor).await, "ensure_indices_as"); assert_denied( db.ensure_indices_on_as("main", actor).await, @@ -1019,7 +1023,11 @@ async fn maintenance_entry_points_enforce_maintain_on_their_branch_before_effect "ensure_indices_on_as(feature)", ); } - assert_no_actor(db.optimize_as(None).await, "optimize_as"); + assert_no_actor( + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await, + "optimize_as", + ); assert_no_actor(db.ensure_indices_as(None).await, "ensure_indices_as"); assert_no_actor( db.ensure_indices_on_as("feature", None).await, @@ -1071,13 +1079,17 @@ async fn maintenance_entry_points_enforce_maintain_on_their_branch_before_effect .graph_commit_id .clone(); let stats = db - .optimize_as(Some("act-maintainer")) + .optimize_as( + Some("act-maintainer"), + omnigraph::db::OptimizeRequest::default(), + ) .await - .expect("maintain on main admits optimize"); + .expect("maintain on main admits optimize") + .tables; assert!( stats .iter() - .any(|stat| stat.type_key == "node:Person" && stat.committed), + .any(|stat| stat.type_key == "node:Person" && stat.committed()), "{stats:?}" ); let main_after = db.list_commits(None).await.unwrap(); diff --git a/crates/omnigraph/tests/schema_apply.rs b/crates/omnigraph/tests/schema_apply.rs index 692b44ddd..e461967d4 100644 --- a/crates/omnigraph/tests/schema_apply.rs +++ b/crates/omnigraph/tests/schema_apply.rs @@ -2124,7 +2124,9 @@ query strong($min: I32) { "an update carries the untouched Blob of an evolved row" ); - db.optimize_as(None).await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); assert_eq!( read_managed_blob_bytes( &db, diff --git a/crates/omnigraph/tests/search.rs b/crates/omnigraph/tests/search.rs index 2715c2b0c..0a21455e6 100644 --- a/crates/omnigraph/tests/search.rs +++ b/crates/omnigraph/tests/search.rs @@ -2426,7 +2426,9 @@ async fn mutation_with_deferred_index_coverage_remains_searchable() { // Ordinary optimize must preserve certified postings, not silently replace // them with an uncertified incremental fold. Both old and tail rows remain // searchable after data compaction and unrelated index maintenance. - db.optimize_as(None).await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); for (term, slug) in [("Quasar", "quasar-notes"), ("Learning", "ml-intro")] { let result = query_main(&db, SEARCH_QUERIES, "text_search", ¶ms(&[("$q", term)])) .await diff --git a/docs/user/operations/maintenance.md b/docs/user/operations/maintenance.md index e59195bef..92ab49884 100644 --- a/docs/user/operations/maintenance.md +++ b/docs/user/operations/maintenance.md @@ -69,32 +69,33 @@ index coverage was uneven before a run may coalesce fully only on the next run, after the rebuilt coverage is in place. Compacting a table with Blob properties reads each managed Blob value into -memory to rewrite it. Optimize sizes the read batches of each compaction task (a -group of neighbouring fragments compacted together) from the task's largest row, -summing that row's managed Blob values, so one batch materializes at most 32 MiB -of managed Blob payload whatever the fragment sizes. External Blob references -are carried without reading the referenced object and count nothing. A task -holding a row over 16 MiB compacts one row per batch, and a single row over 32 -MiB is materialized whole. This bounds the payload of a batch, not the process -heap: Lance's writer copies each inline payload into the arrays it prepares -while it still holds the batch, so memory use exceeds the payload. Every task of -a Blob table gets this batch size explicitly, 1 to 8192 rows, so it takes -precedence over `LANCE_DEFAULT_BATCH_SIZE` there, including when the variable is -smaller. When a fragment being compacted holds a Blob descriptor the engine's -decoder rejects, sizing can refuse the compaction with a Blob integrity error -before that table is rewritten, and nothing is committed; a table with nothing -to compact is not scanned. Optimize works on up to -`OMNIGRAPH_MAINTENANCE_CONCURRENCY` tables at once (default 8), so budget for -that many Blob tables compacting together. +memory. Optimize sizes each compaction batch from its largest row, whatever +`LANCE_DEFAULT_BATCH_SIZE` says, so one batch holds at most 32 MiB of managed +Blob payload; a larger row is read whole, external references are not read, +and the writer's own copies come on top. A Blob descriptor the decoder rejects +refuses the compaction before that table is rewritten. Optimize plans up to +`OMNIGRAPH_MAINTENANCE_CONCURRENCY` tables ahead (default 8) and rewrites up to +that many at once, so budget for that many Blob tables compacting together. + +### Bounding a run + +Writes to `main` wait while optimize holds its writer gates. With +`--max-gate-hold 30s` it starts no table after holding them that long, except +the first: the rest are reported `deferred`, the finished tables are published, +and a writer waits at most the bound plus one table's rewrite. Tables run in +type-key order from the first; `resume_at` names the first deferred table, and +`--tables` runs only the listed tables, in that order. In JSON each table has an +`outcome` (`optimized`, `unchanged`, `deferred` or `skipped`); the report adds +`input_commit`, `graph_commit_id`, `gate_hold_ms` and the `configuration_clear` +and `artifact` outcomes, which never fail a published run. Optimize also persists the traversal-adjacency artifact (`__graph_index/csr-current.bin`), which cold traversal builds load instead of -scanning every edge dataset. The artifact is derived and regenerable: optimize -is its only writer, every load is verified and falls back to the in-memory -build, and a failed write is a warning, not an optimize error. The artifact is -rewritten by optimize runs that advance an edge dataset; on a store with no -edge work it is left as-is, and a stale copy costs only the artifact's speedup, -never correctness. +scanning every edge dataset. It is derived: every load is verified and falls +back to the in-memory build, so a stale copy or a failed write costs only its +speedup, and a failed write is reported as the `artifact` outcome. Optimize +builds it after releasing the writer gates, and only when the run advanced an +edge dataset or no artifact exists. Existing full-text indexes are preserved rather than incrementally merged. Unindexed rows remain searchable, but a growing uncovered tail can cost more to From a0a155dde63ce025f4cc5e77a8f75e794ffb58e5 Mon Sep 17 00:00:00 2001 From: Ragnor Comerford Date: Thu, 29 Oct 2026 09:59:02 -0700 Subject: [PATCH 05/13] feat(engine): call a before-publish hook once before optimize publishes The shared publication helper takes an optional hook and calls it once, after the updates are prepared and before the coordinator lock and the publisher's first attempt, with the lineage intent's nonce and the captured head. Every attempt reads that head and carries that nonce, so a call per attempt would record nothing new. A hook failure publishes nothing and is labelled Unpublished whatever its own evidence says. The helper's other callers pass no hook. OptimizeRequest::with_before_publish passes the hook with the run's per-table outcomes. A run that stages no table never calls it. --- .../optimize-before-publish-hook.added.md | 3 + crates/omnigraph/src/db/mod.rs | 8 +- crates/omnigraph/src/db/omnigraph.rs | 5 +- crates/omnigraph/src/db/omnigraph/optimize.rs | 64 +++++++++++ .../omnigraph/src/db/omnigraph/table_ops.rs | 39 +++++++ .../tests/integration/maintenance.rs | 100 +++++++++++++++++- docs/dev/writes.md | 12 ++- 7 files changed, 223 insertions(+), 8 deletions(-) create mode 100644 changelog.d/optimize-before-publish-hook.added.md diff --git a/changelog.d/optimize-before-publish-hook.added.md b/changelog.d/optimize-before-publish-hook.added.md new file mode 100644 index 000000000..10a42bdcc --- /dev/null +++ b/changelog.d/optimize-before-publish-hook.added.md @@ -0,0 +1,3 @@ +- Embedded hosts can pass `OptimizeRequest::with_before_publish`: it runs once + before an optimize run publishes, with the commit's nonce and input head, and + its failure publishes nothing. diff --git a/crates/omnigraph/src/db/mod.rs b/crates/omnigraph/src/db/mod.rs index efcccdd67..c1deff5bc 100644 --- a/crates/omnigraph/src/db/mod.rs +++ b/crates/omnigraph/src/db/mod.rs @@ -22,10 +22,10 @@ pub use omnigraph::{ ConfigurationClear, DatasetCleanupStats, DatasetOptimizeStats, DatasetRepairStats, DeferReason, EXPORT_CHUNK_MAX_BYTES, ExportCut, FullTextIndexRebuildResult, GraphCreateReconciliation, IndexBuildResult, InitOptions, MergeOutcome, MergeResult, Omnigraph, OpenMode, - OptimizeConcurrency, OptimizeLimits, OptimizeReport, OptimizeRequest, PendingIndex, - PreparedGraphCreate, PreparedSchemaApply, PreparedSchemaSettlement, ReadCheck, - RebuiltFullTextIndex, RepairAction, RepairClassification, RepairStats, - RetainedManifestVersions, SYSTEM_COLUMNS_PREFLIGHT, SchemaApplyReconciliation, + OptimizeBeforePublish, OptimizeConcurrency, OptimizeLimits, OptimizePublication, + OptimizeReport, OptimizeRequest, PendingIndex, PreparedGraphCreate, PreparedSchemaApply, + PreparedSchemaSettlement, ReadCheck, RebuiltFullTextIndex, RepairAction, RepairClassification, + RepairStats, RetainedManifestVersions, SYSTEM_COLUMNS_PREFLIGHT, SchemaApplyReconciliation, SchemaApplyResult, SchemaApplySettlement, SchemaContractDigest, SchemaNonPublicationProof, SkipReason, StagingVerdict, SystemColumnUpgradeFinding, SystemColumnUpgradeOptions, SystemColumnUpgradeOutcome, SystemColumnUpgradeReport, TableCollectionPlan, diff --git a/crates/omnigraph/src/db/omnigraph.rs b/crates/omnigraph/src/db/omnigraph.rs index ce9007f10..0ccca293d 100644 --- a/crates/omnigraph/src/db/omnigraph.rs +++ b/crates/omnigraph/src/db/omnigraph.rs @@ -52,8 +52,8 @@ pub(crate) use export::{ }; pub use optimize::{ ArtifactRefresh, ArtifactSkip, CleanupPolicyOptions, ConfigurationClear, DatasetCleanupStats, - DatasetOptimizeStats, DeferReason, OptimizeConcurrency, OptimizeLimits, OptimizeReport, - OptimizeRequest, SkipReason, TableOptimizeOutcome, + DatasetOptimizeStats, DeferReason, OptimizeBeforePublish, OptimizeConcurrency, OptimizeLimits, + OptimizePublication, OptimizeReport, OptimizeRequest, SkipReason, TableOptimizeOutcome, }; use prepared_create::initial_schema_ir; pub use prepared_create::{GraphCreateReconciliation, PreparedGraphCreate}; @@ -4744,6 +4744,7 @@ impl Omnigraph { txn, lineage_intent, publication, + None, ) .await } diff --git a/crates/omnigraph/src/db/omnigraph/optimize.rs b/crates/omnigraph/src/db/omnigraph/optimize.rs index 32cb7c536..9c13de364 100644 --- a/crates/omnigraph/src/db/omnigraph/optimize.rs +++ b/crates/omnigraph/src/db/omnigraph/optimize.rs @@ -24,8 +24,10 @@ //! explicit confirm flag at the CLI layer. use std::collections::VecDeque; +use std::sync::Arc; use std::time::Duration; +use futures::future::BoxFuture; use futures::stream::{FuturesOrdered, FuturesUnordered, StreamExt}; use lance::dataset::optimize::{CompactionMetrics, CompactionOptions}; use tokio::time::Instant; @@ -362,6 +364,47 @@ impl OptimizeLimits { } } +/// What a run's before-publish hook receives. +#[derive(Debug, Clone)] +#[non_exhaustive] +pub struct OptimizePublication { + /// The nonce of the commit's lineage intent. History stores the commit + /// under an id chosen per publish attempt (`hb1.{block}.{slot}.{nonce}`); + /// `omnigraph_core::graph_commit_id::commit_id_answers` matches such an id + /// to this nonce. + pub nonce: String, + /// The graph head every publish attempt reads: the commit's parent. + pub input_commit: Option, + /// The `__manifest` version of that head. + pub input_manifest_version: u64, + /// One outcome for every table the run considered, as the report lists them. + pub tables: Vec, +} + +/// Called once, after the run's tables are staged and prepared and before the +/// publisher's first attempt; a run that stages no table never calls it. An +/// error publishes nothing and fails the run with `Unpublished` evidence. +#[derive(Clone)] +pub struct OptimizeBeforePublish( + Arc BoxFuture<'static, Result<()>> + Send + Sync>, +); + +impl OptimizeBeforePublish { + pub fn new(hook: F) -> Self + where + F: Fn(OptimizePublication) -> Fut + Send + Sync + 'static, + Fut: std::future::Future> + Send + 'static, + { + Self(Arc::new(move |publication| Box::pin(hook(publication)))) + } +} + +impl std::fmt::Debug for OptimizeBeforePublish { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str("OptimizeBeforePublish(..)") + } +} + /// What one optimize run is asked to do. The default is the direct run as it /// has always been: every table on main, no bound, and both concurrencies from /// `OMNIGRAPH_MAINTENANCE_CONCURRENCY`. @@ -378,6 +421,7 @@ pub struct OptimizeRequest { pub limits: OptimizeLimits, /// `None` reads [`OptimizeConcurrency::from_env`]. pub concurrency: Option, + pub before_publish: Option, } impl OptimizeRequest { @@ -405,6 +449,11 @@ impl OptimizeRequest { self } + pub fn with_before_publish(mut self, hook: OptimizeBeforePublish) -> Self { + self.before_publish = Some(hook); + self + } + /// The tables this request visits, in order, out of the catalog's sorted /// keys. Refuses a malformed request before any effect. fn visit_order(&self, table_keys: &[String]) -> Result> { @@ -627,6 +676,18 @@ pub async fn optimize_all_datasets( .new_lineage_intent_for_branch(None, actor, HistoryReleaseBytes::PRODUCTION) .await .map_err(OmniError::unpublished)?; + let before_publish = request.before_publish.as_ref().map(|hook| { + let hook = hook.0.clone(); + let tables = tables.clone(); + move |intent: super::table_ops::PublicationIntent| { + hook(OptimizePublication { + nonce: intent.nonce, + input_commit: intent.input_commit, + input_manifest_version: intent.input_manifest_version, + tables: tables.clone(), + }) + } + }); let commit = super::table_ops::commit_updates_on_branch_with_expected( db, None, @@ -636,6 +697,9 @@ pub async fn optimize_all_datasets( &authority_txn, lineage, super::table_ops::Publication::Publish, + before_publish + .as_ref() + .map(|hook| hook as super::table_ops::BeforePublishHook<'_>), ) .await?; graph_commit_id = Some(commit.graph_commit_id); diff --git a/crates/omnigraph/src/db/omnigraph/table_ops.rs b/crates/omnigraph/src/db/omnigraph/table_ops.rs index c190084bc..a0a5f3bf7 100644 --- a/crates/omnigraph/src/db/omnigraph/table_ops.rs +++ b/crates/omnigraph/src/db/omnigraph/table_ops.rs @@ -625,6 +625,7 @@ async fn maintain_indices_for_branch( &txn, lineage, Publication::Publish, + None, ) .await?; Some(published.graph_commit_id) @@ -1586,10 +1587,28 @@ async fn publish_prepared( } } +/// What a publication's before-publish hook learns: the lineage intent's +/// nonce, which every attempt's history id carries +/// (`omnigraph_core::graph_commit_id::commit_id_answers` matches one to it), +/// and the head every attempt must read. +#[derive(Debug, Clone)] +pub(crate) struct PublicationIntent { + pub(crate) nonce: String, + pub(crate) input_commit: Option, + pub(crate) input_manifest_version: u64, +} + +/// A hook the shared publication helper calls once before publishing. +pub(crate) type BeforePublishHook<'a> = &'a ( + dyn Fn(PublicationIntent) -> futures::future::BoxFuture<'static, Result<()>> + Send + Sync + ); + /// Commit updates with a publisher-level OCC fence. The /// `expected_table_versions` map asserts the manifest's pre-write per-table /// versions; mismatches surface as `ManifestConflictDetails::PublishedDatasetVersionMismatch`. /// `publication` chooses the head-advancing publish or an unpublished commit. +/// `before_publish`, when given, runs once after the updates are prepared and +/// before the publisher's first attempt; its failure publishes nothing. pub(super) async fn commit_updates_on_branch_with_expected( db: &Omnigraph, branch: Option<&str>, @@ -1599,6 +1618,7 @@ pub(super) async fn commit_updates_on_branch_with_expected( txn: &crate::db::WriteTxn, lineage_intent: crate::db::manifest::LineageIntent, publication: Publication, + before_publish: Option>, ) -> Result { if branch != txn.branch.as_deref() { return Err(OmniError::manifest_internal( @@ -1612,6 +1632,25 @@ pub(super) async fn commit_updates_on_branch_with_expected( .await .map_err(OmniError::unpublished)?; + // Once, not per attempt: every attempt reads the same head and carries + // the same nonce, so another call would record no new fact. The hook runs + // before the coordinator lock, and the publisher has not been called, so + // whatever the hook's own error says, the graph did not move. + if let Some(before_publish) = before_publish { + before_publish(PublicationIntent { + nonce: lineage_intent.graph_commit_id.clone(), + input_commit: txn.authority.graph_head.clone(), + input_manifest_version: txn.base.graph_manifest_version(), + }) + .await + .map_err(|error| { + error + .into_completion_evidence() + .0 + .with_completion_evidence(crate::error::CompletionEvidence::Unpublished) + })?; + } + debug_assert_eq!(lineage_intent.actor_id.as_deref(), actor_id); let changes = prepared .iter() diff --git a/crates/omnigraph/tests/integration/maintenance.rs b/crates/omnigraph/tests/integration/maintenance.rs index 1c582bfa6..344d01df4 100644 --- a/crates/omnigraph/tests/integration/maintenance.rs +++ b/crates/omnigraph/tests/integration/maintenance.rs @@ -16,7 +16,8 @@ use lance::dataset::optimize::{CompactionOptions, compact_files}; use lance_core::datatypes::BlobHandling; use omnigraph::IndexCoverage; use omnigraph::db::{ - ArtifactRefresh, ConfigurationClear, DeferReason, OptimizeLimits, TableOptimizeOutcome, + ArtifactRefresh, ConfigurationClear, DeferReason, OptimizeBeforePublish, OptimizeLimits, + OptimizePublication, OptimizeRequest, TableOptimizeOutcome, }; use omnigraph::db::{ CleanupPolicyOptions, MergeOutcome, Omnigraph, ReadTarget, RepairAction, RepairClassification, @@ -322,6 +323,103 @@ async fn optimize_table_list_runs_only_those_tables_and_refuses_bad_requests() { ); } +fn recording_hook() -> ( + OptimizeBeforePublish, + std::sync::Arc>>, +) { + let seen = std::sync::Arc::new(std::sync::Mutex::new(Vec::new())); + let recorded = std::sync::Arc::clone(&seen); + let hook = OptimizeBeforePublish::new(move |publication| { + recorded.lock().unwrap().push(publication); + async { Ok(()) } + }); + (hook, seen) +} + +#[tokio::test] +async fn optimize_calls_before_publish_once_with_the_nonce_its_commit_answers() { + let dir = tempfile::tempdir().unwrap(); + let db = init_and_load(&dir).await; + add_person_fragments(&db).await; + let head = db.list_commits(None).await.unwrap()[0] + .graph_commit_id + .clone(); + + let (hook, seen) = recording_hook(); + let report = db + .optimize_as(None, OptimizeRequest::default().with_before_publish(hook)) + .await + .unwrap(); + let commit = report.graph_commit_id.clone().expect("the run published"); + let seen = std::mem::take(&mut *seen.lock().unwrap()); + assert_eq!(seen.len(), 1, "the hook runs once: {seen:?}"); + let publication = &seen[0]; + assert!( + omnigraph_core::graph_commit_id::commit_id_answers(&commit, &publication.nonce).unwrap(), + "{commit} answers {}", + publication.nonce + ); + assert_eq!(publication.input_commit.as_deref(), Some(head.as_str())); + assert_eq!(publication.input_commit, report.input_commit); + assert_eq!( + publication.input_manifest_version, + report.input_manifest_version + ); + let outcomes = |tables: &[omnigraph::db::DatasetOptimizeStats]| { + tables + .iter() + .map(|stat| (stat.type_key.clone(), stat.outcome)) + .collect::>() + }; + assert_eq!(outcomes(&publication.tables), outcomes(&report.tables)); + + // A run that stages no table publishes nothing and never calls the hook. + let (hook, seen) = recording_hook(); + let idle = db + .optimize_as(None, OptimizeRequest::default().with_before_publish(hook)) + .await + .unwrap(); + assert_eq!(idle.graph_commit_id, None, "{idle:?}"); + assert!(seen.lock().unwrap().is_empty()); +} + +#[tokio::test] +async fn optimize_publishes_nothing_when_before_publish_fails() { + let dir = tempfile::tempdir().unwrap(); + let db = init_and_load(&dir).await; + add_person_fragments(&db).await; + let head = db.list_commits(None).await.unwrap()[0] + .graph_commit_id + .clone(); + + // The hook's own evidence claims no effect; the staged rewrites exist, so + // the run must report them unpublished instead. + let hook = OptimizeBeforePublish::new(|_| async { + Err(OmniError::manifest("the operation record was not written").before_effect()) + }); + let error = db + .optimize_as(None, OptimizeRequest::default().with_before_publish(hook)) + .await + .unwrap_err(); + assert_eq!( + error.completion_evidence(), + Some(CompletionEvidence::Unpublished), + "{error}" + ); + assert_eq!( + db.list_commits(None).await.unwrap()[0].graph_commit_id, + head, + "a failed hook publishes nothing" + ); + + // The staged rewrites are unreachable; the next run re-plans and publishes. + let retry = db + .optimize_as(None, OptimizeRequest::default()) + .await + .unwrap(); + assert!(retry.graph_commit_id.is_some(), "{retry:?}"); +} + #[tokio::test] async fn optimize_after_load_then_again_is_idempotent() { let dir = tempfile::tempdir().unwrap(); diff --git a/docs/dev/writes.md b/docs/dev/writes.md index 3759ae506..739c2db0f 100644 --- a/docs/dev/writes.md +++ b/docs/dev/writes.md @@ -266,6 +266,16 @@ unpublished staging the collector reclaims once it is dead. A strict mutation prepared before Optimize's publication reports the same read-set conflict as it would after any other writer. +A run with `max_gate_hold` defers every table whose turn comes after the +deadline, measured from gate acquisition, and publishes the tables it finished +(`run_table_pipeline`). A caller's before-publish hook +(`OptimizeRequest::with_before_publish`) belongs to the shared publication +helper, `table_ops::commit_updates_on_branch_with_expected`: it runs once, +after the updates are prepared and before the coordinator lock and the +publisher's first attempt, with the lineage intent's nonce and the captured +head. Its failure publishes nothing and is `Unpublished`. The helper's other +callers pass no hook. + The index writer (`ensure_indices`, which `omnigraph index build` runs on any branch, and the explicit full-text rebuild) follows the same protocol: it opens each productive table at its pin, stages the complete BTREE/FTS/vector @@ -558,7 +568,7 @@ carried stored external reference the policy refuses fails as | Retryable authority movement before effects on a replay-safe adapter | Discard the complete attempt and reprepare boundedly | | Strict read-set movement | `ReadSetChanged` | | Exact duplicate on strict insert | `KeyConflict` | -| Mutation, load, Blob write or merge fails before publication, after detached effects | Typed error; no graph movement; the detached staging is unpublished and the collector reclaims it once its recorded authority is gone. The table store marks an opaque error (storage, I/O, internal) of a staged write or a detached commit `CompletionEvidence::Unpublished` at the source, and the mutation, load, Blob and merge writers mark the steps between, so a server does not treat that logical error as uncertain; native storage settlement is checked independently, and a typed refusal keeps its shape | +| Mutation, load, Blob write, merge or optimize fails before publication, after detached effects | Typed error; no graph movement; the detached staging is unpublished and the collector reclaims it once its recorded authority is gone. The table store marks an opaque error (storage, I/O, internal) of a staged write or a detached commit `CompletionEvidence::Unpublished` at the source, and the mutation, load, Blob, merge and optimize writers mark the steps between (optimize marks a failure before any table's staging starts `BeforeEffect`), so a server does not treat that logical error as uncertain; native storage settlement is checked independently, and a typed refusal keeps its shape | | A manifest request no re-issue settles | The write is reported in doubt and nothing records it. The version is created only if absent, so the request lands before the branch's next publication or never: that publication either loads it as the head and builds on it, or takes the version first, after which the late request fails its precondition (a race between the two is the ordinary version-CAS retry). The attempt's detached effects stay unreachable unless it lands, and the collector keeps them while their recorded publication authority is current | | The publisher fails | Every publisher failure carries its evidence. Its reads, preparation and history archiving, and a manifest request that was never sent (a failed transaction-file put precedes it) or that the store refused outright, are `Unpublished`. A manifest request with an ambiguous answer is re-issued unchanged by the dataset's commit handler until the store answers it; Lance reads a held version back by its transaction. Lance deletes the attempt's transaction file only on a missing version the commit's own request record calls definitive, so a request that may still land keeps the file its manifest would name. A version that may hold the commit after that is read back, and only an outcome the read-back cannot settle is reported in doubt | | Schema apply or the system-column upgrade reports an error after proven publication | `RecoveryRequired` can name the committed outcome; its tables and contract are durable, and refresh/reopen rebuilds only the in-memory view | From 4b9c4a2a19e9fa7652114cbea42df7b43594c9df Mon Sep 17 00:00:00 2001 From: Ragnor Comerford Date: Thu, 29 Oct 2026 09:59:02 -0700 Subject: [PATCH 06/13] feat(engine): fail only the table whose own optimize work fails A table's planning failure is labelled BeforeEffect and its staging failure Unpublished. Either reports the table failed, keeps the error on its stats, and leaves the other tables to publish in the run's one commit. A failure whose evidence stays uncertain still fails the run, as do failures outside one table: authority, catalog, gates and the publisher. The CLI reports a failed table with its error. The DST harnesses treat a failed table as the op's failure, so the convergence obligation still requires a rerun to finish every table. --- ...ptimize-table-failure-isolation.changed.md | 2 + crates/omnigraph-cli/src/output.rs | 52 +++++-- crates/omnigraph-dst/src/concurrent.rs | 4 +- crates/omnigraph-dst/src/harness.rs | 15 +- crates/omnigraph/src/db/omnigraph/optimize.rs | 90 +++++++++--- .../omnigraph/tests/detached_commit_matrix.rs | 8 ++ crates/omnigraph/tests/failpoints.rs | 136 ++++++++++-------- docs/dev/writes.md | 6 +- docs/user/operations/maintenance.md | 14 +- 9 files changed, 217 insertions(+), 110 deletions(-) create mode 100644 changelog.d/optimize-table-failure-isolation.changed.md diff --git a/changelog.d/optimize-table-failure-isolation.changed.md b/changelog.d/optimize-table-failure-isolation.changed.md new file mode 100644 index 000000000..5a6a4ca36 --- /dev/null +++ b/changelog.d/optimize-table-failure-isolation.changed.md @@ -0,0 +1,2 @@ +- A table whose own optimize work fails is reported `failed` with its error, and + the other tables still publish; before, one table's failure failed the run. diff --git a/crates/omnigraph-cli/src/output.rs b/crates/omnigraph-cli/src/output.rs index 3d002047b..0dc107404 100644 --- a/crates/omnigraph-cli/src/output.rs +++ b/crates/omnigraph-cli/src/output.rs @@ -62,6 +62,7 @@ pub(crate) fn write_optimize( "committed": s.committed(), "skipped": s.skipped().map(|r| r.as_str()), "duration_ms": s.duration.as_millis() as u64, + "error": s.failure.as_ref().map(|error| error.to_string()), "pending_indexes": s.pending_indexes.iter().map(|p| serde_json::json!({ "property": p.property, "reason": p.reason, @@ -87,6 +88,14 @@ pub(crate) fn write_optimize( subject, s.fragments_removed, s.fragments_added )?; } + TableOptimizeOutcome::Failed => { + let error = s.failure.as_ref().map(|error| error.to_string()); + let first = error + .as_deref() + .and_then(|error| error.lines().next()) + .unwrap_or("unknown error"); + writeln!(stdout, " {subject:<40} failed ({first})")?; + } _ => writeln!(stdout, " {subject:<40} no-op")?, } for p in &s.pending_indexes { @@ -2033,7 +2042,15 @@ mod tests { let mut noop = skipped.clone(); noop.type_key = "node:Yak".into(); noop.outcome = TableOptimizeOutcome::Unchanged; - report.tables = vec![compacted, skipped, noop, deferred]; + let mut failed = skipped.clone(); + failed.type_key = "node:Xerus".into(); + failed.outcome = TableOptimizeOutcome::Failed; + failed.duration = std::time::Duration::from_millis(5); + failed.failure = Some(std::sync::Arc::new(omnigraph::error::OmniError::manifest( + "rewrite lost", + ))); + let failure = failed.failure.as_ref().unwrap().to_string(); + report.tables = vec![compacted, skipped, noop, failed, deferred]; report.input_commit = Some("hb1.parent".into()); report.graph_commit_id = Some("hb1.child".into()); report.resume_at = Some("node:Zebra".into()); @@ -2041,16 +2058,20 @@ mod tests { report.configuration_clear = ConfigurationClear::Unknown("timed out".into()); report.artifact = ArtifactRefresh::Skipped(ArtifactSkip::NoEdgeTypes); assert_maintenance_report( - concat!( - "optimize graph — 4 datasets\n", - " node type 'Person' frags 3 → 1 ✓\n", - " ↳ index pending on property 'embedding': no trainable vectors\n", - " edge type 'Knows' skipped (published dataset/Lance HEAD drift — run omnigraph repair)\n", - " node type 'Yak' no-op\n", - " node type 'Zebra' deferred (the run reached its writer-gate hold limit)\n", - "graph commit: hb1.child\n", - "resume at: node type 'Zebra'\n", - "configuration clear: unknown (timed out)\n", + &format!( + concat!( + "optimize graph — 5 datasets\n", + " node type 'Person' frags 3 → 1 ✓\n", + " ↳ index pending on property 'embedding': no trainable vectors\n", + " edge type 'Knows' skipped (published dataset/Lance HEAD drift — run omnigraph repair)\n", + " node type 'Yak' no-op\n", + " node type 'Xerus' failed ({failure})\n", + " node type 'Zebra' deferred (the run reached its writer-gate hold limit)\n", + "graph commit: hb1.child\n", + "resume at: node type 'Zebra'\n", + "configuration clear: unknown (timed out)\n", + ), + failure = failure ), serde_json::json!({ "uri":"graph", @@ -2061,10 +2082,11 @@ mod tests { "configuration_clear":{"outcome":"unknown","error":"timed out"}, "artifact":{"outcome":"skipped","reason":"no_edge_types","error":null}, "datasets":[ - {"type_key":"node:Person","outcome":"optimized","reason":null,"fragments_removed":3,"fragments_added":1,"committed":true,"skipped":null,"duration_ms":12,"pending_indexes":[{"property":"embedding","reason":"no trainable vectors"}]}, - {"type_key":"edge:Knows","outcome":"skipped","reason":"drift_needs_repair","fragments_removed":0,"fragments_added":0,"committed":false,"skipped":"drift_needs_repair","duration_ms":3,"pending_indexes":[]}, - {"type_key":"node:Yak","outcome":"unchanged","reason":null,"fragments_removed":0,"fragments_added":0,"committed":false,"skipped":null,"duration_ms":3,"pending_indexes":[]}, - {"type_key":"node:Zebra","outcome":"deferred","reason":"gate_hold_limit","fragments_removed":0,"fragments_added":0,"committed":false,"skipped":null,"duration_ms":0,"pending_indexes":[]}, + {"type_key":"node:Person","outcome":"optimized","reason":null,"fragments_removed":3,"fragments_added":1,"committed":true,"skipped":null,"duration_ms":12,"error":null,"pending_indexes":[{"property":"embedding","reason":"no trainable vectors"}]}, + {"type_key":"edge:Knows","outcome":"skipped","reason":"drift_needs_repair","fragments_removed":0,"fragments_added":0,"committed":false,"skipped":"drift_needs_repair","duration_ms":3,"error":null,"pending_indexes":[]}, + {"type_key":"node:Yak","outcome":"unchanged","reason":null,"fragments_removed":0,"fragments_added":0,"committed":false,"skipped":null,"duration_ms":3,"error":null,"pending_indexes":[]}, + {"type_key":"node:Xerus","outcome":"failed","reason":null,"fragments_removed":0,"fragments_added":0,"committed":false,"skipped":null,"duration_ms":5,"error":failure,"pending_indexes":[]}, + {"type_key":"node:Zebra","outcome":"deferred","reason":"gate_hold_limit","fragments_removed":0,"fragments_added":0,"committed":false,"skipped":null,"duration_ms":0,"error":null,"pending_indexes":[]}, ], }), |json, writer| super::write_optimize("graph", &report, json, writer), diff --git a/crates/omnigraph-dst/src/concurrent.rs b/crates/omnigraph-dst/src/concurrent.rs index efc0a7d5a..a7b83a167 100644 --- a/crates/omnigraph-dst/src/concurrent.rs +++ b/crates/omnigraph-dst/src/concurrent.rs @@ -1754,9 +1754,7 @@ fn maintenance_life( loop { let result = match kind { 0 => Box::pin(db.ensure_indices_as(None)).await.map(|_| ()), - 1 => Box::pin(db.optimize_as(None, omnigraph::db::OptimizeRequest::default())) - .await - .map(|_| ()), + 1 => crate::harness::optimize_to_completion(&db).await, _ => Box::pin(db.cleanup_as( CleanupPolicyOptions { keep_versions: Some(1), diff --git a/crates/omnigraph-dst/src/harness.rs b/crates/omnigraph-dst/src/harness.rs index 70546e3e1..bc3d73a7d 100644 --- a/crates/omnigraph-dst/src/harness.rs +++ b/crates/omnigraph-dst/src/harness.rs @@ -1578,6 +1578,17 @@ fn sample_op(rng: &mut SplitMix64, model: &Model, next_ver: &mut i64, hostile: b } } +/// One full optimize run, whose failed table is the run's failure to the +/// oracles: an op must finish every table it considers. +pub(crate) async fn optimize_to_completion(db: &Session) -> OmniResult<()> { + let report = Box::pin(db.optimize_as(None, omnigraph::db::OptimizeRequest::default())).await?; + match report.tables.into_iter().find_map(|stat| stat.failure) { + Some(failure) => Err(std::sync::Arc::try_unwrap(failure) + .unwrap_or_else(|shared| OmniError::manifest_internal(shared.to_string()))), + None => Ok(()), + } +} + async fn exec_op(db: &Session, branch: &str, op: &Op) -> OmniResult<()> { match op { Op::InsertV { name, age, ver } => mutate_on( @@ -1634,9 +1645,7 @@ async fn exec_op(db: &Session, branch: &str, op: &Op) -> OmniResult<()> { ) .await .map(|_| ()), - Op::Optimize => Box::pin(db.optimize_as(None, omnigraph::db::OptimizeRequest::default())) - .await - .map(|_| ()), + Op::Optimize => optimize_to_completion(db).await, Op::Cleanup => { let stats = Box::pin(db.cleanup_as( omnigraph::db::CleanupPolicyOptions { diff --git a/crates/omnigraph/src/db/omnigraph/optimize.rs b/crates/omnigraph/src/db/omnigraph/optimize.rs index 9c13de364..78352fba9 100644 --- a/crates/omnigraph/src/db/omnigraph/optimize.rs +++ b/crates/omnigraph/src/db/omnigraph/optimize.rs @@ -33,6 +33,7 @@ use lance::dataset::optimize::{CompactionMetrics, CompactionOptions}; use tokio::time::Instant; use super::*; +use crate::error::CompletionEvidence; use crate::seams::{decide_seam, fail}; /// How many datasets to optimize/cleanup concurrently. Each has separate @@ -109,6 +110,10 @@ pub enum TableOptimizeOutcome { Deferred(DeferReason), /// Deliberately left alone. Skipped(SkipReason), + /// A failure of this table's own planning or staging, with definite + /// evidence: its staging is unreachable and the other tables still + /// publish. [`DatasetOptimizeStats::failure`] holds the error. + Failed, } impl TableOptimizeOutcome { @@ -120,6 +125,7 @@ impl TableOptimizeOutcome { TableOptimizeOutcome::Unchanged => "unchanged", TableOptimizeOutcome::Deferred(_) => "deferred", TableOptimizeOutcome::Skipped(_) => "skipped", + TableOptimizeOutcome::Failed => "failed", } } } @@ -173,6 +179,9 @@ pub struct DatasetOptimizeStats { /// Time from the start of the table's planning to the end of its staging; /// zero for a table the run did not start. pub duration: Duration, + /// The error of a [`TableOptimizeOutcome::Failed`] table, carrying its + /// completion evidence (`BeforeEffect` or `Unpublished`). + pub failure: Option>, } impl DatasetOptimizeStats { @@ -188,6 +197,7 @@ impl DatasetOptimizeStats { fragments_added: metrics.fragments_added, pending_indexes: Vec::new(), duration: Duration::ZERO, + failure: None, } } @@ -199,6 +209,19 @@ impl DatasetOptimizeStats { fragments_added: 0, pending_indexes: Vec::new(), duration: Duration::ZERO, + failure: None, + } + } + + fn failed(type_key: String, failure: Arc, duration: Duration) -> Self { + Self { + type_key, + outcome: TableOptimizeOutcome::Failed, + fragments_removed: 0, + fragments_added: 0, + pending_indexes: Vec::new(), + duration, + failure: Some(failure), } } @@ -754,8 +777,9 @@ type PlannedTable = (usize, Instant, Result); /// every earlier table's has, with at most `concurrency.rewrite` at once. /// When a table's turn comes after `deadline`, that table and every later /// one are deferred and their plans, which only read, are dropped; the first -/// table always starts. A failure fails the run before its publication: -/// `BeforeEffect` until a table's staging starts, `Unpublished` after. +/// table always starts. A table's planning failure is `BeforeEffect` and its +/// staging failure `Unpublished`: either fails only that table. A failure +/// whose evidence stays uncertain fails the run. async fn run_table_pipeline( db: &Omnigraph, catalog: &std::sync::Arc, @@ -791,14 +815,6 @@ async fn run_table_pipeline( let mut staging = FuturesUnordered::new(); let mut turns = 0usize; let mut stopped_at = None; - let mut staged_any = false; - let label = |error: OmniError, staged_any: bool| { - if staged_any { - error.unpublished() - } else { - error.before_effect() - } - }; loop { if stopped_at.is_none() { @@ -817,15 +833,26 @@ async fn run_table_pipeline( break; } turns += 1; - match result.map_err(|error| label(error, staged_any))? { - OptimizePreparation::Stat(mut stat) => { + match result { + Ok(OptimizePreparation::Stat(mut stat)) => { stat.duration = started.elapsed(); visited[index] = Some((stat, None)); } - OptimizePreparation::Work(work) => { - staged_any = true; + Ok(OptimizePreparation::Work(work)) => { staging.push(stage(index, started, work)); } + // Planning only reads. + Err(error) => { + let failure = table_failure(&keys[index], error.before_effect())?; + visited[index] = Some(( + DatasetOptimizeStats::failed( + keys[index].clone(), + failure, + started.elapsed(), + ), + None, + )); + } } } if stopped_at.is_some() { @@ -843,9 +870,21 @@ async fn run_table_pipeline( tokio::select! { biased; Some((index, started, result)) = staging.next(), if !staging.is_empty() => { - let mut outcome = result.map_err(OmniError::unpublished)?; - outcome.stat.duration = started.elapsed(); - visited[index] = Some((outcome.stat, outcome.effect)); + visited[index] = Some(match result { + Ok(mut outcome) => { + outcome.stat.duration = started.elapsed(); + (outcome.stat, outcome.effect) + } + // Staging writes only detached links no manifest names. + Err(error) => { + let error = error.with_completion_evidence(CompletionEvidence::Unpublished); + let failure = table_failure(&keys[index], error)?; + ( + DatasetOptimizeStats::failed(keys[index].clone(), failure, started.elapsed()), + None, + ) + } + }); } Some(planned) = planning.next(), if !planning.is_empty() && stopped_at.is_none() => { ready.push_back(planned); @@ -874,6 +913,23 @@ async fn run_table_pipeline( Ok(VisitedTables { tables, resume_at }) } +/// Keep a table's failure with its table when its evidence is definite; any +/// other failure, an uncertain one above all, fails the run. +fn table_failure(table_key: &str, error: OmniError) -> Result> { + match error.completion_evidence() { + Some(CompletionEvidence::BeforeEffect | CompletionEvidence::Unpublished) => { + tracing::warn!( + target: "omnigraph::optimize", + table = table_key, + error = %error, + "optimize left a table unchanged after its own failure" + ); + Ok(Arc::new(error)) + } + _ => Err(error), + } +} + /// Build the full-catalog graph index from a fresh main snapshot and write it /// as the persisted adjacency artifact. Skipped when the catalog declares no /// edge types (nothing to traverse), when any declared edge table is not yet diff --git a/crates/omnigraph/tests/detached_commit_matrix.rs b/crates/omnigraph/tests/detached_commit_matrix.rs index daca0a1a7..166414040 100644 --- a/crates/omnigraph/tests/detached_commit_matrix.rs +++ b/crates/omnigraph/tests/detached_commit_matrix.rs @@ -775,6 +775,14 @@ async fn run_cell( Writer::BlobClear => clear_photo(&db).await, }; acknowledged = outcome.is_ok(); + // A failure after one table's detached effect fails only that + // table, so the run itself succeeds. + if writer == Writer::Optimize && matches!(window, Window::PostDetached(_)) { + assert!( + acknowledged, + "{cell}: a table-local failure does not fail the run: {outcome:?}" + ); + } if let Err(error) = outcome { // A data writer that fails after a detached effect and before // its publication never reports an outcome it cannot diff --git a/crates/omnigraph/tests/failpoints.rs b/crates/omnigraph/tests/failpoints.rs index 87f2e61ca..f22dfc5f1 100644 --- a/crates/omnigraph/tests/failpoints.rs +++ b/crates/omnigraph/tests/failpoints.rs @@ -4768,100 +4768,110 @@ node Embedding { ); } -/// RFC 0067: a failure after one table's detached rewrite and before the -/// batch publishes leaves no residue: no pin moves, no lineage is added, -/// no linear HEAD moves, and the retry compacts every table. +/// A failure after one table's detached rewrite fails only that table: its +/// pin stays, the other table's rewrite publishes in the run's one commit, no +/// linear HEAD moves, and the next run compacts the failed table. #[tokio::test(flavor = "multi_thread", worker_threads = 4)] #[serial(optimize)] -async fn optimize_partial_table_effect_leaves_no_residue() { +async fn optimize_table_failure_fails_only_that_table_and_publishes_the_rest() { let _scenario = FailScenario::setup(); let dir = tempfile::tempdir().unwrap(); let uri = dir.path().to_str().unwrap().to_string(); seed_two_productive_optimize_tables(&uri).await; let db = helpers::session(Omnigraph::open(&uri).await.unwrap()); + let pin = |snapshot: &omnigraph::db::Snapshot, key: &str| { + snapshot.dataset(key).unwrap().published_dataset_version + }; let snapshot_before = db .snapshot_of(omnigraph::db::ReadTarget::branch("main")) .await .unwrap(); - let person_pin = snapshot_before - .dataset("node:Person") - .unwrap() - .published_dataset_version; - let company_pin = snapshot_before - .dataset("node:Company") - .unwrap() - .published_dataset_version; let commits_before = db.list_commits(Some("main")).await.unwrap().len(); - let person_uri = node_table_uri(&db, "Person").await; - let company_uri = node_table_uri(&db, "Company").await; - let person_head = helpers::open_dataset_head_exact(&person_uri, None) - .await - .version() - .version; - let company_head = helpers::open_dataset_head_exact(&company_uri, None) - .await - .version() - .version; - let _failpoint = catalog::OPTIMIZE_POST_TABLE_EFFECT.fire_once_at(1); - let error = db - .optimize_as(None, omnigraph::db::OptimizeRequest::default()) - .await - .unwrap_err(); - assert!( - !matches!(error, OmniError::RecoveryRequired { .. }), - "a partial batch before publication needs no recovery: {error}" + let mut heads = Vec::new(); + for type_name in ["Company", "Person"] { + let table_uri = node_table_uri(&db, type_name).await; + let head = helpers::open_dataset_head_exact(&table_uri, None) + .await + .version() + .version; + heads.push((table_uri, head)); + } + + let report = { + let _failpoint = catalog::OPTIMIZE_POST_TABLE_EFFECT.fire_once_at(1); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap() + }; + let failed = report + .tables + .iter() + .filter(|stat| stat.outcome == omnigraph::db::TableOptimizeOutcome::Failed) + .collect::>(); + assert_eq!(failed.len(), 1, "{report:?}"); + let failed = failed[0]; + assert_eq!( + failed.failure.as_ref().unwrap().completion_evidence(), + Some(omnigraph::error::CompletionEvidence::Unpublished), + "{failed:?}" + ); + let published = report + .tables + .iter() + .find(|stat| stat.committed()) + .expect("the other table publishes"); + assert_ne!(published.type_key, failed.type_key); + assert!(report.graph_commit_id.is_some(), "{report:?}"); + assert_eq!( + db.list_commits(Some("main")).await.unwrap().len(), + commits_before + 1, + "the finished table publishes in one commit" ); - assert_no_recovery_sidecars(dir.path()); let snapshot_after = db .snapshot_of(omnigraph::db::ReadTarget::branch("main")) .await .unwrap(); assert_eq!( - snapshot_after - .dataset("node:Person") - .unwrap() - .published_dataset_version, - person_pin - ); - assert_eq!( - snapshot_after - .dataset("node:Company") - .unwrap() - .published_dataset_version, - company_pin + pin(&snapshot_after, &failed.type_key), + pin(&snapshot_before, &failed.type_key), + "the failed table's pin stays" ); - assert_eq!( - db.list_commits(Some("main")).await.unwrap().len(), - commits_before, - "a partial Optimize publishes no lineage" + assert_ne!( + pin(&snapshot_after, &published.type_key), + pin(&snapshot_before, &published.type_key), + "the other table's rewrite is published" ); - for (uri, head_before) in [(&person_uri, person_head), (&company_uri, company_head)] { - let head = lance::Dataset::open(uri).await.unwrap().version().version; + assert_no_recovery_sidecars(dir.path()); + for (table_uri, head_before) in &heads { + let head = lance::Dataset::open(table_uri) + .await + .unwrap() + .version() + .version; assert_eq!( - head, head_before, + head, *head_before, "a detached rewrite never moves the linear HEAD" ); } + drop(db); let recovered = helpers::session(Omnigraph::open(&uri).await.unwrap()); assert_eq!(helpers::count_rows(&recovered, "node:Person").await, 4); assert_eq!(helpers::count_rows(&recovered, "node:Company").await, 4); - assert_eq!( - recovered.list_commits(Some("main")).await.unwrap().len(), - commits_before - ); - let stats = recovered + let retry = recovered .optimize_as(None, omnigraph::db::OptimizeRequest::default()) .await - .unwrap() - .tables; + .unwrap(); + let compacted = retry + .tables + .iter() + .filter(|stat| stat.committed()) + .map(|stat| stat.type_key.as_str()) + .collect::>(); assert_eq!( - stats - .iter() - .filter(|stat| stat.type_key != "__manifest" && stat.committed()) - .count(), - 2, - "the retry compacts both tables: {stats:?}" + compacted, + [failed.type_key.as_str()], + "the next run compacts the failed table: {retry:?}" ); } diff --git a/docs/dev/writes.md b/docs/dev/writes.md index 739c2db0f..4175103de 100644 --- a/docs/dev/writes.md +++ b/docs/dev/writes.md @@ -268,7 +268,9 @@ it would after any other writer. A run with `max_gate_hold` defers every table whose turn comes after the deadline, measured from gate acquisition, and publishes the tables it finished -(`run_table_pipeline`). A caller's before-publish hook +(`run_table_pipeline`). A table whose own planning or staging fails with +definite evidence is reported `failed` and the others still publish; a +failure whose evidence stays uncertain fails the run. A caller's before-publish hook (`OptimizeRequest::with_before_publish`) belongs to the shared publication helper, `table_ops::commit_updates_on_branch_with_expected`: it runs once, after the updates are prepared and before the coordinator lock and the @@ -568,7 +570,7 @@ carried stored external reference the policy refuses fails as | Retryable authority movement before effects on a replay-safe adapter | Discard the complete attempt and reprepare boundedly | | Strict read-set movement | `ReadSetChanged` | | Exact duplicate on strict insert | `KeyConflict` | -| Mutation, load, Blob write, merge or optimize fails before publication, after detached effects | Typed error; no graph movement; the detached staging is unpublished and the collector reclaims it once its recorded authority is gone. The table store marks an opaque error (storage, I/O, internal) of a staged write or a detached commit `CompletionEvidence::Unpublished` at the source, and the mutation, load, Blob, merge and optimize writers mark the steps between (optimize marks a failure before any table's staging starts `BeforeEffect`), so a server does not treat that logical error as uncertain; native storage settlement is checked independently, and a typed refusal keeps its shape | +| Mutation, load, Blob write, merge or optimize fails before publication, after detached effects | Typed error; no graph movement; the detached staging is unpublished and the collector reclaims it once its recorded authority is gone. The table store marks an opaque error (storage, I/O, internal) of a staged write or a detached commit `CompletionEvidence::Unpublished` at the source, and the mutation, load, Blob, merge and optimize writers mark the steps between (an optimize table's own failure, `BeforeEffect` in planning and `Unpublished` in staging, fails only that table), so a server does not treat that logical error as uncertain; native storage settlement is checked independently, and a typed refusal keeps its shape | | A manifest request no re-issue settles | The write is reported in doubt and nothing records it. The version is created only if absent, so the request lands before the branch's next publication or never: that publication either loads it as the head and builds on it, or takes the version first, after which the late request fails its precondition (a race between the two is the ordinary version-CAS retry). The attempt's detached effects stay unreachable unless it lands, and the collector keeps them while their recorded publication authority is current | | The publisher fails | Every publisher failure carries its evidence. Its reads, preparation and history archiving, and a manifest request that was never sent (a failed transaction-file put precedes it) or that the store refused outright, are `Unpublished`. A manifest request with an ambiguous answer is re-issued unchanged by the dataset's commit handler until the store answers it; Lance reads a held version back by its transaction. Lance deletes the attempt's transaction file only on a missing version the commit's own request record calls definitive, so a request that may still land keeps the file its manifest would name. A version that may hold the commit after that is read back, and only an outcome the read-back cannot settle is reported in doubt | | Schema apply or the system-column upgrade reports an error after proven publication | `RecoveryRequired` can name the committed outcome; its tables and contract are durable, and refresh/reopen rebuilds only the in-memory view | diff --git a/docs/user/operations/maintenance.md b/docs/user/operations/maintenance.md index 92ab49884..34dcc4fe4 100644 --- a/docs/user/operations/maintenance.md +++ b/docs/user/operations/maintenance.md @@ -79,13 +79,13 @@ that many at once, so budget for that many Blob tables compacting together. ### Bounding a run -Writes to `main` wait while optimize holds its writer gates. With -`--max-gate-hold 30s` it starts no table after holding them that long, except -the first: the rest are reported `deferred`, the finished tables are published, -and a writer waits at most the bound plus one table's rewrite. Tables run in -type-key order from the first; `resume_at` names the first deferred table, and -`--tables` runs only the listed tables, in that order. In JSON each table has an -`outcome` (`optimized`, `unchanged`, `deferred` or `skipped`); the report adds +Writes to `main` wait while optimize holds its writer gates. A run given +`--max-gate-hold 30s` starts no table after that long except the first: the +rest are `deferred`, finished tables publish, and a writer waits at most the +bound plus one rewrite. Tables run in type-key order; `resume_at` names the +first deferred one, and `--tables` runs only the listed tables, in order. Each +table's `outcome` is `optimized`, `unchanged`, `deferred`, `skipped` or +`failed` (with its `error`; the other tables still publish). The report adds `input_commit`, `graph_commit_id`, `gate_hold_ms` and the `configuration_clear` and `artifact` outcomes, which never fail a published run. From 50b6d0c8dbc95bc8f4bdd6f65bf0414dfb32710b Mon Sep 17 00:00:00 2001 From: Ragnor Comerford Date: Thu, 29 Oct 2026 09:59:02 -0700 Subject: [PATCH 07/13] feat(engine): stop optimize at its next safe boundary on cancellation OptimizeRequest::with_cancellation takes a shared handle. The run checks it before every table's plan and rewrite and before the publication, and awaits it meanwhile, so a table in progress is dropped where it stands; its detached effects stay unreachable. A cancelled run defers every table as cancelled, publishes nothing, leaves the configuration and the resume position as they were, and skips the artifact scan. A cancellation that arrives after the last check lets the run finish and is reported as having arrived late; only the artifact scan is skipped. --- changelog.d/optimize-cancellation.added.md | 2 + crates/omnigraph/src/db/mod.rs | 17 +- crates/omnigraph/src/db/omnigraph.rs | 5 +- crates/omnigraph/src/db/omnigraph/optimize.rs | 243 +++++++++++++++--- crates/omnigraph/tests/failpoints.rs | 70 +++++ .../tests/integration/maintenance.rs | 89 ++++++- docs/dev/writes.md | 6 +- 7 files changed, 382 insertions(+), 50 deletions(-) create mode 100644 changelog.d/optimize-cancellation.added.md diff --git a/changelog.d/optimize-cancellation.added.md b/changelog.d/optimize-cancellation.added.md new file mode 100644 index 000000000..6fac2560f --- /dev/null +++ b/changelog.d/optimize-cancellation.added.md @@ -0,0 +1,2 @@ +- Embedded hosts can cancel an optimize run with `OptimizeRequest::with_cancellation`: + it stops before the next table or the publication and publishes nothing. diff --git a/crates/omnigraph/src/db/mod.rs b/crates/omnigraph/src/db/mod.rs index c1deff5bc..369d5a6c9 100644 --- a/crates/omnigraph/src/db/mod.rs +++ b/crates/omnigraph/src/db/mod.rs @@ -22,14 +22,15 @@ pub use omnigraph::{ ConfigurationClear, DatasetCleanupStats, DatasetOptimizeStats, DatasetRepairStats, DeferReason, EXPORT_CHUNK_MAX_BYTES, ExportCut, FullTextIndexRebuildResult, GraphCreateReconciliation, IndexBuildResult, InitOptions, MergeOutcome, MergeResult, Omnigraph, OpenMode, - OptimizeBeforePublish, OptimizeConcurrency, OptimizeLimits, OptimizePublication, - OptimizeReport, OptimizeRequest, PendingIndex, PreparedGraphCreate, PreparedSchemaApply, - PreparedSchemaSettlement, ReadCheck, RebuiltFullTextIndex, RepairAction, RepairClassification, - RepairStats, RetainedManifestVersions, SYSTEM_COLUMNS_PREFLIGHT, SchemaApplyReconciliation, - SchemaApplyResult, SchemaApplySettlement, SchemaContractDigest, SchemaNonPublicationProof, - SkipReason, StagingVerdict, SystemColumnUpgradeFinding, SystemColumnUpgradeOptions, - SystemColumnUpgradeOutcome, SystemColumnUpgradeReport, TableCollectionPlan, - TableOptimizeOutcome, UnattachedTagRecord, UnpublishedManifest, unattached_release_order, + OptimizeBeforePublish, OptimizeCancellation, OptimizeConcurrency, OptimizeLimits, + OptimizePublication, OptimizeReport, OptimizeRequest, PendingIndex, PreparedGraphCreate, + PreparedSchemaApply, PreparedSchemaSettlement, ReadCheck, RebuiltFullTextIndex, RepairAction, + RepairClassification, RepairStats, RetainedManifestVersions, RunCancellation, + SYSTEM_COLUMNS_PREFLIGHT, SchemaApplyReconciliation, SchemaApplyResult, SchemaApplySettlement, + SchemaContractDigest, SchemaNonPublicationProof, SkipReason, StagingVerdict, + SystemColumnUpgradeFinding, SystemColumnUpgradeOptions, SystemColumnUpgradeOutcome, + SystemColumnUpgradeReport, TableCollectionPlan, TableOptimizeOutcome, UnattachedTagRecord, + UnpublishedManifest, unattached_release_order, }; pub(crate) use omnigraph::{ LogicalBlobValue, RangedExternalBlobs, logical_row_image, row_blob_values, diff --git a/crates/omnigraph/src/db/omnigraph.rs b/crates/omnigraph/src/db/omnigraph.rs index 0ccca293d..8cf5cd400 100644 --- a/crates/omnigraph/src/db/omnigraph.rs +++ b/crates/omnigraph/src/db/omnigraph.rs @@ -52,8 +52,9 @@ pub(crate) use export::{ }; pub use optimize::{ ArtifactRefresh, ArtifactSkip, CleanupPolicyOptions, ConfigurationClear, DatasetCleanupStats, - DatasetOptimizeStats, DeferReason, OptimizeBeforePublish, OptimizeConcurrency, OptimizeLimits, - OptimizePublication, OptimizeReport, OptimizeRequest, SkipReason, TableOptimizeOutcome, + DatasetOptimizeStats, DeferReason, OptimizeBeforePublish, OptimizeCancellation, + OptimizeConcurrency, OptimizeLimits, OptimizePublication, OptimizeReport, OptimizeRequest, + RunCancellation, SkipReason, TableOptimizeOutcome, }; use prepared_create::initial_schema_ir; pub use prepared_create::{GraphCreateReconciliation, PreparedGraphCreate}; diff --git a/crates/omnigraph/src/db/omnigraph/optimize.rs b/crates/omnigraph/src/db/omnigraph/optimize.rs index 78352fba9..39fcb87da 100644 --- a/crates/omnigraph/src/db/omnigraph/optimize.rs +++ b/crates/omnigraph/src/db/omnigraph/optimize.rs @@ -138,6 +138,8 @@ pub enum DeferReason { /// table's turn came. This table and every later one wait for a later /// run, which [`OptimizeReport::resume_at`] starts at. GateHoldLimit, + /// The run was cancelled before this table's work could land. + Cancelled, } impl DeferReason { @@ -146,6 +148,7 @@ impl DeferReason { pub fn as_str(&self) -> &'static str { match self { DeferReason::GateHoldLimit => "gate_hold_limit", + DeferReason::Cancelled => "cancelled", } } } @@ -154,6 +157,7 @@ impl std::fmt::Display for DeferReason { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { f.write_str(match self { DeferReason::GateHoldLimit => "the run reached its writer-gate hold limit", + DeferReason::Cancelled => "the run was cancelled", }) } } @@ -256,6 +260,8 @@ pub enum ConfigurationClear { Failed(String), /// Whether the removal committed has no definite answer. Unknown(String), + /// The run was cancelled before its publication, so the clear did not run. + NotRun, } impl ConfigurationClear { @@ -267,6 +273,7 @@ impl ConfigurationClear { ConfigurationClear::NothingToClear => "nothing_to_clear", ConfigurationClear::Failed(_) => "failed", ConfigurationClear::Unknown(_) => "unknown", + ConfigurationClear::NotRun => "not_run", } } } @@ -309,6 +316,8 @@ pub enum ArtifactSkip { /// A declared edge type has no table yet, so the artifact has no complete /// identity to stamp. IncompleteEdgeTables, + /// The run was cancelled, so the edge scan did not start. + Cancelled, } impl ArtifactSkip { @@ -318,6 +327,7 @@ impl ArtifactSkip { match self { ArtifactSkip::NoEdgeTypes => "no_edge_types", ArtifactSkip::IncompleteEdgeTables => "incomplete_edge_tables", + ArtifactSkip::Cancelled => "cancelled", } } } @@ -346,6 +356,78 @@ pub struct OptimizeReport { pub configuration_clear: ConfigurationClear, /// The adjacency artifact refresh, after the gates are released. pub artifact: ArtifactRefresh, + /// Whether a cancellation stopped the run. + pub cancellation: RunCancellation, +} + +/// What a run's cancellation handle did. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[non_exhaustive] +pub enum RunCancellation { + /// No cancellation was requested. + NotRequested, + /// The run stopped at a boundary before its publication and published + /// nothing: every table it had not finished, and every staged one, is + /// `deferred(cancelled)`. + Cancelled, + /// The cancellation arrived after the run passed its last check before + /// the publication, so the run finished; only the artifact scan was + /// skipped. + ArrivedLate, +} + +impl RunCancellation { + /// Stable machine-readable token for serialized output. Once emitted this + /// is part of the output contract — keep it stable. + pub fn as_str(&self) -> &'static str { + match self { + RunCancellation::NotRequested => "not_requested", + RunCancellation::Cancelled => "cancelled", + RunCancellation::ArrivedLate => "arrived_late", + } + } +} + +/// A request to stop a run at its next safe boundary: before a table's plan, +/// before its rewrite, and before the publication. A table in progress is +/// dropped where it stands; its detached effects stay unreachable. Clones +/// share one request. +#[derive(Debug, Clone, Default)] +pub struct OptimizeCancellation(Arc); + +#[derive(Debug, Default)] +struct CancellationState { + requested: std::sync::atomic::AtomicBool, + notify: tokio::sync::Notify, +} + +impl OptimizeCancellation { + pub fn new() -> Self { + Self::default() + } + + pub fn cancel(&self) { + self.0 + .requested + .store(true, std::sync::atomic::Ordering::SeqCst); + self.0.notify.notify_waiters(); + } + + pub fn is_cancelled(&self) -> bool { + self.0.requested.load(std::sync::atomic::Ordering::SeqCst) + } + + async fn cancelled(&self) { + loop { + let notified = self.0.notify.notified(); + tokio::pin!(notified); + notified.as_mut().enable(); + if self.is_cancelled() { + return; + } + notified.await; + } + } } /// How many tables a run plans ahead of the ones it rewrites, and how many it @@ -445,6 +527,7 @@ pub struct OptimizeRequest { /// `None` reads [`OptimizeConcurrency::from_env`]. pub concurrency: Option, pub before_publish: Option, + pub cancellation: Option, } impl OptimizeRequest { @@ -477,6 +560,11 @@ impl OptimizeRequest { self } + pub fn with_cancellation(mut self, cancellation: OptimizeCancellation) -> Self { + self.cancellation = Some(cancellation); + self + } + /// The tables this request visits, in order, out of the catalog's sorted /// keys. Refuses a malformed request before any effect. fn visit_order(&self, table_keys: &[String]) -> Result> { @@ -670,13 +758,29 @@ pub async fn optimize_all_datasets( .limits .max_gate_hold .map(|hold| gates_acquired + hold); - let visited = run_table_pipeline(db, &catalog, table_tasks, concurrency, deadline).await?; + let cancellation = request.cancellation.as_ref(); + let cancel_requested = || cancellation.is_some_and(OptimizeCancellation::is_cancelled); + let visited = run_table_pipeline( + db, + &catalog, + table_tasks, + concurrency, + deadline, + cancellation, + ) + .await?; + // The last cancellation point: once past it, the run publishes. + let cancelled = visited.cancelled || cancel_requested(); let mut updates = Vec::new(); let mut expected_versions = crate::db::manifest::ExpectedTableVersions::new(); let mut tables = Vec::with_capacity(visited.tables.len()); - for (stat, effect) in visited.tables { - if let Some(effect) = effect { + for (mut stat, effect) in visited.tables { + if cancelled && effect.is_some() { + let duration = stat.duration; + stat = DatasetOptimizeStats::deferred(stat.type_key, DeferReason::Cancelled); + stat.duration = duration; + } else if let Some(effect) = effect { expected_versions.insert( effect.update.identity, crate::db::manifest::TableVersionExpectation { @@ -732,22 +836,36 @@ pub async fn optimize_all_datasets( } } - // The graph commit is durable from here: nothing below fails the run. - let configuration_clear = clear_manifest_auto_cleanup(db).await; + // The graph commit is durable from here: nothing below fails the run. A + // cancelled run published nothing and leaves the configuration as is. + let configuration_clear = if cancelled { + ConfigurationClear::NotRun + } else { + clear_manifest_auto_cleanup(db).await + }; let gate_hold = gates_acquired.elapsed(); drop(table_guards); drop(main_branch_guard); drop(schema_permit); - let artifact = match persist_graph_index_artifact(db, &catalog, edge_tables_committed).await { - Ok(artifact) => artifact, - Err(error) => { - tracing::warn!( - target: "omnigraph::optimize", - error = %error, - "graph index artifact persist failed; traversals keep building in memory" - ); - ArtifactRefresh::Failed(error.to_string()) + let cancellation = match (cancelled, cancel_requested()) { + (true, _) => RunCancellation::Cancelled, + (false, true) => RunCancellation::ArrivedLate, + (false, false) => RunCancellation::NotRequested, + }; + let artifact = if cancellation != RunCancellation::NotRequested { + ArtifactRefresh::Skipped(ArtifactSkip::Cancelled) + } else { + match persist_graph_index_artifact(db, &catalog, edge_tables_committed).await { + Ok(artifact) => artifact, + Err(error) => { + tracing::warn!( + target: "omnigraph::optimize", + error = %error, + "graph index artifact persist failed; traversals keep building in memory" + ); + ArtifactRefresh::Failed(error.to_string()) + } } }; @@ -756,18 +874,32 @@ pub async fn optimize_all_datasets( input_manifest_version, graph_commit_id, tables, - resume_at: visited.resume_at.filter(|_| full_catalog), + // A cancelled run moved nothing, so the next run starts where this one did. + resume_at: visited.resume_at.filter(|_| full_catalog && !cancelled), gate_hold, configuration_clear, artifact, + cancellation, }) } /// The tables a run visited, in visit order, each with its staged effect if -/// it has one, and the first table a run-wide bound deferred. +/// it has one, the first table the gate hold deferred, and whether a +/// cancellation stopped the pipeline. struct VisitedTables { tables: Vec<(DatasetOptimizeStats, Option)>, resume_at: Option, + cancelled: bool, +} + +/// Why the pipeline stopped starting tables. +#[derive(Clone, Copy)] +enum PipelineStop { + /// The gate hold reached its deadline at this table's turn; rewrites + /// already started finish. + GateHold(usize), + /// A cancellation: every table in progress is dropped where it stands. + Cancelled, } type PlannedTable = (usize, Instant, Result); @@ -777,15 +909,18 @@ type PlannedTable = (usize, Instant, Result); /// every earlier table's has, with at most `concurrency.rewrite` at once. /// When a table's turn comes after `deadline`, that table and every later /// one are deferred and their plans, which only read, are dropped; the first -/// table always starts. A table's planning failure is `BeforeEffect` and its -/// staging failure `Unpublished`: either fails only that table. A failure -/// whose evidence stays uncertain fails the run. +/// table always starts. A cancellation, checked before every plan and every +/// rewrite and awaited meanwhile, drops every table in progress. A table's +/// planning failure is `BeforeEffect` and its staging failure `Unpublished`: +/// either fails only that table. A failure whose evidence stays uncertain +/// fails the run. async fn run_table_pipeline( db: &Omnigraph, catalog: &std::sync::Arc, tasks: Vec, concurrency: OptimizeConcurrency, deadline: Option, + cancellation: Option<&OptimizeCancellation>, ) -> Result { let keys = tasks .iter() @@ -814,22 +949,31 @@ async fn run_table_pipeline( let mut ready: VecDeque = VecDeque::new(); let mut staging = FuturesUnordered::new(); let mut turns = 0usize; - let mut stopped_at = None; + let mut stop = None; + let cancel_requested = || cancellation.is_some_and(OptimizeCancellation::is_cancelled); loop { - if stopped_at.is_none() { + if stop.is_none() { while planning.len() + ready.len() < concurrency.planning { + if cancel_requested() { + stop = Some(PipelineStop::Cancelled); + break; + } let Some((index, task)) = unplanned.next() else { break; }; planning.push_back(plan(index, task)); } - while staging.len() < concurrency.rewrite { + while stop.is_none() && staging.len() < concurrency.rewrite { let Some((index, started, result)) = ready.pop_front() else { break; }; + if cancel_requested() { + stop = Some(PipelineStop::Cancelled); + break; + } if turns > 0 && deadline.is_some_and(|deadline| Instant::now() >= deadline) { - stopped_at = Some(index); + stop = Some(PipelineStop::GateHold(index)); break; } turns += 1; @@ -855,9 +999,17 @@ async fn run_table_pipeline( } } } - if stopped_at.is_some() { - planning.clear(); - ready.clear(); + match stop { + Some(PipelineStop::GateHold(_)) => { + planning.clear(); + ready.clear(); + } + Some(PipelineStop::Cancelled) => { + planning.clear(); + ready.clear(); + staging.clear(); + } + None => {} } } // Filling planning above leaves it non-empty while any table is @@ -866,7 +1018,9 @@ async fn run_table_pipeline( break; } // The deadline stops only a table whose turn has not come. - let timer = deadline.filter(|_| turns > 0 && turns < keys.len() && stopped_at.is_none()); + let timer = deadline.filter(|_| turns > 0 && turns < keys.len() && stop.is_none()); + let awaiting_cancel = + cancellation.filter(|_| !matches!(stop, Some(PipelineStop::Cancelled))); tokio::select! { biased; Some((index, started, result)) = staging.next(), if !staging.is_empty() => { @@ -886,31 +1040,46 @@ async fn run_table_pipeline( } }); } - Some(planned) = planning.next(), if !planning.is_empty() && stopped_at.is_none() => { + Some(planned) = planning.next(), if !planning.is_empty() && stop.is_none() => { ready.push_back(planned); } () = tokio::time::sleep_until(timer.unwrap_or_else(Instant::now)), if timer.is_some() => { - stopped_at = Some(turns); + stop = Some(PipelineStop::GateHold(turns)); planning.clear(); ready.clear(); } + () = async { + if let Some(cancellation) = awaiting_cancel { + cancellation.cancelled().await; + } + }, if awaiting_cancel.is_some() => { + stop = Some(PipelineStop::Cancelled); + planning.clear(); + ready.clear(); + staging.clear(); + } } } - let resume_at = stopped_at.map(|index| keys[index].clone()); + let (resume_at, unvisited) = match stop { + Some(PipelineStop::GateHold(index)) => { + (Some(keys[index].clone()), DeferReason::GateHoldLimit) + } + Some(PipelineStop::Cancelled) => (None, DeferReason::Cancelled), + None => (None, DeferReason::GateHoldLimit), + }; let tables = visited .into_iter() .zip(keys) .map(|(entry, key)| { - entry.unwrap_or_else(|| { - ( - DatasetOptimizeStats::deferred(key, DeferReason::GateHoldLimit), - None, - ) - }) + entry.unwrap_or_else(|| (DatasetOptimizeStats::deferred(key, unvisited), None)) }) .collect(); - Ok(VisitedTables { tables, resume_at }) + Ok(VisitedTables { + tables, + resume_at, + cancelled: matches!(stop, Some(PipelineStop::Cancelled)), + }) } /// Keep a table's failure with its table when its evidence is definite; any diff --git a/crates/omnigraph/tests/failpoints.rs b/crates/omnigraph/tests/failpoints.rs index f22dfc5f1..945ac4a2e 100644 --- a/crates/omnigraph/tests/failpoints.rs +++ b/crates/omnigraph/tests/failpoints.rs @@ -4875,6 +4875,76 @@ async fn optimize_table_failure_fails_only_that_table_and_publishes_the_rest() { ); } +/// A cancellation while a table's staging is under way publishes nothing: +/// every staged and unstarted table is deferred as cancelled, no pin moves, +/// and the next run publishes both tables. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial(optimize)] +async fn optimize_cancelled_mid_run_publishes_nothing_and_the_next_run_does() { + let _scenario = FailScenario::setup(); + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap().to_string(); + seed_two_productive_optimize_tables(&uri).await; + let db = helpers::session(Omnigraph::open(&uri).await.unwrap()); + let commits_before = db.list_commits(Some("main")).await.unwrap().len(); + + let cancellation = omnigraph::db::OptimizeCancellation::new(); + let rendezvous = helpers::failpoint::Rendezvous::park_first(&catalog::OPTIMIZE_BEFORE_COMPACT); + let run = { + let uri = uri.clone(); + let cancellation = cancellation.clone(); + tokio::spawn(async move { + let db = helpers::session(Omnigraph::open(&uri).await.unwrap()); + db.optimize_as( + None, + omnigraph::db::OptimizeRequest::default().with_cancellation(cancellation), + ) + .await + }) + }; + rendezvous.wait_until_reached().await; + cancellation.cancel(); + rendezvous.release(); + let report = tokio::time::timeout(std::time::Duration::from_secs(20), run) + .await + .expect("optimize task hung") + .unwrap() + .unwrap(); + assert_eq!( + report.cancellation, + omnigraph::db::RunCancellation::Cancelled + ); + assert_eq!(report.graph_commit_id, None, "{report:?}"); + for key in ["node:Company", "node:Person"] { + let stat = report + .tables + .iter() + .find(|stat| stat.type_key == key) + .unwrap(); + assert_eq!( + stat.outcome, + omnigraph::db::TableOptimizeOutcome::Deferred(omnigraph::db::DeferReason::Cancelled), + "{stat:?}" + ); + } + assert!(report.tables.iter().all(|stat| !stat.committed())); + assert_eq!( + db.list_commits(Some("main")).await.unwrap().len(), + commits_before, + "a cancelled run publishes nothing" + ); + + let next = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); + assert_eq!( + next.tables.iter().filter(|stat| stat.committed()).count(), + 2, + "{next:?}" + ); +} + /// Optimize captures a complete authority token before entering any writer /// gate, then revalidates it after schema -> main -> table (`optimize.rs`, the /// comment above `open_write_txn`). That revalidation is the only thing that diff --git a/crates/omnigraph/tests/integration/maintenance.rs b/crates/omnigraph/tests/integration/maintenance.rs index 344d01df4..316316c08 100644 --- a/crates/omnigraph/tests/integration/maintenance.rs +++ b/crates/omnigraph/tests/integration/maintenance.rs @@ -16,8 +16,9 @@ use lance::dataset::optimize::{CompactionOptions, compact_files}; use lance_core::datatypes::BlobHandling; use omnigraph::IndexCoverage; use omnigraph::db::{ - ArtifactRefresh, ConfigurationClear, DeferReason, OptimizeBeforePublish, OptimizeLimits, - OptimizePublication, OptimizeRequest, TableOptimizeOutcome, + ArtifactRefresh, ArtifactSkip, ConfigurationClear, DeferReason, OptimizeBeforePublish, + OptimizeCancellation, OptimizeLimits, OptimizePublication, OptimizeRequest, RunCancellation, + TableOptimizeOutcome, }; use omnigraph::db::{ CleanupPolicyOptions, MergeOutcome, Omnigraph, ReadTarget, RepairAction, RepairClassification, @@ -420,6 +421,90 @@ async fn optimize_publishes_nothing_when_before_publish_fails() { assert!(retry.graph_commit_id.is_some(), "{retry:?}"); } +#[tokio::test] +async fn optimize_cancelled_before_it_starts_publishes_nothing() { + let dir = tempfile::tempdir().unwrap(); + let db = init_and_load(&dir).await; + add_person_fragments(&db).await; + let head = db.list_commits(None).await.unwrap()[0] + .graph_commit_id + .clone(); + + let cancellation = OptimizeCancellation::new(); + cancellation.cancel(); + let (hook, seen) = recording_hook(); + let report = db + .optimize_as( + None, + OptimizeRequest::default() + .with_cancellation(cancellation) + .with_before_publish(hook), + ) + .await + .unwrap(); + assert_eq!(report.cancellation, RunCancellation::Cancelled); + assert!( + report + .tables + .iter() + .all(|stat| stat.outcome == TableOptimizeOutcome::Deferred(DeferReason::Cancelled)), + "{report:?}" + ); + assert_eq!(report.graph_commit_id, None); + assert_eq!(report.resume_at, None); + assert_eq!(report.configuration_clear, ConfigurationClear::NotRun); + assert_eq!( + report.artifact, + ArtifactRefresh::Skipped(ArtifactSkip::Cancelled) + ); + assert!( + seen.lock().unwrap().is_empty(), + "a cancelled run never calls the hook" + ); + assert_eq!( + db.list_commits(None).await.unwrap()[0].graph_commit_id, + head + ); +} + +#[tokio::test] +async fn optimize_finishes_a_cancellation_that_arrives_after_its_last_check() { + let dir = tempfile::tempdir().unwrap(); + let db = init_and_load(&dir).await; + add_person_fragments(&db).await; + + // The hook runs after the last check, inside the publication helper. + let cancellation = OptimizeCancellation::new(); + let late = cancellation.clone(); + let hook = OptimizeBeforePublish::new(move |_| { + late.cancel(); + async { Ok(()) } + }); + let report = db + .optimize_as( + None, + OptimizeRequest::default() + .with_cancellation(cancellation) + .with_before_publish(hook), + ) + .await + .unwrap(); + assert_eq!(report.cancellation, RunCancellation::ArrivedLate); + assert!(report.graph_commit_id.is_some(), "{report:?}"); + assert!( + report + .tables + .iter() + .any(|stat| stat.type_key == "node:Person" && stat.committed()), + "{report:?}" + ); + assert_eq!( + report.artifact, + ArtifactRefresh::Skipped(ArtifactSkip::Cancelled), + "the edge scan starts only without a cancellation" + ); +} + #[tokio::test] async fn optimize_after_load_then_again_is_idempotent() { let dir = tempfile::tempdir().unwrap(); diff --git a/docs/dev/writes.md b/docs/dev/writes.md index 4175103de..52e9c3d35 100644 --- a/docs/dev/writes.md +++ b/docs/dev/writes.md @@ -270,7 +270,11 @@ A run with `max_gate_hold` defers every table whose turn comes after the deadline, measured from gate acquisition, and publishes the tables it finished (`run_table_pipeline`). A table whose own planning or staging fails with definite evidence is reported `failed` and the others still publish; a -failure whose evidence stays uncertain fails the run. A caller's before-publish hook +failure whose evidence stays uncertain fails the run. A cancellation +(`OptimizeRequest::with_cancellation`) is checked before every table's plan +and rewrite and before the publication: it drops the tables in progress, +defers every table as `cancelled` and publishes nothing; one that arrives +after the last check lets the run finish and skips only the artifact scan. A caller's before-publish hook (`OptimizeRequest::with_before_publish`) belongs to the shared publication helper, `table_ops::commit_updates_on_branch_with_expected`: it runs once, after the updates are prepared and before the coordinator lock and the From 352126f09258529cc33c8e28b5356933d0cc7939 Mon Sep 17 00:00:00 2001 From: Ragnor Comerford Date: Thu, 29 Oct 2026 09:59:03 -0700 Subject: [PATCH 08/13] feat(engine): bound optimize by per-table ceilings and a run byte budget OptimizeLimits gains TableCeilings (source fragments, rows and bytes, managed Blob bytes, and the rows an index rebuild or build would scan) and run_source_bytes. Each set ceiling is measured from the table's compaction plan before its first write, fragments first so they bound the size requests, and counting stops once a value passes its ceiling. A table over one, or one whose size cannot be read, is deferred as resource_limit with the measure, and nothing is written for it. Source bytes are a data file's recorded size or a HEAD of the file, and a HEAD of each deletion file, which Lance 13 records no size for. Files resolve through the collector's base rule, now shared as data_file_location and deletion_file_location. Blob bytes come from the descriptor scan compaction sizing already runs, refactored into one visitor; external references count nothing. The run budget is also the default table byte ceiling, and a larger table ceiling is refused before any effect, so a first table within its ceilings always fits. The first table that would pass the remaining budget is deferred as byte_budget with every later one, and resume_at names it. Each table's measured bytes are reported on its stats. --- .../optimize-resource-ceilings.added.md | 3 + crates/omnigraph/src/db/mod.rs | 12 +- crates/omnigraph/src/db/omnigraph.rs | 3 +- .../omnigraph/src/db/omnigraph/collector.rs | 49 ++- crates/omnigraph/src/db/omnigraph/optimize.rs | 317 +++++++++++++++++- crates/omnigraph/src/table_store.rs | 46 ++- crates/omnigraph/tests/forbidden_apis.rs | 4 +- .../tests/integration/maintenance.rs | 226 ++++++++++++- docs/dev/writes.md | 12 +- 9 files changed, 630 insertions(+), 42 deletions(-) create mode 100644 changelog.d/optimize-resource-ceilings.added.md diff --git a/changelog.d/optimize-resource-ceilings.added.md b/changelog.d/optimize-resource-ceilings.added.md new file mode 100644 index 000000000..383348d74 --- /dev/null +++ b/changelog.d/optimize-resource-ceilings.added.md @@ -0,0 +1,3 @@ +- Embedded hosts can bound optimize per table (`TableCeilings`) and per run + (`run_source_bytes`); a table over a ceiling is deferred before its first + write, with nothing written for it. diff --git a/crates/omnigraph/src/db/mod.rs b/crates/omnigraph/src/db/mod.rs index 369d5a6c9..0562bfac3 100644 --- a/crates/omnigraph/src/db/mod.rs +++ b/crates/omnigraph/src/db/mod.rs @@ -25,12 +25,12 @@ pub use omnigraph::{ OptimizeBeforePublish, OptimizeCancellation, OptimizeConcurrency, OptimizeLimits, OptimizePublication, OptimizeReport, OptimizeRequest, PendingIndex, PreparedGraphCreate, PreparedSchemaApply, PreparedSchemaSettlement, ReadCheck, RebuiltFullTextIndex, RepairAction, - RepairClassification, RepairStats, RetainedManifestVersions, RunCancellation, - SYSTEM_COLUMNS_PREFLIGHT, SchemaApplyReconciliation, SchemaApplyResult, SchemaApplySettlement, - SchemaContractDigest, SchemaNonPublicationProof, SkipReason, StagingVerdict, - SystemColumnUpgradeFinding, SystemColumnUpgradeOptions, SystemColumnUpgradeOutcome, - SystemColumnUpgradeReport, TableCollectionPlan, TableOptimizeOutcome, UnattachedTagRecord, - UnpublishedManifest, unattached_release_order, + RepairClassification, RepairStats, ResourceExceeded, ResourceMeasure, RetainedManifestVersions, + RunCancellation, SYSTEM_COLUMNS_PREFLIGHT, SchemaApplyReconciliation, SchemaApplyResult, + SchemaApplySettlement, SchemaContractDigest, SchemaNonPublicationProof, SkipReason, + StagingVerdict, SystemColumnUpgradeFinding, SystemColumnUpgradeOptions, + SystemColumnUpgradeOutcome, SystemColumnUpgradeReport, TableCeilings, TableCollectionPlan, + TableOptimizeOutcome, UnattachedTagRecord, UnpublishedManifest, unattached_release_order, }; pub(crate) use omnigraph::{ LogicalBlobValue, RangedExternalBlobs, logical_row_image, row_blob_values, diff --git a/crates/omnigraph/src/db/omnigraph.rs b/crates/omnigraph/src/db/omnigraph.rs index 8cf5cd400..fb904e369 100644 --- a/crates/omnigraph/src/db/omnigraph.rs +++ b/crates/omnigraph/src/db/omnigraph.rs @@ -54,7 +54,8 @@ pub use optimize::{ ArtifactRefresh, ArtifactSkip, CleanupPolicyOptions, ConfigurationClear, DatasetCleanupStats, DatasetOptimizeStats, DeferReason, OptimizeBeforePublish, OptimizeCancellation, OptimizeConcurrency, OptimizeLimits, OptimizePublication, OptimizeReport, OptimizeRequest, - RunCancellation, SkipReason, TableOptimizeOutcome, + ResourceExceeded, ResourceMeasure, RunCancellation, SkipReason, TableCeilings, + TableOptimizeOutcome, }; use prepared_create::initial_schema_ir; pub use prepared_create::{GraphCreateReconciliation, PreparedGraphCreate}; diff --git a/crates/omnigraph/src/db/omnigraph/collector.rs b/crates/omnigraph/src/db/omnigraph/collector.rs index 24c168196..54c50e305 100644 --- a/crates/omnigraph/src/db/omnigraph/collector.rs +++ b/crates/omnigraph/src/db/omnigraph/collector.rs @@ -521,6 +521,42 @@ fn probe_directory( }) } +/// The store and path of one data or overlay file of `dataset`, under the +/// directory Lance's base rule gives it. +pub(crate) async fn data_file_location( + dataset: &lance::Dataset, + file: &lance_table::format::DataFile, +) -> Result<( + std::sync::Arc, + object_store::path::Path, +)> { + let store = dataset + .object_store(file.base_id) + .await + .map_err(OmniError::storage)?; + let path = probe_directory(dataset, file.base_id, "data")?.join(file.path.as_str()); + Ok((store, path)) +} + +/// The store and path of fragment `fragment_id`'s deletion file, under the +/// dataset root or its base, which must be a dataset root. +pub(crate) async fn deletion_file_location( + dataset: &lance::Dataset, + fragment_id: u64, + deletion: &lance_table::format::DeletionFile, +) -> Result<( + std::sync::Arc, + object_store::path::Path, +)> { + let store = dataset + .object_store(deletion.base_id) + .await + .map_err(OmniError::storage)?; + let base = probe_directory(dataset, deletion.base_id, "")?; + let path = lance_table::io::deletion::deletion_file_path(&base, fragment_id, deletion); + Ok((store, path)) +} + async fn capture_index_paths( snapshot: &mut CollectorPathSnapshot, dataset: &lance::Dataset, @@ -576,20 +612,11 @@ pub(crate) async fn capture_dataset_paths( ); for fragment in dataset.manifest().fragments.iter() { for file in fragment.referenced_lance_files() { - let store = dataset - .object_store(file.base_id) - .await - .map_err(OmniError::storage)?; - let path = probe_directory(dataset, file.base_id, "data")?.join(file.path.as_str()); + let (store, path) = data_file_location(dataset, file).await?; snapshot.insert(location, store, path); } if let Some(deletion) = &fragment.deletion_file { - let store = dataset - .object_store(deletion.base_id) - .await - .map_err(OmniError::storage)?; - let base = probe_directory(dataset, deletion.base_id, "")?; - let path = lance_table::io::deletion::deletion_file_path(&base, fragment.id, deletion); + let (store, path) = deletion_file_location(dataset, fragment.id, deletion).await?; snapshot.insert(location, store, path); } } diff --git a/crates/omnigraph/src/db/omnigraph/optimize.rs b/crates/omnigraph/src/db/omnigraph/optimize.rs index 39fcb87da..35462d575 100644 --- a/crates/omnigraph/src/db/omnigraph/optimize.rs +++ b/crates/omnigraph/src/db/omnigraph/optimize.rs @@ -140,6 +140,54 @@ pub enum DeferReason { GateHoldLimit, /// The run was cancelled before this table's work could land. Cancelled, + /// The table's plan passed a per-table ceiling, measured before its first + /// write; nothing was written for it, and later tables still run. + ResourceLimit(ResourceExceeded), + /// The table's source bytes would pass the run's remaining byte budget. + /// This table and every later one wait for a later run, which + /// [`OptimizeReport::resume_at`] starts at. + ByteBudget, +} + +/// What a per-table ceiling measures, all from the table's plan. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[non_exhaustive] +pub enum ResourceMeasure { + /// The fragments the compaction plan rewrites. + SourceFragments, + /// Their rows. + SourceRows, + /// The bytes of their data, overlay and deletion files. + SourceBytes, + /// Their managed Blob payload, from descriptors; external references + /// count nothing. + BlobBytes, + /// The table's rows, when the run would rebuild or build an index over it. + IndexRows, +} + +impl ResourceMeasure { + /// Stable machine-readable token for serialized output. Once emitted this + /// is part of the output contract — keep it stable. + pub fn as_str(&self) -> &'static str { + match self { + ResourceMeasure::SourceFragments => "source_fragments", + ResourceMeasure::SourceRows => "source_rows", + ResourceMeasure::SourceBytes => "source_bytes", + ResourceMeasure::BlobBytes => "blob_bytes", + ResourceMeasure::IndexRows => "index_rows", + } + } +} + +/// A table's plan against one ceiling. `measured` is `None` when the size +/// could not be measured; otherwise counting stopped once it passed the +/// ceiling, so it is a lower bound. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct ResourceExceeded { + pub measure: ResourceMeasure, + pub measured: Option, + pub ceiling: u64, } impl DeferReason { @@ -149,6 +197,8 @@ impl DeferReason { match self { DeferReason::GateHoldLimit => "gate_hold_limit", DeferReason::Cancelled => "cancelled", + DeferReason::ResourceLimit(_) => "resource_limit", + DeferReason::ByteBudget => "byte_budget", } } } @@ -158,6 +208,25 @@ impl std::fmt::Display for DeferReason { f.write_str(match self { DeferReason::GateHoldLimit => "the run reached its writer-gate hold limit", DeferReason::Cancelled => "the run was cancelled", + DeferReason::ResourceLimit(ResourceExceeded { + measure, + measured: Some(measured), + ceiling, + }) => { + return write!( + f, + "{} of at least {measured} pass the table ceiling of {ceiling}", + measure.as_str() + ); + } + DeferReason::ResourceLimit(ResourceExceeded { + measure, + measured: None, + .. + }) => { + return write!(f, "{} could not be measured", measure.as_str()); + } + DeferReason::ByteBudget => "the run's byte budget is spent", }) } } @@ -186,6 +255,9 @@ pub struct DatasetOptimizeStats { /// The error of a [`TableOptimizeOutcome::Failed`] table, carrying its /// completion evidence (`BeforeEffect` or `Unpublished`). pub failure: Option>, + /// The plan's source bytes, when a byte ceiling or the run's byte budget + /// measured them; what the table spent of the budget. + pub source_bytes: Option, } impl DatasetOptimizeStats { @@ -202,6 +274,7 @@ impl DatasetOptimizeStats { pending_indexes: Vec::new(), duration: Duration::ZERO, failure: None, + source_bytes: None, } } @@ -214,6 +287,7 @@ impl DatasetOptimizeStats { pending_indexes: Vec::new(), duration: Duration::ZERO, failure: None, + source_bytes: None, } } @@ -226,6 +300,7 @@ impl DatasetOptimizeStats { pending_indexes: Vec::new(), duration, failure: Some(failure), + source_bytes: None, } } @@ -460,6 +535,12 @@ pub struct OptimizeLimits { /// writer wait is at most this, plus that table, the publication and the /// configuration clear. pub max_gate_hold: Option, + /// Ceilings on one table's plan, checked before its first write. + pub table: TableCeilings, + /// Total source bytes across the tables the run starts. It is also the + /// table byte ceiling when that is unset, and refuses a larger one, so a + /// first table within its ceilings always fits. + pub run_source_bytes: Option, } impl OptimizeLimits { @@ -467,6 +548,60 @@ impl OptimizeLimits { self.max_gate_hold = Some(hold); self } + + pub fn with_table_ceilings(mut self, ceilings: TableCeilings) -> Self { + self.table = ceilings; + self + } + + pub fn with_run_source_bytes(mut self, budget: u64) -> Self { + self.run_source_bytes = Some(budget); + self + } + + /// The table byte ceiling in force: the explicit one, else the budget. + fn table_source_bytes(&self) -> Option { + self.table.source_bytes.or(self.run_source_bytes) + } +} + +/// Ceilings on one table's plan; an absent one is not measured. A table over +/// any of them is `deferred(resource_limit)` and nothing is written for it. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +#[non_exhaustive] +pub struct TableCeilings { + pub source_fragments: Option, + pub source_rows: Option, + pub source_bytes: Option, + pub blob_bytes: Option, + pub index_rows: Option, +} + +impl TableCeilings { + pub fn with_source_fragments(mut self, ceiling: u64) -> Self { + self.source_fragments = Some(ceiling); + self + } + + pub fn with_source_rows(mut self, ceiling: u64) -> Self { + self.source_rows = Some(ceiling); + self + } + + pub fn with_source_bytes(mut self, ceiling: u64) -> Self { + self.source_bytes = Some(ceiling); + self + } + + pub fn with_blob_bytes(mut self, ceiling: u64) -> Self { + self.blob_bytes = Some(ceiling); + self + } + + pub fn with_index_rows(mut self, ceiling: u64) -> Self { + self.index_rows = Some(ceiling); + self + } } /// What a run's before-publish hook receives. @@ -575,6 +710,14 @@ impl OptimizeRequest { "optimize concurrency must be at least 1 for planning and rewriting", )); } + if let (Some(table), Some(budget)) = + (self.limits.table.source_bytes, self.limits.run_source_bytes) + && table > budget + { + return Err(OmniError::manifest(format!( + "optimize table byte ceiling {table} exceeds the run byte budget {budget}" + ))); + } let Some(tables) = &self.tables else { let start = self .start_at @@ -647,6 +790,9 @@ struct PreparedOptimizeTable { initial_snapshot: crate::storage_layer::SnapshotHandle, witness: crate::table_store::StagingWitness, last_linear_version: Option, + /// The plan's source bytes, measured when a byte ceiling or the run's + /// byte budget is set. + source_bytes: Option, } enum OptimizePreparation { @@ -766,6 +912,7 @@ pub async fn optimize_all_datasets( table_tasks, concurrency, deadline, + request.limits, cancellation, ) .await?; @@ -895,9 +1042,9 @@ struct VisitedTables { /// Why the pipeline stopped starting tables. #[derive(Clone, Copy)] enum PipelineStop { - /// The gate hold reached its deadline at this table's turn; rewrites - /// already started finish. - GateHold(usize), + /// A run-wide bound (the gate hold or the byte budget) stopped the run at + /// this table's turn; rewrites already started finish. + Bound(usize, DeferReason), /// A cancellation: every table in progress is dropped where it stands. Cancelled, } @@ -920,6 +1067,7 @@ async fn run_table_pipeline( tasks: Vec, concurrency: OptimizeConcurrency, deadline: Option, + limits: OptimizeLimits, cancellation: Option<&OptimizeCancellation>, ) -> Result { let keys = tasks @@ -932,7 +1080,7 @@ async fn run_table_pipeline( let catalog = std::sync::Arc::clone(catalog); async move { let started = Instant::now(); - let result = prepare_optimize_table(db, catalog.as_ref(), task).await; + let result = prepare_optimize_table(db, catalog.as_ref(), task, &limits).await; (index, started, result) } }; @@ -949,6 +1097,7 @@ async fn run_table_pipeline( let mut ready: VecDeque = VecDeque::new(); let mut staging = FuturesUnordered::new(); let mut turns = 0usize; + let mut spent = 0_u64; let mut stop = None; let cancel_requested = || cancellation.is_some_and(OptimizeCancellation::is_cancelled); @@ -973,7 +1122,16 @@ async fn run_table_pipeline( break; } if turns > 0 && deadline.is_some_and(|deadline| Instant::now() >= deadline) { - stop = Some(PipelineStop::GateHold(index)); + stop = Some(PipelineStop::Bound(index, DeferReason::GateHoldLimit)); + break; + } + // A table's bytes never pass the budget alone, so the first + // table with work always fits. + if let (Some(budget), Ok(OptimizePreparation::Work(work))) = + (limits.run_source_bytes, &result) + && spent.saturating_add(work.source_bytes.unwrap_or(0)) > budget + { + stop = Some(PipelineStop::Bound(index, DeferReason::ByteBudget)); break; } turns += 1; @@ -983,6 +1141,7 @@ async fn run_table_pipeline( visited[index] = Some((stat, None)); } Ok(OptimizePreparation::Work(work)) => { + spent = spent.saturating_add(work.source_bytes.unwrap_or(0)); staging.push(stage(index, started, work)); } // Planning only reads. @@ -1000,7 +1159,7 @@ async fn run_table_pipeline( } } match stop { - Some(PipelineStop::GateHold(_)) => { + Some(PipelineStop::Bound(..)) => { planning.clear(); ready.clear(); } @@ -1044,7 +1203,7 @@ async fn run_table_pipeline( ready.push_back(planned); } () = tokio::time::sleep_until(timer.unwrap_or_else(Instant::now)), if timer.is_some() => { - stop = Some(PipelineStop::GateHold(turns)); + stop = Some(PipelineStop::Bound(turns, DeferReason::GateHoldLimit)); planning.clear(); ready.clear(); } @@ -1062,9 +1221,7 @@ async fn run_table_pipeline( } let (resume_at, unvisited) = match stop { - Some(PipelineStop::GateHold(index)) => { - (Some(keys[index].clone()), DeferReason::GateHoldLimit) - } + Some(PipelineStop::Bound(index, reason)) => (Some(keys[index].clone()), reason), Some(PipelineStop::Cancelled) => (None, DeferReason::Cancelled), None => (None, DeferReason::GateHoldLimit), }; @@ -1178,6 +1335,7 @@ async fn prepare_optimize_table( db: &Omnigraph, catalog: &omnigraph_compiler::catalog::Catalog, task: OptimizeTableTask, + limits: &OptimizeLimits, ) -> Result { let snapshot = db .open_pinned_for_write(&task.full_path, &task.entry) @@ -1187,10 +1345,8 @@ async fn prepare_optimize_table( // dropped property's values is compaction work even when Lance's planner // would leave the table alone. let options = CompactionOptions::default(); - let will_compact = TableStore::plan_table_compaction(snapshot.dataset(), &options) - .await? - .num_tasks() - > 0; + let plan = TableStore::plan_table_compaction(snapshot.dataset(), &options).await?; + let will_compact = plan.num_tasks() > 0; let needs_reindex = TableStore::has_foldable_unindexed_fragments(snapshot.dataset()).await?; let index_work = super::table_ops::index_work_status_on_dataset_for_catalog( db, @@ -1214,6 +1370,22 @@ async fn prepare_optimize_table( return Ok(OptimizePreparation::Stat(stat)); } + let source = plan + .tasks + .iter() + .flat_map(|task| task.fragments.iter().cloned()) + .collect::>(); + let index_work = needs_reindex || index_work.needs_commit; + let source_bytes = match measure_plan(snapshot.dataset(), &source, index_work, limits).await? { + Ok(source_bytes) => source_bytes, + Err(exceeded) => { + return Ok(OptimizePreparation::Stat(DatasetOptimizeStats::deferred( + task.table_key, + DeferReason::ResourceLimit(exceeded), + ))); + } + }; + Ok(OptimizePreparation::Work(PreparedOptimizeTable { identity: task.identity, table_key: task.table_key, @@ -1222,9 +1394,123 @@ async fn prepare_optimize_table( initial_snapshot: snapshot, witness: task.witness, last_linear_version: task.entry.version_metadata.last_linear_version(), + source_bytes, })) } +/// Measure a table's plan against its ceilings, in order, and stop at the +/// first one it passes; the fragment count comes first, so it bounds the size +/// requests. Only a set ceiling is measured. On success, the source bytes when +/// a byte ceiling or the run's byte budget needs them. +async fn measure_plan( + ds: &lance::Dataset, + source: &[lance_table::format::Fragment], + index_work: bool, + limits: &OptimizeLimits, +) -> Result, ResourceExceeded>> { + let ceilings = &limits.table; + let over = |measure, measured, ceiling| { + Ok(Err(ResourceExceeded { + measure, + measured, + ceiling, + })) + }; + if let Some(ceiling) = ceilings.source_fragments + && source.len() as u64 > ceiling + { + return over( + ResourceMeasure::SourceFragments, + Some(source.len() as u64), + ceiling, + ); + } + if let Some(ceiling) = ceilings.source_rows { + let mut rows = 0_u64; + for fragment in source { + let Some(physical) = fragment.physical_rows else { + return over(ResourceMeasure::SourceRows, None, ceiling); + }; + rows = rows.saturating_add(physical as u64); + if rows > ceiling { + return over(ResourceMeasure::SourceRows, Some(rows), ceiling); + } + } + } + let mut source_bytes = None; + if let Some(ceiling) = limits.table_source_bytes() { + match source_file_bytes(ds, source, ceiling).await { + Some(bytes) if bytes > ceiling => { + return over(ResourceMeasure::SourceBytes, Some(bytes), ceiling); + } + Some(bytes) => source_bytes = Some(bytes), + None => return over(ResourceMeasure::SourceBytes, None, ceiling), + } + } + if let Some(ceiling) = ceilings.blob_bytes + && !source.is_empty() + { + match TableStore::managed_blob_bytes(ds, source.to_vec(), ceiling).await? { + Some(bytes) if bytes > ceiling => { + return over(ResourceMeasure::BlobBytes, Some(bytes), ceiling); + } + Some(_) => {} + None => return over(ResourceMeasure::BlobBytes, None, ceiling), + } + } + if let Some(ceiling) = ceilings.index_rows + && index_work + { + match ds.count_rows(None).await { + Ok(rows) if rows as u64 > ceiling => { + return over(ResourceMeasure::IndexRows, Some(rows as u64), ceiling); + } + Ok(_) => {} + Err(_) => return over(ResourceMeasure::IndexRows, None, ceiling), + } + } + Ok(Ok(source_bytes)) +} + +/// The bytes of `source`'s data, overlay and deletion files: a data file's +/// recorded size, or a HEAD of the file when none is recorded, and a HEAD of +/// each deletion file, which records no size in Lance 13. Each file resolves +/// through the collector's base rule. Counting stops once it passes +/// `ceiling`. `None` when a size cannot be read. +async fn source_file_bytes( + ds: &lance::Dataset, + source: &[lance_table::format::Fragment], + ceiling: u64, +) -> Option { + let mut total = 0_u64; + for fragment in source { + for file in fragment.referenced_lance_files() { + let size = match file.file_size_bytes.get() { + Some(size) => size.get(), + None => { + let (store, path) = + super::collector::data_file_location(ds, file).await.ok()?; + store.size(&path).await.ok()? + } + }; + total = total.saturating_add(size); + if total > ceiling { + return Some(total); + } + } + if let Some(deletion) = &fragment.deletion_file { + let (store, path) = super::collector::deletion_file_location(ds, fragment.id, deletion) + .await + .ok()?; + total = total.saturating_add(store.size(&path).await.ok()?); + if total > ceiling { + return Some(total); + } + } + } + Some(total) +} + decide_seam! { /// After one table's detached rewrite or index link committed, before the /// next link or table (RFC 0067). Nothing is published yet. @@ -1250,6 +1536,7 @@ async fn apply_optimize_table_effects( let full_path = work.full_path; let base = work.initial_snapshot; let witness = work.witness; + let source_bytes = work.source_bytes; let declared_full_text = super::table_ops::declared_full_text_columns(catalog, &table_key); fail(&OPTIMIZE_BEFORE_COMPACT)?; let options = CompactionOptions::default(); @@ -1321,6 +1608,7 @@ async fn apply_optimize_table_effects( } let Some(identity) = tip_identity else { let mut stat = DatasetOptimizeStats::planned(table_key, &metrics, false); + stat.source_bytes = source_bytes; stat.pending_indexes = index_work.pending; super::table_ops::append_deferred_full_text_indexes( &tip, @@ -1333,6 +1621,7 @@ async fn apply_optimize_table_effects( return Ok(OptimizeEffectOutcome { stat, effect: None }); }; let mut stat = DatasetOptimizeStats::planned(table_key.clone(), &metrics, true); + stat.source_bytes = source_bytes; stat.pending_indexes = index_work.pending; super::table_ops::append_deferred_full_text_indexes( &tip, diff --git a/crates/omnigraph/src/table_store.rs b/crates/omnigraph/src/table_store.rs index 1acf1d746..5e72959e2 100644 --- a/crates/omnigraph/src/table_store.rs +++ b/crates/omnigraph/src/table_store.rs @@ -4499,6 +4499,41 @@ impl TableStore { /// nothing); after an early stop, a lower bound that already derives one-row batches. /// `None` when a nested or non-Blob-v2 Blob field cannot be sized. async fn max_row_blob_bytes(ds: &Dataset, fragments: Vec) -> Result> { + let mut max = 0_u64; + let sized = Self::visit_row_blob_bytes(ds, fragments, |rows| { + max = rows.iter().copied().fold(max, u64::max); + compaction_blob_batch_rows(max) != 1 + }) + .await?; + Ok(sized.then_some(max)) + } + + /// Managed Blob bytes in `fragments`, from descriptors (externals count nothing and + /// are never fetched); the count stops once it passes `ceiling`, so a value above + /// `ceiling` is a lower bound. `None` when a nested or non-Blob-v2 Blob field cannot + /// be sized. + pub(crate) async fn managed_blob_bytes( + ds: &Dataset, + fragments: Vec, + ceiling: u64, + ) -> Result> { + let mut total = 0_u64; + let sized = Self::visit_row_blob_bytes(ds, fragments, |rows| { + total = rows.iter().fold(total, |sum, row| sum.saturating_add(*row)); + total <= ceiling + }) + .await?; + Ok(sized.then_some(total)) + } + + /// Scan the Blob descriptors of `fragments` and hand each batch's per-row managed + /// bytes to `visit`, which returns false to stop. Returns false, scanning nothing, + /// when a nested or non-Blob-v2 Blob field cannot be sized. + async fn visit_row_blob_bytes( + ds: &Dataset, + fragments: Vec, + mut visit: impl FnMut(&[u64]) -> bool, + ) -> Result { if nested_blob_field(ds.schema()).is_some() || ds .schema() @@ -4506,7 +4541,7 @@ impl TableStore { .iter() .any(|field| field.is_blob() && !field.is_blob_v2()) { - return Ok(None); + return Ok(false); } let blob_columns = ds .schema() @@ -4515,13 +4550,15 @@ impl TableStore { .filter(|field| field.is_blob()) .map(|field| field.name.as_str()) .collect::>(); + if blob_columns.is_empty() { + return Ok(true); + } let mut stream = Self::scan_stream_with(ds, Some(&blob_columns), None, None, false, |scanner| { scanner.with_fragments(fragments); Ok(()) }) .await?; - let mut max = 0_u64; while let Some(batch) = stream.try_next().await.map_err(OmniError::storage)? { let mut row_bytes = vec![0_u64; batch.num_rows()]; for (field, column) in batch.schema().fields().iter().zip(batch.columns()) { @@ -4542,12 +4579,11 @@ impl TableStore { } } } - max = row_bytes.into_iter().fold(max, u64::max); - if compaction_blob_batch_rows(max) == 1 { + if !visit(&row_bytes) { break; } } - Ok(Some(max)) + Ok(true) } /// Stage creation of a new dataset without publishing its first manifest. diff --git a/crates/omnigraph/tests/forbidden_apis.rs b/crates/omnigraph/tests/forbidden_apis.rs index 0e20efc90..940e05e1c 100644 --- a/crates/omnigraph/tests/forbidden_apis.rs +++ b/crates/omnigraph/tests/forbidden_apis.rs @@ -907,7 +907,7 @@ gateway_surfaces! { "predicted_materialized_blob_batch_bytes", "materialize_blob_batch_bounded_with_preflight_cache", "managed_blob_payloads", "can_fold_index", "has_foldable_unindexed_fragments", "index_is_vector", - "plan_table_compaction", + "plan_table_compaction", "managed_blob_bytes", ], "table_store.rs" => "TableStore" => GatewayDisposition::StageOnly => [ "stage_create", "stage_keyed_write", "stage_proven_strict_insert", "stage_overwrite", @@ -1146,7 +1146,7 @@ durable_calls! { ("db/omnigraph/repair.rs", ".dataset()", 1, WriteProtocol::ReadOnlyAccess), // The sixth accessor reports deferred FTS coverage from an immutable // snapshot; it only reads index metadata and never stages or publishes. - ("db/omnigraph/optimize.rs", ".dataset()", 10, WriteProtocol::Composed("Optimize planning + read-only coverage and native-fork inventory + the configuration clear's read-back after a failed removal + physical cleanup, including the release of expired proposal tags and of expired unattached tags descendants first, each confirmed by a readback")), + ("db/omnigraph/optimize.rs", ".dataset()", 11, WriteProtocol::Composed("Optimize planning, including each plan's measure against its table ceilings, + read-only coverage and native-fork inventory + the configuration clear's read-back after a failed removal + physical cleanup, including the release of expired proposal tags and of expired unattached tags descendants first, each confirmed by a readback")), ("db/omnigraph/optimize.rs", ".into_dataset()", 1, WriteProtocol::PhysicalOnly), ("exec/merge.rs", "SnapshotHandle::new(", 4, MERGE_V9), } diff --git a/crates/omnigraph/tests/integration/maintenance.rs b/crates/omnigraph/tests/integration/maintenance.rs index 316316c08..73d3bf3c7 100644 --- a/crates/omnigraph/tests/integration/maintenance.rs +++ b/crates/omnigraph/tests/integration/maintenance.rs @@ -17,8 +17,8 @@ use lance_core::datatypes::BlobHandling; use omnigraph::IndexCoverage; use omnigraph::db::{ ArtifactRefresh, ArtifactSkip, ConfigurationClear, DeferReason, OptimizeBeforePublish, - OptimizeCancellation, OptimizeLimits, OptimizePublication, OptimizeRequest, RunCancellation, - TableOptimizeOutcome, + OptimizeCancellation, OptimizeLimits, OptimizePublication, OptimizeRequest, ResourceMeasure, + RunCancellation, TableCeilings, TableOptimizeOutcome, }; use omnigraph::db::{ CleanupPolicyOptions, MergeOutcome, Omnigraph, ReadTarget, RepairAction, RepairClassification, @@ -309,6 +309,11 @@ async fn optimize_table_list_runs_only_those_tables_and_refuses_bad_requests() { .with_start_at(keys[1].clone()), omnigraph::db::OptimizeRequest::default() .with_concurrency(omnigraph::db::OptimizeConcurrency::new(0, 1)), + omnigraph::db::OptimizeRequest::default().with_limits( + OptimizeLimits::default() + .with_table_ceilings(TableCeilings::default().with_source_bytes(2)) + .with_run_source_bytes(1), + ), ] { let error = db.optimize_as(None, request.clone()).await.unwrap_err(); assert_eq!( @@ -505,6 +510,196 @@ async fn optimize_finishes_a_cancellation_that_arrives_after_its_last_check() { ); } +async fn person_pin(db: &Omnigraph) -> u64 { + db.snapshot_of(ReadTarget::branch("main")) + .await + .unwrap() + .dataset("node:Person") + .unwrap() + .published_dataset_version +} + +/// Each per-table ceiling defers a table over it with nothing written; the +/// plan holds a deletion file, which records no size, so the byte measure +/// reads it from storage. +#[tokio::test] +async fn optimize_defers_a_table_over_a_ceiling_and_writes_nothing_for_it() { + let dir = tempfile::tempdir().unwrap(); + let db = init_and_load(&dir).await; + add_person_fragments(&db).await; + mutate_main( + &db, + MUTATION_QUERIES, + "remove_person", + &helpers::params(&[("$name", "Eve")]), + ) + .await + .unwrap(); + let pin = person_pin(&db).await; + + for (ceilings, measure) in [ + ( + TableCeilings::default().with_source_fragments(1), + ResourceMeasure::SourceFragments, + ), + ( + TableCeilings::default().with_source_rows(1), + ResourceMeasure::SourceRows, + ), + ( + TableCeilings::default().with_source_bytes(1), + ResourceMeasure::SourceBytes, + ), + ( + TableCeilings::default().with_index_rows(1), + ResourceMeasure::IndexRows, + ), + ] { + let report = db + .optimize_as( + None, + OptimizeRequest::default() + .with_limits(OptimizeLimits::default().with_table_ceilings(ceilings)), + ) + .await + .unwrap(); + let person = report + .tables + .iter() + .find(|stat| stat.type_key == "node:Person") + .unwrap(); + let TableOptimizeOutcome::Deferred(DeferReason::ResourceLimit(exceeded)) = person.outcome + else { + panic!("{measure:?}: {person:?}"); + }; + assert_eq!(exceeded.measure, measure); + assert_eq!(exceeded.ceiling, 1); + assert!( + exceeded.measured.is_some_and(|measured| measured > 1), + "{exceeded:?}" + ); + assert_eq!( + report.resume_at, None, + "a resource limit never moves the resume position" + ); + assert_eq!(person_pin(&db).await, pin, "{measure:?} wrote nothing"); + } + + // Measured whole, including the deletion file's size, the plan fits. + let report = db + .optimize_as( + None, + OptimizeRequest::default() + .with_limits(OptimizeLimits::default().with_run_source_bytes(u64::MAX)), + ) + .await + .unwrap(); + let person = report + .tables + .iter() + .find(|stat| stat.type_key == "node:Person") + .unwrap(); + assert!(person.committed(), "{person:?}"); + assert!( + person.source_bytes.is_some_and(|bytes| bytes > 0), + "{person:?}" + ); +} + +/// The run budget stops the run at the first table that would pass it: +/// that table and every later one are deferred, and the next run resumes +/// there. +#[tokio::test] +async fn optimize_byte_budget_defers_the_tables_after_it_and_resumes_there() { + let dir = tempfile::tempdir().unwrap(); + let db = init_and_load(&dir).await; + add_person_fragments(&db).await; + for name in ["Acme", "Beta", "Cygnus", "Delta"] { + db.load_jsonl( + &serde_json::json!({"type": "Company", "data": {"name": name}}).to_string(), + LoadMode::Merge, + ) + .await + .unwrap(); + } + + // Measure every table's source bytes in a run whose hook refuses to + // publish it. + let seen = std::sync::Arc::new(std::sync::Mutex::new(Vec::new())); + let recorded = std::sync::Arc::clone(&seen); + let probe = OptimizeBeforePublish::new(move |publication| { + recorded.lock().unwrap().push(publication); + async { Err(OmniError::manifest("measuring only")) } + }); + db.optimize_as( + None, + OptimizeRequest::default() + .with_limits(OptimizeLimits::default().with_run_source_bytes(u64::MAX)) + .with_before_publish(probe), + ) + .await + .unwrap_err(); + let measured = seen.lock().unwrap()[0] + .tables + .iter() + .filter_map(|stat| Some((stat.type_key.clone(), stat.source_bytes?))) + .collect::>(); + assert_eq!( + measured + .iter() + .map(|(key, _)| key.as_str()) + .collect::>(), + ["node:Company", "node:Person"], + "{measured:?}" + ); + // Each table fits alone, both together do not. + let budget = measured[0].1.max(measured[1].1); + let limits = OptimizeLimits::default().with_run_source_bytes(budget); + + let report = db + .optimize_as(None, OptimizeRequest::default().with_limits(limits)) + .await + .unwrap(); + let outcome = |report: &omnigraph::db::OptimizeReport, key: &str| { + report + .tables + .iter() + .find(|stat| stat.type_key == key) + .unwrap() + .outcome + }; + assert!( + matches!( + outcome(&report, "node:Company"), + TableOptimizeOutcome::Optimized + ), + "{report:?}" + ); + assert_eq!( + outcome(&report, "node:Person"), + TableOptimizeOutcome::Deferred(DeferReason::ByteBudget) + ); + assert_eq!(report.resume_at.as_deref(), Some("node:Person")); + + let next = db + .optimize_as( + None, + OptimizeRequest::default() + .with_limits(limits) + .with_start_at("node:Person"), + ) + .await + .unwrap(); + assert!( + matches!( + outcome(&next, "node:Person"), + TableOptimizeOutcome::Optimized + ), + "{next:?}" + ); + assert_eq!(next.resume_at, None); +} + #[tokio::test] async fn optimize_after_load_then_again_is_idempotent() { let dir = tempfile::tempdir().unwrap(); @@ -1572,6 +1767,33 @@ async fn optimize_does_not_size_a_blob_batch_by_external_references() { "test precondition: the load keeps the external reference as a descriptor" ); + // Nor toward the Blob ceiling: only the 3 managed bytes count. + let bounded = db + .optimize_as( + None, + omnigraph::db::OptimizeRequest::default().with_limits( + OptimizeLimits::default() + .with_table_ceilings(TableCeilings::default().with_blob_bytes(2)), + ), + ) + .await + .unwrap(); + let doc = bounded + .tables + .iter() + .find(|stat| stat.type_key == "node:Doc") + .unwrap(); + assert_eq!( + doc.outcome, + TableOptimizeOutcome::Deferred(DeferReason::ResourceLimit( + omnigraph::db::ResourceExceeded { + measure: ResourceMeasure::BlobBytes, + measured: Some(3), + ceiling: 2, + } + )) + ); + let probes = omnigraph::instrumentation::MergeWriteProbes::default(); let stats = omnigraph::instrumentation::with_merge_write_probes( probes.clone(), diff --git a/docs/dev/writes.md b/docs/dev/writes.md index 52e9c3d35..6c3412899 100644 --- a/docs/dev/writes.md +++ b/docs/dev/writes.md @@ -274,7 +274,17 @@ failure whose evidence stays uncertain fails the run. A cancellation (`OptimizeRequest::with_cancellation`) is checked before every table's plan and rewrite and before the publication: it drops the tables in progress, defers every table as `cancelled` and publishes nothing; one that arrives -after the last check lets the run finish and skips only the artifact scan. A caller's before-publish hook +after the last check lets the run finish and skips only the artifact scan. +Per-table ceilings (`TableCeilings`) are measured from the table's compaction +plan before its first write, fragments first, and only when set: rows from +fragment metadata, bytes from recorded data-file sizes or a HEAD resolved by +the collector's base rule (`collector::data_file_location`, +`deletion_file_location`; Lance 13 records no deletion-file size), managed Blob +bytes from the descriptor scan compaction sizing uses, and the table's rows +when an index would be rebuilt or built. A table over one, or one whose size +cannot be read, is `deferred(resource_limit)` with nothing written. The run +byte budget (`run_source_bytes`, also the default table byte ceiling) defers +the first table that would pass it, and every later one, as `byte_budget`. A caller's before-publish hook (`OptimizeRequest::with_before_publish`) belongs to the shared publication helper, `table_ops::commit_updates_on_branch_with_expected`: it runs once, after the updates are prepared and before the coordinator lock and the From 345760dd1d8bb86472cf7b146fddfe599c731c86 Mon Sep 17 00:00:00 2001 From: Ragnor Comerford Date: Thu, 29 Oct 2026 09:59:03 -0700 Subject: [PATCH 09/13] feat(engine): bound the adjacency artifact optimize writes OptimizeLimits::artifact_bytes bounds the whole object. Before the edge scan, a lower bound (the 20-byte prefix, every section prelude, one extra offset entry per array, the header at its shortest, and 8 bytes per live edge row) skips the artifact when it already passes the limit. During the scan, GraphIndex::build_counted reads each edge table one batch at a time and adds what the batch grows the object by, counted from the widest header: 4 bytes plus the UTF-8 length per new id, 4 bytes in every offset array its type sizes, and 8 bytes per edge. Once the count passes the limit it stops before keeping another batch or allocating the adjacency arrays. Either stop reports skipped(size_limit) and writes nothing. The unbounded build streams the same way instead of collecting every batch first. The artifact write runs in a spawned task the run awaits, so a dropped run cannot drop an issued write on the mutable key. --- .../optimize-artifact-size-limit.added.md | 3 + crates/omnigraph/src/db/omnigraph.rs | 6 + crates/omnigraph/src/db/omnigraph/optimize.rs | 90 +++++++- crates/omnigraph/src/graph_index/mod.rs | 44 +++- crates/omnigraph/src/graph_index/persist.rs | 197 ++++++++++++++++-- crates/omnigraph/tests/forbidden_apis.rs | 25 +-- .../tests/integration/maintenance.rs | 58 ++++++ docs/dev/writes.md | 11 +- 8 files changed, 389 insertions(+), 45 deletions(-) create mode 100644 changelog.d/optimize-artifact-size-limit.added.md diff --git a/changelog.d/optimize-artifact-size-limit.added.md b/changelog.d/optimize-artifact-size-limit.added.md new file mode 100644 index 000000000..544c66051 --- /dev/null +++ b/changelog.d/optimize-artifact-size-limit.added.md @@ -0,0 +1,3 @@ +- Embedded hosts can bound the adjacency artifact optimize writes with + `OptimizeLimits::artifact_bytes`; a larger one is skipped, before or during + its edge scan, and nothing is written. diff --git a/crates/omnigraph/src/db/omnigraph.rs b/crates/omnigraph/src/db/omnigraph.rs index fb904e369..5716a6a61 100644 --- a/crates/omnigraph/src/db/omnigraph.rs +++ b/crates/omnigraph/src/db/omnigraph.rs @@ -1481,6 +1481,12 @@ impl Omnigraph { self.storage.as_ref() } + /// A shared handle on the storage adapter, for work that must outlive the + /// call that starts it. + pub(crate) fn storage_adapter_handle(&self) -> Arc { + Arc::clone(&self.storage) + } + /// Root-scoped writer queues (schema, branch, and `(table, branch)` gates). /// /// Engine-internal writers (mutation finalize, schema_apply, diff --git a/crates/omnigraph/src/db/omnigraph/optimize.rs b/crates/omnigraph/src/db/omnigraph/optimize.rs index 35462d575..1581b7d8d 100644 --- a/crates/omnigraph/src/db/omnigraph/optimize.rs +++ b/crates/omnigraph/src/db/omnigraph/optimize.rs @@ -393,6 +393,10 @@ pub enum ArtifactSkip { IncompleteEdgeTables, /// The run was cancelled, so the edge scan did not start. Cancelled, + /// The artifact would pass `OptimizeLimits::artifact_bytes`: its lower + /// bound did before the scan, or the counting build's running bound did + /// during it. `counted` is that bound; nothing was written. + SizeLimit { counted: u64, limit: u64 }, } impl ArtifactSkip { @@ -403,6 +407,7 @@ impl ArtifactSkip { ArtifactSkip::NoEdgeTypes => "no_edge_types", ArtifactSkip::IncompleteEdgeTables => "incomplete_edge_tables", ArtifactSkip::Cancelled => "cancelled", + ArtifactSkip::SizeLimit { .. } => "size_limit", } } } @@ -541,6 +546,10 @@ pub struct OptimizeLimits { /// table byte ceiling when that is unset, and refuses a larger one, so a /// first table within its ceilings always fits. pub run_source_bytes: Option, + /// The largest adjacency artifact the run writes, in bytes of the whole + /// object. Over it, the artifact is `skipped(size_limit)` and a load + /// rebuilds the adjacency in memory. + pub artifact_bytes: Option, } impl OptimizeLimits { @@ -559,6 +568,11 @@ impl OptimizeLimits { self } + pub fn with_artifact_bytes(mut self, limit: u64) -> Self { + self.artifact_bytes = Some(limit); + self + } + /// The table byte ceiling in force: the explicit one, else the budget. fn table_source_bytes(&self) -> Option { self.table.source_bytes.or(self.run_source_bytes) @@ -1003,7 +1017,14 @@ pub async fn optimize_all_datasets( let artifact = if cancellation != RunCancellation::NotRequested { ArtifactRefresh::Skipped(ArtifactSkip::Cancelled) } else { - match persist_graph_index_artifact(db, &catalog, edge_tables_committed).await { + match persist_graph_index_artifact( + db, + &catalog, + edge_tables_committed, + request.limits.artifact_bytes, + ) + .await + { Ok(artifact) => artifact, Err(error) => { tracing::warn!( @@ -1277,6 +1298,7 @@ async fn persist_graph_index_artifact( db: &Omnigraph, catalog: &omnigraph_compiler::catalog::Catalog, edge_tables_committed: bool, + limit: Option, ) -> Result { if catalog.edge_types.is_empty() { return Ok(ArtifactRefresh::Skipped(ArtifactSkip::NoEdgeTypes)); @@ -1307,15 +1329,67 @@ async fn persist_graph_index_artifact( ); return Ok(ArtifactRefresh::Skipped(ArtifactSkip::IncompleteEdgeTables)); } - let index = - crate::graph_index::GraphIndex::build(&snapshot, &edge_types, db.catalog().system_columns) - .await?; - let written = - crate::graph_index::persist::save(&snapshot, db.storage_adapter(), &edge_types, &index) - .await?; - let Some(uri) = written else { + let mut count = None; + if let Some(limit) = limit { + let Some((shortest, widest)) = + crate::graph_index::persist::fixed_bytes(&snapshot, &edge_types)? + else { + return Ok(ArtifactRefresh::Skipped(ArtifactSkip::IncompleteEdgeTables)); + }; + // A lower bound: every edge is two `u32` targets, and the live row + // count is the number of rows the scan returns. + let mut lower = shortest; + for edge in edge_types.keys() { + let rows = snapshot + .open_lance_dataset(&format!("edge:{edge}")) + .await? + .count_rows(None) + .await + .map_err(OmniError::storage)?; + lower = lower.saturating_add((rows as u64).saturating_mul(8)); + } + if lower > limit { + return Ok(ArtifactRefresh::Skipped(ArtifactSkip::SizeLimit { + counted: lower, + limit, + })); + } + count = Some(crate::graph_index::persist::ArtifactCount::new( + widest, + limit, + &edge_types, + )); + } + let Some(index) = crate::graph_index::GraphIndex::build_counted( + &snapshot, + &edge_types, + db.catalog().system_columns, + count.as_mut(), + ) + .await? + else { + let counted = count.as_ref().map_or(0, |count| count.counted()); + return Ok(ArtifactRefresh::Skipped(ArtifactSkip::SizeLimit { + counted, + limit: limit.unwrap_or(0), + })); + }; + let Some((uri, body)) = + crate::graph_index::persist::artifact_body(&snapshot, &edge_types, &index)? + else { return Ok(ArtifactRefresh::Skipped(ArtifactSkip::IncompleteEdgeTables)); }; + drop(index); + // The artifact key is mutable, so a write dropped mid-request would leave + // its outcome unknown: once issued, the write runs to its end even when + // this run is dropped. + let adapter = db.storage_adapter_handle(); + let write_uri = uri.clone(); + tokio::spawn(async move { adapter.write_bytes(&write_uri, &body).await }) + .await + .map_err(|error| { + OmniError::manifest_internal(format!("graph index artifact write task: {error}")) + })??; // A fresh artifact may serve scope keys whose earlier load attempt // failed; those negative verdicts are stamp-keyed, and when this // optimize advanced no table version the stamps (hence keys) are diff --git a/crates/omnigraph/src/graph_index/mod.rs b/crates/omnigraph/src/graph_index/mod.rs index 5971116b5..92b1c7969 100644 --- a/crates/omnigraph/src/graph_index/mod.rs +++ b/crates/omnigraph/src/graph_index/mod.rs @@ -183,6 +183,22 @@ impl GraphIndex { edge_types: &HashMap, // edge_name → (from_type, to_type) system_columns: SystemColumns, ) -> Result { + Self::build_counted(snapshot, edge_types, system_columns, None) + .await? + .ok_or_else(|| OmniError::manifest_internal("an uncounted graph index build stopped")) + } + + /// Build as [`Self::build`] does, reading each edge table one scan batch + /// at a time. With `count`, each batch's growth of the persisted artifact + /// is added to it, and once the count passes its limit the build drops + /// what it holds and returns `None`, before it keeps another batch and + /// before the adjacency arrays are allocated. + pub(crate) async fn build_counted( + snapshot: &Snapshot, + edge_types: &HashMap, + system_columns: SystemColumns, + mut count: Option<&mut persist::ArtifactCount>, + ) -> Result> { // Counted here — not at the cache-miss site — so the probe counts // actual edge-table scan builds, never persisted-artifact loads. crate::instrumentation::record_graph_build(edge_types.len()); @@ -209,15 +225,12 @@ impl GraphIndex { let ds = snapshot.open_lance_dataset(&table_key).await?; - let batches: Vec = ds + let mut batches = ds .scan() .project(&[system_columns.src, system_columns.dst]) .map_err(OmniError::storage)? .try_into_stream() .await - .map_err(OmniError::storage)? - .try_collect() - .await .map_err(OmniError::storage)?; type_indices @@ -228,9 +241,11 @@ impl GraphIndex { .or_insert_with(TypeIndex::new); let mut edges: Vec<(u32, u32)> = Vec::new(); - for batch in &batches { - let srcs = string_column(batch, system_columns.src)?; - let dsts = string_column(batch, system_columns.dst)?; + while let Some(batch) = batches.try_next().await.map_err(OmniError::storage)? { + let from_before = type_indices[from_type].len(); + let to_before = type_indices[to_type].len(); + let srcs = string_column(&batch, system_columns.src)?; + let dsts = string_column(&batch, system_columns.dst)?; for i in 0..batch.num_rows() { let src_dense = type_indices @@ -243,6 +258,17 @@ impl GraphIndex { .get_or_insert(dsts.value(i)); edges.push((src_dense, dst_dense)); } + if let Some(count) = count.as_deref_mut() { + count.add_ids(from_type, &type_indices[from_type].ids()[from_before..]); + // A type on both ends of the edge has one dictionary. + if to_type != from_type { + count.add_ids(to_type, &type_indices[to_type].ids()[to_before..]); + } + count.add_edges(batch.num_rows() as u64); + if count.passed() { + return Ok(None); + } + } } edge_pairs.insert(edge_name.clone(), edges); } @@ -264,11 +290,11 @@ impl GraphIndex { csc.insert(edge_name.clone(), CsrIndex::build(dst_count, &reversed)); } - Ok(Self { + Ok(Some(Self { type_indices, csr, csc, - }) + })) } /// Load the persisted adjacency artifact when one matches this snapshot's diff --git a/crates/omnigraph/src/graph_index/persist.rs b/crates/omnigraph/src/graph_index/persist.rs index 18890cd0c..89ef34ab4 100644 --- a/crates/omnigraph/src/graph_index/persist.rs +++ b/crates/omnigraph/src/graph_index/persist.rs @@ -638,31 +638,167 @@ pub(crate) async fn load( } } -/// Persist `index` (covering `edge_types`) for this snapshot. Called by -/// `optimize` only (see the module contract). `Ok(None)` when any covered -/// edge table is absent from the snapshot — a partially-materialized store -/// has no complete identity to stamp. Errors are the caller's to log; a -/// failed write costs nothing but the next cold build. -pub(crate) async fn save( +/// One stamp per covered edge type, sorted; `None` when any covered edge +/// table is absent from the snapshot, which has then no complete identity to +/// stamp. +fn snapshot_stamps( snapshot: &Snapshot, - adapter: &dyn StorageAdapter, edge_types: &HashMap, - index: &GraphIndex, -) -> Result> { +) -> Option> { let mut tables = Vec::with_capacity(edge_types.len()); for (edge_name, (from_type, to_type)) in edge_types { - match snapshot_stamp(snapshot, edge_name, from_type, to_type) { - Some(stamp) => tables.push(stamp), - None => return Ok(None), - } + tables.push(snapshot_stamp(snapshot, edge_name, from_type, to_type)?); } tables.sort_by(|a, b| a.edge_name.cmp(&b.edge_name)); - let uri = artifact_uri(snapshot.root_uri()); - let body = serialize(index, tables)?; + Some(tables) +} + +/// The artifact's URI and bytes for `index` (covering `edge_types`) at this +/// snapshot; `Ok(None)` when any covered edge table is absent from the +/// snapshot. +pub(crate) fn artifact_body( + snapshot: &Snapshot, + edge_types: &HashMap, + index: &GraphIndex, +) -> Result)>> { + let Some(tables) = snapshot_stamps(snapshot, edge_types) else { + return Ok(None); + }; + Ok(Some(( + artifact_uri(snapshot.root_uri()), + serialize(index, tables)?, + ))) +} + +/// Persist `index` (covering `edge_types`) for this snapshot in one call. +/// `Ok(None)` when any covered edge table is absent from the snapshot — a +/// partially-materialized store has no complete identity to stamp. Tests +/// only: `optimize`, the one writer (see the module contract), writes +/// [`artifact_body`] from a task it awaits, so a dropped run cannot drop the +/// write. +#[cfg(test)] +pub(crate) async fn save( + snapshot: &Snapshot, + adapter: &dyn StorageAdapter, + edge_types: &HashMap, + index: &GraphIndex, +) -> Result> { + let Some((uri, body)) = artifact_body(snapshot, edge_types, index)? else { + return Ok(None); + }; adapter.write_bytes(&uri, &body).await?; Ok(Some(uri)) } +/// The bytes `serialize` writes for `edge_types` at this snapshot before any +/// edge: the 20-byte prefix, every section prelude, one extra `u32` per offset +/// array, and the header. The first value has the header's count and length +/// fields at their shortest, a lower bound on the object; the second at their +/// widest, a safe start for [`ArtifactCount`]. `None` when any covered edge +/// table is absent. +pub(crate) fn fixed_bytes( + snapshot: &Snapshot, + edge_types: &HashMap, +) -> Result> { + let Some(tables) = snapshot_stamps(snapshot, edge_types) else { + return Ok(None); + }; + fixed_bytes_for(&tables).map(Some) +} + +fn fixed_bytes_for(tables: &[TableStamp]) -> Result<(u64, u64)> { + let mut type_names = tables + .iter() + .flat_map(|table| [table.from_type.as_str(), table.to_type.as_str()]) + .collect::>(); + type_names.sort_unstable(); + type_names.dedup(); + let mut fixed = (MAGIC.len() + 4 + 8) as u64; + for name in &type_names { + fixed += 1 + 4 + name.len() as u64; + } + for table in tables { + let stamp_json = serde_json::to_vec(table).map_err(|e| { + OmniError::manifest(format!("graph index artifact stamp serialize: {e}")) + })?; + // The prelude, then the extra offset entry of the CSR and the CSC. + fixed += 1 + 4 + table.edge_name.len() as u64 + 4 + stamp_json.len() as u64 + 2 * 4; + } + let header = |field: u64| -> Result { + serde_json::to_vec(&Header { + tables: tables.to_vec(), + payload_sha256_b64: base64::engine::general_purpose::STANDARD.encode([0u8; 32]), + dicts: type_names + .iter() + .map(|name| DictMeta { + type_name: (*name).to_string(), + count: field, + len: field, + }) + .collect(), + adjacencies: tables + .iter() + .map(|table| AdjMeta { + edge_name: table.edge_name.clone(), + lens: [field; 4], + }) + .collect(), + }) + .map(|header| header.len() as u64) + .map_err(|e| OmniError::manifest(format!("graph index artifact header serialize: {e}"))) + }; + Ok((fixed + header(0)?, fixed + header(u64::MAX)?)) +} + +/// A running upper bound on the artifact a counting build would write. It +/// starts at the fixed bytes with the widest header and only grows: each new +/// id of a node type adds its dictionary entry and one `u32` in every CSR +/// offset array whose source type it is and every CSC offset array whose +/// destination type it is; each edge adds its two targets. +pub(crate) struct ArtifactCount { + counted: u64, + limit: u64, + per_id: HashMap, +} + +impl ArtifactCount { + pub(crate) fn new( + fixed_widest: u64, + limit: u64, + edge_types: &HashMap, + ) -> Self { + let mut per_id = HashMap::new(); + for (from_type, to_type) in edge_types.values() { + *per_id.entry(from_type.clone()).or_insert(0) += 4; + *per_id.entry(to_type.clone()).or_insert(0) += 4; + } + Self { + counted: fixed_widest, + limit, + per_id, + } + } + + pub(crate) fn add_ids(&mut self, type_name: &str, ids: &[String]) { + let widen = self.per_id.get(type_name).copied().unwrap_or(0); + for id in ids { + self.counted = self.counted.saturating_add(4 + id.len() as u64 + widen); + } + } + + pub(crate) fn add_edges(&mut self, edges: u64) { + self.counted = self.counted.saturating_add(edges.saturating_mul(8)); + } + + pub(crate) fn passed(&self) -> bool { + self.counted > self.limit + } + + pub(crate) fn counted(&self) -> u64 { + self.counted + } +} + #[cfg(test)] mod tests { use super::*; @@ -713,6 +849,37 @@ mod tests { } } + /// The pre-check's lower bound and the counting build's bound bracket the + /// bytes `serialize` writes. + #[test] + fn size_bounds_bracket_the_serialized_artifact() { + let index = sample_index(); + let tables = vec![sample_stamp()]; + let actual = serialize(&index, tables.clone()).unwrap().len() as u64; + let (shortest, widest) = fixed_bytes_for(&tables).unwrap(); + let edges = 3; + assert!( + shortest + 8 * edges <= actual, + "{shortest} + edges vs {actual}" + ); + let mut edge_types = HashMap::new(); + edge_types.insert( + "Knows".to_string(), + ("Person".to_string(), "Person".to_string()), + ); + let mut count = ArtifactCount::new(widest, u64::MAX, &edge_types); + count.add_ids("Person", index.parts().0["Person"].ids()); + count.add_edges(edges); + assert!(actual <= count.counted(), "{actual} vs {}", count.counted()); + // Exact apart from the header's count and length digits. + let digits = 19 * (2 + 4); + assert!( + count.counted() - actual <= digits, + "{}", + count.counted() - actual + ); + } + /// Header/payload halves round-trip. Stamp verification against a live /// snapshot rides the integration tests in `tests/integration/traversal_adaptive.rs`. #[test] diff --git a/crates/omnigraph/tests/forbidden_apis.rs b/crates/omnigraph/tests/forbidden_apis.rs index 940e05e1c..0cf9cf838 100644 --- a/crates/omnigraph/tests/forbidden_apis.rs +++ b/crates/omnigraph/tests/forbidden_apis.rs @@ -1013,12 +1013,6 @@ durable_calls! { ("omnigraph-catalog/commit.rs", "InsertBuilder::new(", 1, WriteProtocol::Exact("lowest manifest publisher gateway")), ("omnigraph-catalog/commit.rs", ".execute_uncommitted(", 1, WriteProtocol::Exact("lowest manifest publisher gateway")), ("omnigraph-catalog/commit.rs", "CommitBuilder::new(", 2, WriteProtocol::Exact("lowest manifest publisher gateway: the linear version with zero retries, or a detached candidate version whose only retry is the random version id")), - // The persisted CSR/CSC adjacency artifact (`__graph_index/csr-current.bin`): - // derived, regenerable topology written ONLY from `optimize`'s tail (never - // the query path, which only loads), outside graph visibility — a stale or - // partial object is rejected by its identity stamps + payload digest and - // rebuilt in memory, so this write can never change a query's result. - ("graph_index/persist.rs", ".write_bytes(", 1, WriteProtocol::PhysicalOnly), ("omnigraph-core/instrumentation.rs", ".write_bytes(", 1, WriteProtocol::Composed("instrumented storage forwarding")), ("omnigraph-core/instrumentation.rs", ".write_text(", 1, WriteProtocol::Composed("instrumented storage forwarding")), ("omnigraph-core/instrumentation.rs", ".write_text_if_absent(", 1, WriteProtocol::Composed("instrumented storage forwarding")), @@ -1148,6 +1142,13 @@ durable_calls! { // snapshot; it only reads index metadata and never stages or publishes. ("db/omnigraph/optimize.rs", ".dataset()", 11, WriteProtocol::Composed("Optimize planning, including each plan's measure against its table ceilings, + read-only coverage and native-fork inventory + the configuration clear's read-back after a failed removal + physical cleanup, including the release of expired proposal tags and of expired unattached tags descendants first, each confirmed by a readback")), ("db/omnigraph/optimize.rs", ".into_dataset()", 1, WriteProtocol::PhysicalOnly), + // The persisted CSR/CSC adjacency artifact (`__graph_index/csr-current.bin`): + // derived, regenerable topology written ONLY from `optimize`'s tail (never + // the query path, which only loads), outside graph visibility — a stale or + // partial object is rejected by its identity stamps + payload digest and + // rebuilt in memory, so this write can never change a query's result. It + // runs in a task the run awaits, so a dropped run cannot drop it. + ("db/omnigraph/optimize.rs", ".write_bytes(", 1, WriteProtocol::PhysicalOnly), ("exec/merge.rs", "SnapshotHandle::new(", 4, MERGE_V9), } @@ -2564,6 +2565,12 @@ const MUTABLE_KEY_REQUESTS: &[(&str, &str, usize, &str)] = &[ 1, "cleanup reclaims a pre-v11 table fork no registration names; fork names carry a commit ULID and are never reused", ), + ( + "db/omnigraph/optimize.rs", + ".write_bytes(", + 1, + "derived topology artifact stamped with its snapshot identity; a late older copy fails the stamp check and the reader builds in memory", + ), ( "db/omnigraph/prepared_create.rs", ".delete(", @@ -2576,12 +2583,6 @@ const MUTABLE_KEY_REQUESTS: &[(&str, &str, usize, &str)] = &[ 2, "reconciles an unpublished graph create: its tables and manifest have no published version, and nothing recreates them under this claim", ), - ( - "graph_index/persist.rs", - ".write_bytes(", - 1, - "derived topology artifact stamped with its snapshot identity; a late older copy fails the stamp check and the reader builds in memory", - ), ( "omnigraph-catalog/retention.rs", ".delete(", diff --git a/crates/omnigraph/tests/integration/maintenance.rs b/crates/omnigraph/tests/integration/maintenance.rs index 73d3bf3c7..3ac180334 100644 --- a/crates/omnigraph/tests/integration/maintenance.rs +++ b/crates/omnigraph/tests/integration/maintenance.rs @@ -700,6 +700,64 @@ async fn optimize_byte_budget_defers_the_tables_after_it_and_resumes_there() { assert_eq!(next.resume_at, None); } +/// The artifact limit holds before the scan (a lower bound) and during it +/// (the counting build), writes nothing when it stops, and admits the whole +/// object once there is room for it. +#[tokio::test] +async fn optimize_skips_an_artifact_over_its_size_limit() { + use std::sync::atomic::{AtomicU64, Ordering}; + let dir = tempfile::tempdir().unwrap(); + let db = init_and_load(&dir).await; + let artifact = dir.path().join("__graph_index/csr-current.bin"); + let report = db + .optimize_as(None, OptimizeRequest::default()) + .await + .unwrap(); + assert_eq!(report.artifact, ArtifactRefresh::Refreshed); + let size = std::fs::metadata(&artifact).unwrap().len(); + std::fs::remove_file(&artifact).unwrap(); + + let bounded = |limit: u64| { + let builds = std::sync::Arc::new(AtomicU64::new(0)); + let probes = omnigraph::instrumentation::QueryIoProbes { + graph_build_count: std::sync::Arc::clone(&builds), + ..Default::default() + }; + let run = omnigraph::instrumentation::with_query_io_probes( + probes, + db.optimize_as( + None, + OptimizeRequest::default() + .with_limits(OptimizeLimits::default().with_artifact_bytes(limit)), + ), + ); + async move { (run.await.unwrap().artifact, builds.load(Ordering::Relaxed)) } + }; + + let (outcome, builds) = bounded(1).await; + assert!( + matches!(outcome, ArtifactRefresh::Skipped(ArtifactSkip::SizeLimit { counted, limit: 1 }) if counted > 1), + "{outcome:?}" + ); + assert_eq!(builds, 0, "a lower bound over the limit starts no scan"); + assert!(!artifact.exists()); + + let (outcome, builds) = bounded(size - 1).await; + let ArtifactRefresh::Skipped(ArtifactSkip::SizeLimit { counted, limit }) = outcome else { + panic!("{outcome:?}"); + }; + assert_eq!(limit, size - 1); + assert!(counted > limit, "{counted} vs {limit}"); + assert_eq!(builds, 1, "the counting build started and stopped"); + assert!(!artifact.exists(), "a stopped build writes nothing"); + + // The counted header is the widest, so admitting the whole object needs + // room for its count and length digits. + let (outcome, _) = bounded(size + 4096).await; + assert_eq!(outcome, ArtifactRefresh::Refreshed); + assert_eq!(std::fs::metadata(&artifact).unwrap().len(), size); +} + #[tokio::test] async fn optimize_after_load_then_again_is_idempotent() { let dir = tempfile::tempdir().unwrap(); diff --git a/docs/dev/writes.md b/docs/dev/writes.md index 6c3412899..4f4d44f7e 100644 --- a/docs/dev/writes.md +++ b/docs/dev/writes.md @@ -284,7 +284,16 @@ bytes from the descriptor scan compaction sizing uses, and the table's rows when an index would be rebuilt or built. A table over one, or one whose size cannot be read, is `deferred(resource_limit)` with nothing written. The run byte budget (`run_source_bytes`, also the default table byte ceiling) defers -the first table that would pass it, and every later one, as `byte_budget`. A caller's before-publish hook +the first table that would pass it, and every later one, as `byte_budget`. +With `artifact_bytes`, the adjacency artifact is bounded twice: before the +edge scan by a lower bound (`persist::fixed_bytes` at the shortest header plus +8 bytes per live edge row), and during it by `GraphIndex::build_counted`, which +reads one batch at a time, adds each batch's growth of the object to +`persist::ArtifactCount` (from the widest header), and stops before keeping +another batch or allocating the adjacency arrays. Either stop is +`skipped(size_limit)` and writes nothing. The artifact write runs in a spawned +task the run awaits, so a dropped run cannot drop an issued write on the +mutable key. A caller's before-publish hook (`OptimizeRequest::with_before_publish`) belongs to the shared publication helper, `table_ops::commit_updates_on_branch_with_expected`: it runs once, after the updates are prepared and before the coordinator lock and the From 982672c5abd0499fec1ecbcc8cfc01af4097db83 Mon Sep 17 00:00:00 2001 From: Ragnor Comerford Date: Thu, 29 Oct 2026 10:05:50 -0700 Subject: [PATCH 10/13] feat(server): run optimize through the serving owner with a durable record A cluster server can run optimize on a graph's main when OMNIGRAPH_MAINTENANCE_ROUTES is true: POST /graphs/{graph_id}/maintenance/optimize submits under a caller-chosen ULID, GET /graphs/{graph_id}/maintenance/{maintenance_id} looks it up and POST .../cancel stops it at its next safe boundary. Every route needs maintain on main. The record is a new cluster control object, __cluster/maintenance.json, separate from state.json and written only by the lock holder, each write a conditional put: at acceptance, when the run starts, in optimize's before-publish hook (publishing, with the nonce and input head) and at the end. It holds one active operation, at most four unresolved publications, 32 terminal operations and each graph's resume position, and it names the cluster's ledger id; another ledger's record is refused, never merged. A write with no definite answer stops the run's writes. The next admission or lookup reads the record back: an accepted or running operation no live run holds is interrupted, a publishing one waits on main's history, and a lost terminal write is re-issued. Omnigraph::resolve_publication walks main's parent chain from a fresh head to the input head's child and judges it against the nonce. The run is an owned server operation that holds its graph request; shutdown cancels it, and a deployment that affects the graph cancels it before its drain. A repeated submission of the same request answers the existing record; the same id with another request, an outstanding deployment and a fifth unresolved publication are 409, another active operation 429. --- changelog.d/served-optimize-routes.added.md | 5 + crates/omnigraph-api-types/src/operations.rs | 31 + crates/omnigraph-cluster/src/admission.rs | 54 + crates/omnigraph-cluster/src/deployment.rs | 2 +- crates/omnigraph-cluster/src/lib.rs | 8 + crates/omnigraph-cluster/src/maintenance.rs | 851 ++++++++++++ crates/omnigraph-cluster/src/store.rs | 59 + crates/omnigraph-server/src/deployment.rs | 12 + .../omnigraph-server/src/http_operations.rs | 11 +- crates/omnigraph-server/src/lib.rs | 24 +- crates/omnigraph-server/src/maintenance.rs | 1076 ++++++++++++++++ .../tests/integration/openapi.rs | 3 + .../tests/maintenance_routes.rs | 358 ++++++ crates/omnigraph/src/db/mod.rs | 15 +- crates/omnigraph/src/db/omnigraph.rs | 2 + .../omnigraph/src/db/omnigraph/resolution.rs | 109 ++ crates/omnigraph/tests/forbidden_apis.rs | 2 + .../tests/integration/maintenance.rs | 58 + docs/dev/control-plane.md | 22 + docs/dev/http-api.md | 2 +- docs/user/index.md | 1 + docs/user/operations/maintenance.md | 4 +- docs/user/operations/served-maintenance.md | 87 ++ openapi.json | 1135 +++++++++++++++++ 24 files changed, 3918 insertions(+), 13 deletions(-) create mode 100644 changelog.d/served-optimize-routes.added.md create mode 100644 crates/omnigraph-cluster/src/maintenance.rs create mode 100644 crates/omnigraph-server/src/maintenance.rs create mode 100644 crates/omnigraph-server/tests/maintenance_routes.rs create mode 100644 crates/omnigraph/src/db/omnigraph/resolution.rs create mode 100644 docs/user/operations/served-maintenance.md diff --git a/changelog.d/served-optimize-routes.added.md b/changelog.d/served-optimize-routes.added.md new file mode 100644 index 000000000..5d7d62f8a --- /dev/null +++ b/changelog.d/served-optimize-routes.added.md @@ -0,0 +1,5 @@ +- A cluster server can run optimize on a graph's `main` when + `OMNIGRAPH_MAINTENANCE_ROUTES=true`, keeping a durable record a caller looks + up by its own id. See [served maintenance][served-maintenance]. + +[served-maintenance]: ../docs/user/operations/served-maintenance.md diff --git a/crates/omnigraph-api-types/src/operations.rs b/crates/omnigraph-api-types/src/operations.rs index 10a8278c5..391d1f4f0 100644 --- a/crates/omnigraph-api-types/src/operations.rs +++ b/crates/omnigraph-api-types/src/operations.rs @@ -145,6 +145,10 @@ const PROBE: &str = "Process probe, not a graph command."; const OAUTH: &str = "Protocol-defined OAuth discovery, not a graph operation."; const DEPLOYMENT: &str = "Deployment control is not an MCP agent tool."; const BYTE_STREAM: &str = "MCP tools do not carry this byte-stream contract."; +const MAINTENANCE: &str = + "Served maintenance runs under a durable operation record, not as an MCP agent tool."; +const MAINTENANCE_CLI: &str = + "The CLI's optimize runs against direct storage; served maintenance is submitted over HTTP."; /// Every operation in the generated HTTP specification. Protocol endpoints /// (`/mcp`, `/openapi.json`) and the catalogue's reserved-id dispatch retain @@ -410,6 +414,33 @@ pub const HTTP_OPERATIONS: &[HttpOperation] = &[ Surface::bound(&["commits_publish"]), Surface::bound(&["commit publish"]), ), + operation( + "start optimize", + HttpMethod::Post, + "/graphs/{graph_id}/maintenance/optimize", + Capability::Write, + None, + Surface::absent(MAINTENANCE), + Surface::absent(MAINTENANCE_CLI), + ), + operation( + "get maintenance operation", + HttpMethod::Get, + "/graphs/{graph_id}/maintenance/{maintenance_id}", + Capability::Read, + None, + Surface::absent(MAINTENANCE), + Surface::absent(MAINTENANCE_CLI), + ), + operation( + "cancel maintenance operation", + HttpMethod::Post, + "/graphs/{graph_id}/maintenance/{maintenance_id}/cancel", + Capability::Write, + None, + Surface::absent(MAINTENANCE), + Surface::absent(MAINTENANCE_CLI), + ), operation( "diff", HttpMethod::Get, diff --git a/crates/omnigraph-cluster/src/admission.rs b/crates/omnigraph-cluster/src/admission.rs index 259634eb2..ac2d72dbc 100644 --- a/crates/omnigraph-cluster/src/admission.rs +++ b/crates/omnigraph-cluster/src/admission.rs @@ -158,6 +158,60 @@ impl ClusterAdmission { Ok(()) } + /// The cluster ledger id and the outstanding deployment's id, for + /// maintenance admission. Reading them needs no configuration grant: they + /// decide whether a maintenance operation may start, and say nothing about + /// the deployment beyond its id. + pub async fn maintenance_context(&self) -> Result<(String, Option), Diagnostic> { + let snapshot = self + .owner() + .store + .read_state(&mut self.owner().store.observations()) + .await?; + let state = snapshot.state.ok_or_else(|| { + Diagnostic::error( + "state_missing", + CLUSTER_STATE_FILE, + "a served cluster has a ledger", + ) + })?; + let ledger_id = state.ledger_id.clone().ok_or_else(|| { + Diagnostic::error( + "ledger_upgrade_required", + CLUSTER_STATE_FILE, + "maintenance requires a v2 ledger", + ) + })?; + Ok(( + ledger_id, + state + .outstanding + .as_ref() + .map(|outstanding| outstanding.id.clone()), + )) + } + + /// The maintenance record and its version token. + pub async fn read_maintenance( + &self, + ) -> Result, Diagnostic> { + self.owner().store.read_maintenance().await + } + + /// One conditional put of the maintenance record. Only this admission's + /// exact lock holder writes it, so the lock is checked first. + pub async fn write_maintenance( + &self, + ledger: &crate::MaintenanceLedger, + expected_version: Option<&str>, + ) -> Result { + self.validate_current_lock().await?; + self.owner() + .store + .write_maintenance(ledger, expected_version) + .await + } + /// A recovery/deployment owner cannot be reinterpreted as a serving owner. pub fn validate_serving(&self) -> Result<(), Diagnostic> { if self.owner().purpose != ClusterAdmissionPurpose::Serve { diff --git a/crates/omnigraph-cluster/src/deployment.rs b/crates/omnigraph-cluster/src/deployment.rs index 03bce7807..c90906700 100644 --- a/crates/omnigraph-cluster/src/deployment.rs +++ b/crates/omnigraph-cluster/src/deployment.rs @@ -310,7 +310,7 @@ struct DeploymentId { sequence: u64, } -fn canonical_ulid(part: &str) -> bool { +pub(crate) fn canonical_ulid(part: &str) -> bool { part.parse::().is_ok_and(|id| id.to_string() == part) } diff --git a/crates/omnigraph-cluster/src/lib.rs b/crates/omnigraph-cluster/src/lib.rs index eeeb5bad6..833807db5 100644 --- a/crates/omnigraph-cluster/src/lib.rs +++ b/crates/omnigraph-cluster/src/lib.rs @@ -30,6 +30,7 @@ mod deployment; mod diff; mod graph_read; mod lease; +mod maintenance; mod serve; mod state_lock; mod store; @@ -56,6 +57,13 @@ use config::{ pub use deployment::*; use diff::{ResourceKind, compute_blast_radius, resource_kind}; pub use graph_read::GraphReadAuthority; +pub use maintenance::{ + Admitted, CLUSTER_MAINTENANCE_FILE, MAX_MAINTENANCE_BYTES, MAX_MAINTENANCE_RECORD_BYTES, + MAX_MAINTENANCE_TERMINAL, MAX_MAINTENANCE_UNRESOLVED, MaintenanceKind, MaintenanceLedger, + MaintenanceOperation, MaintenanceRefusal, MaintenanceState, MaintenanceTableOutcome, + MaintenanceTailOutcome, MaintenanceTimings, MaintenanceWrite, OptimizeOptions, Resolution, + canonical_maintenance_id, maintenance_request_digest, +}; #[cfg(any(test, feature = "test-util"))] pub use serve::read_serving_snapshot_with_display_root; pub use serve::{ diff --git a/crates/omnigraph-cluster/src/maintenance.rs b/crates/omnigraph-cluster/src/maintenance.rs new file mode 100644 index 000000000..aff5c0be1 --- /dev/null +++ b/crates/omnigraph-cluster/src/maintenance.rs @@ -0,0 +1,851 @@ +//! The maintenance operation record, `__cluster/maintenance.json`. +//! +//! One control object per cluster root, separate from `state.json` (whose +//! `deny_unknown_fields` ledger an older binary must keep reading). Only the +//! process that holds the cluster lock writes it, and every write is a +//! conditional put against the version the writer read: at acceptance, when +//! execution starts, before the publication, and at the end. A write whose +//! answer is lost is in doubt; the caller then starts no table work and no +//! publication until a re-read settles the key. +//! +//! The ledger holds at most one active operation, the operations whose +//! publication is in doubt (`unresolved`), a bounded list of terminal +//! operations newest first, and each graph's resume position. A terminal +//! record is never rewritten, and an unresolved one is never evicted. + +use std::collections::BTreeMap; + +use serde::{Deserialize, Serialize}; + +use crate::Diagnostic; +use crate::deployment::canonical_ulid; + +pub const CLUSTER_MAINTENANCE_FILE: &str = "__cluster/maintenance.json"; + +/// Terminal operations kept, newest first; the oldest is evicted first. +pub const MAX_MAINTENANCE_TERMINAL: usize = 32; +/// Operations whose publication is in doubt; a fifth refuses admission. +pub const MAX_MAINTENANCE_UNRESOLVED: usize = 4; +pub const MAX_MAINTENANCE_RECORD_BYTES: usize = 1024 * 1024; +pub const MAX_MAINTENANCE_BYTES: usize = 4 * 1024 * 1024; + +const MAINTENANCE_VERSION: u32 = 1; + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct MaintenanceLedger { + pub version: u32, + /// The cluster ledger the record belongs to. A record naming another + /// ledger is from an earlier ledger at this root and is never merged. + pub ledger_id: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub active: Option, + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub unresolved: Vec, + /// Newest first. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub terminal: Vec, + /// Graph id to the table key its next full-catalog run starts at. + #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] + pub resume_at: BTreeMap, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum MaintenanceKind { + Optimize, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum MaintenanceState { + Accepted, + Running, + /// The before-publish write landed: the run may have sent a manifest + /// request. Only main's history settles it (`Resolution`). + Publishing, + Succeeded, + Failed, + /// Stopped before any manifest request; published nothing. + Cancelled, + /// No live run holds it any more, and it never reached `publishing`, so + /// no attempt sent a manifest request. + Interrupted, +} + +impl MaintenanceState { + pub fn is_terminal(self) -> bool { + matches!( + self, + Self::Succeeded | Self::Failed | Self::Cancelled | Self::Interrupted + ) + } +} + +/// The options the caller asked for, exactly; the request digest covers them. +#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct OptimizeOptions { + #[serde(default, skip_serializing_if = "Option::is_none")] + pub tables: Option>, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub max_gate_hold_ms: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub max_source_fragments: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub max_source_rows: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub max_source_bytes: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub max_blob_bytes: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub max_index_rows: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub run_source_bytes: Option, +} + +/// One table's outcome, as the engine reported it. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct MaintenanceTableOutcome { + pub type_key: String, + /// `optimized`, `unchanged`, `deferred`, `skipped` or `failed`. + pub outcome: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub reason: Option, + /// The human detail of a deferral or a failure. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub detail: Option, + pub fragments_removed: u64, + pub fragments_added: u64, + pub duration_ms: u64, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub source_bytes: Option, +} + +/// An outcome after the graph commit, which never fails the operation. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct MaintenanceTailOutcome { + pub outcome: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub detail: Option, +} + +#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct MaintenanceTimings { + /// Acceptance to the start of execution. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub queued_ms: Option, + /// How long the run held main's writer gates. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub gate_hold_ms: Option, + /// The start of execution to the end of the run. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub compute_ms: Option, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct MaintenanceOperation { + /// The caller's canonical ULID. + pub maintenance_id: String, + /// `sha256:` over the canonical request: graph, branch, kind, options and + /// principal. The same id with another digest is a different request. + pub request_digest: String, + pub graph: String, + pub branch: String, + pub kind: MaintenanceKind, + pub options: OptimizeOptions, + pub principal: String, + pub accepted_at: String, + pub state: MaintenanceState, + /// The resume position the server supplied for a full-catalog run. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub start_at: Option, + /// The lineage intent's nonce, written before the publication. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub nonce: Option, + /// The graph head every publish attempt reads, written with the nonce. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub input_commit: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub input_manifest_version: Option, + /// The published commit's history id, from the publisher's answer or + /// from resolution. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub published_commit: Option, + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub tables: Vec, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub resume_at: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub configuration_clear: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub artifact: Option, + #[serde(default)] + pub timings: MaintenanceTimings, + /// Why a terminal operation ended as it did, or where an unresolved + /// publication's resolution stopped. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub reason: Option, +} + +/// Why admission refused a request, before any write. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum MaintenanceRefusal { + /// The same id names a different request. + DifferentRequest, + /// Another operation holds the cluster's active slot. + Active { maintenance_id: String }, + /// Four publications are already in doubt. + Unresolved, +} + +/// Admission's answer for a request it does not refuse. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Admitted { + /// The id is new: it now holds the active slot, `accepted`. + New, + /// The same request was already accepted; nothing changed. + Existing(Box), +} + +/// How main's history settles an unresolved publication. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Resolution { + /// main's graph head is still the input head, or the walk could not + /// prove the commit absent. The record stays `publishing`; `reason` says + /// where the walk stopped. + InDoubt { reason: String }, + /// The input head's child answers the nonce. + Committed { published_commit: String }, + /// The input head's child does not answer the nonce. + NotCommitted, +} + +/// Whether `id` is a canonical ULID, the only spelling a maintenance id has. +pub fn canonical_maintenance_id(id: &str) -> bool { + canonical_ulid(id) +} + +impl MaintenanceOperation { + /// A new `accepted` operation, stamped now, with its request digest. + pub fn accepted( + maintenance_id: impl Into, + graph: impl Into, + options: OptimizeOptions, + principal: impl Into, + start_at: Option, + ) -> Self { + let graph = graph.into(); + let principal = principal.into(); + let branch = "main".to_string(); + Self { + maintenance_id: maintenance_id.into(), + request_digest: maintenance_request_digest( + &graph, + &branch, + MaintenanceKind::Optimize, + &options, + &principal, + ), + graph, + branch, + kind: MaintenanceKind::Optimize, + options, + principal, + accepted_at: crate::now_rfc3339(), + state: MaintenanceState::Accepted, + start_at, + nonce: None, + input_commit: None, + input_manifest_version: None, + published_commit: None, + tables: Vec::new(), + resume_at: None, + configuration_clear: None, + artifact: None, + timings: MaintenanceTimings::default(), + reason: None, + } + } +} + +impl MaintenanceLedger { + pub fn new(ledger_id: impl Into) -> Self { + Self { + version: MAINTENANCE_VERSION, + ledger_id: ledger_id.into(), + active: None, + unresolved: Vec::new(), + terminal: Vec::new(), + resume_at: BTreeMap::new(), + } + } + + /// The operation this record holds for `maintenance_id` and `graph`. + pub fn find(&self, graph: &str, maintenance_id: &str) -> Option<&MaintenanceOperation> { + self.active + .iter() + .chain(&self.unresolved) + .chain(&self.terminal) + .find(|op| op.maintenance_id == maintenance_id && op.graph == graph) + } + + /// Hold `operation` in the active slot, or answer the existing record of + /// the same request. Callers resolve orphans first + /// ([`Self::interrupt_orphans`]) so a record that landed after an in-doubt + /// acceptance never holds the slot. + pub fn admit( + &mut self, + operation: MaintenanceOperation, + ) -> Result { + let same_id = self + .active + .iter() + .chain(&self.unresolved) + .chain(&self.terminal) + .find(|op| op.maintenance_id == operation.maintenance_id); + if let Some(existing) = same_id { + return if existing.request_digest == operation.request_digest + && existing.graph == operation.graph + { + Ok(Admitted::Existing(Box::new(existing.clone()))) + } else { + Err(MaintenanceRefusal::DifferentRequest) + }; + } + if let Some(active) = &self.active { + return Err(MaintenanceRefusal::Active { + maintenance_id: active.maintenance_id.clone(), + }); + } + if self.unresolved.len() >= MAX_MAINTENANCE_UNRESOLVED { + return Err(MaintenanceRefusal::Unresolved); + } + self.active = Some(operation); + Ok(Admitted::New) + } + + /// Settle the active operation when no live run holds it (`live` is the + /// id this process is running, if any): `accepted` or `running` becomes + /// `interrupted`, since no attempt sent a manifest request, and + /// `publishing` moves to the unresolved list for main's history to + /// settle. Returns whether anything changed. + pub fn interrupt_orphans(&mut self, live: Option<&str>) -> bool { + let Some(active) = &self.active else { + return false; + }; + if live == Some(active.maintenance_id.as_str()) { + return false; + } + let mut operation = self.active.take().expect("checked above"); + match operation.state { + MaintenanceState::Publishing => self.unresolved.push(operation), + _ => { + operation.state = MaintenanceState::Interrupted; + operation.reason = Some(format!("no live run held it at {}", crate::now_rfc3339())); + self.push_terminal(operation); + } + } + true + } + + /// The active operation, if `maintenance_id` holds the slot. + pub fn active_mut(&mut self, maintenance_id: &str) -> Option<&mut MaintenanceOperation> { + self.active + .as_mut() + .filter(|op| op.maintenance_id == maintenance_id) + } + + /// End the active operation with its terminal record, and keep the + /// resume position a full-catalog run reports. A `publishing` operation + /// whose publication is in doubt goes to [`Self::defer_publication`] + /// instead. + pub fn finish(&mut self, operation: MaintenanceOperation) -> Result<(), Diagnostic> { + if !operation.state.is_terminal() { + return Err(invalid("a finished maintenance operation must be terminal")); + } + if self.active.as_ref().map(|active| &active.maintenance_id) + != Some(&operation.maintenance_id) + { + return Err(invalid("only the active maintenance operation can finish")); + } + self.active = None; + self.keep_resume_position(&operation); + self.push_terminal(operation); + Ok(()) + } + + /// Move the active operation, its publication in doubt, to the + /// unresolved list, freeing the slot. Its work is never run again. + pub fn defer_publication( + &mut self, + maintenance_id: &str, + reason: String, + ) -> Result<(), Diagnostic> { + let Some(mut operation) = self + .active + .take_if(|op| op.maintenance_id == maintenance_id) + else { + return Err(invalid("only the active maintenance operation can defer")); + }; + operation.state = MaintenanceState::Publishing; + operation.reason = Some(reason); + self.unresolved.push(operation); + Ok(()) + } + + /// Apply main's history to one unresolved operation. A settled one + /// becomes terminal; an operation still in doubt records where the walk + /// stopped. Returns whether anything changed. + pub fn settle(&mut self, maintenance_id: &str, resolution: Resolution) -> bool { + let Some(index) = self + .unresolved + .iter() + .position(|op| op.maintenance_id == maintenance_id) + else { + return false; + }; + match resolution { + Resolution::InDoubt { reason } => { + let operation = &mut self.unresolved[index]; + if operation.reason.as_deref() == Some(reason.as_str()) { + return false; + } + operation.reason = Some(reason); + } + Resolution::Committed { published_commit } => { + let mut operation = self.unresolved.remove(index); + operation.state = MaintenanceState::Succeeded; + operation.published_commit = Some(published_commit); + operation.reason = None; + self.keep_resume_position(&operation); + self.push_terminal(operation); + } + Resolution::NotCommitted => { + let mut operation = self.unresolved.remove(index); + operation.state = MaintenanceState::Failed; + operation.reason = Some("not_committed".to_string()); + self.push_terminal(operation); + } + } + true + } + + fn keep_resume_position(&mut self, operation: &MaintenanceOperation) { + // Only a full-catalog run that succeeded moves the position. + if operation.state != MaintenanceState::Succeeded || operation.options.tables.is_some() { + return; + } + match &operation.resume_at { + Some(key) => { + self.resume_at.insert(operation.graph.clone(), key.clone()); + } + None => { + self.resume_at.remove(&operation.graph); + } + } + } + + /// Newest first; the oldest terminal records go once the count or the + /// total size passes its bound. Unresolved records are never evicted. + fn push_terminal(&mut self, operation: MaintenanceOperation) { + self.terminal.insert(0, operation); + self.terminal.truncate(MAX_MAINTENANCE_TERMINAL); + while self.terminal.len() > 1 && self.encoded_len() > MAX_MAINTENANCE_BYTES { + self.terminal.pop(); + } + } + + fn encoded_len(&self) -> usize { + serde_json::to_vec(self).map_or(usize::MAX, |bytes| bytes.len()) + } + + /// The bounds and identities every read and write checks. + pub fn validate(&self) -> Result<(), Diagnostic> { + if self.version != MAINTENANCE_VERSION { + return Err(invalid(format!( + "unsupported maintenance record version {}", + self.version + ))); + } + if !canonical_ulid(&self.ledger_id) { + return Err(invalid( + "maintenance record ledger id is not a canonical ULID", + )); + } + if self.unresolved.len() > MAX_MAINTENANCE_UNRESOLVED { + return Err(invalid( + "maintenance record holds too many unresolved operations", + )); + } + if self.terminal.len() > MAX_MAINTENANCE_TERMINAL { + return Err(invalid( + "maintenance record holds too many terminal operations", + )); + } + let mut ids = std::collections::BTreeSet::new(); + for (place, operation) in self + .active + .iter() + .map(|op| ("active", op)) + .chain(self.unresolved.iter().map(|op| ("unresolved", op))) + .chain(self.terminal.iter().map(|op| ("terminal", op))) + { + if !canonical_ulid(&operation.maintenance_id) { + return Err(invalid("maintenance id is not a canonical ULID")); + } + if !ids.insert(operation.maintenance_id.as_str()) { + return Err(invalid(format!( + "maintenance id {} appears twice", + operation.maintenance_id + ))); + } + let fits = match place { + "active" => !operation.state.is_terminal(), + "unresolved" => operation.state == MaintenanceState::Publishing, + _ => operation.state.is_terminal(), + }; + if !fits { + return Err(invalid(format!( + "maintenance operation {} is {:?} in the {place} section", + operation.maintenance_id, operation.state + ))); + } + if serde_json::to_vec(operation).map_or(usize::MAX, |bytes| bytes.len()) + > MAX_MAINTENANCE_RECORD_BYTES + { + return Err(invalid(format!( + "maintenance operation {} exceeds {MAX_MAINTENANCE_RECORD_BYTES} bytes", + operation.maintenance_id + ))); + } + } + if self.encoded_len() > MAX_MAINTENANCE_BYTES { + return Err(invalid(format!( + "maintenance record exceeds {MAX_MAINTENANCE_BYTES} bytes" + ))); + } + Ok(()) + } +} + +/// The digest of a request: what makes two submissions of one id the same. +pub fn maintenance_request_digest( + graph: &str, + branch: &str, + kind: MaintenanceKind, + options: &OptimizeOptions, + principal: &str, +) -> String { + #[derive(Serialize)] + struct Canonical<'a> { + graph: &'a str, + branch: &'a str, + kind: MaintenanceKind, + options: &'a OptimizeOptions, + principal: &'a str, + } + let bytes = serde_json::to_vec(&Canonical { + graph, + branch, + kind, + options, + principal, + }) + .expect("a maintenance request serializes"); + format!("sha256:{}", crate::sha256_hex(&bytes)) +} + +/// The answer to one conditional write of the record. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum MaintenanceWrite { + /// The write landed; the new version is the next write's precondition. + Written { version: String }, + /// The precondition failed: another write changed the record. + Conflict, + /// No definite answer. The key is unsettled until a re-read settles it. + InDoubt { error: String }, +} + +pub(crate) fn decode(text: &str) -> Result { + let ledger: MaintenanceLedger = serde_json::from_str(text).map_err(|error| { + Diagnostic::error( + "maintenance_record_invalid", + CLUSTER_MAINTENANCE_FILE, + format!("could not parse the maintenance record: {error}"), + ) + })?; + ledger.validate()?; + Ok(ledger) +} + +pub(crate) fn encode(ledger: &MaintenanceLedger) -> Result { + ledger.validate()?; + serde_json::to_string(ledger).map_err(|error| { + Diagnostic::error( + "maintenance_record_invalid", + CLUSTER_MAINTENANCE_FILE, + format!("could not encode the maintenance record: {error}"), + ) + }) +} + +fn invalid(message: impl Into) -> Diagnostic { + Diagnostic::error( + "maintenance_record_invalid", + CLUSTER_MAINTENANCE_FILE, + message.into(), + ) +} + +#[cfg(test)] +mod tests { + use super::*; + + const LEDGER: &str = "01JBQZ0000000000000000ABCD"; + + fn id(n: u8) -> String { + format!("01JBQZ00000000000000000{:03}", n) + } + + fn operation(n: u8, graph: &str) -> MaintenanceOperation { + let options = OptimizeOptions::default(); + MaintenanceOperation { + maintenance_id: id(n), + request_digest: maintenance_request_digest( + graph, + "main", + MaintenanceKind::Optimize, + &options, + "act-op", + ), + graph: graph.to_string(), + branch: "main".to_string(), + kind: MaintenanceKind::Optimize, + options, + principal: "act-op".to_string(), + accepted_at: "2026-10-29T00:00:00Z".to_string(), + state: MaintenanceState::Accepted, + start_at: None, + nonce: None, + input_commit: None, + input_manifest_version: None, + published_commit: None, + tables: Vec::new(), + resume_at: None, + configuration_clear: None, + artifact: None, + timings: MaintenanceTimings::default(), + reason: None, + } + } + + fn finished(mut op: MaintenanceOperation, state: MaintenanceState) -> MaintenanceOperation { + op.state = state; + op + } + + #[test] + fn ids_used_by_these_tests_are_canonical() { + assert!(canonical_ulid(LEDGER), "{LEDGER}"); + assert!(canonical_ulid(&id(1)), "{}", id(1)); + } + + #[test] + fn admission_holds_one_active_operation_and_answers_a_repeat() { + let mut ledger = MaintenanceLedger::new(LEDGER); + assert_eq!(ledger.admit(operation(1, "g")), Ok(Admitted::New)); + assert_eq!( + ledger.admit(operation(1, "g")), + Ok(Admitted::Existing(Box::new(operation(1, "g")))) + ); + let mut other = operation(1, "g"); + other.options.tables = Some(vec!["node:Person".to_string()]); + other.request_digest = maintenance_request_digest( + "g", + "main", + MaintenanceKind::Optimize, + &other.options, + "act-op", + ); + assert_eq!( + ledger.admit(other), + Err(MaintenanceRefusal::DifferentRequest) + ); + assert_eq!( + ledger.admit(operation(2, "h")), + Err(MaintenanceRefusal::Active { + maintenance_id: id(1) + }) + ); + ledger.validate().unwrap(); + } + + #[test] + fn an_orphan_is_interrupted_unless_it_was_publishing() { + let mut ledger = MaintenanceLedger::new(LEDGER); + ledger.admit(operation(1, "g")).unwrap(); + assert!( + !ledger.interrupt_orphans(Some(&id(1))), + "a live run keeps its slot" + ); + assert!(ledger.interrupt_orphans(None)); + assert_eq!(ledger.terminal[0].state, MaintenanceState::Interrupted); + assert!(ledger.active.is_none()); + + ledger.admit(operation(2, "g")).unwrap(); + ledger.active_mut(&id(2)).unwrap().state = MaintenanceState::Publishing; + assert!(ledger.interrupt_orphans(None)); + assert_eq!(ledger.unresolved[0].maintenance_id, id(2)); + ledger.validate().unwrap(); + } + + #[test] + fn a_fifth_unresolved_publication_refuses_admission() { + let mut ledger = MaintenanceLedger::new(LEDGER); + for n in 1..=MAX_MAINTENANCE_UNRESOLVED as u8 { + ledger.admit(operation(n, "g")).unwrap(); + ledger + .defer_publication(&id(n), "head unchanged".to_string()) + .unwrap(); + } + assert_eq!( + ledger.admit(operation(9, "g")), + Err(MaintenanceRefusal::Unresolved) + ); + ledger.validate().unwrap(); + } + + #[test] + fn history_settles_an_unresolved_publication_once() { + let mut ledger = MaintenanceLedger::new(LEDGER); + ledger.admit(operation(1, "g")).unwrap(); + ledger.active_mut(&id(1)).unwrap().resume_at = Some("node:Zebra".to_string()); + ledger + .defer_publication(&id(1), "head unchanged".to_string()) + .unwrap(); + assert!(!ledger.settle( + &id(1), + Resolution::InDoubt { + reason: "head unchanged".to_string() + } + )); + assert!(ledger.settle( + &id(1), + Resolution::Committed { + published_commit: "hb1.child".to_string() + } + )); + assert!(ledger.unresolved.is_empty()); + assert_eq!(ledger.terminal[0].state, MaintenanceState::Succeeded); + assert_eq!( + ledger.resume_at.get("g").map(String::as_str), + Some("node:Zebra") + ); + assert!( + !ledger.settle(&id(1), Resolution::NotCommitted), + "a terminal record is never rewritten" + ); + + ledger.admit(operation(2, "g")).unwrap(); + ledger + .defer_publication(&id(2), "head unchanged".to_string()) + .unwrap(); + assert!(ledger.settle(&id(2), Resolution::NotCommitted)); + assert_eq!(ledger.terminal[0].state, MaintenanceState::Failed); + assert_eq!(ledger.terminal[0].reason.as_deref(), Some("not_committed")); + ledger.validate().unwrap(); + } + + #[test] + fn terminal_records_are_bounded_newest_first() { + let mut ledger = MaintenanceLedger::new(LEDGER); + for n in 1..=(MAX_MAINTENANCE_TERMINAL as u8 + 3) { + ledger.admit(operation(n, "g")).unwrap(); + ledger + .finish(finished(operation(n, "g"), MaintenanceState::Succeeded)) + .unwrap(); + } + assert_eq!(ledger.terminal.len(), MAX_MAINTENANCE_TERMINAL); + assert_eq!( + ledger.terminal[0].maintenance_id, + id(MAX_MAINTENANCE_TERMINAL as u8 + 3) + ); + assert!(ledger.find("g", &id(1)).is_none(), "the oldest is evicted"); + ledger.validate().unwrap(); + } + + #[test] + fn a_table_list_run_never_moves_the_resume_position() { + let mut ledger = MaintenanceLedger::new(LEDGER); + ledger + .resume_at + .insert("g".to_string(), "node:Kept".to_string()); + let mut listed = operation(1, "g"); + listed.options.tables = Some(vec!["node:Person".to_string()]); + listed.resume_at = Some("node:Other".to_string()); + ledger.admit(listed.clone()).unwrap(); + ledger + .finish(finished(listed, MaintenanceState::Succeeded)) + .unwrap(); + assert_eq!( + ledger.resume_at.get("g").map(String::as_str), + Some("node:Kept") + ); + } + + #[tokio::test] + async fn a_conditional_write_lands_once_and_refuses_a_stale_version() { + let dir = tempfile::tempdir().unwrap(); + std::fs::create_dir_all(dir.path().join(crate::CLUSTER_STATE_DIR)).unwrap(); + let store = crate::store::ClusterStore::for_config_dir(dir.path()); + assert!(store.read_maintenance().await.unwrap().is_none()); + + let mut ledger = MaintenanceLedger::new(LEDGER); + let MaintenanceWrite::Written { version: first } = + store.write_maintenance(&ledger, None).await.unwrap() + else { + panic!("the first create lands"); + }; + assert_eq!( + store.write_maintenance(&ledger, None).await.unwrap(), + MaintenanceWrite::Conflict, + "a create over an existing record conflicts" + ); + ledger.admit(operation(1, "g")).unwrap(); + let MaintenanceWrite::Written { version: second } = store + .write_maintenance(&ledger, Some(&first)) + .await + .unwrap() + else { + panic!("a write at the read version lands"); + }; + assert_ne!(first, second); + assert_eq!( + store + .write_maintenance(&MaintenanceLedger::new(LEDGER), Some(&first)) + .await + .unwrap(), + MaintenanceWrite::Conflict, + "a stale version conflicts" + ); + assert_eq!( + store.read_maintenance().await.unwrap(), + Some((ledger, second)) + ); + } + + #[test] + fn decoding_refuses_an_operation_in_the_wrong_section() { + let mut ledger = MaintenanceLedger::new(LEDGER); + ledger.terminal.push(operation(1, "g")); + let text = serde_json::to_string(&ledger).unwrap(); + assert!(decode(&text).is_err()); + let valid = MaintenanceLedger::new(LEDGER); + assert_eq!(decode(&encode(&valid).unwrap()).unwrap(), valid); + } +} diff --git a/crates/omnigraph-cluster/src/store.rs b/crates/omnigraph-cluster/src/store.rs index 43cb4ee8b..6057487f5 100644 --- a/crates/omnigraph-cluster/src/store.rs +++ b/crates/omnigraph-cluster/src/store.rs @@ -1132,6 +1132,65 @@ impl ClusterStore { Ok(()) } + // ---- maintenance record ---- + + /// The record and its backend version token, or `None` when the root has + /// none yet. One bounded GET; a read error stays loud. + pub(crate) async fn read_maintenance( + &self, + ) -> Result, Diagnostic> { + let uri = self.uri(crate::maintenance::CLUSTER_MAINTENANCE_FILE); + match self.read_versioned_opt(&uri).await { + Ok(Some((text, version))) => Ok(Some((crate::maintenance::decode(&text)?, version))), + Ok(None) => Ok(None), + Err(error) => Err(Diagnostic::error( + "maintenance_record_read_error", + crate::maintenance::CLUSTER_MAINTENANCE_FILE, + format!("could not read the maintenance record: {error}"), + )), + } + } + + /// One conditional put of the record against the version the writer read + /// (`None`: the record must not exist yet). A backend error is no definite + /// answer, so it is reported in doubt, never as a conflict. + pub(crate) async fn write_maintenance( + &self, + ledger: &crate::maintenance::MaintenanceLedger, + expected_version: Option<&str>, + ) -> Result { + use crate::maintenance::MaintenanceWrite; + let payload = crate::maintenance::encode(ledger)?; + let uri = self.uri(crate::maintenance::CLUSTER_MAINTENANCE_FILE); + let answer = match expected_version { + None => self + .adapter + .write_text_if_absent(&uri, &payload) + .await + .map(|created| created.then_some(None)), + Some(version) => self + .adapter + .write_text_if_match(&uri, &payload, version) + .await + .map(|written| written.map(Some)), + }; + Ok(match answer { + Ok(Some(Some(version))) => MaintenanceWrite::Written { version }, + // A create answers no token; the next write reads one back. + Ok(Some(None)) => match self.read_versioned_opt(&uri).await { + Ok(Some((text, version))) if text == payload => { + MaintenanceWrite::Written { version } + } + Ok(_) => MaintenanceWrite::Conflict, + Err(error) => MaintenanceWrite::InDoubt { error }, + }, + Ok(None) => MaintenanceWrite::Conflict, + Err(error) => MaintenanceWrite::InDoubt { + error: error.to_string(), + }, + }) + } + // ---- lock ---- pub(crate) async fn acquire_lock( diff --git a/crates/omnigraph-server/src/deployment.rs b/crates/omnigraph-server/src/deployment.rs index 587a308d6..c6065dc84 100644 --- a/crates/omnigraph-server/src/deployment.rs +++ b/crates/omnigraph-server/src/deployment.rs @@ -563,6 +563,18 @@ async fn execute( } } } + // A maintenance run on an affected graph stops at its next safe boundary; + // the drain then waits for at most one table's rewrite, the publication, + // the configuration clear and an issued artifact write. + let affected = keys + .iter() + .map(|key| key.graph_id.to_string()) + .collect::>(); + state.maintenance.cancel( + None, + Some(&affected), + &format!("deployment {}", request.deployment_id), + ); let transition = state .routing .registry diff --git a/crates/omnigraph-server/src/http_operations.rs b/crates/omnigraph-server/src/http_operations.rs index fe17da96b..494751d5a 100644 --- a/crates/omnigraph-server/src/http_operations.rs +++ b/crates/omnigraph-server/src/http_operations.rs @@ -173,7 +173,16 @@ pub(crate) fn graph() -> HttpRoutes { .route("commit changes", handlers::server_commit_changes) .route("diff", handlers::server_diff) .route("changes poll", handlers::server_changes_feed) - .route("changes baseline", handlers::server_changes_baseline); + .route("changes baseline", handlers::server_changes_baseline) + .route("start optimize", crate::maintenance::start_optimize) + .route( + "get maintenance operation", + crate::maintenance::get_maintenance_operation, + ) + .route( + "cancel maintenance operation", + crate::maintenance::cancel_maintenance_operation, + ); #[cfg(test)] let routes = { let mut routes = routes; diff --git a/crates/omnigraph-server/src/lib.rs b/crates/omnigraph-server/src/lib.rs index 6354b3676..19690ae97 100644 --- a/crates/omnigraph-server/src/lib.rs +++ b/crates/omnigraph-server/src/lib.rs @@ -11,6 +11,8 @@ mod handlers; mod http_contract; mod http_operations; mod ingress; +mod maintenance; +pub use maintenance::MaintenanceSettings; mod mcp; pub mod operations; mod redacted_cause; @@ -123,6 +125,7 @@ fn hash_bearer_token(token: &str) -> BearerTokenHash { (name = "cluster", description = "Cluster deployment planning and execution"), (name = "commits", description = "Graph commit history and publication"), (name = "health", description = "Process health and readiness probes"), + (name = "maintenance", description = "Served maintenance operations and their records"), (name = "management", description = "Server graph inventory and OAuth resource metadata"), (name = "mutations", description = "Graph mutations and data loading"), (name = "queries", description = "Inline and stored queries, and graph exports"), @@ -170,6 +173,9 @@ fn hash_bearer_token(token: &str) -> BearerTokenHash { handlers::server_merge_base, handlers::server_changes_feed, handlers::server_changes_baseline, + maintenance::start_optimize, + maintenance::get_maintenance_operation, + maintenance::cancel_maintenance_operation, ), components(schemas(api::EntityKind, api::CommitListing, api::ChangeBaselineRecord, api::ChangeErrorOutput)), modifiers(&SecurityAddon), @@ -421,6 +427,8 @@ pub struct AppState { // drain is not qualified native settlement, so the server never unlocks. cluster_admission: Option, deployments: Arc, + /// Served maintenance: the record gate and the run this process holds. + maintenance: Arc, /// Runtime routing and availability for every configured graph. /// Middleware injects an admitted `serving::GraphRequest` or refuses a blocked /// graph before collecting its request body. @@ -860,6 +868,7 @@ impl AppState { Self { cluster_admission: None, deployments: Arc::new(deployment::DeploymentRuntime::default()), + maintenance: Arc::new(maintenance::MaintenanceRuntime::default()), routing: GraphRouting { registry, config_path: None, @@ -914,6 +923,7 @@ impl AppState { Ok(Self { cluster_admission: None, deployments: Arc::new(deployment::DeploymentRuntime::default()), + maintenance: Arc::new(maintenance::MaintenanceRuntime::default()), routing: GraphRouting { registry, config_path, @@ -1037,6 +1047,14 @@ impl AppState { self } + /// Attach the served maintenance settings; the routes are off without + /// them. + #[must_use] + pub fn with_maintenance_settings(mut self, settings: MaintenanceSettings) -> Self { + self.maintenance = Arc::new(maintenance::MaintenanceRuntime::new(settings)); + self + } + pub(crate) fn mcp_settings(&self) -> &McpSettings { &self.mcp } @@ -4048,7 +4066,11 @@ async fn serve_config( shutdown_grace, ) .with_process_defaults(process_defaults) - .with_mcp_settings(mcp); + .with_mcp_settings(mcp) + .with_maintenance_settings( + MaintenanceSettings::from_env() + .map_err(|error| eyre!("maintenance setting refused at startup: {error}"))?, + ); deployment::initialize_boot_activation(&state); let retained_admission = state.cluster_admission.clone(); if let Some(admission) = retained_admission.as_ref() { diff --git a/crates/omnigraph-server/src/maintenance.rs b/crates/omnigraph-server/src/maintenance.rs new file mode 100644 index 000000000..b47b0aea4 --- /dev/null +++ b/crates/omnigraph-server/src/maintenance.rs @@ -0,0 +1,1076 @@ +//! Served maintenance: a remote optimize run through the serving owner. +//! +//! A caller submits an optimize under its own maintenance id. Admission and +//! every change of the operation's state go through the cluster's record, +//! `__cluster/maintenance.json`, which this process, the lock holder, writes +//! with one conditional put each time: at acceptance, when the run starts, +//! before its publication and at its end. The run is an owned server +//! operation and holds its graph request, so a deployment that affects the +//! graph cancels it before draining it, and shutdown cancels it too. +//! +//! A write whose answer is lost stops the run's writes. The next admission or +//! lookup reads the record back: an `accepted` or `running` operation no live +//! run holds is `interrupted`, a `publishing` one waits on main's history, +//! and a lost terminal write is re-issued before any other. + +use std::sync::{Arc, Mutex as SyncMutex, PoisonError}; +use std::time::Duration; + +use axum::{Extension, Json, extract::State, http::StatusCode}; +use omnigraph::db::{ + ArtifactRefresh, ConfigurationClear, OptimizeBeforePublish, OptimizeCancellation, + OptimizeLimits, OptimizeReport, PublicationLookup, RunCancellation, TableCeilings, + TableOptimizeOutcome, +}; +use omnigraph_cluster::{ + Admitted, ClusterAdmission, MaintenanceLedger, MaintenanceOperation, MaintenanceRefusal, + MaintenanceState, MaintenanceTableOutcome, MaintenanceTailOutcome, MaintenanceWrite, + OptimizeOptions, Resolution, +}; +use serde::{Deserialize, Serialize}; +use tokio::sync::{Mutex, MutexGuard}; +use tokio::time::Instant; +use tracing::Instrument; +use utoipa::ToSchema; + +use crate::extract::{ApiJson, ApiPath}; +use crate::operations::OwnedResult; +use crate::serving::GraphRequest; +use crate::workload::IngressLease; +use crate::{ApiError, AppState, AuthenticatedActor, ErrorCode, PolicyAction, PolicyRequest}; + +type Actor = Option>; + +const MAIN: &str = "main"; + +/// Operator settings for the maintenance routes, read once at startup. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct MaintenanceSettings { + /// The routes answer 503 unless an operator enables them. + pub enabled: bool, + /// The gate hold of a request that names none. + pub default_gate_hold: Duration, + /// The largest gate hold a request may ask for. + pub max_gate_hold: Duration, + /// The adjacency artifact bound every served run applies. + pub artifact_write_limit: u64, +} + +impl Default for MaintenanceSettings { + fn default() -> Self { + Self { + enabled: false, + default_gate_hold: Duration::from_secs(60), + max_gate_hold: Duration::from_secs(300), + artifact_write_limit: 512 * 1024 * 1024, + } + } +} + +impl MaintenanceSettings { + /// `OMNIGRAPH_MAINTENANCE_ROUTES` (`true` or `false`), + /// `OMNIGRAPH_MAINTENANCE_GATE_HOLD_SECONDS`, + /// `OMNIGRAPH_MAINTENANCE_MAX_GATE_HOLD_SECONDS` and + /// `OMNIGRAPH_MAINTENANCE_ARTIFACT_WRITE_LIMIT_BYTES`, over the defaults. + pub fn from_env() -> Result { + Self::from_lookup(|name| std::env::var(name).ok()) + } + + fn from_lookup(lookup: impl Fn(&str) -> Option) -> Result { + let mut settings = Self::default(); + if let Some(value) = lookup("OMNIGRAPH_MAINTENANCE_ROUTES") { + settings.enabled = match value.as_str() { + "true" | "1" => true, + "false" | "0" => false, + other => { + return Err(format!( + "OMNIGRAPH_MAINTENANCE_ROUTES must be true or false, not '{other}'" + )); + } + }; + } + let number = |name: &str| -> Result, String> { + lookup(name) + .map(|value| { + value + .parse::() + .map_err(|_| format!("{name} must be a whole number, not '{value}'")) + }) + .transpose() + }; + if let Some(seconds) = number("OMNIGRAPH_MAINTENANCE_GATE_HOLD_SECONDS")? { + settings.default_gate_hold = Duration::from_secs(seconds); + } + if let Some(seconds) = number("OMNIGRAPH_MAINTENANCE_MAX_GATE_HOLD_SECONDS")? { + settings.max_gate_hold = Duration::from_secs(seconds); + } + if let Some(bytes) = number("OMNIGRAPH_MAINTENANCE_ARTIFACT_WRITE_LIMIT_BYTES")? { + settings.artifact_write_limit = bytes; + } + if settings.default_gate_hold > settings.max_gate_hold { + return Err("the default maintenance gate hold exceeds its ceiling".to_string()); + } + Ok(settings) + } +} + +/// The run this process holds, and how to stop it. +struct LiveRun { + maintenance_id: String, + graph: String, + cancellation: OptimizeCancellation, + /// Why a cancellation was requested, kept as the terminal reason. + cancel_reason: Option, +} + +#[derive(Default)] +struct RecordState { + /// A terminal write whose answer was lost. This process alone writes the + /// record and knows the outcome, so it is re-issued against whatever a + /// re-read shows before any other write. + unsettled: Option, +} + +#[derive(Default)] +pub(crate) struct MaintenanceRuntime { + settings: MaintenanceSettings, + /// Serializes every read and write of the record in this process. + record: Mutex, + live: SyncMutex>, +} + +impl MaintenanceRuntime { + pub(crate) fn new(settings: MaintenanceSettings) -> Self { + Self { + settings, + ..Self::default() + } + } + + fn live_id(&self) -> Option { + self.live + .lock() + .unwrap_or_else(PoisonError::into_inner) + .as_ref() + .map(|live| live.maintenance_id.clone()) + } + + fn clear_live(&self, maintenance_id: &str) -> Option { + let mut live = self.live.lock().unwrap_or_else(PoisonError::into_inner); + if live + .as_ref() + .is_some_and(|run| run.maintenance_id == maintenance_id) + { + live.take().and_then(|run| run.cancel_reason) + } else { + None + } + } + + /// Cancel the live run, if it is `maintenance_id` (any run when `None`) + /// on one of `graphs` (any graph when `None`), recording `reason`. + pub(crate) fn cancel( + &self, + maintenance_id: Option<&str>, + graphs: Option<&[String]>, + reason: &str, + ) -> bool { + let mut live = self.live.lock().unwrap_or_else(PoisonError::into_inner); + let Some(run) = live.as_mut() else { + return false; + }; + if maintenance_id.is_some_and(|id| id != run.maintenance_id) + || graphs.is_some_and(|graphs| !graphs.contains(&run.graph)) + { + return false; + } + run.cancel_reason.get_or_insert_with(|| reason.to_string()); + run.cancellation.cancel(); + true + } + + fn require_enabled(&self) -> Result<(), ApiError> { + if self.settings.enabled { + Ok(()) + } else { + Err(ApiError::new( + ErrorCode::ServiceUnavailable, + "the maintenance routes are not enabled on this server", + )) + } + } +} + +/// `POST /graphs/{graph_id}/maintenance/optimize`. +#[derive(Debug, Clone, Deserialize, ToSchema)] +#[serde(deny_unknown_fields)] +pub(crate) struct StartOptimizeRequest { + /// A canonical ULID the caller chose; a repeated submission with the same + /// request answers the existing record. + maintenance_id: String, + /// Only these table keys (`node:`, `edge:`), in this order. + #[serde(default)] + tables: Option>, + #[serde(default)] + max_gate_hold_ms: Option, + #[serde(default)] + max_source_fragments: Option, + #[serde(default)] + max_source_rows: Option, + #[serde(default)] + max_source_bytes: Option, + #[serde(default)] + max_blob_bytes: Option, + #[serde(default)] + max_index_rows: Option, + #[serde(default)] + run_source_bytes: Option, +} + +impl StartOptimizeRequest { + fn options(&self) -> OptimizeOptions { + OptimizeOptions { + tables: self.tables.clone(), + max_gate_hold_ms: self.max_gate_hold_ms, + max_source_fragments: self.max_source_fragments, + max_source_rows: self.max_source_rows, + max_source_bytes: self.max_source_bytes, + max_blob_bytes: self.max_blob_bytes, + max_index_rows: self.max_index_rows, + run_source_bytes: self.run_source_bytes, + } + } +} + +#[derive(Debug, Clone, Copy, Serialize, ToSchema)] +#[serde(rename_all = "snake_case")] +pub(crate) enum MaintenanceLookup { + Found, + /// The record does not hold this id for this graph. It proves neither + /// that the operation never ran nor that it may be replayed. + Unknown, +} + +#[derive(Debug, Clone, Serialize, ToSchema)] +pub(crate) struct MaintenanceOperationOutput { + lookup: MaintenanceLookup, + /// The operation's record; absent for an `unknown` lookup. + #[schema(value_type = Option)] + operation: Option, + /// Whether this server is running it now. + in_progress: bool, +} + +#[derive(Deserialize)] +pub(crate) struct MaintenancePath { + maintenance_id: String, +} + +fn authenticated(actor: Actor) -> Result { + actor.map(|Extension(actor)| actor).ok_or_else(|| { + ApiError::forbidden( + "maintenance routes require an authenticated actor; this server resolves none", + ) + }) +} + +fn authorize(handle: &GraphRequest, actor: &AuthenticatedActor) -> Result<(), ApiError> { + crate::handlers::authorize_request( + Some(actor), + handle.policy.as_deref(), + PolicyRequest { + action: PolicyAction::Maintain, + branch: Some(MAIN.to_string()), + target_branch: None, + }, + ) +} + +fn admission(state: &AppState) -> Result { + state.cluster_admission.clone().ok_or_else(|| { + ApiError::conflict("served maintenance requires a v2 cluster with serving admission") + }) +} + +fn refusal(error: omnigraph_cluster::Diagnostic) -> ApiError { + ApiError::conflict(format!("[{}] {}", error.code, error.message)) +} + +fn outcome_unknown(maintenance_id: &str, error: &str) -> ApiError { + ApiError::new( + ErrorCode::OutcomeUnknown, + format!( + "the maintenance record write for {maintenance_id} has no definite answer ({error}); look the operation up by its id before submitting another" + ), + ) +} + +/// The record, read under the gate after settling a lost terminal write, and +/// checked against the cluster's ledger. +struct Loaded { + ledger: MaintenanceLedger, + version: Option, + outstanding_deployment: Option, +} + +async fn load(owner: &ClusterAdmission, record: &mut RecordState) -> Result { + let (ledger_id, outstanding_deployment) = owner.maintenance_context().await.map_err(refusal)?; + let mut read = owner.read_maintenance().await.map_err(refusal)?; + if let Some(intended) = record.unsettled.as_ref() { + match &read { + Some((current, _)) if current == intended => {} + _ => { + let expected = read.as_ref().map(|(_, version)| version.as_str()); + match owner + .write_maintenance(intended, expected) + .await + .map_err(refusal)? + { + MaintenanceWrite::Written { version } => { + read = Some((intended.clone(), version)); + } + MaintenanceWrite::Conflict => { + return Err(ApiError::conflict( + "the maintenance record changed while a terminal write was unsettled", + )); + } + MaintenanceWrite::InDoubt { error } => { + return Err(ApiError::new( + ErrorCode::OutcomeUnknown, + format!( + "a terminal maintenance record write is still unsettled ({error}); retry later" + ), + )); + } + } + } + } + record.unsettled = None; + } + let (ledger, version) = match read { + Some((ledger, version)) => (ledger, Some(version)), + None => (MaintenanceLedger::new(ledger_id.clone()), None), + }; + if ledger.ledger_id != ledger_id { + return Err(ApiError::conflict(format!( + "[maintenance_record_foreign] the maintenance record names ledger {} but this cluster's ledger is {ledger_id}; move the record aside", + ledger.ledger_id + ))); + } + Ok(Loaded { + ledger, + version, + outstanding_deployment, + }) +} + +/// Settle operations no live run holds, then apply main's history to each +/// unresolved publication whose graph this server serves. +async fn resolve(state: &AppState, ledger: &mut MaintenanceLedger) -> bool { + let live = state.maintenance.live_id(); + let mut changed = ledger.interrupt_orphans(live.as_deref()); + for operation in ledger.unresolved.clone() { + let (Some(nonce), Some(version)) = (&operation.nonce, operation.input_manifest_version) + else { + continue; + }; + let Some(engine) = state + .routing + .registry + .list() + .into_iter() + .find(|view| view.key.graph_id.as_str() == operation.graph) + .map(|view| Arc::clone(&view.engine)) + else { + continue; + }; + let resolution = match engine + .resolve_publication(nonce, operation.input_commit.as_deref(), version) + .await + { + Ok(PublicationLookup::Committed { commit_id }) => Resolution::Committed { + published_commit: commit_id, + }, + Ok(PublicationLookup::NotCommitted) => Resolution::NotCommitted, + Ok(PublicationLookup::HeadUnchanged) => Resolution::InDoubt { + reason: "main's graph head is still the input head".to_string(), + }, + Ok(PublicationLookup::Unproven { reason }) => Resolution::InDoubt { reason }, + Ok(other) => Resolution::InDoubt { + reason: format!("{other:?}"), + }, + Err(error) => Resolution::InDoubt { + reason: format!("reading main's history: {error}"), + }, + }; + changed |= ledger.settle(&operation.maintenance_id, resolution); + } + changed +} + +/// Load and resolve under the gate, writing the resolution when it changed +/// the record. +async fn load_resolved<'a>( + state: &AppState, + owner: &ClusterAdmission, + record: &mut MutexGuard<'a, RecordState>, +) -> Result { + let mut loaded = load(owner, record).await?; + if resolve(state, &mut loaded.ledger).await { + match owner + .write_maintenance(&loaded.ledger, loaded.version.as_deref()) + .await + .map_err(refusal)? + { + MaintenanceWrite::Written { version } => loaded.version = Some(version), + MaintenanceWrite::Conflict => { + return Err(ApiError::conflict( + "the maintenance record changed during resolution; retry", + )); + } + MaintenanceWrite::InDoubt { error } => { + return Err(ApiError::new( + ErrorCode::OutcomeUnknown, + format!( + "resolving the maintenance record has no definite answer ({error}); retry later" + ), + )); + } + } + } + Ok(loaded) +} + +fn output( + state: &AppState, + operation: Option, +) -> (StatusCode, Json) { + let live = state.maintenance.live_id(); + let in_progress = operation + .as_ref() + .is_some_and(|op| live.as_deref() == Some(op.maintenance_id.as_str())); + let status = match &operation { + Some(op) if !op.state.is_terminal() => StatusCode::ACCEPTED, + _ => StatusCode::OK, + }; + ( + status, + Json(MaintenanceOperationOutput { + lookup: if operation.is_some() { + MaintenanceLookup::Found + } else { + MaintenanceLookup::Unknown + }, + operation, + in_progress, + }), + ) +} + +fn validate(settings: &MaintenanceSettings, input: &StartOptimizeRequest) -> Result<(), ApiError> { + if !omnigraph_cluster::canonical_maintenance_id(&input.maintenance_id) { + return Err(ApiError::bad_request( + "maintenance_id must be a canonical ULID", + )); + } + if let Some(hold) = input.max_gate_hold_ms + && Duration::from_millis(hold) > settings.max_gate_hold + { + return Err(ApiError::new( + ErrorCode::PayloadTooLarge, + format!( + "max_gate_hold_ms {hold} exceeds this server's ceiling of {} ms", + settings.max_gate_hold.as_millis() + ), + )); + } + if let (Some(table), Some(budget)) = (input.max_source_bytes, input.run_source_bytes) + && table > budget + { + return Err(ApiError::bad_request( + "max_source_bytes may not exceed run_source_bytes", + )); + } + Ok(()) +} + +#[utoipa::path( + post, path = "/maintenance/optimize", tag = "maintenance", + request_body = StartOptimizeRequest, + responses( + (status = 200, description = "The operation is terminal", body = MaintenanceOperationOutput), + (status = 202, description = "Durably accepted, or still running; look it up by its id until it is terminal", body = MaintenanceOperationOutput), + (status = 400, body = crate::api::ErrorOutput), + (status = 403, body = crate::api::ErrorOutput), + (status = 409, description = "The same id with a different request, an outstanding deployment, four unresolved publications, or a record from another ledger", body = crate::api::ErrorOutput), + (status = 413, description = "A limit above this server's ceiling", body = crate::api::ErrorOutput), + (status = 429, description = "Another maintenance operation is active in the cluster; its id is in the message", body = crate::api::ErrorOutput), + (status = 500, description = "Internal failure (code `internal`), or a record write with no definite answer (code `outcome_unknown`); look the operation up by its id", body = crate::api::ErrorOutput), + (status = 503, description = "The routes are not enabled, or admission is closed", body = crate::api::ErrorOutput), + ), + security(("bearer_token" = [])), +)] +/// Submit a served optimize run on the graph's main branch. +pub(crate) async fn start_optimize( + State(state): State, + Extension(handle): Extension, + Extension(ingress): Extension, + actor: Actor, + request: Result, ApiError>, +) -> Result<(StatusCode, Json), ApiError> { + let actor = authenticated(actor)?; + authorize(&handle, &actor)?; + state.maintenance.require_enabled()?; + let ApiJson(input) = request?; + validate(&state.maintenance.settings, &input)?; + let owner = admission(&state)?; + let graph = handle.key.graph_id.to_string(); + let principal = actor.actor_id.to_string(); + + let mut record = state.maintenance.record.lock().await; + let mut loaded = load_resolved(&state, &owner, &mut record).await?; + let start_at = if input.tables.is_none() { + loaded.ledger.resume_at.get(&graph).cloned() + } else { + None + }; + let operation = MaintenanceOperation::accepted( + input.maintenance_id.clone(), + graph.clone(), + input.options(), + principal, + start_at, + ); + if let Some(existing) = loaded.ledger.find(&graph, &input.maintenance_id) { + return if existing.request_digest == operation.request_digest { + Ok(output(&state, Some(existing.clone()))) + } else { + Err(ApiError::conflict(format!( + "maintenance id {} names a different request", + input.maintenance_id + ))) + }; + } + if let Some(deployment) = &loaded.outstanding_deployment { + return Err(ApiError::conflict(format!( + "deployment {deployment} is outstanding; submit maintenance after it completes" + ))); + } + let accepted_at = Instant::now(); + match loaded.ledger.admit(operation.clone()) { + Ok(Admitted::New) => {} + Ok(Admitted::Existing(existing)) => return Ok(output(&state, Some(*existing))), + Err(MaintenanceRefusal::DifferentRequest) => { + return Err(ApiError::conflict(format!( + "maintenance id {} names a different request", + input.maintenance_id + ))); + } + Err(MaintenanceRefusal::Active { maintenance_id }) => { + return Err(ApiError::too_many_requests(format!( + "maintenance operation {maintenance_id} is active in this cluster; retry after it finishes" + ))); + } + Err(MaintenanceRefusal::Unresolved) => { + return Err(ApiError::conflict( + "four maintenance publications are unresolved; a write on their graphs' main settles them", + )); + } + } + match owner + .write_maintenance(&loaded.ledger, loaded.version.as_deref()) + .await + .map_err(refusal)? + { + MaintenanceWrite::Written { .. } => {} + MaintenanceWrite::Conflict => { + return Err(ApiError::conflict( + "the maintenance record changed during admission; retry", + )); + } + MaintenanceWrite::InDoubt { error } => { + return Err(outcome_unknown(&input.maintenance_id, &error)); + } + } + + let cancellation = OptimizeCancellation::new(); + *state + .maintenance + .live + .lock() + .unwrap_or_else(PoisonError::into_inner) = Some(LiveRun { + maintenance_id: input.maintenance_id.clone(), + graph, + cancellation: cancellation.clone(), + cancel_reason: None, + }); + drop(record); + let run_state = state.clone(); + let run_handle = handle.clone(); + let span = tracing::info_span!( + "maintenance_optimize", + maintenance_id = %input.maintenance_id, + graph = %operation.graph, + ); + let submitted = state.operations.submit( + (ingress, handle), + run( + run_state, + owner, + run_handle, + operation, + cancellation, + accepted_at, + ) + .instrument(span), + ); + if let Err(error) = submitted { + // The accepted record has no live run; the next admission or lookup + // records it `interrupted`. + state.maintenance.clear_live(&input.maintenance_id); + return Err(error); + } + let accepted = { + let _record = state.maintenance.record.lock().await; + owner_record(&state, &input.maintenance_id).await + }; + Ok(output(&state, accepted)) +} + +/// The operation as the record holds it, read for a response. +async fn owner_record(state: &AppState, maintenance_id: &str) -> Option { + let owner = state.cluster_admission.as_ref()?; + let (ledger, _) = owner.read_maintenance().await.ok()??; + ledger + .active + .iter() + .chain(&ledger.unresolved) + .chain(&ledger.terminal) + .find(|op| op.maintenance_id == maintenance_id) + .cloned() +} + +/// Apply `change` to the active operation and write the record once. +async fn transition( + state: &AppState, + owner: &ClusterAdmission, + maintenance_id: &str, + change: impl FnOnce(&mut MaintenanceOperation), +) -> Result<(), String> { + let mut record = state.maintenance.record.lock().await; + let mut loaded = load(owner, &mut record) + .await + .map_err(|error| error.message.to_string())?; + let Some(active) = loaded.ledger.active_mut(maintenance_id) else { + return Err(format!( + "maintenance operation {maintenance_id} no longer holds the active slot" + )); + }; + change(active); + match owner + .write_maintenance(&loaded.ledger, loaded.version.as_deref()) + .await + { + Ok(MaintenanceWrite::Written { .. }) => Ok(()), + Ok(MaintenanceWrite::Conflict) => Err("the maintenance record changed".to_string()), + Ok(MaintenanceWrite::InDoubt { error }) => Err(error), + Err(error) => Err(error.message), + } +} + +/// How the run ends in the record. +enum Ending { + Terminal(Box), + /// The publication may have landed: main's history settles it. + Unresolved(String), + /// A record write was in doubt: write nothing more; a later read settles. + Silent, +} + +async fn run( + state: AppState, + owner: ClusterAdmission, + handle: GraphRequest, + mut operation: MaintenanceOperation, + cancellation: OptimizeCancellation, + accepted_at: Instant, +) -> OwnedResult<()> { + let started = Instant::now(); + let id = operation.maintenance_id.clone(); + + if let Err(error) = transition(&state, &owner, &id, |active| { + active.state = MaintenanceState::Running; + }) + .await + { + tracing::warn!(maintenance_id = %id, %error, "maintenance run did not start"); + state.maintenance.clear_live(&id); + return Ok(()).into(); + } + + // Shutdown and containment close admission; the run stops at its next + // safe boundary. + let watch = { + let operations = state.operations.clone(); + let maintenance = Arc::clone(&state.maintenance); + let id = id.clone(); + tokio::spawn(async move { + operations.wait_closed().await; + maintenance.cancel(Some(&id), None, "the server is shutting down"); + }) + }; + + let publishing_landed = Arc::new(std::sync::atomic::AtomicBool::new(false)); + let publishing_in_doubt = Arc::new(std::sync::atomic::AtomicBool::new(false)); + let hook = { + let state = state.clone(); + let owner = owner.clone(); + let id = id.clone(); + let landed = Arc::clone(&publishing_landed); + let in_doubt = Arc::clone(&publishing_in_doubt); + OptimizeBeforePublish::new(move |publication| { + let state = state.clone(); + let owner = owner.clone(); + let id = id.clone(); + let landed = Arc::clone(&landed); + let in_doubt = Arc::clone(&in_doubt); + async move { + let tables = publication.tables.iter().map(table_outcome).collect(); + match transition(&state, &owner, &id, |active| { + active.state = MaintenanceState::Publishing; + active.nonce = Some(publication.nonce.clone()); + active.input_commit = publication.input_commit.clone(); + active.input_manifest_version = Some(publication.input_manifest_version); + active.tables = tables; + }) + .await + { + Ok(()) => { + landed.store(true, std::sync::atomic::Ordering::SeqCst); + Ok(()) + } + Err(error) => { + in_doubt.store(true, std::sync::atomic::Ordering::SeqCst); + Err(omnigraph::error::OmniError::manifest_internal(format!( + "the maintenance record write before publication failed: {error}" + ))) + } + } + } + }) + }; + + let settings = &state.maintenance.settings; + let options = &operation.options; + let limits = OptimizeLimits::default() + .with_max_gate_hold( + options + .max_gate_hold_ms + .map_or(settings.default_gate_hold, Duration::from_millis), + ) + .with_artifact_bytes(settings.artifact_write_limit) + .with_table_ceilings(table_ceilings(options)); + let limits = match options.run_source_bytes { + Some(budget) => limits.with_run_source_bytes(budget), + None => limits, + }; + let mut request = omnigraph::db::OptimizeRequest::default() + .with_limits(limits) + .with_cancellation(cancellation) + .with_before_publish(hook); + if let Some(tables) = &options.tables { + request = request.with_tables(tables.clone()); + } else if let Some(start_at) = &operation.start_at { + request = request.with_start_at(start_at.clone()); + } + let result = handle + .engine + .optimize_as(Some(operation.principal.as_str()), request) + .await; + watch.abort(); + let cancel_reason = state.maintenance.clear_live(&id); + + operation.timings.queued_ms = Some(started.duration_since(accepted_at).as_millis() as u64); + operation.timings.compute_ms = Some(started.elapsed().as_millis() as u64); + let (ending, failure) = match result { + Ok(report) => ( + Ending::Terminal(Box::new(terminal_from_report( + operation, + &report, + cancel_reason, + ))), + None, + ), + Err(error) => { + let landed = publishing_landed.load(std::sync::atomic::Ordering::SeqCst); + let in_doubt = publishing_in_doubt.load(std::sync::atomic::Ordering::SeqCst); + let uncertain = matches!( + error.completion_evidence(), + Some(omnigraph::error::CompletionEvidence::Uncertain) | None + ); + let message = error.to_string(); + let ending = if in_doubt { + Ending::Silent + } else if landed && uncertain { + Ending::Unresolved(format!("the publication has no definite answer: {message}")) + } else { + operation.state = MaintenanceState::Failed; + operation.reason = Some(message); + Ending::Terminal(Box::new(operation)) + }; + (ending, Some(ApiError::from_omni(error))) + } + }; + { + let mut record = state.maintenance.record.lock().await; + let written = async { + let mut loaded = load(&owner, &mut record).await.ok()?; + match &ending { + Ending::Terminal(terminal) => { + loaded.ledger.finish((**terminal).clone()).ok()?; + } + Ending::Unresolved(reason) => { + loaded.ledger.defer_publication(&id, reason.clone()).ok()?; + } + Ending::Silent => return Some(()), + } + match owner + .write_maintenance(&loaded.ledger, loaded.version.as_deref()) + .await + { + Ok(MaintenanceWrite::Written { .. }) => Some(()), + Ok(MaintenanceWrite::InDoubt { .. }) if matches!(ending, Ending::Terminal(_)) => { + record.unsettled = Some(loaded.ledger); + Some(()) + } + _ => None, + } + } + .await; + if written.is_none() { + tracing::warn!(maintenance_id = %id, "the maintenance run's final record write did not land; a later read settles it"); + } + } + match failure { + // A failure that leaves storage in doubt contains the process, as any + // other owned write's does; the record already holds the outcome. + Some(error) if error.contains_process() => Err(error).into(), + _ => Ok(()).into(), + } +} + +fn table_ceilings(options: &OptimizeOptions) -> TableCeilings { + let mut ceilings = TableCeilings::default(); + if let Some(value) = options.max_source_fragments { + ceilings = ceilings.with_source_fragments(value); + } + if let Some(value) = options.max_source_rows { + ceilings = ceilings.with_source_rows(value); + } + if let Some(value) = options.max_source_bytes { + ceilings = ceilings.with_source_bytes(value); + } + if let Some(value) = options.max_blob_bytes { + ceilings = ceilings.with_blob_bytes(value); + } + if let Some(value) = options.max_index_rows { + ceilings = ceilings.with_index_rows(value); + } + ceilings +} + +fn table_outcome(stat: &omnigraph::db::DatasetOptimizeStats) -> MaintenanceTableOutcome { + let (reason, detail) = match stat.outcome { + TableOptimizeOutcome::Deferred(reason) => (Some(reason.as_str()), Some(reason.to_string())), + TableOptimizeOutcome::Skipped(reason) => (Some(reason.as_str()), Some(reason.to_string())), + TableOptimizeOutcome::Failed => { + (None, stat.failure.as_ref().map(|error| error.to_string())) + } + _ => (None, None), + }; + MaintenanceTableOutcome { + type_key: stat.type_key.clone(), + outcome: stat.outcome.as_str().to_string(), + reason: reason.map(str::to_string), + detail, + fragments_removed: stat.fragments_removed as u64, + fragments_added: stat.fragments_added as u64, + duration_ms: stat.duration.as_millis() as u64, + source_bytes: stat.source_bytes, + } +} + +fn terminal_from_report( + mut operation: MaintenanceOperation, + report: &OptimizeReport, + cancel_reason: Option, +) -> MaintenanceOperation { + operation.tables = report.tables.iter().map(table_outcome).collect(); + operation.input_commit = report.input_commit.clone(); + operation.input_manifest_version = Some(report.input_manifest_version); + operation.published_commit = report.graph_commit_id.clone(); + operation.resume_at = report.resume_at.clone(); + operation.timings.gate_hold_ms = Some(report.gate_hold.as_millis() as u64); + operation.configuration_clear = Some(MaintenanceTailOutcome { + outcome: report.configuration_clear.as_str().to_string(), + detail: match &report.configuration_clear { + ConfigurationClear::Failed(error) | ConfigurationClear::Unknown(error) => { + Some(error.clone()) + } + _ => None, + }, + }); + operation.artifact = Some(MaintenanceTailOutcome { + outcome: report.artifact.as_str().to_string(), + detail: match &report.artifact { + ArtifactRefresh::Skipped(reason) => Some(reason.as_str().to_string()), + ArtifactRefresh::Failed(error) => Some(error.clone()), + _ => None, + }, + }); + match report.cancellation { + RunCancellation::Cancelled => { + operation.state = MaintenanceState::Cancelled; + operation.reason = Some(cancel_reason.unwrap_or_else(|| "cancelled".to_string())); + } + RunCancellation::ArrivedLate => { + operation.state = MaintenanceState::Succeeded; + operation.reason = Some(format!( + "a cancellation arrived after the publication started{}", + cancel_reason + .map(|reason| format!(": {reason}")) + .unwrap_or_default() + )); + } + _ => operation.state = MaintenanceState::Succeeded, + } + operation +} + +#[utoipa::path( + get, path = "/maintenance/{maintenance_id}", tag = "maintenance", + params(("maintenance_id" = String, Path, description = "The id the caller submitted")), + responses( + (status = 200, description = "The operation's record, or `lookup: unknown`", body = MaintenanceOperationOutput), + (status = 403, body = crate::api::ErrorOutput), + (status = 409, description = "A record from another ledger at this root, or the record changed during resolution", body = crate::api::ErrorOutput), + (status = 500, description = "Internal failure (code `internal`), or a resolution write with no definite answer (code `outcome_unknown`); retry the lookup", body = crate::api::ErrorOutput), + (status = 503, description = "The routes are not enabled", body = crate::api::ErrorOutput), + ), + security(("bearer_token" = [])), +)] +/// Look a served maintenance operation up by its original id. +pub(crate) async fn get_maintenance_operation( + State(state): State, + Extension(handle): Extension, + actor: Actor, + ApiPath(path): ApiPath, +) -> Result, ApiError> { + let actor = authenticated(actor)?; + authorize(&handle, &actor)?; + state.maintenance.require_enabled()?; + let owner = admission(&state)?; + let graph = handle.key.graph_id.to_string(); + let mut record = state.maintenance.record.lock().await; + let loaded = load_resolved(&state, &owner, &mut record).await?; + let operation = loaded.ledger.find(&graph, &path.maintenance_id).cloned(); + drop(record); + Ok(output(&state, operation).1) +} + +#[utoipa::path( + post, path = "/maintenance/{maintenance_id}/cancel", tag = "maintenance", + params(("maintenance_id" = String, Path, description = "The id the caller submitted")), + responses( + (status = 200, description = "The operation is terminal, unchanged; or `lookup: unknown`", body = MaintenanceOperationOutput), + (status = 202, description = "Cancellation requested; a `publishing` operation is unchanged", body = MaintenanceOperationOutput), + (status = 403, body = crate::api::ErrorOutput), + (status = 409, description = "A record from another ledger at this root, or the record changed during resolution", body = crate::api::ErrorOutput), + (status = 500, description = "Internal failure (code `internal`), or a resolution write with no definite answer (code `outcome_unknown`); look the operation up by its id", body = crate::api::ErrorOutput), + (status = 503, description = "The routes are not enabled", body = crate::api::ErrorOutput), + ), + security(("bearer_token" = [])), +)] +/// Ask a served maintenance operation to stop at its next safe boundary. +pub(crate) async fn cancel_maintenance_operation( + State(state): State, + Extension(handle): Extension, + actor: Actor, + ApiPath(path): ApiPath, +) -> Result<(StatusCode, Json), ApiError> { + let actor = authenticated(actor)?; + authorize(&handle, &actor)?; + state.maintenance.require_enabled()?; + let owner = admission(&state)?; + let graph = handle.key.graph_id.to_string(); + let mut record = state.maintenance.record.lock().await; + let loaded = load_resolved(&state, &owner, &mut record).await?; + let operation = loaded.ledger.find(&graph, &path.maintenance_id).cloned(); + drop(record); + if let Some(op) = &operation + && !op.state.is_terminal() + && op.state != MaintenanceState::Publishing + { + state.maintenance.cancel( + Some(&op.maintenance_id), + None, + &format!("cancelled by {}", actor.actor_id), + ); + } + Ok(output(&state, operation)) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn settings_default_off_and_read_their_variables() { + assert_eq!( + MaintenanceSettings::from_lookup(|_| None).unwrap(), + MaintenanceSettings::default() + ); + let settings = MaintenanceSettings::from_lookup(|name| match name { + "OMNIGRAPH_MAINTENANCE_ROUTES" => Some("true".to_string()), + "OMNIGRAPH_MAINTENANCE_GATE_HOLD_SECONDS" => Some("30".to_string()), + _ => None, + }) + .unwrap(); + assert!(settings.enabled); + assert_eq!(settings.default_gate_hold, Duration::from_secs(30)); + assert!( + MaintenanceSettings::from_lookup( + |name| (name == "OMNIGRAPH_MAINTENANCE_ROUTES").then(|| "yes".to_string()) + ) + .is_err() + ); + assert!( + MaintenanceSettings::from_lookup(|name| (name + == "OMNIGRAPH_MAINTENANCE_GATE_HOLD_SECONDS") + .then(|| "600".to_string())) + .is_err(), + "a default above the ceiling is refused" + ); + } + + #[test] + fn cancellation_reaches_only_the_named_run() { + let runtime = MaintenanceRuntime::default(); + let cancellation = OptimizeCancellation::new(); + *runtime.live.lock().unwrap() = Some(LiveRun { + maintenance_id: "01JBQZ00000000000000000001".to_string(), + graph: "g".to_string(), + cancellation: cancellation.clone(), + cancel_reason: None, + }); + assert!(!runtime.cancel(Some("01JBQZ00000000000000000002"), None, "other")); + assert!(!runtime.cancel(None, Some(&["h".to_string()][..]), "other graph")); + assert!(!cancellation.is_cancelled()); + assert!(runtime.cancel(None, Some(&["g".to_string()][..]), "deployment d")); + assert!(cancellation.is_cancelled()); + assert_eq!( + runtime.clear_live("01JBQZ00000000000000000001").as_deref(), + Some("deployment d") + ); + } +} diff --git a/crates/omnigraph-server/tests/integration/openapi.rs b/crates/omnigraph-server/tests/integration/openapi.rs index d6186fbd5..954996d94 100644 --- a/crates/omnigraph-server/tests/integration/openapi.rs +++ b/crates/omnigraph-server/tests/integration/openapi.rs @@ -220,6 +220,9 @@ const EXPECTED_PATHS: &[&str] = &[ "/graphs/{graph_id}/branches/diff", "/graphs/{graph_id}/branches/merge-base", "/graphs/{graph_id}/diff", + "/graphs/{graph_id}/maintenance/optimize", + "/graphs/{graph_id}/maintenance/{maintenance_id}", + "/graphs/{graph_id}/maintenance/{maintenance_id}/cancel", "/graphs/{graph_id}/commits", "/graphs/{graph_id}/commits/{commit_id}", "/graphs/{graph_id}/commits/{commit_id}/publish", diff --git a/crates/omnigraph-server/tests/maintenance_routes.rs b/crates/omnigraph-server/tests/maintenance_routes.rs new file mode 100644 index 000000000..ddfee47c1 --- /dev/null +++ b/crates/omnigraph-server/tests/maintenance_routes.rs @@ -0,0 +1,358 @@ +//! Served maintenance routes: submit, look up and cancel a served optimize +//! run, and the operation record that answers them. +//! +//! A separate binary: the journey re-executes it to run a production server +//! that holds the cluster's admission, which the record requires. + +use std::fs; +use std::io::{BufRead, BufReader}; +use std::path::{Path, PathBuf}; +use std::process::{Child, Command, Stdio}; +use std::time::{Duration, Instant}; + +use axum::body::Body; +use axum::http::{Request, StatusCode}; +use omnigraph_server::api::{HTTP_API_CONTRACT, HTTP_API_CONTRACT_HEADER}; +use omnigraph_server::{AppState, MaintenanceSettings, build_app}; +use serde_json::{Value, json}; + +mod support; +use support::*; + +const ROOT: &str = "OMNIGRAPH_MAINTENANCE_TEST_ROOT"; +const MAINTENANCE_ID: &str = "01JC0000000000000000000001"; + +const POLICY: &str = r#" +version: 1 +groups: + operators: [operator] + readers: [reader] +rules: + - id: maintain + allow: + actors: { group: operators } + actions: [maintain, read] + branch_scope: any + - id: read + allow: + actors: { group: readers } + actions: [read] + branch_scope: any +"#; + +fn post(uri: &str, token: &str, body: Value) -> Request { + Request::post(uri) + .header(HTTP_API_CONTRACT_HEADER, HTTP_API_CONTRACT) + .header("authorization", format!("Bearer {token}")) + .header("content-type", "application/json") + .body(Body::from(serde_json::to_vec(&body).unwrap())) + .unwrap() +} + +async fn single_graph_state( + settings: Option, +) -> (tempfile::TempDir, AppState) { + let temp = init_loaded_graph().await; + let policy = temp.path().join("policy.yaml"); + fs::write(&policy, POLICY).unwrap(); + let state = AppState::open_with_bearer_tokens_and_policy( + graph_path(temp.path()).to_string_lossy().to_string(), + vec![ + ("operator".into(), "op-secret".into()), + ("reader".into(), "read-secret".into()), + ], + Some(&policy), + ) + .await + .unwrap(); + let state = match settings { + Some(settings) => state.with_maintenance_settings(settings), + None => state, + }; + (temp, state) +} + +/// Off by default; an actor without `maintain` is refused first; a server +/// without cluster admission has no record to keep. +#[tokio::test] +async fn routes_refuse_without_a_grant_an_operator_switch_or_a_cluster() { + let route = "/graphs/default/maintenance/optimize"; + let body = json!({ "maintenance_id": MAINTENANCE_ID }); + + let (_dir, state) = single_graph_state(None).await; + let app = build_app(state); + let (status, error) = json_response(&app, post(route, "read-secret", body.clone())).await; + assert_eq!(status, StatusCode::FORBIDDEN, "{error}"); + let (status, error) = json_response(&app, post(route, "op-secret", body.clone())).await; + assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE, "{error}"); + assert_eq!(error["code"], "service_unavailable", "{error}"); + + let enabled = MaintenanceSettings { + enabled: true, + ..MaintenanceSettings::default() + }; + let (_dir, state) = single_graph_state(Some(enabled)).await; + let app = build_app(state); + let (status, error) = json_response( + &app, + post( + route, + "op-secret", + json!({ "maintenance_id": "not-a-ulid" }), + ), + ) + .await; + assert_eq!(status, StatusCode::BAD_REQUEST, "{error}"); + let (status, error) = json_response( + &app, + post( + route, + "op-secret", + json!({ "maintenance_id": MAINTENANCE_ID, "max_gate_hold_ms": 3_600_000 }), + ), + ) + .await; + assert_eq!(status, StatusCode::PAYLOAD_TOO_LARGE, "{error}"); + let (status, error) = json_response(&app, post(route, "op-secret", body)).await; + assert_eq!(status, StatusCode::CONFLICT, "{error}"); + assert!( + error["error"].as_str().unwrap().contains("v2 cluster"), + "{error}" + ); +} + +struct ContainedChild(Child); + +impl Drop for ContainedChild { + fn drop(&mut self) { + let _ = self.0.kill(); + let _ = self.0.wait(); + } +} + +fn applied_cluster() -> tempfile::TempDir { + let temp = tempfile::tempdir().unwrap(); + fs::write( + temp.path().join("people.pg"), + "node Person {\n name: String @key\n}\n", + ) + .unwrap(); + fs::write(temp.path().join("policy.yaml"), POLICY).unwrap(); + fs::write( + temp.path().join("cluster.yaml"), + "version: 1\ngraphs:\n knowledge:\n schema: ./people.pg\npolicies:\n access:\n file: ./policy.yaml\n applies_to: [knowledge]\n", + ) + .unwrap(); + temp +} + +fn spawn_server(root: &Path) -> (ContainedChild, String) { + let mut child = ContainedChild( + Command::new(std::env::current_exe().unwrap()) + .args(["--exact", "served_child", "--ignored", "--nocapture"]) + .env(ROOT, root) + .env("OMNIGRAPH_MAINTENANCE_ROUTES", "true") + .env( + "OMNIGRAPH_SERVER_BEARER_TOKENS_JSON", + r#"{"operator":"op-secret","reader":"read-secret"}"#, + ) + .env_remove("OMNIGRAPH_SERVER_BEARER_TOKEN") + .env_remove("OMNIGRAPH_SERVER_BEARER_TOKENS_FILE") + .env_remove("OMNIGRAPH_SERVER_BEARER_TOKENS_AWS_SECRET") + .stdout(Stdio::piped()) + .stderr(Stdio::inherit()) + .spawn() + .unwrap(), + ); + let stdout = child.0.stdout.take().unwrap(); + let (listen_tx, listen_rx) = std::sync::mpsc::channel(); + std::thread::spawn(move || { + for line in BufReader::new(stdout).lines().map_while(Result::ok) { + if let Some(address) = line.strip_prefix(omnigraph_server::LISTEN_ADDR_PREFIX) { + let _ = listen_tx.send(address.to_string()); + } + } + }); + let address = listen_rx + .recv_timeout(Duration::from_secs(30)) + .expect("the served child starts its listener"); + (child, address) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[ignore = "instrument: subprocess helper for served_optimize_keeps_a_record_a_caller_reads_by_id"] +async fn served_child() { + let Some(root) = std::env::var_os(ROOT).map(PathBuf::from) else { + return; + }; + let mut config = cluster_settings(&root).await.unwrap(); + config.bind = "127.0.0.1:0".into(); + omnigraph_server::serve(config).await.unwrap(); +} + +struct Client { + http: reqwest::Client, + base: String, +} + +impl Client { + async fn send(&self, request: reqwest::RequestBuilder) -> (u16, Value) { + let response = request + .header(HTTP_API_CONTRACT_HEADER, HTTP_API_CONTRACT) + .send() + .await + .unwrap(); + let status = response.status().as_u16(); + (status, response.json().await.unwrap_or(Value::Null)) + } + + async fn submit(&self, token: &str, body: Value) -> (u16, Value) { + self.send( + self.http + .post(format!( + "{}/graphs/knowledge/maintenance/optimize", + self.base + )) + .bearer_auth(token) + .json(&body), + ) + .await + } + + async fn lookup(&self, id: &str) -> (u16, Value) { + self.send( + self.http + .get(format!("{}/graphs/knowledge/maintenance/{id}", self.base)) + .bearer_auth("op-secret"), + ) + .await + } + + async fn cancel(&self, id: &str) -> (u16, Value) { + self.send( + self.http + .post(format!( + "{}/graphs/knowledge/maintenance/{id}/cancel", + self.base + )) + .bearer_auth("op-secret"), + ) + .await + } + + async fn wait_ready(&self, child: &mut Child) { + let deadline = Instant::now() + Duration::from_secs(30); + loop { + if let Ok(response) = self.http.get(format!("{}/readyz", self.base)).send().await + && response.status().is_success() + { + return; + } + assert!(child.try_wait().unwrap().is_none(), "server exited"); + assert!(Instant::now() < deadline, "server never became ready"); + tokio::time::sleep(Duration::from_millis(20)).await; + } + } +} + +/// The journey: a served run is accepted, runs under its owner and ends +/// `succeeded` with its commit; the same request answers the same record, +/// another request under the id is refused, an unknown id is `unknown`, and +/// the record on storage names the cluster's ledger. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn served_optimize_keeps_a_record_a_caller_reads_by_id() { + let temp = applied_cluster(); + apply_cluster_fixture(temp.path()).await; + { + // Three loads leave three small fragments for the run to compact. + let db = session( + omnigraph::db::Omnigraph::open( + temp.path().join("graphs/knowledge.omni").to_str().unwrap(), + ) + .await + .unwrap(), + ); + for name in ["Ada", "Grace", "Edsger"] { + db.load_jsonl( + &json!({"type": "Person", "data": {"name": name}}).to_string(), + omnigraph::loader::LoadMode::Merge, + ) + .await + .unwrap(); + } + } + let (mut child, address) = spawn_server(temp.path()); + let client = Client { + http: reqwest::Client::new(), + base: format!("http://{address}"), + }; + client.wait_ready(&mut child.0).await; + + let body = json!({ "maintenance_id": MAINTENANCE_ID, "max_gate_hold_ms": 30_000 }); + let (status, refused) = client.submit("read-secret", body.clone()).await; + assert_eq!(status, 403, "{refused}"); + + let (status, accepted) = client.submit("op-secret", body.clone()).await; + assert!(matches!(status, 200 | 202), "{status}: {accepted}"); + assert_eq!(accepted["lookup"], "found", "{accepted}"); + + let deadline = Instant::now() + Duration::from_secs(60); + let terminal = loop { + let (status, found) = client.lookup(MAINTENANCE_ID).await; + assert_eq!(status, 200, "{found}"); + let state = found["operation"]["state"].as_str().unwrap().to_string(); + if matches!( + state.as_str(), + "succeeded" | "failed" | "cancelled" | "interrupted" + ) { + break found; + } + assert!(Instant::now() < deadline, "the run never finished: {found}"); + tokio::time::sleep(Duration::from_millis(50)).await; + }; + let operation = &terminal["operation"]; + assert_eq!(operation["state"], "succeeded", "{terminal}"); + assert_eq!(terminal["in_progress"], false, "{terminal}"); + assert!(operation["published_commit"].is_string(), "{terminal}"); + assert!( + operation["tables"] + .as_array() + .unwrap() + .iter() + .any(|table| table["type_key"] == "node:Person" && table["outcome"] == "optimized"), + "{terminal}" + ); + assert_eq!(operation["principal"], "operator", "{terminal}"); + + let (status, repeated) = client.submit("op-secret", body.clone()).await; + assert_eq!(status, 200, "{repeated}"); + assert_eq!( + repeated["operation"], *operation, + "a repeat answers the record" + ); + let (status, refused) = client + .submit( + "op-secret", + json!({ "maintenance_id": MAINTENANCE_ID, "tables": ["node:Person"] }), + ) + .await; + assert_eq!(status, 409, "{refused}"); + + let (status, unknown) = client.lookup("01JC0000000000000000000009").await; + assert_eq!(status, 200, "{unknown}"); + assert_eq!(unknown["lookup"], "unknown", "{unknown}"); + let (status, unchanged) = client.cancel(MAINTENANCE_ID).await; + assert_eq!(status, 200, "{unchanged}"); + assert_eq!(unchanged["operation"], *operation); + + let record: Value = + serde_json::from_slice(&fs::read(temp.path().join("__cluster/maintenance.json")).unwrap()) + .unwrap(); + let ledger: Value = + serde_json::from_slice(&fs::read(temp.path().join("__cluster/state.json")).unwrap()) + .unwrap(); + assert_eq!(record["ledger_id"], ledger["ledger_id"]); + assert_eq!(record["terminal"][0]["maintenance_id"], MAINTENANCE_ID); + assert!(record.get("active").is_none(), "{record}"); + drop(child); +} diff --git a/crates/omnigraph/src/db/mod.rs b/crates/omnigraph/src/db/mod.rs index 0562bfac3..efb7129fb 100644 --- a/crates/omnigraph/src/db/mod.rs +++ b/crates/omnigraph/src/db/mod.rs @@ -24,13 +24,14 @@ pub use omnigraph::{ IndexBuildResult, InitOptions, MergeOutcome, MergeResult, Omnigraph, OpenMode, OptimizeBeforePublish, OptimizeCancellation, OptimizeConcurrency, OptimizeLimits, OptimizePublication, OptimizeReport, OptimizeRequest, PendingIndex, PreparedGraphCreate, - PreparedSchemaApply, PreparedSchemaSettlement, ReadCheck, RebuiltFullTextIndex, RepairAction, - RepairClassification, RepairStats, ResourceExceeded, ResourceMeasure, RetainedManifestVersions, - RunCancellation, SYSTEM_COLUMNS_PREFLIGHT, SchemaApplyReconciliation, SchemaApplyResult, - SchemaApplySettlement, SchemaContractDigest, SchemaNonPublicationProof, SkipReason, - StagingVerdict, SystemColumnUpgradeFinding, SystemColumnUpgradeOptions, - SystemColumnUpgradeOutcome, SystemColumnUpgradeReport, TableCeilings, TableCollectionPlan, - TableOptimizeOutcome, UnattachedTagRecord, UnpublishedManifest, unattached_release_order, + PreparedSchemaApply, PreparedSchemaSettlement, PublicationLookup, ReadCheck, + RebuiltFullTextIndex, RepairAction, RepairClassification, RepairStats, ResourceExceeded, + ResourceMeasure, RetainedManifestVersions, RunCancellation, SYSTEM_COLUMNS_PREFLIGHT, + SchemaApplyReconciliation, SchemaApplyResult, SchemaApplySettlement, SchemaContractDigest, + SchemaNonPublicationProof, SkipReason, StagingVerdict, SystemColumnUpgradeFinding, + SystemColumnUpgradeOptions, SystemColumnUpgradeOutcome, SystemColumnUpgradeReport, + TableCeilings, TableCollectionPlan, TableOptimizeOutcome, UnattachedTagRecord, + UnpublishedManifest, unattached_release_order, }; pub(crate) use omnigraph::{ LogicalBlobValue, RangedExternalBlobs, logical_row_image, row_blob_values, diff --git a/crates/omnigraph/src/db/omnigraph.rs b/crates/omnigraph/src/db/omnigraph.rs index 5716a6a61..1926ca66e 100644 --- a/crates/omnigraph/src/db/omnigraph.rs +++ b/crates/omnigraph/src/db/omnigraph.rs @@ -36,6 +36,7 @@ pub(crate) mod optimize; mod prepared_create; pub(crate) mod promotion; mod repair; +mod resolution; pub(crate) mod schema_apply; pub(crate) mod system_column_upgrade; pub(crate) mod table_ops; @@ -60,6 +61,7 @@ pub use optimize::{ use prepared_create::initial_schema_ir; pub use prepared_create::{GraphCreateReconciliation, PreparedGraphCreate}; pub use repair::{DatasetRepairStats, RepairAction, RepairClassification, RepairStats}; +pub use resolution::PublicationLookup; pub use schema_apply::{ PreparedSchemaApply, PreparedSchemaSettlement, SchemaApplyReconciliation, SchemaApplySettlement, SchemaContractDigest, SchemaNonPublicationProof, diff --git a/crates/omnigraph/src/db/omnigraph/resolution.rs b/crates/omnigraph/src/db/omnigraph/resolution.rs new file mode 100644 index 000000000..140c90f6c --- /dev/null +++ b/crates/omnigraph/src/db/omnigraph/resolution.rs @@ -0,0 +1,109 @@ +//! Settling a publication whose answer was lost, from main's history. +//! +//! A run that publishes under an exact graph-head expectation can only land +//! as the commit whose parent is the head every attempt read. Once main's +//! graph head moves, the input head's child is that commit or proves the run +//! did not commit: every `__manifest` version an attempt could have taken is +//! occupied, and a later attempt reads the moved head and refuses. + +use super::*; +use crate::db::graph_coordinator::GraphCoordinator; + +/// What main's history says about a publication with a given intent nonce. +#[derive(Debug, Clone, PartialEq, Eq)] +#[non_exhaustive] +pub enum PublicationLookup { + /// main's graph head is still the input head. A configuration-only + /// `__manifest` version does not move it, and an attempt at the version + /// after one can still be in flight. + HeadUnchanged, + /// The input head's child answers the nonce: the run committed as it. + Committed { commit_id: String }, + /// The input head's child does not answer the nonce. + NotCommitted, + /// The walk could not reach the input head's child, so nothing is proven + /// either way; `reason` says where it stopped. + Unproven { reason: String }, +} + +impl Omnigraph { + /// Walk main's `parent_commit_id` chain from the head of a fresh read of + /// its `__manifest` back to the child of `input_commit`, and judge that + /// child against `nonce`. Each step looks its parent up where a + /// commit-addressed read does: the head row and the commit buffer, then + /// `__history`, then the live branches. Every commit on main is written + /// on main, so each step's `graph_manifest_version` is below the one + /// before it, and the walk stops at the first link not above + /// `input_manifest_version`. + pub async fn resolve_publication( + &self, + nonce: &str, + input_commit: Option<&str>, + input_manifest_version: u64, + ) -> Result { + let coordinator = self.open_coordinator_for_branch(None).await?; + walk_to_child(&coordinator, nonce, input_commit, input_manifest_version).await + } +} + +async fn walk_to_child( + coordinator: &GraphCoordinator, + nonce: &str, + input_commit: Option<&str>, + input_manifest_version: u64, +) -> Result { + let mut current = coordinator.head_commit(); + if Some(current.graph_commit_id.as_str()) == input_commit { + return Ok(PublicationLookup::HeadUnchanged); + } + loop { + if current.parent_commit_id.as_deref() == input_commit { + return Ok( + match omnigraph_core::graph_commit_id::commit_id_answers( + ¤t.graph_commit_id, + nonce, + ) { + Ok(true) => PublicationLookup::Committed { + commit_id: current.graph_commit_id, + }, + Ok(false) => PublicationLookup::NotCommitted, + Err(error) => PublicationLookup::Unproven { + reason: format!( + "the input head's child {} has no readable id: {error}", + current.graph_commit_id + ), + }, + }, + ); + } + if current.graph_manifest_version <= input_manifest_version { + return Ok(PublicationLookup::Unproven { + reason: format!( + "the walk passed the input head's version at {}", + current.graph_commit_id + ), + }); + } + let Some(parent) = current.parent_commit_id.clone() else { + return Ok(PublicationLookup::Unproven { + reason: "the walk reached the first commit without meeting the input head" + .to_string(), + }); + }; + current = match coordinator.published_record(&parent).await { + Ok(Some(record)) => { + crate::db::commit_graph::graph_commit_from_manifest_row(record.commit) + } + Ok(None) => { + return Ok(PublicationLookup::Unproven { + reason: format!("no record of commit {parent}"), + }); + } + Err(error) => { + return Ok(PublicationLookup::Unproven { + reason: format!("reading commit {parent}: {error}"), + }); + } + }; + } +} diff --git a/crates/omnigraph/tests/forbidden_apis.rs b/crates/omnigraph/tests/forbidden_apis.rs index 0cf9cf838..a0dbf8242 100644 --- a/crates/omnigraph/tests/forbidden_apis.rs +++ b/crates/omnigraph/tests/forbidden_apis.rs @@ -343,6 +343,8 @@ const READ_ONLY_SURFACES: &[(&str, &str)] = &[ ("db/omnigraph.rs", "get_commit"), ("db/omnigraph.rs", "commit_target_branch"), ("db/omnigraph.rs", "list_commits"), + // A fresh read of main's head and its parent chain; no write. + ("db/omnigraph/resolution.rs", "resolve_publication"), ("db/omnigraph.rs", "list_commits_page"), ("db/omnigraph.rs", "list_commits_page_readable"), ("db/omnigraph.rs", "list_unpublished_commits"), diff --git a/crates/omnigraph/tests/integration/maintenance.rs b/crates/omnigraph/tests/integration/maintenance.rs index 3ac180334..c6ae45109 100644 --- a/crates/omnigraph/tests/integration/maintenance.rs +++ b/crates/omnigraph/tests/integration/maintenance.rs @@ -758,6 +758,64 @@ async fn optimize_skips_an_artifact_over_its_size_limit() { assert_eq!(std::fs::metadata(&artifact).unwrap().len(), size); } +/// main's history settles a publication whose answer was lost: the input +/// head's child answers the run's nonce, another nonce did not commit, a +/// later commit on main changes neither, and an unmoved head stays in doubt. +#[tokio::test] +async fn resolve_publication_judges_the_input_heads_child() { + use omnigraph::db::PublicationLookup; + let dir = tempfile::tempdir().unwrap(); + let db = init_and_load(&dir).await; + add_person_fragments(&db).await; + let (hook, seen) = recording_hook(); + let report = db + .optimize_as(None, OptimizeRequest::default().with_before_publish(hook)) + .await + .unwrap(); + let commit = report.graph_commit_id.clone().expect("the run published"); + let publication = seen.lock().unwrap()[0].clone(); + let lookup = |nonce: String| { + let input = publication.input_commit.clone(); + let version = publication.input_manifest_version; + let db = &db; + async move { + db.resolve_publication(&nonce, input.as_deref(), version) + .await + .unwrap() + } + }; + let committed = PublicationLookup::Committed { + commit_id: commit.clone(), + }; + assert_eq!(lookup(publication.nonce.clone()).await, committed); + assert_eq!( + lookup("01ARZ3NDEKTSV4RRFFQ69G5FAV".to_string()).await, + PublicationLookup::NotCommitted + ); + + mutate_main( + &db, + MUTATION_QUERIES, + "insert_person", + &mixed_params(&[("$name", "Zed")], &[("$age", 50)]), + ) + .await + .unwrap(); + assert_eq!(lookup(publication.nonce.clone()).await, committed); + + let head = db.list_commits(None).await.unwrap()[0].clone(); + assert_eq!( + db.resolve_publication( + &publication.nonce, + Some(&head.graph_commit_id), + head.graph_manifest_version + ) + .await + .unwrap(), + PublicationLookup::HeadUnchanged + ); +} + #[tokio::test] async fn optimize_after_load_then_again_is_idempotent() { let dir = tempfile::tempdir().unwrap(); diff --git a/docs/dev/control-plane.md b/docs/dev/control-plane.md index 360665e71..b8b041d29 100644 --- a/docs/dev/control-plane.md +++ b/docs/dev/control-plane.md @@ -86,6 +86,7 @@ scope, response validation and refusal tests. | `__cluster/recoveries/` | Legacy evidence read only during explicit conversion checks | | `__cluster/approvals/` | Preserved legacy audit artifacts; no operational approval writer | | `__cluster/lock.json` | Persisted control lock; v2 lifetime cluster admission | +| `__cluster/maintenance.json` | Served maintenance record: the active operation, unresolved publications, bounded terminal results and each graph's resume position | | `graphs/.omni/` | Derived graph roots managed through apply | All stored control objects use the shared storage adapter. Filesystem replacement and object-store PUT/CAS details stay below that boundary; higher layers deal in versioned reads, conditional writes, and normalized roots. @@ -95,6 +96,27 @@ lineage publish together in `__manifest`. Ledger v2 stores prepared schema and graph-birth and graph-deletion authority in one outstanding record. Legacy control evidence is preserved for explicit conversion checks, never executed by a fallback sweep. +## Maintenance record + +`__cluster/maintenance.json` (`omnigraph_cluster::MaintenanceLedger`) is +separate from `state.json`, whose `deny_unknown_fields` ledger older binaries +must keep reading. It names the cluster's ledger id; a record naming another +ledger is refused with `maintenance_record_foreign` and never merged. Only +the lock holder writes it (`ClusterAdmission::write_maintenance` checks the +lock first), each write a conditional put against the version read: at +acceptance, when the run starts, in the optimize run's before-publish hook +(`publishing`, with the intent nonce and input head), and at the end. A +write with no definite answer (`MaintenanceWrite::InDoubt`) stops the run's +writes; the server's next admission or lookup reads the record back. An +`accepted` or `running` operation no live run holds becomes `interrupted`, +and a `publishing` one joins at most four unresolved operations, settled by +`Omnigraph::resolve_publication` once main's graph head moves: the input +head's child answers the nonce or proves the run did not commit. A lost +terminal write is re-issued first. The server side is +`crates/omnigraph-server/src/maintenance.rs`: one active operation per +cluster, an owned operation that holds its graph request, and cancellation +on shutdown and before a deployment's drain of the graph. + ## Deployment ledger `deployment.rs` and `deployment/execution.rs` own the sole v2 protocol. Fresh diff --git a/docs/dev/http-api.md b/docs/dev/http-api.md index e84429f5e..efef7b8ed 100644 --- a/docs/dev/http-api.md +++ b/docs/dev/http-api.md @@ -32,7 +32,7 @@ commit-page default remains with the separate pagination design. | `flat-path` | `/readyz` | `get=readiness` | Process probe. | | `flat-path` | `/.well-known/oauth-protected-resource` | `get=oauthProtectedResourceMetadata` | RFC 9728 path; retain the SDK name for compatibility. | | `flat-path` | `/.well-known/oauth-protected-resource/{resource_path}` | `get=oauthProtectedResourceMetadataAtResourcePath` | RFC 9728 path; retain the SDK name for compatibility. | -| `path-parameters` | New path parameters | `graph_id branch_name commit_id query_name deployment_id resource_path` | Resource-qualified names; OAuth keeps its protocol name. | +| `path-parameters` | New path parameters | `graph_id branch_name commit_id query_name deployment_id maintenance_id resource_path` | Resource-qualified names; OAuth keeps its protocol name. | | `query-parameters` | Query parameter names | `snake_case` | Choices use one enum; commit listing uses history, unpublished or unattached. | | `parameter-descriptions` | Every parameter | `required` | Document meaning, defaults, and refusal conditions. | | `header-names` | Parameters and response headers | `Title-Case` | Standard headers keep their HTTP meaning. | diff --git a/docs/user/index.md b/docs/user/index.md index 112c3aeb8..daf06d1b3 100644 --- a/docs/user/index.md +++ b/docs/user/index.md @@ -49,6 +49,7 @@ deployment. - [Server logs and traces](operations/observability.md) - [Authorization and actors](operations/policy.md) - [Maintenance](operations/maintenance.md) +- [Served maintenance](operations/served-maintenance.md) - [Upgrade from 0.13 to 0.14](operations/upgrade-0.14.md): checklist, YAML and CLI replacements - [Storage-format upgrades](operations/upgrade.md) diff --git a/docs/user/operations/maintenance.md b/docs/user/operations/maintenance.md index 34dcc4fe4..1df23aefd 100644 --- a/docs/user/operations/maintenance.md +++ b/docs/user/operations/maintenance.md @@ -9,8 +9,8 @@ OmniGraph provides five direct-storage maintenance commands: - `cleanup` permanently removes unretained table versions and unused table forks left by branches created before storage format v11. -They do not run through the HTTP server. Address a standalone graph directly, -or select a graph from a cluster root: +They do not run through the HTTP server; a cluster server runs optimize through +[served maintenance](served-maintenance.md). Address a graph directly, or from a cluster root: ```bash omnigraph optimize --target ./graph.omni diff --git a/docs/user/operations/served-maintenance.md b/docs/user/operations/served-maintenance.md new file mode 100644 index 000000000..2190fa846 --- /dev/null +++ b/docs/user/operations/served-maintenance.md @@ -0,0 +1,87 @@ +# Served maintenance + +A cluster server can run `optimize` on a graph's `main` itself, so a scheduler +or an operator can start it with one HTTP request and look the result up later +by an id it chose. The run uses the same engine as +[`omnigraph optimize`](maintenance.md#optimize) and publishes one graph commit. +Every request and every outcome is kept in a durable operation record, so a +caller that lost a response learns what happened instead of guessing. + +The routes are off by default. A standalone server, which has no cluster +record, refuses them; direct `omnigraph optimize` stays available there. + +## Enable it + +| Variable | Default | Meaning | +|---|---|---| +| `OMNIGRAPH_MAINTENANCE_ROUTES` | `false` | `true` serves the routes; otherwise they answer 503. | +| `OMNIGRAPH_MAINTENANCE_GATE_HOLD_SECONDS` | `60` | The gate hold of a request that names none. | +| `OMNIGRAPH_MAINTENANCE_MAX_GATE_HOLD_SECONDS` | `300` | The largest gate hold a request may ask for. | +| `OMNIGRAPH_MAINTENANCE_ARTIFACT_WRITE_LIMIT_BYTES` | `536870912` | The largest adjacency artifact a served run writes. | + +A value that does not parse, or a default above its ceiling, stops the server +at startup. Every route needs the [`maintain`](policy.md#actions) action on the +graph's `main` branch, for the actor the bearer token resolves to. + +## Submit, look up, cancel + +```bash +curl -X POST "$SERVER/graphs/knowledge/maintenance/optimize" \ + -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \ + -d '{"maintenance_id": "01JC0000000000000000000000", "max_gate_hold_ms": 30000}' +curl "$SERVER/graphs/knowledge/maintenance/01JC0000000000000000000000" \ + -H "Authorization: Bearer $TOKEN" +curl -X POST "$SERVER/graphs/knowledge/maintenance/01JC0000000000000000000000/cancel" \ + -H "Authorization: Bearer $TOKEN" +``` + +`maintenance_id` is a canonical ULID the caller generates. The body also takes +`tables` (run only those table keys, in that order) and the per-table ceilings +`max_source_fragments`, `max_source_rows`, `max_source_bytes`, +`max_blob_bytes` and `max_index_rows`, and the run budget `run_source_bytes`; +see [bounding a run](maintenance.md#bounding-a-run). + +A submission answers 202 with the record once it is durably accepted, and 200 +once the operation is terminal. Submitting the same id with the same request +answers the existing record; it never starts the work again. A lookup answers +200 with the record, or `"lookup": "unknown"` when the record does not hold +the id. `unknown` does not prove the operation never ran, and it is never a +reason to resubmit the same work under the same id. + +| Refusal | Status | +|---|---| +| The same id with a different request; an outstanding deployment; four unresolved publications; a record from another ledger at this root | 409 `conflict` | +| Another maintenance operation is active in the cluster (its id is in the message) | 429 `too_many_requests` | +| A gate hold above the server's ceiling | 413 `payload_too_large` | +| The routes are off, or the server is draining | 503 `service_unavailable` | +| A record write without a definite answer | 500 `outcome_unknown`; look the id up | + +## States + +An operation moves through `accepted`, `running` and `publishing` to one +terminal state: + +- `succeeded`: it published a commit (`published_commit`), or found nothing to + publish. A succeeded run may still have deferred or failed tables; read each + table's outcome. +- `failed`, `cancelled` and `interrupted`: nothing was published, and the + staged work is reclaimed by a later `cleanup`. + +`publishing` means a commit may have landed. When the server cannot tell, the +operation stays `publishing` and leaves the active slot. Once that graph's +`main` has a newer commit, the next submission or lookup settles it from +`main`'s history: `succeeded` if the run's commit is there, otherwise +`failed`. An operation that was `accepted` or `running` when its server +stopped is `interrupted` at the next submission or lookup. + +A full-catalog run that a gate hold or the byte budget stopped records the +table to start at; the next full-catalog submission for that graph starts +there, so successive bounded runs reach every table. + +Cancellation stops the run at its next safe boundary. A run that already +started its publication finishes it and reports that the cancellation arrived +late. A shutdown and a deployment that affects the graph cancel the running +operation the same way, and the record names the reason. + +The record keeps the 32 newest terminal operations; an older id answers +`unknown`. diff --git a/openapi.json b/openapi.json index e25be99be..727f1289f 100644 --- a/openapi.json +++ b/openapi.json @@ -9950,6 +9950,1027 @@ ] } }, + "/graphs/{graph_id}/maintenance/optimize": { + "post": { + "tags": [ + "maintenance" + ], + "summary": "Submit a served optimize run on the graph's main branch.", + "operationId": "startOptimize", + "parameters": [ + { + "name": "graph_id", + "in": "path", + "description": "Graph id to route the request to. The `og_` prefix is reserved for system graphs. `og_catalog` is the built-in, storage-free catalogue: it serves only POST /query and GET /schema with their ordinary response bodies. Its query refuses `branch` and `snapshot`; every other route under /graphs/og_catalog returns 405 `method_not_allowed` before effects.", + "required": true, + "schema": { + "type": "string" + } + }, + { + "name": "Omnigraph-Http-Api", + "in": "header", + "description": "exactly one Omnigraph-Http-Api header with value 0.14 is required", + "required": true, + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + } + }, + { + "name": "Omnigraph-If-Actor", + "in": "header", + "description": "The actor the caller expects its credential to resolve to. A mismatch is refused with 412 actor_mismatch before graph lookup or effects; a match changes nothing.", + "required": false, + "schema": { + "type": "string" + } + } + ], + "requestBody": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/StartOptimizeRequest" + } + } + }, + "required": true + }, + "responses": { + "200": { + "description": "The operation is terminal", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/MaintenanceOperationOutput" + } + } + } + }, + "202": { + "description": "Durably accepted, or still running; look it up by its id until it is terminal", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/MaintenanceOperationOutput" + } + } + } + }, + "400": { + "description": "; api_contract_mismatch when the required API contract header is missing, duplicated or unsupported", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "401": { + "description": "unauthorized when no bearer credential is presented or the server refuses it", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + }, + "WWW-Authenticate": { + "schema": { + "type": "string" + }, + "description": "Bearer challenge (RFC 6750 §3). It carries error=\"invalid_token\" when a presented bearer credential was refused, and resource_metadata naming the OAuth protected-resource metadata when the server has an OIDC resource identity." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "403": { + "description": "", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "404": { + "description": "not_found when no graph `graph_id` is served or the caller may not see it, or when no route serves the path", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "405": { + "description": "method_not_allowed when the path is served, but not with this method", + "headers": { + "Allow": { + "schema": { + "type": "string" + }, + "description": "The methods the path serves." + }, + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "408": { + "description": "request_timeout when the request body deadline passes before operation admission", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "409": { + "description": "The same id with a different request, an outstanding deployment, four unresolved publications, or a record from another ledger", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "412": { + "description": "actor_mismatch when Omnigraph-If-Actor names another actor than the credential resolves to", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "413": { + "description": "A limit above this server's ceiling; payload_too_large when the request body exceeds its route limit before operation admission", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "429": { + "description": "Another maintenance operation is active in the cluster; its id is in the message; too_many_requests when server observer, ingress or operation admission capacity is exhausted; honor Retry-After", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + }, + "Retry-After": { + "schema": { + "type": "string" + }, + "description": "Minimum delay before a caller-bounded retry of this refused request; not a scheduled server retry." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "500": { + "description": "Internal failure (code `internal`), or a record write with no definite answer (code `outcome_unknown`); look the operation up by its id", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "503": { + "description": "The routes are not enabled, or admission is closed; graph_unavailable for a known graph that is not served, or service_unavailable when server operation admission is closed; honor Retry-After, and reconcile any earlier write before retrying", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + }, + "Retry-After": { + "schema": { + "type": "string" + }, + "description": "Minimum delay before a caller-bounded retry of this refused request; not a scheduled server retry." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + } + }, + "security": [ + { + "bearer_token": [] + } + ] + } + }, + "/graphs/{graph_id}/maintenance/{maintenance_id}": { + "get": { + "tags": [ + "maintenance" + ], + "summary": "Look a served maintenance operation up by its original id.", + "operationId": "getMaintenanceOperation", + "parameters": [ + { + "name": "graph_id", + "in": "path", + "description": "Graph id to route the request to. The `og_` prefix is reserved for system graphs. `og_catalog` is the built-in, storage-free catalogue: it serves only POST /query and GET /schema with their ordinary response bodies. Its query refuses `branch` and `snapshot`; every other route under /graphs/og_catalog returns 405 `method_not_allowed` before effects.", + "required": true, + "schema": { + "type": "string" + } + }, + { + "name": "maintenance_id", + "in": "path", + "description": "The id the caller submitted", + "required": true, + "schema": { + "type": "string" + } + }, + { + "name": "Omnigraph-Http-Api", + "in": "header", + "description": "exactly one Omnigraph-Http-Api header with value 0.14 is required", + "required": true, + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + } + }, + { + "name": "Omnigraph-If-Actor", + "in": "header", + "description": "The actor the caller expects its credential to resolve to. A mismatch is refused with 412 actor_mismatch before graph lookup or effects; a match changes nothing.", + "required": false, + "schema": { + "type": "string" + } + } + ], + "responses": { + "200": { + "description": "The operation's record, or `lookup: unknown`", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/MaintenanceOperationOutput" + } + } + } + }, + "400": { + "description": "api_contract_mismatch when the required API contract header is missing, duplicated or unsupported", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "401": { + "description": "unauthorized when no bearer credential is presented or the server refuses it", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + }, + "WWW-Authenticate": { + "schema": { + "type": "string" + }, + "description": "Bearer challenge (RFC 6750 §3). It carries error=\"invalid_token\" when a presented bearer credential was refused, and resource_metadata naming the OAuth protected-resource metadata when the server has an OIDC resource identity." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "403": { + "description": "", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "404": { + "description": "not_found when no graph `graph_id` is served or the caller may not see it, or when no route serves the path", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "405": { + "description": "method_not_allowed when the path is served, but not with this method", + "headers": { + "Allow": { + "schema": { + "type": "string" + }, + "description": "The methods the path serves." + }, + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "409": { + "description": "A record from another ledger at this root, or the record changed during resolution", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "412": { + "description": "actor_mismatch when Omnigraph-If-Actor names another actor than the credential resolves to", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "429": { + "description": "too_many_requests when server observer, ingress or operation admission capacity is exhausted; honor Retry-After", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + }, + "Retry-After": { + "schema": { + "type": "string" + }, + "description": "Minimum delay before a caller-bounded retry of this refused request; not a scheduled server retry." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "500": { + "description": "Internal failure (code `internal`), or a resolution write with no definite answer (code `outcome_unknown`); retry the lookup", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "503": { + "description": "The routes are not enabled; graph_unavailable for a known graph that is not served, or service_unavailable when server operation admission is closed; honor Retry-After, and reconcile any earlier write before retrying", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + }, + "Retry-After": { + "schema": { + "type": "string" + }, + "description": "Minimum delay before a caller-bounded retry of this refused request; not a scheduled server retry." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + } + }, + "security": [ + { + "bearer_token": [] + } + ] + } + }, + "/graphs/{graph_id}/maintenance/{maintenance_id}/cancel": { + "post": { + "tags": [ + "maintenance" + ], + "summary": "Ask a served maintenance operation to stop at its next safe boundary.", + "operationId": "cancelMaintenanceOperation", + "parameters": [ + { + "name": "graph_id", + "in": "path", + "description": "Graph id to route the request to. The `og_` prefix is reserved for system graphs. `og_catalog` is the built-in, storage-free catalogue: it serves only POST /query and GET /schema with their ordinary response bodies. Its query refuses `branch` and `snapshot`; every other route under /graphs/og_catalog returns 405 `method_not_allowed` before effects.", + "required": true, + "schema": { + "type": "string" + } + }, + { + "name": "maintenance_id", + "in": "path", + "description": "The id the caller submitted", + "required": true, + "schema": { + "type": "string" + } + }, + { + "name": "Omnigraph-Http-Api", + "in": "header", + "description": "exactly one Omnigraph-Http-Api header with value 0.14 is required", + "required": true, + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + } + }, + { + "name": "Omnigraph-If-Actor", + "in": "header", + "description": "The actor the caller expects its credential to resolve to. A mismatch is refused with 412 actor_mismatch before graph lookup or effects; a match changes nothing.", + "required": false, + "schema": { + "type": "string" + } + } + ], + "responses": { + "200": { + "description": "The operation is terminal, unchanged; or `lookup: unknown`", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/MaintenanceOperationOutput" + } + } + } + }, + "202": { + "description": "Cancellation requested; a `publishing` operation is unchanged", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/MaintenanceOperationOutput" + } + } + } + }, + "400": { + "description": "api_contract_mismatch when the required API contract header is missing, duplicated or unsupported", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "401": { + "description": "unauthorized when no bearer credential is presented or the server refuses it", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + }, + "WWW-Authenticate": { + "schema": { + "type": "string" + }, + "description": "Bearer challenge (RFC 6750 §3). It carries error=\"invalid_token\" when a presented bearer credential was refused, and resource_metadata naming the OAuth protected-resource metadata when the server has an OIDC resource identity." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "403": { + "description": "", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "404": { + "description": "not_found when no graph `graph_id` is served or the caller may not see it, or when no route serves the path", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "405": { + "description": "method_not_allowed when the path is served, but not with this method", + "headers": { + "Allow": { + "schema": { + "type": "string" + }, + "description": "The methods the path serves." + }, + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "409": { + "description": "A record from another ledger at this root, or the record changed during resolution", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "412": { + "description": "actor_mismatch when Omnigraph-If-Actor names another actor than the credential resolves to", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "429": { + "description": "too_many_requests when server observer, ingress or operation admission capacity is exhausted; honor Retry-After", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + }, + "Retry-After": { + "schema": { + "type": "string" + }, + "description": "Minimum delay before a caller-bounded retry of this refused request; not a scheduled server retry." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "500": { + "description": "Internal failure (code `internal`), or a resolution write with no definite answer (code `outcome_unknown`); look the operation up by its id", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "503": { + "description": "The routes are not enabled; graph_unavailable for a known graph that is not served, or service_unavailable when server operation admission is closed; honor Retry-After, and reconcile any earlier write before retrying", + "headers": { + "Omnigraph-Http-Api": { + "schema": { + "type": "string", + "enum": [ + "0.14" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + }, + "Retry-After": { + "schema": { + "type": "string" + }, + "description": "Minimum delay before a caller-bounded retry of this refused request; not a scheduled server retry." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + } + }, + "security": [ + { + "bearer_token": [] + } + ] + } + }, "/graphs/{graph_id}/mutate": { "post": { "tags": [ @@ -15131,6 +16152,36 @@ } } }, + "MaintenanceLookup": { + "type": "string", + "enum": [ + "found", + "unknown" + ] + }, + "MaintenanceOperationOutput": { + "type": "object", + "required": [ + "lookup", + "in_progress" + ], + "properties": { + "in_progress": { + "type": "boolean", + "description": "Whether this server is running it now." + }, + "lookup": { + "$ref": "#/components/schemas/MaintenanceLookup" + }, + "operation": { + "type": [ + "object", + "null" + ], + "description": "The operation's record; absent for an `unknown` lookup." + } + } + }, "MergeBaseOutput": { "type": "object", "description": "The commit a merge of `source` into `target` would classify against, and\nthe two heads it was computed from.", @@ -15823,6 +16874,86 @@ } } }, + "StartOptimizeRequest": { + "type": "object", + "description": "`POST /graphs/{graph_id}/maintenance/optimize`.", + "required": [ + "maintenance_id" + ], + "properties": { + "maintenance_id": { + "type": "string", + "description": "A canonical ULID the caller chose; a repeated submission with the same\nrequest answers the existing record." + }, + "max_blob_bytes": { + "type": [ + "integer", + "null" + ], + "format": "int64", + "minimum": 0 + }, + "max_gate_hold_ms": { + "type": [ + "integer", + "null" + ], + "format": "int64", + "minimum": 0 + }, + "max_index_rows": { + "type": [ + "integer", + "null" + ], + "format": "int64", + "minimum": 0 + }, + "max_source_bytes": { + "type": [ + "integer", + "null" + ], + "format": "int64", + "minimum": 0 + }, + "max_source_fragments": { + "type": [ + "integer", + "null" + ], + "format": "int64", + "minimum": 0 + }, + "max_source_rows": { + "type": [ + "integer", + "null" + ], + "format": "int64", + "minimum": 0 + }, + "run_source_bytes": { + "type": [ + "integer", + "null" + ], + "format": "int64", + "minimum": 0 + }, + "tables": { + "type": [ + "array", + "null" + ], + "items": { + "type": "string" + }, + "description": "Only these table keys (`node:`, `edge:`), in this order." + } + }, + "additionalProperties": false + }, "SuggestionOutput": { "type": "object", "description": "Non-overlapping edits against the original request source.", @@ -15921,6 +17052,10 @@ "name": "health", "description": "Process health and readiness probes" }, + { + "name": "maintenance", + "description": "Served maintenance operations and their records" + }, { "name": "management", "description": "Server graph inventory and OAuth resource metadata" From 2936dffdc25e53a146eeb1416307c98bce6e1b24 Mon Sep 17 00:00:00 2001 From: Ragnor Comerford Date: Thu, 29 Oct 2026 10:17:52 -0700 Subject: [PATCH 11/13] feat(server): name a different request under a maintenance id with its own code The same maintenance id submitted with another request is refused with a [maintenance_request_mismatch] prefix, so a client tells it apart from the other 409 refusals without reading prose. --- crates/omnigraph-server/src/maintenance.rs | 4 ++-- crates/omnigraph-server/tests/maintenance_routes.rs | 7 +++++++ docs/user/operations/served-maintenance.md | 2 +- 3 files changed, 10 insertions(+), 3 deletions(-) diff --git a/crates/omnigraph-server/src/maintenance.rs b/crates/omnigraph-server/src/maintenance.rs index b47b0aea4..896755186 100644 --- a/crates/omnigraph-server/src/maintenance.rs +++ b/crates/omnigraph-server/src/maintenance.rs @@ -546,7 +546,7 @@ pub(crate) async fn start_optimize( Ok(output(&state, Some(existing.clone()))) } else { Err(ApiError::conflict(format!( - "maintenance id {} names a different request", + "[maintenance_request_mismatch] maintenance id {} names a different request", input.maintenance_id ))) }; @@ -562,7 +562,7 @@ pub(crate) async fn start_optimize( Ok(Admitted::Existing(existing)) => return Ok(output(&state, Some(*existing))), Err(MaintenanceRefusal::DifferentRequest) => { return Err(ApiError::conflict(format!( - "maintenance id {} names a different request", + "[maintenance_request_mismatch] maintenance id {} names a different request", input.maintenance_id ))); } diff --git a/crates/omnigraph-server/tests/maintenance_routes.rs b/crates/omnigraph-server/tests/maintenance_routes.rs index ddfee47c1..7bd381446 100644 --- a/crates/omnigraph-server/tests/maintenance_routes.rs +++ b/crates/omnigraph-server/tests/maintenance_routes.rs @@ -337,6 +337,13 @@ async fn served_optimize_keeps_a_record_a_caller_reads_by_id() { ) .await; assert_eq!(status, 409, "{refused}"); + assert!( + refused["error"] + .as_str() + .unwrap() + .starts_with("[maintenance_request_mismatch]"), + "{refused}" + ); let (status, unknown) = client.lookup("01JC0000000000000000000009").await; assert_eq!(status, 200, "{unknown}"); diff --git a/docs/user/operations/served-maintenance.md b/docs/user/operations/served-maintenance.md index 2190fa846..cf4065983 100644 --- a/docs/user/operations/served-maintenance.md +++ b/docs/user/operations/served-maintenance.md @@ -50,7 +50,7 @@ reason to resubmit the same work under the same id. | Refusal | Status | |---|---| -| The same id with a different request; an outstanding deployment; four unresolved publications; a record from another ledger at this root | 409 `conflict` | +| The same id with a different request (message prefix `[maintenance_request_mismatch]`); an outstanding deployment; four unresolved publications; a record from another ledger at this root | 409 `conflict` | | Another maintenance operation is active in the cluster (its id is in the message) | 429 `too_many_requests` | | A gate hold above the server's ceiling | 413 `payload_too_large` | | The routes are off, or the server is draining | 503 `service_unavailable` | From 3646a79a187394c90240da17d4d20ac136e651f4 Mon Sep 17 00:00:00 2001 From: Ragnor Comerford Date: Thu, 29 Oct 2026 10:39:49 -0700 Subject: [PATCH 12/13] chore(server): classify the served optimize request's storage terms The served optimize request names tables and source and index rows: the physical rewrite it bounds. The vocabulary inventory records those four OpenAPI occurrences as accurate physical terminology. --- .../omnigraph-vocabulary-guard/graph-vocabulary-inventory.tsv | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/tools/omnigraph-vocabulary-guard/graph-vocabulary-inventory.tsv b/tools/omnigraph-vocabulary-guard/graph-vocabulary-inventory.tsv index a5ed9ca35..9771c50d3 100644 --- a/tools/omnigraph-vocabulary-guard/graph-vocabulary-inventory.tsv +++ b/tools/omnigraph-vocabulary-guard/graph-vocabulary-inventory.tsv @@ -30,6 +30,10 @@ schema_version occurrence_id surface source_path boundary site_kind current_term 2 openapi:%2Fcomponents%2Fschemas%2FSchemaOutput%2Fproperties%2Fcatalogue_source%2Fdescription:prose:0696683a88c0dca6b24c2b09b98e92df084fedda03798653c52baeef5a3439b6:rows:1 openapi openapi.json /components/schemas/SchemaOutput/properties/catalogue_source/description prose rows retain result_projection retain query-result vocabulary none omnigraph-vocabulary-guard::openapi_inventory_matches Rows names the tabular query-result projection a read of an engine catalogue type returns; the catalogue is described as what a query answers. 2 openapi:%2Fcomponents%2Fschemas%2FSchemaOutput%2Fproperties%2Fsystem_columns%2FoneOf%2F1%2Fdescription:prose:e8bc9446b94b2d885da68f180b01d0c0e34877ad8e34a21854edcd1c74a8d072:column:1 openapi openapi.json /components/schemas/SchemaOutput/properties/system_columns/oneOf/1/description prose column retain physical_storage retain accurate physical/storage terminology none omnigraph-vocabulary-guard::openapi_inventory_matches The system column spellings field names physical storage columns; the terminology is the mechanism being described. 2 openapi:%2Fcomponents%2Fschemas%2FSchemaOutput%2Fproperties%2Fsystem_columns:property_name:162bd2a9950898bf827635f9fd45a45919fab6a503a3d9e807a94fac973d7a1f:columns:1 openapi openapi.json /components/schemas/SchemaOutput/properties/system_columns property_name columns retain physical_storage retain accurate physical/storage terminology none omnigraph-vocabulary-guard::openapi_inventory_matches The system column spellings field names physical storage columns; the terminology is the mechanism being described. +2 openapi:%2Fcomponents%2Fschemas%2FStartOptimizeRequest%2Fproperties%2Fmax_index_rows:property_name:c87da789c57e41b2788352f151d88394f5f66fb0e0e9cc520dc74c4701a9e30d:rows:1 openapi openapi.json /components/schemas/StartOptimizeRequest/properties/max_index_rows property_name rows retain physical_storage retain accurate physical/storage terminology none omnigraph-vocabulary-guard::openapi_inventory_matches A served optimize run bounds a physical rewrite: the tables it compacts and the source and index rows it may read are the mechanism the request limits. +2 openapi:%2Fcomponents%2Fschemas%2FStartOptimizeRequest%2Fproperties%2Fmax_source_rows:property_name:2a23da0a67a774ca9cbb0faac3c06a6184148b9ad2c6a3fe51052710894ed00d:rows:1 openapi openapi.json /components/schemas/StartOptimizeRequest/properties/max_source_rows property_name rows retain physical_storage retain accurate physical/storage terminology none omnigraph-vocabulary-guard::openapi_inventory_matches A served optimize run bounds a physical rewrite: the tables it compacts and the source and index rows it may read are the mechanism the request limits. +2 openapi:%2Fcomponents%2Fschemas%2FStartOptimizeRequest%2Fproperties%2Ftables%2Fdescription:prose:4a648bb8497e9d61439c2fc72696ac79493d5abac3961f64430196a84abe828c:table:1 openapi openapi.json /components/schemas/StartOptimizeRequest/properties/tables/description prose table retain physical_storage retain accurate physical/storage terminology none omnigraph-vocabulary-guard::openapi_inventory_matches A served optimize run bounds a physical rewrite: the tables it compacts and the source and index rows it may read are the mechanism the request limits. +2 openapi:%2Fcomponents%2Fschemas%2FStartOptimizeRequest%2Fproperties%2Ftables:property_name:d98a034f5cb8a0e3f07d5d8b5dc3c43723ea8d8b95199e18a82c52e072eca4dd:tables:1 openapi openapi.json /components/schemas/StartOptimizeRequest/properties/tables property_name tables retain physical_storage retain accurate physical/storage terminology none omnigraph-vocabulary-guard::openapi_inventory_matches A served optimize run bounds a physical rewrite: the tables it compacts and the source and index rows it may read are the mechanism the request limits. 2 openapi:%2Fcomponents%2Fschemas%2FSystemColumnsOutput%2Fdescription:prose:33634b754f85b52bddac2927f8409f5c1575c3cd7537a7f150f1c25b26fc3401:column:1 openapi openapi.json /components/schemas/SystemColumnsOutput/description prose column retain physical_storage retain accurate physical/storage terminology none omnigraph-vocabulary-guard::openapi_inventory_matches The system column spellings field names physical storage columns; the terminology is the mechanism being described. 2 openapi:%2Fcomponents%2Fschemas%2FSystemColumnsOutput%2Fdescription:prose:33634b754f85b52bddac2927f8409f5c1575c3cd7537a7f150f1c25b26fc3401:columns:1 openapi openapi.json /components/schemas/SystemColumnsOutput/description prose columns retain physical_storage retain accurate physical/storage terminology none omnigraph-vocabulary-guard::openapi_inventory_matches The system column spellings field names physical storage columns; the terminology is the mechanism being described. 2 openapi:%2Fcomponents%2Fschemas%2FSystemColumnsOutput:component_name:dedc4b8727bbc36f7bac392cd29282ded5d6303a41393caf573cf982e54b6eb0:Columns:1 openapi openapi.json /components/schemas/SystemColumnsOutput component_name Columns retain physical_storage retain accurate physical/storage terminology none omnigraph-vocabulary-guard::openapi_inventory_matches The system column spellings field names physical storage columns; the terminology is the mechanism being described. From 889a694e15a316179f9cf755ee6fb81a3300d432 Mon Sep 17 00:00:00 2001 From: Ragnor Comerford Date: Thu, 29 Oct 2026 11:03:36 -0700 Subject: [PATCH 13/13] fix(server): keep a deployment's new graphs in its serving snapshot The maintenance cancel hook bound its graph list to `affected`, shadowing the preview's affected graphs. The serving snapshot read after the apply then left out every graph the deployment created, so a live apply that added a graph completed without activating it. The hook now reads the preview's list directly; a graph the deployment creates has no run to cancel. --- crates/omnigraph-server/src/deployment.rs | 4 ---- 1 file changed, 4 deletions(-) diff --git a/crates/omnigraph-server/src/deployment.rs b/crates/omnigraph-server/src/deployment.rs index c6065dc84..92a2883aa 100644 --- a/crates/omnigraph-server/src/deployment.rs +++ b/crates/omnigraph-server/src/deployment.rs @@ -566,10 +566,6 @@ async fn execute( // A maintenance run on an affected graph stops at its next safe boundary; // the drain then waits for at most one table's rewrite, the publication, // the configuration clear and an issued artifact write. - let affected = keys - .iter() - .map(|key| key.graph_id.to_string()) - .collect::>(); state.maintenance.cancel( None, Some(&affected),