diff --git a/changelog.d/ensure-indices-refreshes-coverage.changed.md b/changelog.d/ensure-indices-refreshes-coverage.changed.md index 0401eca82..24027f1d3 100644 --- a/changelog.d/ensure-indices-refreshes-coverage.changed.md +++ b/changelog.d/ensure-indices-refreshes-coverage.changed.md @@ -1,3 +1,3 @@ -- `ensure_indices` and `ensure_indices_on` also rebuild lagging BTREE/vector - indexes, report lagging full-text coverage as pending, and under a policy - need an actor with `Change`: use `ensure_indices_on_as`. +- The engine's index build (`ensure_indices_as`, `ensure_indices_on_as`) also + rebuilds lagging BTREE and vector indexes and reports lagging full-text + coverage as pending. diff --git a/changelog.d/full-text-rebuild-change-grant.deprecated.md b/changelog.d/full-text-rebuild-change-grant.deprecated.md new file mode 100644 index 000000000..42839a66c --- /dev/null +++ b/changelog.d/full-text-rebuild-change-grant.deprecated.md @@ -0,0 +1,5 @@ +- The full-text rebuild accepts `change` on its branch for one more release; + after that it needs `maintain`. Grant `maintain` to the accounts that rebuild + full-text indexes. See [rebuild full-text indexes][full-text-rebuild-change-grant]. + +[full-text-rebuild-change-grant]: ../docs/user/operations/maintenance.md#rebuild-full-text-indexes diff --git a/changelog.d/maintenance-entry-points-take-an-actor.breaking.md b/changelog.d/maintenance-entry-points-take-an-actor.breaking.md new file mode 100644 index 000000000..1953dc6cc --- /dev/null +++ b/changelog.d/maintenance-entry-points-take-an-actor.breaking.md @@ -0,0 +1,10 @@ +- An embedded host with a policy checker refuses maintenance until its policy + grants `maintain` or `reclaim`. Engine maintenance methods take an actor: + `optimize_as`, `cleanup_as`, `ensure_indices_as`, `ensure_indices_on_as` and + `rebuild_full_text_indices_on_as`. `repair` is the read-only + `repair_report_as`, without `RepairOptions`, `healed`, `forced` or + `graph_manifest_version`. See [policy actions][maintenance-actions-upgrade] + and [repair][maintenance-entry-points-repair]. + +[maintenance-actions-upgrade]: ../docs/user/operations/policy.md#actions +[maintenance-entry-points-repair]: ../docs/user/operations/maintenance.md#repair diff --git a/changelog.d/maintenance-policy-actions.added.md b/changelog.d/maintenance-policy-actions.added.md new file mode 100644 index 000000000..5f4ed1ff3 --- /dev/null +++ b/changelog.d/maintenance-policy-actions.added.md @@ -0,0 +1,5 @@ +- Cedar policy has two maintenance actions: `maintain` for optimize, index + builds, the full-text rebuild and repair, and `reclaim` for cleanup. `change` + implies neither. See [policy actions][maintenance-policy-actions]. + +[maintenance-policy-actions]: ../docs/user/operations/policy.md#actions diff --git a/changelog.d/optimize-artifact-size-limit.added.md b/changelog.d/optimize-artifact-size-limit.added.md new file mode 100644 index 000000000..544c66051 --- /dev/null +++ b/changelog.d/optimize-artifact-size-limit.added.md @@ -0,0 +1,3 @@ +- Embedded hosts can bound the adjacency artifact optimize writes with + `OptimizeLimits::artifact_bytes`; a larger one is skipped, before or during + its edge scan, and nothing is written. diff --git a/changelog.d/optimize-before-publish-hook.added.md b/changelog.d/optimize-before-publish-hook.added.md new file mode 100644 index 000000000..10a42bdcc --- /dev/null +++ b/changelog.d/optimize-before-publish-hook.added.md @@ -0,0 +1,3 @@ +- Embedded hosts can pass `OptimizeRequest::with_before_publish`: it runs once + before an optimize run publishes, with the commit's nonce and input head, and + its failure publishes nothing. diff --git a/changelog.d/optimize-cancellation.added.md b/changelog.d/optimize-cancellation.added.md new file mode 100644 index 000000000..6fac2560f --- /dev/null +++ b/changelog.d/optimize-cancellation.added.md @@ -0,0 +1,2 @@ +- Embedded hosts can cancel an optimize run with `OptimizeRequest::with_cancellation`: + it stops before the next table or the publication and publishes nothing. diff --git a/changelog.d/optimize-gate-hold-bound.added.md b/changelog.d/optimize-gate-hold-bound.added.md new file mode 100644 index 000000000..edd40f7c8 --- /dev/null +++ b/changelog.d/optimize-gate-hold-bound.added.md @@ -0,0 +1,3 @@ +- `optimize --max-gate-hold` bounds how long a run holds main's writer gates and names the table the next run resumes at; `--tables` runs only the listed tables. See [bounding a run][optimize-bound]. + +[optimize-bound]: ../docs/user/operations/maintenance.md#bounding-a-run diff --git a/changelog.d/optimize-report-outcomes.changed.md b/changelog.d/optimize-report-outcomes.changed.md new file mode 100644 index 000000000..160cd393e --- /dev/null +++ b/changelog.d/optimize-report-outcomes.changed.md @@ -0,0 +1,3 @@ +- `optimize` reports one outcome per table, both graph commits and the gate hold; a failed configuration clear or adjacency artifact no longer fails a published run. See [bounding a run][optimize-report]. + +[optimize-report]: ../docs/user/operations/maintenance.md#bounding-a-run diff --git a/changelog.d/optimize-resource-ceilings.added.md b/changelog.d/optimize-resource-ceilings.added.md new file mode 100644 index 000000000..383348d74 --- /dev/null +++ b/changelog.d/optimize-resource-ceilings.added.md @@ -0,0 +1,3 @@ +- Embedded hosts can bound optimize per table (`TableCeilings`) and per run + (`run_source_bytes`); a table over a ceiling is deferred before its first + write, with nothing written for it. diff --git a/changelog.d/optimize-table-failure-isolation.changed.md b/changelog.d/optimize-table-failure-isolation.changed.md new file mode 100644 index 000000000..5a6a4ca36 --- /dev/null +++ b/changelog.d/optimize-table-failure-isolation.changed.md @@ -0,0 +1,2 @@ +- A table whose own optimize work fails is reported `failed` with its error, and + the other tables still publish; before, one table's failure failed the run. diff --git a/changelog.d/optimize-takes-a-request.breaking.md b/changelog.d/optimize-takes-a-request.breaking.md new file mode 100644 index 000000000..3b28f3cf7 --- /dev/null +++ b/changelog.d/optimize-takes-a-request.breaking.md @@ -0,0 +1,8 @@ +- Embedded hosts: `optimize_as` takes an `OptimizeRequest` (`OptimizeRequest::default()` + is a full run) and returns an `OptimizeReport` whose `tables` holds the per-table + stats; `committed` and `skipped` are methods. `optimize --json` reports the + `__manifest` row as `configuration_clear` and drops the always-null + `published_dataset_version` and `lance_head_version`. See + [bounding a run][optimize-request-upgrade]. + +[optimize-request-upgrade]: ../docs/user/operations/maintenance.md#bounding-a-run diff --git a/crates/omnigraph-api-types/src/lib.rs b/crates/omnigraph-api-types/src/lib.rs index 34d2c7fc0..eeed8a896 100644 --- a/crates/omnigraph-api-types/src/lib.rs +++ b/crates/omnigraph-api-types/src/lib.rs @@ -2058,6 +2058,9 @@ fn access_slot(action: omnigraph_policy::PolicyAction) -> Option { P::InvokeQuery => Some(AccessSlot::GraphWide), // Server- or operator-scoped: no graph's access report names them. P::Admin | P::GraphList | P::ConfigManage => None, + // Maintenance runs only through the direct CLI and embedded callers; + // no served operation performs it, so the report does not name it. + P::Maintain | P::Reclaim => None, } } diff --git a/crates/omnigraph-bench/src/dataset_worker.rs b/crates/omnigraph-bench/src/dataset_worker.rs index f66ae24af..3999d6e98 100644 --- a/crates/omnigraph-bench/src/dataset_worker.rs +++ b/crates/omnigraph-bench/src/dataset_worker.rs @@ -803,7 +803,10 @@ pub(crate) async fn build_dataset( Engine::V2, )?; if plan.needs_indices { - session.ensure_indices().await.map_err(|e| e.to_string())?; + session + .ensure_indices_as(None) + .await + .map_err(|e| e.to_string())?; } (session, case, Some(exact)) } diff --git a/crates/omnigraph-bench/src/legacy/branch_merge_oracle.rs b/crates/omnigraph-bench/src/legacy/branch_merge_oracle.rs index b5151f265..e33438f05 100644 --- a/crates/omnigraph-bench/src/legacy/branch_merge_oracle.rs +++ b/crates/omnigraph-bench/src/legacy/branch_merge_oracle.rs @@ -28,7 +28,7 @@ pub async fn initialize_local_fixture( let optimized_user_tables = match plan.compaction_recency { CompactionRecency::Optimized => { let outcomes = db - .optimize() + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) .await .map_err(|error| fixture_error(format!("optimize fixture main branch: {error}")))?; let intended_keys = (0..plan.node_tables()) @@ -36,6 +36,7 @@ pub async fn initialize_local_fixture( .chain((0..plan.edge_tables()).map(edge_table_key)); for key in intended_keys { let outcome = outcomes + .tables .iter() .find(|outcome| outcome.type_key == key) .ok_or_else(|| { @@ -43,17 +44,17 @@ pub async fn initialize_local_fixture( "optimized fixture returned no outcome for intended user table {key}" )) })?; - if outcome.skipped.is_some() - || !outcome.committed + if outcome.skipped().is_some() + || !outcome.committed() || outcome.fragments_removed == 0 || outcome.fragments_added == 0 { return Err(fixture_error(format!( "optimized fixture did not productively compact {key}: committed={}, fragments_removed={}, fragments_added={}, skipped={:?}", - outcome.committed, + outcome.committed(), outcome.fragments_removed, outcome.fragments_added, - outcome.skipped + outcome.skipped() ))); } } diff --git a/crates/omnigraph-cli/src/commands/maintenance/args.rs b/crates/omnigraph-cli/src/commands/maintenance/args.rs index fb4f3aebf..3cc6c7e51 100644 --- a/crates/omnigraph-cli/src/commands/maintenance/args.rs +++ b/crates/omnigraph-cli/src/commands/maintenance/args.rs @@ -60,6 +60,15 @@ pub(crate) struct UpgradeArgs { #[derive(Debug, Args)] pub(crate) struct OptimizeArgs { + /// Only these tables, in this order: `node:` or `edge:`, + /// comma-separated or repeated. Default: every table on main. + #[arg(long, value_delimiter = ',')] + pub(crate) tables: Vec, + /// Stop starting tables once the run has held main's writer gates this + /// long, and report the rest deferred. The first table always runs. + /// Accepts Go-style durations: `30s`, `5m`. + #[arg(long)] + pub(crate) max_gate_hold: Option, #[arg(long)] pub(crate) json: bool, } diff --git a/crates/omnigraph-cli/src/commands/maintenance/mod.rs b/crates/omnigraph-cli/src/commands/maintenance/mod.rs index 481dd375a..6ea603145 100644 --- a/crates/omnigraph-cli/src/commands/maintenance/mod.rs +++ b/crates/omnigraph-cli/src/commands/maintenance/mod.rs @@ -3,7 +3,7 @@ use std::io::Write; use color_eyre::eyre::{Result, bail}; -use omnigraph::db::{CleanupPolicyOptions, Omnigraph, RepairAction, RepairOptions}; +use omnigraph::db::{CleanupPolicyOptions, Omnigraph, RepairAction}; use crate::app::Completion; use crate::helpers::{confirm_destructive, parse_duration_arg, resolve_actor}; @@ -42,17 +42,31 @@ pub(crate) async fn run_optimize( stdout: &mut dyn Write, stderr: &mut dyn Write, ) -> Result { - let OptimizeArgs { json } = args; + let OptimizeArgs { + tables, + max_gate_hold, + json, + } = args; + let mut request = omnigraph::db::OptimizeRequest::default(); + if !tables.is_empty() { + request = request.with_tables(tables); + } + if let Some(hold) = max_gate_hold { + request = request.with_limits( + omnigraph::db::OptimizeLimits::default().with_max_gate_hold(parse_duration_arg(&hold)?), + ); + } let uri = crate::scope::direct_graph_uri(scope.target, scope.graph).await?; let binding = crate::scope::capture_storage_binding(scope.target)?.for_uri(&uri)?; + let actor = resolve_actor(scope.operator, scope.actor, stderr)?; crate::admission::ensure_graph_with_binding(&binding, stderr).await?; let db = Omnigraph::open_with_storage( binding.uri(), omnigraph::storage::storage_for_binding(&binding)?, ) .await?; - let stats = db.optimize().await?; - write_optimize(&uri, &stats, json, stdout)?; + let report = db.optimize_as(actor.as_deref(), request).await?; + write_optimize(&uri, &report, json, stdout)?; Ok(Completion::success()) } @@ -141,13 +155,14 @@ pub(crate) async fn run_repair( let RepairArgs { json } = args; let uri = crate::scope::direct_graph_uri(scope.target, scope.graph).await?; let binding = crate::scope::capture_storage_binding(scope.target)?.for_uri(&uri)?; + let actor = resolve_actor(scope.operator, scope.actor, stderr)?; crate::admission::ensure_graph_with_binding(&binding, stderr).await?; let db = Omnigraph::open_with_storage( binding.uri(), omnigraph::storage::storage_for_binding(&binding)?, ) .await?; - let stats = db.repair(RepairOptions::default()).await?; + let stats = db.repair_report_as(actor.as_deref()).await?; let refused_count = stats .datasets .iter() @@ -207,13 +222,14 @@ pub(crate) async fn run_cleanup( keep_versions: keep, older_than: older_than_dur, }; + let actor = resolve_actor(scope.operator, scope.actor, stderr)?; crate::admission::ensure_graph_with_binding(&binding, stderr).await?; let db = Omnigraph::open_with_storage( binding.uri(), omnigraph::storage::storage_for_binding(&binding)?, ) .await?; - let stats = db.cleanup(options.clone()).await?; + let stats = db.cleanup_as(options.clone(), actor.as_deref()).await?; write_cleanup(&uri, &options, &policy_desc, &stats, json, stdout)?; Ok(Completion::success()) } diff --git a/crates/omnigraph-cli/src/output.rs b/crates/omnigraph-cli/src/output.rs index de5a97c3c..0dc107404 100644 --- a/crates/omnigraph-cli/src/output.rs +++ b/crates/omnigraph-cli/src/output.rs @@ -17,21 +17,52 @@ pub(crate) fn graph_type_subject(type_key: &str) -> String { pub(crate) fn write_optimize( uri: &str, - stats: &[omnigraph::db::DatasetOptimizeStats], + report: &omnigraph::db::OptimizeReport, json: bool, stdout: &mut dyn Write, ) -> Result<()> { + use omnigraph::db::{ArtifactRefresh, ConfigurationClear, TableOptimizeOutcome}; + let stats = &report.tables; + let reason = |outcome: &TableOptimizeOutcome| match outcome { + TableOptimizeOutcome::Deferred(reason) => Some(reason.as_str()), + TableOptimizeOutcome::Skipped(reason) => Some(reason.as_str()), + _ => None, + }; + let clear_error = match &report.configuration_clear { + ConfigurationClear::Failed(error) | ConfigurationClear::Unknown(error) => Some(error), + _ => None, + }; + let (artifact_reason, artifact_error) = match &report.artifact { + ArtifactRefresh::Skipped(reason) => (Some(reason.as_str()), None), + ArtifactRefresh::Failed(error) => (None, Some(error)), + _ => (None, None), + }; if json { let value = serde_json::json!({ "uri": uri, + "input_commit": report.input_commit, + "graph_commit_id": report.graph_commit_id, + "resume_at": report.resume_at, + "gate_hold_ms": report.gate_hold.as_millis() as u64, + "configuration_clear": { + "outcome": report.configuration_clear.as_str(), + "error": clear_error, + }, + "artifact": { + "outcome": report.artifact.as_str(), + "reason": artifact_reason, + "error": artifact_error, + }, "datasets": stats.iter().map(|s| serde_json::json!({ "type_key": s.type_key, + "outcome": s.outcome.as_str(), + "reason": reason(&s.outcome), "fragments_removed": s.fragments_removed, "fragments_added": s.fragments_added, - "committed": s.committed, - "skipped": s.skipped.map(|r| r.as_str()), - "published_dataset_version": s.published_dataset_version, - "lance_head_version": s.lance_head_version, + "committed": s.committed(), + "skipped": s.skipped().map(|r| r.as_str()), + "duration_ms": s.duration.as_millis() as u64, + "error": s.failure.as_ref().map(|error| error.to_string()), "pending_indexes": s.pending_indexes.iter().map(|p| serde_json::json!({ "property": p.property, "reason": p.reason, @@ -43,16 +74,29 @@ pub(crate) fn write_optimize( writeln!(stdout, "optimize {} — {} datasets", uri, stats.len())?; for s in stats { let subject = graph_type_subject(&s.type_key); - if let Some(reason) = s.skipped { - writeln!(stdout, " {subject:<40} skipped ({reason})")?; - } else if s.committed { - writeln!( - stdout, - " {:<40} frags {} → {} ✓", - subject, s.fragments_removed, s.fragments_added - )?; - } else { - writeln!(stdout, " {subject:<40} no-op")?; + match s.outcome { + TableOptimizeOutcome::Skipped(reason) => { + writeln!(stdout, " {subject:<40} skipped ({reason})")?; + } + TableOptimizeOutcome::Deferred(reason) => { + writeln!(stdout, " {subject:<40} deferred ({reason})")?; + } + TableOptimizeOutcome::Optimized => { + writeln!( + stdout, + " {:<40} frags {} → {} ✓", + subject, s.fragments_removed, s.fragments_added + )?; + } + TableOptimizeOutcome::Failed => { + let error = s.failure.as_ref().map(|error| error.to_string()); + let first = error + .as_deref() + .and_then(|error| error.lines().next()) + .unwrap_or("unknown error"); + writeln!(stdout, " {subject:<40} failed ({first})")?; + } + _ => writeln!(stdout, " {subject:<40} no-op")?, } for p in &s.pending_indexes { writeln!( @@ -62,6 +106,28 @@ pub(crate) fn write_optimize( )?; } } + if let Some(commit) = &report.graph_commit_id { + writeln!(stdout, "graph commit: {commit}")?; + } + if let Some(key) = &report.resume_at { + writeln!(stdout, "resume at: {}", graph_type_subject(key))?; + } + match clear_error { + Some(error) => writeln!( + stdout, + "configuration clear: {} ({error})", + report.configuration_clear.as_str() + )?, + None if report.configuration_clear == ConfigurationClear::Cleared => { + writeln!(stdout, "configuration clear: cleared")? + } + None => {} + } + match (&report.artifact, artifact_error) { + (_, Some(error)) => writeln!(stdout, "adjacency artifact: failed ({error})")?, + (ArtifactRefresh::Refreshed, _) => writeln!(stdout, "adjacency artifact: refreshed")?, + _ => {} + } } Ok(()) } @@ -199,7 +265,6 @@ pub(crate) fn write_repair( if json { let value = serde_json::json!({ "uri": uri, - "graph_manifest_version": stats.graph_manifest_version, "datasets": stats.datasets.iter().map(|s| serde_json::json!({ "type_key": s.type_key, "published_dataset_version": s.published_dataset_version, @@ -1933,7 +1998,8 @@ mod tests { #[tokio::test] async fn maintenance_optimize_and_repair_reports_preserve_all_result_fields() { use omnigraph::db::{ - Omnigraph, PendingIndex, RepairAction, RepairClassification, RepairOptions, SkipReason, + ArtifactRefresh, ArtifactSkip, ConfigurationClear, DeferReason, Omnigraph, + PendingIndex, RepairAction, RepairClassification, SkipReason, TableOptimizeOutcome, }; // These engine result types are non_exhaustive. Obtain one valid value // from a tiny graph, then set its public presentation fields; no engine @@ -1947,14 +2013,16 @@ mod tests { let db = Omnigraph::init(&uri, "node Person {\n slug: String @key\n}\n") .await .unwrap(); - let mut compacted = db.optimize().await.unwrap().pop().unwrap(); + let mut report = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); + let mut compacted = report.tables.pop().unwrap(); compacted.type_key = "node:Person".into(); compacted.fragments_removed = 3; compacted.fragments_added = 1; - compacted.committed = true; - compacted.skipped = None; - compacted.published_dataset_version = None; - compacted.lance_head_version = None; + compacted.outcome = TableOptimizeOutcome::Optimized; + compacted.duration = std::time::Duration::from_millis(12); compacted.pending_indexes = vec![PendingIndex { type_key: "node:Person".into(), property: "embedding".into(), @@ -1964,42 +2032,89 @@ mod tests { skipped.type_key = "edge:Knows".into(); skipped.fragments_removed = 0; skipped.fragments_added = 0; - skipped.committed = false; - skipped.skipped = Some(SkipReason::DriftNeedsRepair); - skipped.published_dataset_version = Some(4); - skipped.lance_head_version = Some(5); + skipped.outcome = TableOptimizeOutcome::Skipped(SkipReason::DriftNeedsRepair); + skipped.duration = std::time::Duration::from_millis(3); skipped.pending_indexes.clear(); + let mut deferred = skipped.clone(); + deferred.type_key = "node:Zebra".into(); + deferred.outcome = TableOptimizeOutcome::Deferred(DeferReason::GateHoldLimit); + deferred.duration = std::time::Duration::ZERO; let mut noop = skipped.clone(); - noop.type_key = "__manifest".into(); - noop.skipped = None; - noop.published_dataset_version = None; - noop.lance_head_version = None; - let optimize = [compacted, skipped, noop]; + noop.type_key = "node:Yak".into(); + noop.outcome = TableOptimizeOutcome::Unchanged; + let mut failed = skipped.clone(); + failed.type_key = "node:Xerus".into(); + failed.outcome = TableOptimizeOutcome::Failed; + failed.duration = std::time::Duration::from_millis(5); + failed.failure = Some(std::sync::Arc::new(omnigraph::error::OmniError::manifest( + "rewrite lost", + ))); + let failure = failed.failure.as_ref().unwrap().to_string(); + report.tables = vec![compacted, skipped, noop, failed, deferred]; + report.input_commit = Some("hb1.parent".into()); + report.graph_commit_id = Some("hb1.child".into()); + report.resume_at = Some("node:Zebra".into()); + report.gate_hold = std::time::Duration::from_millis(40); + report.configuration_clear = ConfigurationClear::Unknown("timed out".into()); + report.artifact = ArtifactRefresh::Skipped(ArtifactSkip::NoEdgeTypes); assert_maintenance_report( - concat!( - "optimize graph — 3 datasets\n", - " node type 'Person' frags 3 → 1 ✓\n", - " ↳ index pending on property 'embedding': no trainable vectors\n", - " edge type 'Knows' skipped (published dataset/Lance HEAD drift — run omnigraph repair)\n", - " dataset '__manifest' no-op\n", + &format!( + concat!( + "optimize graph — 5 datasets\n", + " node type 'Person' frags 3 → 1 ✓\n", + " ↳ index pending on property 'embedding': no trainable vectors\n", + " edge type 'Knows' skipped (published dataset/Lance HEAD drift — run omnigraph repair)\n", + " node type 'Yak' no-op\n", + " node type 'Xerus' failed ({failure})\n", + " node type 'Zebra' deferred (the run reached its writer-gate hold limit)\n", + "graph commit: hb1.child\n", + "resume at: node type 'Zebra'\n", + "configuration clear: unknown (timed out)\n", + ), + failure = failure ), serde_json::json!({ "uri":"graph", + "input_commit":"hb1.parent", + "graph_commit_id":"hb1.child", + "resume_at":"node:Zebra", + "gate_hold_ms":40, + "configuration_clear":{"outcome":"unknown","error":"timed out"}, + "artifact":{"outcome":"skipped","reason":"no_edge_types","error":null}, "datasets":[ - {"type_key":"node:Person","fragments_removed":3,"fragments_added":1,"committed":true,"skipped":null,"published_dataset_version":null,"lance_head_version":null,"pending_indexes":[{"property":"embedding","reason":"no trainable vectors"}]}, - {"type_key":"edge:Knows","fragments_removed":0,"fragments_added":0,"committed":false,"skipped":"drift_needs_repair","published_dataset_version":4,"lance_head_version":5,"pending_indexes":[]}, - {"type_key":"__manifest","fragments_removed":0,"fragments_added":0,"committed":false,"skipped":null,"published_dataset_version":null,"lance_head_version":null,"pending_indexes":[]}, + {"type_key":"node:Person","outcome":"optimized","reason":null,"fragments_removed":3,"fragments_added":1,"committed":true,"skipped":null,"duration_ms":12,"error":null,"pending_indexes":[{"property":"embedding","reason":"no trainable vectors"}]}, + {"type_key":"edge:Knows","outcome":"skipped","reason":"drift_needs_repair","fragments_removed":0,"fragments_added":0,"committed":false,"skipped":"drift_needs_repair","duration_ms":3,"error":null,"pending_indexes":[]}, + {"type_key":"node:Yak","outcome":"unchanged","reason":null,"fragments_removed":0,"fragments_added":0,"committed":false,"skipped":null,"duration_ms":3,"error":null,"pending_indexes":[]}, + {"type_key":"node:Xerus","outcome":"failed","reason":null,"fragments_removed":0,"fragments_added":0,"committed":false,"skipped":null,"duration_ms":5,"error":failure,"pending_indexes":[]}, + {"type_key":"node:Zebra","outcome":"deferred","reason":"gate_hold_limit","fragments_removed":0,"fragments_added":0,"committed":false,"skipped":null,"duration_ms":0,"error":null,"pending_indexes":[]}, ], }), - |json, writer| super::write_optimize("graph", &optimize, json, writer), + |json, writer| super::write_optimize("graph", &report, json, writer), ); + let mut empty = report.clone(); + empty.tables.clear(); + empty.input_commit = None; + empty.graph_commit_id = None; + empty.resume_at = None; + empty.gate_hold = std::time::Duration::ZERO; + empty.configuration_clear = ConfigurationClear::NothingToClear; + empty.artifact = ArtifactRefresh::Unchanged; assert_maintenance_report( "optimize graph — 0 datasets\n", - serde_json::json!({"uri":"graph","datasets":[]}), - |json, writer| super::write_optimize("graph", &[], json, writer), + serde_json::json!({ + "uri":"graph", + "input_commit":null, + "graph_commit_id":null, + "resume_at":null, + "gate_hold_ms":0, + "configuration_clear":{"outcome":"nothing_to_clear","error":null}, + "artifact":{"outcome":"unchanged","reason":null,"error":null}, + "datasets":[], + }), + |json, writer| super::write_optimize("graph", &empty, json, writer), ); - let mut repair = db.repair(RepairOptions::default()).await.unwrap(); + let mut repair = db.repair_report_as(None).await.unwrap(); let mut clean = repair.datasets.pop().unwrap(); clean.type_key = "node:Person".into(); clean.published_dataset_version = 4; @@ -2016,7 +2131,6 @@ mod tests { refused.operations = vec!["Append".into(), "Delete".into()]; refused.error = Some("foreign write".into()); repair.datasets = vec![clean, refused]; - repair.graph_manifest_version = Some(9); assert_maintenance_report( concat!( "repair graph — 2 datasets\n", @@ -2024,7 +2138,7 @@ mod tests { " edge type 'Knows' refused foreign_drift 4 → 6 [Append, Delete] (foreign write)\n", ), serde_json::json!({ - "uri":"graph", "graph_manifest_version":9, + "uri":"graph", "datasets":[ {"type_key":"node:Person","published_dataset_version":4,"lance_head_version":4,"classification":"no_drift","action":"no_op","operations":[],"error":null}, {"type_key":"edge:Knows","published_dataset_version":4,"lance_head_version":6,"classification":"foreign_drift","action":"refused","operations":["Append","Delete"],"error":"foreign write"}, @@ -2033,10 +2147,9 @@ mod tests { |json, writer| super::write_repair("graph", &repair, json, writer), ); repair.datasets.clear(); - repair.graph_manifest_version = None; assert_maintenance_report( "repair graph — 0 datasets\n", - serde_json::json!({"uri":"graph","graph_manifest_version":null,"datasets":[]}), + serde_json::json!({"uri":"graph","datasets":[]}), |json, writer| super::write_repair("graph", &repair, json, writer), ); } diff --git a/crates/omnigraph-cli/tests/integration/cli_data.rs b/crates/omnigraph-cli/tests/integration/cli_data.rs index 9bd14985f..70554828b 100644 --- a/crates/omnigraph-cli/tests/integration/cli_data.rs +++ b/crates/omnigraph-cli/tests/integration/cli_data.rs @@ -1384,7 +1384,8 @@ fn repair_json_reports_noop_on_clean_graph() { assert!(payload.get("confirm").is_none()); assert!(payload.get("force").is_none()); - assert_eq!(payload["graph_manifest_version"], Value::Null); + // The report publishes nothing, so it names no graph manifest version. + assert!(payload.get("graph_manifest_version").is_none()); assert!(payload.get("manifest_version").is_none()); assert!(payload.get("tables").is_none()); let datasets = payload["datasets"].as_array().unwrap(); @@ -1469,7 +1470,7 @@ fn repair_reports_foreign_drift_and_refuses_removed_flags() { .arg(&graph), ); let payload: Value = serde_json::from_slice(&output.stdout).unwrap(); - assert_eq!(payload["graph_manifest_version"], Value::Null); + assert!(payload.get("graph_manifest_version").is_none()); let person = payload["datasets"] .as_array() .unwrap() diff --git a/crates/omnigraph-dst/src/concurrent.rs b/crates/omnigraph-dst/src/concurrent.rs index a05a5274a..a7b83a167 100644 --- a/crates/omnigraph-dst/src/concurrent.rs +++ b/crates/omnigraph-dst/src/concurrent.rs @@ -1753,12 +1753,15 @@ fn maintenance_life( let mut occ_retries = 0usize; loop { let result = match kind { - 0 => Box::pin(db.ensure_indices()).await.map(|_| ()), - 1 => Box::pin(db.optimize()).await.map(|_| ()), - _ => Box::pin(db.cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - })) + 0 => Box::pin(db.ensure_indices_as(None)).await.map(|_| ()), + 1 => crate::harness::optimize_to_completion(&db).await, + _ => Box::pin(db.cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + )) .await .map(|_| ()), }; diff --git a/crates/omnigraph-dst/src/harness.rs b/crates/omnigraph-dst/src/harness.rs index f2742f0ca..bc3d73a7d 100644 --- a/crates/omnigraph-dst/src/harness.rs +++ b/crates/omnigraph-dst/src/harness.rs @@ -1578,6 +1578,17 @@ fn sample_op(rng: &mut SplitMix64, model: &Model, next_ver: &mut i64, hostile: b } } +/// One full optimize run, whose failed table is the run's failure to the +/// oracles: an op must finish every table it considers. +pub(crate) async fn optimize_to_completion(db: &Session) -> OmniResult<()> { + let report = Box::pin(db.optimize_as(None, omnigraph::db::OptimizeRequest::default())).await?; + match report.tables.into_iter().find_map(|stat| stat.failure) { + Some(failure) => Err(std::sync::Arc::try_unwrap(failure) + .unwrap_or_else(|shared| OmniError::manifest_internal(shared.to_string()))), + None => Ok(()), + } +} + async fn exec_op(db: &Session, branch: &str, op: &Op) -> OmniResult<()> { match op { Op::InsertV { name, age, ver } => mutate_on( @@ -1634,12 +1645,15 @@ async fn exec_op(db: &Session, branch: &str, op: &Op) -> OmniResult<()> { ) .await .map(|_| ()), - Op::Optimize => Box::pin(db.optimize()).await.map(|_| ()), + Op::Optimize => optimize_to_completion(db).await, Op::Cleanup => { - let stats = Box::pin(db.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - })) + let stats = Box::pin(db.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + )) .await?; let deferred: BTreeMap = stats .iter() @@ -1655,7 +1669,7 @@ async fn exec_op(db: &Session, branch: &str, op: &Op) -> OmniResult<()> { .insert(db.uri().to_string(), deferred); Ok(()) } - Op::EnsureIndices => Box::pin(db.ensure_indices()).await.map(|_| ()), + Op::EnsureIndices => Box::pin(db.ensure_indices_as(None)).await.map(|_| ()), Op::SchemaAddProperty { count } => Box::pin(db.apply_schema(&schema_with_extras(*count))) .await .map(|_| ()), @@ -6432,7 +6446,7 @@ impl UniverseScenario for Scenario { tokio::time::resume(); let lively = tokio::time::timeout( std::time::Duration::from_secs(120), - Box::pin(db.ensure_indices()), + Box::pin(db.ensure_indices_as(None)), ) .await; tokio::time::pause(); diff --git a/crates/omnigraph-dst/tests/scenarios.rs b/crates/omnigraph-dst/tests/scenarios.rs index 50e885aa1..c79a5fb1a 100644 --- a/crates/omnigraph-dst/tests/scenarios.rs +++ b/crates/omnigraph-dst/tests/scenarios.rs @@ -1045,7 +1045,7 @@ async fn dst_memory_graph_end_to_end() { db.load_jsonl(TEST_DATA, LoadMode::Overwrite) .await .expect("load test data"); - db.ensure_indices().await.expect("ensure indices"); + db.ensure_indices_as(None).await.expect("ensure indices"); let persons = count_rows(&db, "node:Person").await; assert!(persons > 0, "loaded graph must have Person rows"); @@ -3669,7 +3669,9 @@ fn reborn_branch_cache_poison_body() { .expect("op1"); db.branch_create("b0").await.expect("op2"); world_read(&db).await; // ARMING READ 1 (the universe's op-2 verify) - db.optimize().await.expect("op3"); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .expect("op3"); m!( db, "b0", @@ -3683,7 +3685,7 @@ fn reborn_branch_cache_poison_body() { Box::pin(db.branch_merge("b0", "main")) .await .expect("op6 merge"); - db.ensure_indices().await.expect("op7"); + db.ensure_indices_as(None).await.expect("op7"); db.branch_delete("b0").await.expect("op8"); m!( db, @@ -3717,7 +3719,7 @@ fn reborn_branch_cache_poison_body() { &[] ) .expect("op12"); - db.ensure_indices().await.expect("op13"); + db.ensure_indices_as(None).await.expect("op13"); m!( db, "main", @@ -3758,10 +3760,13 @@ fn reborn_branch_cache_poison_body() { &[("$age", 20)] ) .expect("op20"); - db.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .expect("op21"); m!( @@ -3780,10 +3785,13 @@ fn reborn_branch_cache_poison_body() { &[("$age", 14), ("$ver", 8)] ) .expect("op23"); - db.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .expect("op24"); db.branch_create("b0").await.expect("op25"); @@ -5198,7 +5206,7 @@ fn dst_classa_ablation_matrix_body() { db.branch_create("b0").await.expect("bc b0 life1"); } if o { - db.optimize().await.expect("optimize"); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()).await.expect("optimize"); } if r { upsert(&db, "b0", "w6", 45, 2).await; @@ -5211,20 +5219,20 @@ fn dst_classa_ablation_matrix_body() { upsert(&db, "main", "w1", 19, 3).await; upsert(&db, "main", "w7", 7, 6).await; if c { - db.cleanup(omnigraph::db::CleanupPolicyOptions { + db.cleanup_as(omnigraph::db::CleanupPolicyOptions { keep_versions: Some(1), older_than: None, - }) + }, None) .await .expect("cleanup 1"); } upsert(&db, "main", "w7", 17, 7).await; upsert(&db, "main", "w5", 14, 8).await; if c { - db.cleanup(omnigraph::db::CleanupPolicyOptions { + db.cleanup_as(omnigraph::db::CleanupPolicyOptions { keep_versions: Some(1), older_than: None, - }) + }, None) .await .expect("cleanup 2"); } @@ -5285,26 +5293,26 @@ fn dst_classa_ablation_matrix_body() { m!(db, "main", "set_age_v", &[("$name", "w5")], &[("$age", 70), ("$ver", 1)]).expect("op0"); m!(db, "main", "insert_person", &[("$name", "w6")], &[("$age", 7)]).expect("op1"); db.branch_create("b0").await.expect("op2"); - db.optimize().await.expect("op3"); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()).await.expect("op3"); m!(db, "b0", "set_age_v", &[("$name", "w6")], &[("$age", 45), ("$ver", 2)]).expect("op4"); m!(db, "b0", "remove_friendships_from", &[("$from", "w6")], &[]).expect("op5"); Box::pin(db.branch_merge("b0", "main")).await.expect("op6 merge"); - db.ensure_indices().await.expect("op7"); + db.ensure_indices_as(None).await.expect("op7"); db.branch_delete("b0").await.expect("op8"); m!(db, "main", "remove_friendships_from", &[("$from", "w2")], &[]).expect("op9"); m!(db, "main", "insert_person_v", &[("$name", "w1")], &[("$age", 19), ("$ver", 3)]).expect("op10"); m!(db, "main", "set_age_v", &[("$name", "w2")], &[("$age", 3), ("$ver", 4)]).expect("op11"); m!(db, "main", "add_friend", &[("$from", "Charlie"), ("$to", "w6")], &[]).expect("op12"); - db.ensure_indices().await.expect("op13"); + db.ensure_indices_as(None).await.expect("op13"); m!(db, "main", "insert_person", &[("$name", "w1")], &[("$age", 52)]).expect("op15"); m!(db, "main", "remove_friendships_from", &[("$from", "w3")], &[]).expect("op16"); m!(db, "main", "set_age_v", &[("$name", "w3")], &[("$age", 24), ("$ver", 5)]).expect("op18"); m!(db, "main", "set_age_v", &[("$name", "w7")], &[("$age", 7), ("$ver", 6)]).expect("op19"); m!(db, "main", "insert_person", &[("$name", "w7")], &[("$age", 20)]).expect("op20"); - db.cleanup(omnigraph::db::CleanupPolicyOptions { keep_versions: Some(1), older_than: None }).await.expect("op21"); + db.cleanup_as(omnigraph::db::CleanupPolicyOptions { keep_versions: Some(1), older_than: None }, None).await.expect("op21"); m!(db, "main", "set_age_v", &[("$name", "w7")], &[("$age", 17), ("$ver", 7)]).expect("op22"); m!(db, "main", "set_age_v", &[("$name", "w5")], &[("$age", 14), ("$ver", 8)]).expect("op23"); - db.cleanup(omnigraph::db::CleanupPolicyOptions { keep_versions: Some(1), older_than: None }).await.expect("op24"); + db.cleanup_as(omnigraph::db::CleanupPolicyOptions { keep_versions: Some(1), older_than: None }, None).await.expect("op24"); db.branch_create("b0").await.expect("op25"); let empty_merge = Box::pin(db.branch_merge("b0", "main")).await; println!("faithful op26 empty merge: {:?}", empty_merge.map(|_| "ok")); @@ -5374,7 +5382,7 @@ fn dst_liveness_oracle_survives_cross_thread_work() { tokio::time::resume(); let lively = tokio::time::timeout( std::time::Duration::from_secs(120), - Box::pin(db.ensure_indices()), + Box::pin(db.ensure_indices_as(None)), ) .await; tokio::time::pause(); @@ -6339,7 +6347,11 @@ fn dst_optimize_races_branch_delete_minimal_two_thread_negative() { let _ = mutate_main(&db, MUTATION_QUERIES, "set_age", ¶ms).await; } - match Box::pin(db.optimize()).await { + match Box::pin( + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()), + ) + .await + { Ok(_) => {} Err(err) => { let rendered = format!("{err:?}"); diff --git a/crates/omnigraph-gqt-core/src/lib.rs b/crates/omnigraph-gqt-core/src/lib.rs index 9f5d9f91f..1157ebf41 100644 --- a/crates/omnigraph-gqt-core/src/lib.rs +++ b/crates/omnigraph-gqt-core/src/lib.rs @@ -4322,7 +4322,7 @@ pub async fn seed_case(session: &Session, seed: &Seed, needs_indices: bool) -> R seed.load(session).await?; if needs_indices { session - .ensure_indices() + .ensure_indices_as(None) .await .map_err(|e| format!("ensure_indices failed: {e}"))?; } diff --git a/crates/omnigraph-policy/src/lib.rs b/crates/omnigraph-policy/src/lib.rs index 3e1c18422..c0ea6200a 100644 --- a/crates/omnigraph-policy/src/lib.rs +++ b/crates/omnigraph-policy/src/lib.rs @@ -30,6 +30,16 @@ pub enum PolicyAction { /// Drop an unpublished graph commit (`commit drop ""`). Target-branch /// scoped: the branch the proposal would have landed on. CommitDrop, + /// Content-preserving maintenance: optimize, index builds, the full-text + /// rebuild and the repair report. Branch scoped like `change`: the scope + /// is the branch whose tables the operation maintains, `main` for + /// optimize and repair. It grants no right to change data or to delete + /// history. + Maintain, + /// Reclaim storage: cleanup, which permanently deletes table versions and + /// files no retained commit needs. A run covers every branch's history, + /// so the action is graph scoped and takes no branch scope. + Reclaim, /// Reserved action; no current engine or HTTP operation uses this grant. Admin, /// List the server's graph registry. This is server-scoped, with no branch @@ -57,6 +67,8 @@ impl PolicyAction { Self::BranchMerge => "branch_merge", Self::CommitPublish => "commit_publish", Self::CommitDrop => "commit_drop", + Self::Maintain => "maintain", + Self::Reclaim => "reclaim", Self::Admin => "admin", Self::GraphList => "graph_list", Self::InvokeQuery => "invoke_query", @@ -65,7 +77,10 @@ impl PolicyAction { } fn uses_branch_scope(self) -> bool { - matches!(self, Self::Read | Self::Export | Self::Change) + matches!( + self, + Self::Read | Self::Export | Self::Change | Self::Maintain + ) } fn uses_target_branch_scope(self) -> bool { @@ -98,6 +113,8 @@ impl PolicyAction { | Self::BranchMerge | Self::CommitPublish | Self::CommitDrop + | Self::Maintain + | Self::Reclaim | Self::Admin | Self::InvokeQuery => PolicyResourceKind::Graph, } @@ -160,6 +177,8 @@ impl FromStr for PolicyAction { "branch_merge" => Ok(Self::BranchMerge), "commit_publish" => Ok(Self::CommitPublish), "commit_drop" => Ok(Self::CommitDrop), + "maintain" => Ok(Self::Maintain), + "reclaim" => Ok(Self::Reclaim), "admin" => Ok(Self::Admin), "graph_list" => Ok(Self::GraphList), "invoke_query" => Ok(Self::InvokeQuery), @@ -942,6 +961,8 @@ namespace Omnigraph { action "branch_merge" appliesTo { principal: Actor, resource: Graph, context: RequestContext }; action "commit_publish" appliesTo { principal: Actor, resource: Graph, context: RequestContext }; action "commit_drop" appliesTo { principal: Actor, resource: Graph, context: RequestContext }; + action "maintain" appliesTo { principal: Actor, resource: Graph, context: RequestContext }; + action "reclaim" appliesTo { principal: Actor, resource: Graph, context: RequestContext }; action "admin" appliesTo { principal: Actor, resource: Graph, context: RequestContext }; action "invoke_query" appliesTo { principal: Actor, resource: Graph, context: RequestContext }; @@ -1095,6 +1116,35 @@ pub trait PolicyChecker: Send + Sync { scope: &ResourceScope, actor: &str, ) -> Result<(), PolicyError>; + + /// Gate for an operation that any one of `actions` admits on `scope`, + /// such as the full-text rebuild, which `maintain` and its older grant + /// `change` both admit. Each action is decided as [`Self::check`] decides + /// it, in order, and the first allowed one admits the operation. An + /// evaluation failure is returned at once rather than passed over for + /// the next action, so a broken rule never widens access. When every + /// action is denied, the denial names each one. + fn check_any( + &self, + actions: &[PolicyAction], + scope: &ResourceScope, + actor: &str, + ) -> Result<(), PolicyError> { + if actions.is_empty() { + return Err(PolicyError::Internal( + "no policy action to check".to_string(), + )); + } + let mut denials = Vec::with_capacity(actions.len()); + for &action in actions { + match self.check(action, scope, actor) { + Ok(()) => return Ok(()), + Err(PolicyError::Denied(message)) => denials.push(message), + Err(internal) => return Err(internal), + } + } + Err(PolicyError::Denied(denials.join("; "))) + } } impl PolicyChecker for PolicyEngine { @@ -1951,4 +2001,225 @@ rules: assert!(PolicyEngine::load_cluster_from_source(&mixed).is_err()); } } + + #[test] + fn every_action_round_trips_through_its_name() { + use clap::ValueEnum; + for &action in PolicyAction::value_variants() { + let name = action.as_str(); + assert_eq!(action.to_string(), name); + assert_eq!(name.parse::().unwrap(), action, "{name}"); + assert_eq!( + serde_json::to_value(action).unwrap(), + serde_json::Value::String(name.to_string()) + ); + assert_eq!( + serde_json::from_value::(serde_json::json!(name)).unwrap(), + action + ); + } + assert_eq!( + "maintain".parse::().unwrap(), + PolicyAction::Maintain + ); + assert_eq!( + "reclaim".parse::().unwrap(), + PolicyAction::Reclaim + ); + assert!("maintenance".parse::().is_err()); + } + + /// A rule granting any one action compiles under the Cedar schema and + /// allows that action, so no action can be added to the enum without its + /// declaration. + #[test] + fn every_action_is_declared_in_the_cedar_schema() { + use clap::ValueEnum; + for &action in PolicyAction::value_variants() { + let source = format!( + "version: 1\ngroups:\n g: [act]\nrules:\n - id: r\n allow:\n actors: {{ group: g }}\n actions: [{}]\n", + action.as_str() + ); + let engine = match action.resource_kind() { + PolicyResourceKind::Graph => PolicyEngine::load_graph_from_source(&source, "graph"), + PolicyResourceKind::Server => PolicyEngine::load_server_from_source(&source), + PolicyResourceKind::Cluster => PolicyEngine::load_cluster_from_source(&source), + } + .unwrap_or_else(|error| panic!("{action}: {error}")); + let decision = engine + .authorize( + "act", + &PolicyRequest { + action, + branch: None, + target_branch: None, + }, + ) + .unwrap(); + assert!(decision.allowed, "{action}: {}", decision.message); + } + } + + #[test] + fn maintain_is_branch_scoped_and_reclaim_takes_no_branch_scope() { + let policy: PolicyConfig = yaml_serde::from_str( + r#" +version: 1 +groups: + automation: [act-bot] + writers: [act-writer] +protected_branches: [main] +rules: + - id: automation-maintains-unprotected + allow: + actors: { group: automation } + actions: [maintain] + branch_scope: unprotected + - id: writers-change + allow: + actors: { group: writers } + actions: [change] + branch_scope: any +"#, + ) + .unwrap(); + let engine = PolicyCompiler::compile(&policy, "graph").unwrap(); + let decide = |actor: &str, action, branch: Option<&str>| { + engine + .authorize( + actor, + &PolicyRequest { + action, + branch: branch.map(str::to_string), + target_branch: None, + }, + ) + .unwrap() + .allowed + }; + assert!(decide("act-bot", PolicyAction::Maintain, Some("feature"))); + assert!(!decide("act-bot", PolicyAction::Maintain, Some("main"))); + // Maintenance grants neither data writes nor reclamation. + assert!(!decide("act-bot", PolicyAction::Change, Some("feature"))); + assert!(!decide("act-bot", PolicyAction::Reclaim, None)); + // And `change` grants no maintenance. + assert!(!decide( + "act-writer", + PolicyAction::Maintain, + Some("feature") + )); + assert!(!decide("act-writer", PolicyAction::Reclaim, None)); + + let reclaim = r#" +version: 1 +groups: + automation: [act-bot] +rules: + - id: automation-reclaims + allow: + actors: { group: automation } + actions: [reclaim] +"#; + let engine = PolicyEngine::load_graph_from_source(reclaim, "graph").unwrap(); + let allowed = engine + .authorize( + "act-bot", + &PolicyRequest { + action: PolicyAction::Reclaim, + branch: None, + target_branch: None, + }, + ) + .unwrap(); + assert!(allowed.allowed); + assert_eq!( + PolicyAction::Reclaim.resource_kind(), + PolicyResourceKind::Graph + ); + for scope in ["branch_scope", "target_branch_scope"] { + let scoped = reclaim.replace( + "actions: [reclaim]", + &format!("actions: [reclaim]\n {scope}: any"), + ); + let Err(error) = PolicyEngine::load_graph_from_source(&scoped, "graph") else { + panic!("{scope} on reclaim must be refused"); + }; + assert!( + error.to_string().contains("unsupported action 'reclaim'"), + "{scope}: {error}" + ); + } + let target_scoped_maintain = reclaim.replace("[reclaim]", "[maintain]").replace( + "actions: [maintain]", + "actions: [maintain]\n target_branch_scope: any", + ); + assert!(PolicyEngine::load_graph_from_source(&target_scoped_maintain, "graph").is_err()); + } + + #[test] + fn check_any_admits_on_any_allowed_action_and_names_every_denial() { + use super::{PolicyChecker, PolicyError, ResourceScope}; + + let policy = PolicyEngine::load_graph_from_source( + r#" +version: 1 +groups: + maintainers: [act-maintainer] + writers: [act-writer] + readers: [act-reader] +rules: + - id: maintainers + allow: + actors: { group: maintainers } + actions: [maintain] + - id: writers + allow: + actors: { group: writers } + actions: [change] +"#, + "graph", + ) + .unwrap(); + let either = [PolicyAction::Maintain, PolicyAction::Change]; + let scope = ResourceScope::Branch("main".to_string()); + for actor in ["act-maintainer", "act-writer"] { + policy.check_any(&either, &scope, actor).unwrap(); + } + match policy.check_any(&either, &scope, "act-reader") { + Err(PolicyError::Denied(message)) => { + assert!( + message.contains("'maintain'") && message.contains("'change'"), + "{message}" + ); + } + other => panic!("expected a denial naming both actions, got {other:?}"), + } + assert!(matches!( + policy.check_any(&[], &scope, "act-maintainer"), + Err(PolicyError::Internal(_)) + )); + + /// Fails to evaluate `maintain` and allows everything else. + struct BrokenMaintain; + impl PolicyChecker for BrokenMaintain { + fn check( + &self, + action: PolicyAction, + _scope: &ResourceScope, + _actor: &str, + ) -> Result<(), PolicyError> { + match action { + PolicyAction::Maintain => Err(PolicyError::Internal("broken".to_string())), + _ => Ok(()), + } + } + } + assert!( + matches!( + BrokenMaintain.check_any(&either, &scope, "act"), + Err(PolicyError::Internal(message)) if message == "broken" + ), + "an evaluation failure must not fall through to the next action" + ); + } } diff --git a/crates/omnigraph/src/db/graph_coordinator.rs b/crates/omnigraph/src/db/graph_coordinator.rs index 30d23dad5..cb2eda996 100644 --- a/crates/omnigraph/src/db/graph_coordinator.rs +++ b/crates/omnigraph/src/db/graph_coordinator.rs @@ -12,8 +12,10 @@ use super::commit_graph::{ CommitGraph, FirstParentPage, GraphCommit, HistoryCache, Lineage, graph_commit_from_manifest_row, }; +#[cfg(any(test, feature = "failpoints"))] +use super::manifest::DatasetUpdate; use super::manifest::{ - BranchRecords, CapturedManifestProbe, CommitBuffer, DatasetUpdate, ExpectedTableVersions, + BranchRecords, CapturedManifestProbe, CommitBuffer, ExpectedTableVersions, GenesisManifestAttempt, GraphHeadExpectation, HistoryRecord, HistoryReleaseBytes, LineageIntent, ManifestChange, ManifestCoordinator, ManifestIncarnation, ManifestInitError, PublishPrecondition, SchemaContractRow, UnattachedCommit, UnattachedIntent, @@ -896,7 +898,10 @@ impl GraphCoordinator { /// each immutable table identity matches what the caller observed before /// writing; the diagnostic alias is checked as part of the expectation. /// Mismatches surface as `OmniError::Manifest` with - /// `ManifestConflictDetails::PublishedDatasetVersionMismatch`. + /// `ManifestConflictDetails::PublishedDatasetVersionMismatch`. It publishes + /// over whatever head the branch has (`PublishPrecondition::Any`), so no + /// production writer uses it; tests and failpoint fixtures do. + #[cfg(any(test, feature = "failpoints"))] pub(crate) async fn commit_updates_with_actor_with_expected( &mut self, updates: &[DatasetUpdate], @@ -915,6 +920,7 @@ impl GraphCoordinator { /// no separate write, no `commit_graph.refresh()` to pick a parent (the /// publisher resolves it under the CAS). The in-memory commit cache is then /// updated from the intent + the resolved parent without a re-read. + #[cfg(any(test, feature = "failpoints"))] async fn commit_changes_with_actor_with_expected( &mut self, changes: &[ManifestChange], @@ -1104,6 +1110,7 @@ pub(crate) fn commit_not_found(snapshot_id: &SnapshotId) -> OmniError { } /// Wrap each `DatasetUpdate` as a `ManifestChange::Update` for the publisher. +#[cfg(any(test, feature = "failpoints"))] fn updates_to_changes(updates: &[DatasetUpdate]) -> Vec { updates .iter() diff --git a/crates/omnigraph/src/db/mod.rs b/crates/omnigraph/src/db/mod.rs index 8217af43a..0562bfac3 100644 --- a/crates/omnigraph/src/db/mod.rs +++ b/crates/omnigraph/src/db/mod.rs @@ -16,19 +16,21 @@ pub use commit_graph::{GraphCommit, GraphCommitKind}; pub use graph_coordinator::{ReadTarget, ResolvedTarget, SnapshotId}; pub use manifest::{DatasetEntry, DatasetUpdate}; pub use omnigraph::{ - BranchCreated, BuiltIndex, BuiltIndexAction, BuiltIndexKind, COMMIT_LIST_DEFAULT_PAGE_COMMITS, - COMMIT_LIST_MAX_PAGE_COMMITS, CleanupPolicyOptions, CollectorCost, CollectorPathSnapshot, - CollectorReport, CollectorRowSummary, CommitListPage, DatasetCleanupStats, - DatasetOptimizeStats, DatasetRepairStats, EXPORT_CHUNK_MAX_BYTES, ExportCut, - FullTextIndexRebuildResult, GraphCreateReconciliation, IndexBuildResult, InitOptions, - MergeOutcome, MergeResult, Omnigraph, OpenMode, PendingIndex, PreparedGraphCreate, + ArtifactRefresh, ArtifactSkip, BranchCreated, BuiltIndex, BuiltIndexAction, BuiltIndexKind, + COMMIT_LIST_DEFAULT_PAGE_COMMITS, COMMIT_LIST_MAX_PAGE_COMMITS, CleanupPolicyOptions, + CollectorCost, CollectorPathSnapshot, CollectorReport, CollectorRowSummary, CommitListPage, + ConfigurationClear, DatasetCleanupStats, DatasetOptimizeStats, DatasetRepairStats, DeferReason, + EXPORT_CHUNK_MAX_BYTES, ExportCut, FullTextIndexRebuildResult, GraphCreateReconciliation, + IndexBuildResult, InitOptions, MergeOutcome, MergeResult, Omnigraph, OpenMode, + OptimizeBeforePublish, OptimizeCancellation, OptimizeConcurrency, OptimizeLimits, + OptimizePublication, OptimizeReport, OptimizeRequest, PendingIndex, PreparedGraphCreate, PreparedSchemaApply, PreparedSchemaSettlement, ReadCheck, RebuiltFullTextIndex, RepairAction, - RepairClassification, RepairOptions, RepairStats, RetainedManifestVersions, - SYSTEM_COLUMNS_PREFLIGHT, SchemaApplyReconciliation, SchemaApplyResult, SchemaApplySettlement, - SchemaContractDigest, SchemaNonPublicationProof, SkipReason, StagingVerdict, - SystemColumnUpgradeFinding, SystemColumnUpgradeOptions, SystemColumnUpgradeOutcome, - SystemColumnUpgradeReport, TableCollectionPlan, UnattachedTagRecord, UnpublishedManifest, - unattached_release_order, + RepairClassification, RepairStats, ResourceExceeded, ResourceMeasure, RetainedManifestVersions, + RunCancellation, SYSTEM_COLUMNS_PREFLIGHT, SchemaApplyReconciliation, SchemaApplyResult, + SchemaApplySettlement, SchemaContractDigest, SchemaNonPublicationProof, SkipReason, + StagingVerdict, SystemColumnUpgradeFinding, SystemColumnUpgradeOptions, + SystemColumnUpgradeOutcome, SystemColumnUpgradeReport, TableCeilings, TableCollectionPlan, + TableOptimizeOutcome, UnattachedTagRecord, UnpublishedManifest, unattached_release_order, }; pub(crate) use omnigraph::{ LogicalBlobValue, RangedExternalBlobs, logical_row_image, row_blob_values, diff --git a/crates/omnigraph/src/db/omnigraph.rs b/crates/omnigraph/src/db/omnigraph.rs index 195e49a2a..5716a6a61 100644 --- a/crates/omnigraph/src/db/omnigraph.rs +++ b/crates/omnigraph/src/db/omnigraph.rs @@ -50,12 +50,16 @@ pub use export::{EXPORT_CHUNK_MAX_BYTES, ExportCut}; pub(crate) use export::{ LogicalBlobValue, RangedExternalBlobs, logical_row_image, row_blob_values, }; -pub use optimize::{CleanupPolicyOptions, DatasetCleanupStats, DatasetOptimizeStats, SkipReason}; +pub use optimize::{ + ArtifactRefresh, ArtifactSkip, CleanupPolicyOptions, ConfigurationClear, DatasetCleanupStats, + DatasetOptimizeStats, DeferReason, OptimizeBeforePublish, OptimizeCancellation, + OptimizeConcurrency, OptimizeLimits, OptimizePublication, OptimizeReport, OptimizeRequest, + ResourceExceeded, ResourceMeasure, RunCancellation, SkipReason, TableCeilings, + TableOptimizeOutcome, +}; use prepared_create::initial_schema_ir; pub use prepared_create::{GraphCreateReconciliation, PreparedGraphCreate}; -pub use repair::{ - DatasetRepairStats, RepairAction, RepairClassification, RepairOptions, RepairStats, -}; +pub use repair::{DatasetRepairStats, RepairAction, RepairClassification, RepairStats}; pub use schema_apply::{ PreparedSchemaApply, PreparedSchemaSettlement, SchemaApplyReconciliation, SchemaApplySettlement, SchemaContractDigest, SchemaNonPublicationProof, @@ -1133,6 +1137,19 @@ impl Omnigraph { action: omnigraph_policy::PolicyAction, scope: &omnigraph_policy::ResourceScope, actor: Option<&str>, + ) -> Result<()> { + self.enforce_any(&[action], scope, actor) + } + + /// [`Self::enforce`] for an operation that any one of `actions` admits on + /// `scope`, decided by [`omnigraph_policy::PolicyChecker::check_any`]: the + /// first allowed action admits it, an evaluation failure refuses it, and a + /// refusal names every denied action. + pub(crate) fn enforce_any( + &self, + actions: &[omnigraph_policy::PolicyAction], + scope: &omnigraph_policy::ResourceScope, + actor: Option<&str>, ) -> Result<()> { let (branch, target_branch) = scope.to_branch_pair(); for name in [branch, target_branch].into_iter().flatten() { @@ -1150,7 +1167,7 @@ impl Omnigraph { )); }; checker - .check(action, scope, actor) + .check_any(actions, scope, actor) .map_err(|err| OmniError::Policy(err.to_string())) } @@ -1464,6 +1481,12 @@ impl Omnigraph { self.storage.as_ref() } + /// A shared handle on the storage adapter, for work that must outlive the + /// call that starts it. + pub(crate) fn storage_adapter_handle(&self) -> Arc { + Arc::clone(&self.storage) + } + /// Root-scoped writer queues (schema, branch, and `(table, branch)` gates). /// /// Engine-internal writers (mutation finalize, schema_apply, @@ -3514,15 +3537,10 @@ impl Omnigraph { table_ops::graph_index_for_resolved(self, resolved, edge_types, system_columns).await } - /// [`Self::ensure_indices_on`] on this handle's current branch. - pub async fn ensure_indices(&self) -> Result> { - Ok(table_ops::ensure_indices(self).await?.pending) - } - - /// [`Self::ensure_indices_on_as`] with no actor, returning only what it - /// left pending. With a policy installed, use the actor-aware counterpart. - pub async fn ensure_indices_on(&self, branch: &str) -> Result> { - Ok(self.ensure_indices_on_as(branch, None).await?.pending) + /// [`Self::ensure_indices_on_as`] on this handle's current branch, + /// returning only what it left pending. + pub async fn ensure_indices_as(&self, actor: Option<&str>) -> Result> { + Ok(table_ops::ensure_indices_as(self, actor).await?.pending) } /// Bring the selected branch's declared indexes up to its rows: build @@ -3534,7 +3552,7 @@ impl Omnigraph { /// keep their indexes. Idempotent: a branch with nothing to do publishes /// nothing. /// - /// Requires `Change` on the selected branch before any effect and + /// Requires `maintain` on the selected branch before any effect and /// records the actor in the publication. Plans from one graph-manifest /// and schema token and builds the artifacts before taking the schema → /// branch → sorted table gates, then revalidates that authority under @@ -3560,19 +3578,13 @@ impl Omnigraph { /// Rebuilt full-text indexes use the engine's default analyzer, including /// indexes originally created externally with custom tokenizer settings. /// - /// Unlike [`Self::ensure_indices_on`], this replaces existing FTS indexes - /// even when their row coverage is complete. It establishes the supported - /// analyzer generation after an upgrade; it does not rewrite old history. - /// With a policy installed, use the actor-aware counterpart. - pub async fn rebuild_full_text_indices_on( - &self, - branch: &str, - ) -> Result { - self.rebuild_full_text_indices_on_as(branch, None).await - } - - /// Actor-aware full-text rebuilding. Requires `Change` on the selected - /// branch before any effects and records the actor in the graph publication. + /// Unlike [`Self::ensure_indices_on_as`], this replaces existing FTS + /// indexes even when their row coverage is complete. It establishes the + /// supported analyzer generation after an upgrade; it does not rewrite old + /// history. + /// + /// Requires `maintain` or `change` on the selected branch before any + /// effect and records the actor in the graph publication. pub async fn rebuild_full_text_indices_on_as( &self, branch: &str, @@ -3598,27 +3610,36 @@ impl Omnigraph { .await } - /// Compact small Lance fragments into fewer larger ones across every - /// backing dataset for a node or edge type on `main`. See [`optimize`] for details. - pub async fn optimize(&self) -> Result> { - optimize::optimize_all_datasets(self).await + /// Compact small Lance fragments into fewer larger ones across the + /// backing datasets `request` selects on `main`, and catch up their + /// indexes. See [`optimize`] for details. Requires `maintain` on `main` + /// before any effect and records `actor` in the publication. + pub async fn optimize_as( + &self, + actor: Option<&str>, + request: optimize::OptimizeRequest, + ) -> Result { + optimize::optimize_all_datasets(self, actor, request).await } - /// Classify and explicitly repair uncovered graph-manifest/Lance-HEAD drift. See - /// [`repair`] for the distinction between safe maintenance drift and - /// suspicious/unverifiable drift. - pub async fn repair(&self, options: repair::RepairOptions) -> Result { - repair::repair_all_datasets(self, options).await + /// Report, per table on `main`, how its Lance linear history stands + /// against its registration. See [`repair`] for the classifications. The + /// report publishes nothing and adopts no drift. Requires `maintain` on + /// `main`. + pub async fn repair_report_as(&self, actor: Option<&str>) -> Result { + repair::repair_report(self, actor).await } /// Remove Lance manifests (and the fragments they uniquely own) per the /// given [`optimize::CleanupPolicyOptions`]. Destructive to version - /// history. See [`optimize`] for details. - pub async fn cleanup( + /// history. See [`optimize`] for details. Requires `reclaim` before any + /// effect; a run covers every branch, so the check names none. + pub async fn cleanup_as( &self, options: optimize::CleanupPolicyOptions, + actor: Option<&str>, ) -> Result> { - optimize::cleanup_all_datasets(self, options).await + optimize::cleanup_all_datasets(self, options, actor).await } /// The tracing collector's plan for `options`, deleting nothing and taking @@ -4731,6 +4752,7 @@ impl Omnigraph { txn, lineage_intent, publication, + None, ) .await } @@ -6200,7 +6222,7 @@ edge WorksAt: Person -> Company ); // The reconciler materializes it (Person has a row). - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let snapshot = db.snapshot().await; let ds = db .storage() @@ -6232,7 +6254,7 @@ edge WorksAt: Person -> Company db.apply_schema(&desired).await.unwrap(); // After the rewrite the reconciler restores index coverage. - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let snapshot = db.snapshot().await; let ds = db .storage() diff --git a/crates/omnigraph/src/db/omnigraph/collector.rs b/crates/omnigraph/src/db/omnigraph/collector.rs index f803fd81d..54c50e305 100644 --- a/crates/omnigraph/src/db/omnigraph/collector.rs +++ b/crates/omnigraph/src/db/omnigraph/collector.rs @@ -521,6 +521,42 @@ fn probe_directory( }) } +/// The store and path of one data or overlay file of `dataset`, under the +/// directory Lance's base rule gives it. +pub(crate) async fn data_file_location( + dataset: &lance::Dataset, + file: &lance_table::format::DataFile, +) -> Result<( + std::sync::Arc, + object_store::path::Path, +)> { + let store = dataset + .object_store(file.base_id) + .await + .map_err(OmniError::storage)?; + let path = probe_directory(dataset, file.base_id, "data")?.join(file.path.as_str()); + Ok((store, path)) +} + +/// The store and path of fragment `fragment_id`'s deletion file, under the +/// dataset root or its base, which must be a dataset root. +pub(crate) async fn deletion_file_location( + dataset: &lance::Dataset, + fragment_id: u64, + deletion: &lance_table::format::DeletionFile, +) -> Result<( + std::sync::Arc, + object_store::path::Path, +)> { + let store = dataset + .object_store(deletion.base_id) + .await + .map_err(OmniError::storage)?; + let base = probe_directory(dataset, deletion.base_id, "")?; + let path = lance_table::io::deletion::deletion_file_path(&base, fragment_id, deletion); + Ok((store, path)) +} + async fn capture_index_paths( snapshot: &mut CollectorPathSnapshot, dataset: &lance::Dataset, @@ -576,20 +612,11 @@ pub(crate) async fn capture_dataset_paths( ); for fragment in dataset.manifest().fragments.iter() { for file in fragment.referenced_lance_files() { - let store = dataset - .object_store(file.base_id) - .await - .map_err(OmniError::storage)?; - let path = probe_directory(dataset, file.base_id, "data")?.join(file.path.as_str()); + let (store, path) = data_file_location(dataset, file).await?; snapshot.insert(location, store, path); } if let Some(deletion) = &fragment.deletion_file { - let store = dataset - .object_store(deletion.base_id) - .await - .map_err(OmniError::storage)?; - let base = probe_directory(dataset, deletion.base_id, "")?; - let path = lance_table::io::deletion::deletion_file_path(&base, fragment.id, deletion); + let (store, path) = deletion_file_location(dataset, fragment.id, deletion).await?; snapshot.insert(location, store, path); } } @@ -2262,7 +2289,7 @@ mod borrowed_origin_tests { load_company(&db, "initial-company").await; db.branch_create("legacy").await.unwrap(); load_person(&db, "borrowed", 41).await; - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let source_snapshot = db.snapshot_of(ReadTarget::branch("main")).await.unwrap(); let entry = source_snapshot.dataset("node:Person").unwrap().clone(); let mut source = source_snapshot @@ -2376,7 +2403,7 @@ mod borrowed_origin_tests { assert!(!company_old.is_empty()); load_company(&db, "unrelated-current").await; load_person(&db, "main-current", 99).await; - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let options = super::super::optimize::CleanupPolicyOptions { keep_versions: Some(1), older_than: None, @@ -2391,7 +2418,7 @@ mod borrowed_origin_tests { source_plan.would_remove().contains(&source_version), "borrowed source manifest is reclaimable" ); - let rows = db.cleanup(options).await.unwrap(); + let rows = db.cleanup_as(options, None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); let source_store = source.object_store(None).await.unwrap(); for path in source_data.iter().chain(&source_indices) { @@ -2504,7 +2531,7 @@ mod borrowed_origin_tests { .any(|table| table.would_remove().contains(&version)), "old source pin must be reclaimable without the native tag's borrowed files" ); - let rows = db.cleanup(options).await.unwrap(); + let rows = db.cleanup_as(options, None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); drop(db); // forbidden-api-allow: test-only cold native read of a tagged fork absent from graph registrations @@ -2745,7 +2772,7 @@ mod overlay_retention_tests { .any(|object| object.path == overlay_path) ); assert!(expected.missing_paths().await.unwrap().is_empty()); - let rows = db.cleanup(options).await.unwrap(); + let rows = db.cleanup_as(options, None).await.unwrap(); let row = rows .iter() .find(|row| row.type_key == entry.type_key) @@ -2793,7 +2820,7 @@ mod overlay_retention_tests { .any(|object| object.path == overlay_path), "the independent pre-cleanup probe must include overlays" ); - let rows = db.cleanup(options.clone()).await.unwrap(); + let rows = db.cleanup_as(options.clone(), None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); let store = overlay.object_store(None).await.unwrap(); assert!(store.exists(&overlay_path).await.unwrap()); @@ -2842,7 +2869,7 @@ mod overlay_retention_tests { } else { load_person(&db, 99).await; } - let rows = db.cleanup(options).await.unwrap(); + let rows = db.cleanup_as(options, None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); assert!( !store.exists(&overlay_path).await.unwrap(), @@ -3020,7 +3047,7 @@ mod native_table_tag_tests { !plan.would_remove().contains(&version), "an exact native table tag must exclude its version from cleanup: {plan:?}", ); - let rows = db.cleanup(options.clone()).await.unwrap(); + let rows = db.cleanup_as(options.clone(), None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); assert!(store.exists(&manifest_path).await.unwrap()); // forbidden-api-allow: test-only cold native tag read after the graph collector ran @@ -3056,7 +3083,7 @@ mod native_table_tag_tests { .any(|plan| plan.location == location && plan.would_remove().contains(&version)), "removing the tag must make the old version reclaimable" ); - let rows = db.cleanup(options).await.unwrap(); + let rows = db.cleanup_as(options, None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); assert!( !store.exists(&manifest_path).await.unwrap(), diff --git a/crates/omnigraph/src/db/omnigraph/optimize.rs b/crates/omnigraph/src/db/omnigraph/optimize.rs index 6a9e38430..1581b7d8d 100644 --- a/crates/omnigraph/src/db/omnigraph/optimize.rs +++ b/crates/omnigraph/src/db/omnigraph/optimize.rs @@ -23,12 +23,17 @@ //! to unreferenced version history — callers should gate this behind an //! explicit confirm flag at the CLI layer. +use std::collections::VecDeque; +use std::sync::Arc; use std::time::Duration; -use futures::stream::StreamExt; +use futures::future::BoxFuture; +use futures::stream::{FuturesOrdered, FuturesUnordered, StreamExt}; use lance::dataset::optimize::{CompactionMetrics, CompactionOptions}; +use tokio::time::Instant; use super::*; +use crate::error::CompletionEvidence; use crate::seams::{decide_seam, fail}; /// How many datasets to optimize/cleanup concurrently. Each has separate @@ -92,50 +97,672 @@ impl std::fmt::Display for SkipReason { } } -/// Per-dataset outcome of `optimize_all_datasets`. This is a returned result type, +/// What one optimize run did to one table. A run reports exactly one outcome +/// for every table it considered. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[non_exhaustive] +pub enum TableOptimizeOutcome { + /// A rewrite or index link was staged and is in the run's publication. + Optimized, + /// Planning found nothing to do. + Unchanged, + /// Not started: nothing was written for the table. + Deferred(DeferReason), + /// Deliberately left alone. + Skipped(SkipReason), + /// A failure of this table's own planning or staging, with definite + /// evidence: its staging is unreachable and the other tables still + /// publish. [`DatasetOptimizeStats::failure`] holds the error. + Failed, +} + +impl TableOptimizeOutcome { + /// Stable machine-readable token for serialized output. Once emitted this + /// is part of the output contract — keep it stable. + pub fn as_str(&self) -> &'static str { + match self { + TableOptimizeOutcome::Optimized => "optimized", + TableOptimizeOutcome::Unchanged => "unchanged", + TableOptimizeOutcome::Deferred(_) => "deferred", + TableOptimizeOutcome::Skipped(_) => "skipped", + TableOptimizeOutcome::Failed => "failed", + } + } +} + +/// Why a run did not start a table. Typed so callers branch on the reason. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[non_exhaustive] +pub enum DeferReason { + /// The run had held main's writer gates for `max_gate_hold` when the + /// table's turn came. This table and every later one wait for a later + /// run, which [`OptimizeReport::resume_at`] starts at. + GateHoldLimit, + /// The run was cancelled before this table's work could land. + Cancelled, + /// The table's plan passed a per-table ceiling, measured before its first + /// write; nothing was written for it, and later tables still run. + ResourceLimit(ResourceExceeded), + /// The table's source bytes would pass the run's remaining byte budget. + /// This table and every later one wait for a later run, which + /// [`OptimizeReport::resume_at`] starts at. + ByteBudget, +} + +/// What a per-table ceiling measures, all from the table's plan. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[non_exhaustive] +pub enum ResourceMeasure { + /// The fragments the compaction plan rewrites. + SourceFragments, + /// Their rows. + SourceRows, + /// The bytes of their data, overlay and deletion files. + SourceBytes, + /// Their managed Blob payload, from descriptors; external references + /// count nothing. + BlobBytes, + /// The table's rows, when the run would rebuild or build an index over it. + IndexRows, +} + +impl ResourceMeasure { + /// Stable machine-readable token for serialized output. Once emitted this + /// is part of the output contract — keep it stable. + pub fn as_str(&self) -> &'static str { + match self { + ResourceMeasure::SourceFragments => "source_fragments", + ResourceMeasure::SourceRows => "source_rows", + ResourceMeasure::SourceBytes => "source_bytes", + ResourceMeasure::BlobBytes => "blob_bytes", + ResourceMeasure::IndexRows => "index_rows", + } + } +} + +/// A table's plan against one ceiling. `measured` is `None` when the size +/// could not be measured; otherwise counting stopped once it passed the +/// ceiling, so it is a lower bound. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct ResourceExceeded { + pub measure: ResourceMeasure, + pub measured: Option, + pub ceiling: u64, +} + +impl DeferReason { + /// Stable machine-readable token for serialized output. Once emitted this + /// is part of the output contract — keep it stable. + pub fn as_str(&self) -> &'static str { + match self { + DeferReason::GateHoldLimit => "gate_hold_limit", + DeferReason::Cancelled => "cancelled", + DeferReason::ResourceLimit(_) => "resource_limit", + DeferReason::ByteBudget => "byte_budget", + } + } +} + +impl std::fmt::Display for DeferReason { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str(match self { + DeferReason::GateHoldLimit => "the run reached its writer-gate hold limit", + DeferReason::Cancelled => "the run was cancelled", + DeferReason::ResourceLimit(ResourceExceeded { + measure, + measured: Some(measured), + ceiling, + }) => { + return write!( + f, + "{} of at least {measured} pass the table ceiling of {ceiling}", + measure.as_str() + ); + } + DeferReason::ResourceLimit(ResourceExceeded { + measure, + measured: None, + .. + }) => { + return write!(f, "{} could not be measured", measure.as_str()); + } + DeferReason::ByteBudget => "the run's byte budget is spent", + }) + } +} + +/// Per-table outcome of `optimize_all_datasets`. This is a returned result type, /// not built by callers, so it is `#[non_exhaustive]`: future fields stay /// non-breaking and downstream code reads fields rather than constructing it. #[derive(Debug, Clone)] #[non_exhaustive] pub struct DatasetOptimizeStats { - /// Qualified graph type key, or `__manifest` for the system dataset. + /// Qualified graph type key. pub type_key: String, + /// What the run did to this table. + pub outcome: TableOptimizeOutcome, /// Number of source fragments that were rewritten by Lance. pub fragments_removed: usize, /// Number of new, larger fragments Lance produced. pub fragments_added: usize, - /// Whether this dataset advanced to a new Lance version that this run - /// published through the graph manifest. This may result from compaction, - /// index maintenance or materialization, or stale `auto_cleanup` removal. - pub committed: bool, - /// `Some(reason)` if this dataset was deliberately not compacted. When set, - /// `fragments_removed == 0`, `fragments_added == 0`, and `!committed`. - pub skipped: Option, - /// Published dataset version observed by optimize for drift skips. `None` for - /// normal compaction/no-op outcomes. - pub published_dataset_version: Option, - /// Lance HEAD version observed by optimize for drift skips. `None` for - /// normal compaction/no-op outcomes. - pub lance_head_version: Option, /// Index work deferred this run, with the reason and remedy: a vector /// property without trainable vectors, or full-text coverage requiring an /// explicit rebuild. Deferred work alone does not stage or publish. pub pending_indexes: Vec, + /// Time from the start of the table's planning to the end of its staging; + /// zero for a table the run did not start. + pub duration: Duration, + /// The error of a [`TableOptimizeOutcome::Failed`] table, carrying its + /// completion evidence (`BeforeEffect` or `Unpublished`). + pub failure: Option>, + /// The plan's source bytes, when a byte ceiling or the run's byte budget + /// measured them; what the table spent of the budget. + pub source_bytes: Option, } impl DatasetOptimizeStats { - /// Stat for a dataset that Lance actually compacted. - fn compacted(type_key: String, metrics: &CompactionMetrics, committed: bool) -> Self { + fn planned(type_key: String, metrics: &CompactionMetrics, committed: bool) -> Self { Self { type_key, + outcome: if committed { + TableOptimizeOutcome::Optimized + } else { + TableOptimizeOutcome::Unchanged + }, fragments_removed: metrics.fragments_removed, fragments_added: metrics.fragments_added, - committed, - skipped: None, - published_dataset_version: None, - lance_head_version: None, pending_indexes: Vec::new(), + duration: Duration::ZERO, + failure: None, + source_bytes: None, + } + } + + fn deferred(type_key: String, reason: DeferReason) -> Self { + Self { + type_key, + outcome: TableOptimizeOutcome::Deferred(reason), + fragments_removed: 0, + fragments_added: 0, + pending_indexes: Vec::new(), + duration: Duration::ZERO, + failure: None, + source_bytes: None, + } + } + + fn failed(type_key: String, failure: Arc, duration: Duration) -> Self { + Self { + type_key, + outcome: TableOptimizeOutcome::Failed, + fragments_removed: 0, + fragments_added: 0, + pending_indexes: Vec::new(), + duration, + failure: Some(failure), + source_bytes: None, + } + } + + /// Whether this table advanced to a new Lance version that this run + /// published through the graph manifest: compaction, index maintenance or + /// materialization. + pub fn committed(&self) -> bool { + self.outcome == TableOptimizeOutcome::Optimized + } + + /// `Some(reason)` if this table was deliberately not compacted. + pub fn skipped(&self) -> Option { + match self.outcome { + TableOptimizeOutcome::Skipped(reason) => Some(reason), + _ => None, + } + } +} + +/// The configuration clear that follows a run's publication under the writer +/// gates: stale `lance.auto_cleanup.*` keys on `__manifest` are removed with +/// one config-only commit. It never fails the run; the next run clears keys +/// that are still present. +#[derive(Debug, Clone, PartialEq, Eq)] +#[non_exhaustive] +pub enum ConfigurationClear { + /// The keys were removed. + Cleared, + /// No key was present. + NothingToClear, + /// The keys are still present. + Failed(String), + /// Whether the removal committed has no definite answer. + Unknown(String), + /// The run was cancelled before its publication, so the clear did not run. + NotRun, +} + +impl ConfigurationClear { + /// Stable machine-readable token for serialized output. Once emitted this + /// is part of the output contract — keep it stable. + pub fn as_str(&self) -> &'static str { + match self { + ConfigurationClear::Cleared => "cleared", + ConfigurationClear::NothingToClear => "nothing_to_clear", + ConfigurationClear::Failed(_) => "failed", + ConfigurationClear::Unknown(_) => "unknown", + ConfigurationClear::NotRun => "not_run", + } + } +} + +/// The adjacency artifact refresh, after the writer gates are released. The +/// artifact is derived state: a stale or missing one is rejected on load and +/// rebuilt in memory, so no outcome here changes an answer or fails the run. +#[derive(Debug, Clone, PartialEq, Eq)] +#[non_exhaustive] +pub enum ArtifactRefresh { + /// A new artifact was written. + Refreshed, + /// No edge table advanced and an artifact exists. + Unchanged, + /// Deliberately not built. + Skipped(ArtifactSkip), + /// The build or the write failed; traversals keep building in memory. + Failed(String), +} + +impl ArtifactRefresh { + /// Stable machine-readable token for serialized output. Once emitted this + /// is part of the output contract — keep it stable. + pub fn as_str(&self) -> &'static str { + match self { + ArtifactRefresh::Refreshed => "refreshed", + ArtifactRefresh::Unchanged => "unchanged", + ArtifactRefresh::Skipped(_) => "skipped", + ArtifactRefresh::Failed(_) => "failed", + } + } +} + +/// Why a run did not build the adjacency artifact. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[non_exhaustive] +pub enum ArtifactSkip { + /// The catalog declares no edge type. + NoEdgeTypes, + /// A declared edge type has no table yet, so the artifact has no complete + /// identity to stamp. + IncompleteEdgeTables, + /// The run was cancelled, so the edge scan did not start. + Cancelled, + /// The artifact would pass `OptimizeLimits::artifact_bytes`: its lower + /// bound did before the scan, or the counting build's running bound did + /// during it. `counted` is that bound; nothing was written. + SizeLimit { counted: u64, limit: u64 }, +} + +impl ArtifactSkip { + /// Stable machine-readable token for serialized output. Once emitted this + /// is part of the output contract — keep it stable. + pub fn as_str(&self) -> &'static str { + match self { + ArtifactSkip::NoEdgeTypes => "no_edge_types", + ArtifactSkip::IncompleteEdgeTables => "incomplete_edge_tables", + ArtifactSkip::Cancelled => "cancelled", + ArtifactSkip::SizeLimit { .. } => "size_limit", + } + } +} + +/// What one optimize run did. A returned result type, so `#[non_exhaustive]`. +#[derive(Debug, Clone)] +#[non_exhaustive] +pub struct OptimizeReport { + /// main's graph head as the run captured it under the writer gates: the + /// parent of the commit the run publishes. + pub input_commit: Option, + /// The `__manifest` version of that head. + pub input_manifest_version: u64, + /// The graph commit the run published; `None` when it staged no table. + pub graph_commit_id: Option, + /// One entry per table the run considered, in the order it visited them. + pub tables: Vec, + /// The first table a full-catalog run deferred because of a run-wide + /// bound. A full-catalog run that passes it as `start_at` visits it + /// first. `None` when the run stopped on no bound, and for a `tables` + /// request, which neither reads nor moves the resume position. + pub resume_at: Option, + /// How long the run held main's writer gates, from acquisition to release. + pub gate_hold: Duration, + /// The configuration clear, under the gates after the publication. + pub configuration_clear: ConfigurationClear, + /// The adjacency artifact refresh, after the gates are released. + pub artifact: ArtifactRefresh, + /// Whether a cancellation stopped the run. + pub cancellation: RunCancellation, +} + +/// What a run's cancellation handle did. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[non_exhaustive] +pub enum RunCancellation { + /// No cancellation was requested. + NotRequested, + /// The run stopped at a boundary before its publication and published + /// nothing: every table it had not finished, and every staged one, is + /// `deferred(cancelled)`. + Cancelled, + /// The cancellation arrived after the run passed its last check before + /// the publication, so the run finished; only the artifact scan was + /// skipped. + ArrivedLate, +} + +impl RunCancellation { + /// Stable machine-readable token for serialized output. Once emitted this + /// is part of the output contract — keep it stable. + pub fn as_str(&self) -> &'static str { + match self { + RunCancellation::NotRequested => "not_requested", + RunCancellation::Cancelled => "cancelled", + RunCancellation::ArrivedLate => "arrived_late", + } + } +} + +/// A request to stop a run at its next safe boundary: before a table's plan, +/// before its rewrite, and before the publication. A table in progress is +/// dropped where it stands; its detached effects stay unreachable. Clones +/// share one request. +#[derive(Debug, Clone, Default)] +pub struct OptimizeCancellation(Arc); + +#[derive(Debug, Default)] +struct CancellationState { + requested: std::sync::atomic::AtomicBool, + notify: tokio::sync::Notify, +} + +impl OptimizeCancellation { + pub fn new() -> Self { + Self::default() + } + + pub fn cancel(&self) { + self.0 + .requested + .store(true, std::sync::atomic::Ordering::SeqCst); + self.0.notify.notify_waiters(); + } + + pub fn is_cancelled(&self) -> bool { + self.0.requested.load(std::sync::atomic::Ordering::SeqCst) + } + + async fn cancelled(&self) { + loop { + let notified = self.0.notify.notified(); + tokio::pin!(notified); + notified.as_mut().enable(); + if self.is_cancelled() { + return; + } + notified.await; + } + } +} + +/// How many tables a run plans ahead of the ones it rewrites, and how many it +/// rewrites at once. Planning only reads. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct OptimizeConcurrency { + pub planning: usize, + pub rewrite: usize, +} + +impl OptimizeConcurrency { + pub fn new(planning: usize, rewrite: usize) -> Self { + Self { planning, rewrite } + } + + /// The direct path's setting: both from `OMNIGRAPH_MAINTENANCE_CONCURRENCY` + /// (default 8). + pub fn from_env() -> Self { + let concurrency = maint_concurrency(); + Self::new(concurrency, concurrency) + } +} + +/// Bounds on one run. An absent limit does not bound it. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +#[non_exhaustive] +pub struct OptimizeLimits { + /// Stop starting tables once the run has held main's writer gates this + /// long. The first table always starts, so every run makes progress; the + /// writer wait is at most this, plus that table, the publication and the + /// configuration clear. + pub max_gate_hold: Option, + /// Ceilings on one table's plan, checked before its first write. + pub table: TableCeilings, + /// Total source bytes across the tables the run starts. It is also the + /// table byte ceiling when that is unset, and refuses a larger one, so a + /// first table within its ceilings always fits. + pub run_source_bytes: Option, + /// The largest adjacency artifact the run writes, in bytes of the whole + /// object. Over it, the artifact is `skipped(size_limit)` and a load + /// rebuilds the adjacency in memory. + pub artifact_bytes: Option, +} + +impl OptimizeLimits { + pub fn with_max_gate_hold(mut self, hold: Duration) -> Self { + self.max_gate_hold = Some(hold); + self + } + + pub fn with_table_ceilings(mut self, ceilings: TableCeilings) -> Self { + self.table = ceilings; + self + } + + pub fn with_run_source_bytes(mut self, budget: u64) -> Self { + self.run_source_bytes = Some(budget); + self + } + + pub fn with_artifact_bytes(mut self, limit: u64) -> Self { + self.artifact_bytes = Some(limit); + self + } + + /// The table byte ceiling in force: the explicit one, else the budget. + fn table_source_bytes(&self) -> Option { + self.table.source_bytes.or(self.run_source_bytes) + } +} + +/// Ceilings on one table's plan; an absent one is not measured. A table over +/// any of them is `deferred(resource_limit)` and nothing is written for it. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +#[non_exhaustive] +pub struct TableCeilings { + pub source_fragments: Option, + pub source_rows: Option, + pub source_bytes: Option, + pub blob_bytes: Option, + pub index_rows: Option, +} + +impl TableCeilings { + pub fn with_source_fragments(mut self, ceiling: u64) -> Self { + self.source_fragments = Some(ceiling); + self + } + + pub fn with_source_rows(mut self, ceiling: u64) -> Self { + self.source_rows = Some(ceiling); + self + } + + pub fn with_source_bytes(mut self, ceiling: u64) -> Self { + self.source_bytes = Some(ceiling); + self + } + + pub fn with_blob_bytes(mut self, ceiling: u64) -> Self { + self.blob_bytes = Some(ceiling); + self + } + + pub fn with_index_rows(mut self, ceiling: u64) -> Self { + self.index_rows = Some(ceiling); + self + } +} + +/// What a run's before-publish hook receives. +#[derive(Debug, Clone)] +#[non_exhaustive] +pub struct OptimizePublication { + /// The nonce of the commit's lineage intent. History stores the commit + /// under an id chosen per publish attempt (`hb1.{block}.{slot}.{nonce}`); + /// `omnigraph_core::graph_commit_id::commit_id_answers` matches such an id + /// to this nonce. + pub nonce: String, + /// The graph head every publish attempt reads: the commit's parent. + pub input_commit: Option, + /// The `__manifest` version of that head. + pub input_manifest_version: u64, + /// One outcome for every table the run considered, as the report lists them. + pub tables: Vec, +} + +/// Called once, after the run's tables are staged and prepared and before the +/// publisher's first attempt; a run that stages no table never calls it. An +/// error publishes nothing and fails the run with `Unpublished` evidence. +#[derive(Clone)] +pub struct OptimizeBeforePublish( + Arc BoxFuture<'static, Result<()>> + Send + Sync>, +); + +impl OptimizeBeforePublish { + pub fn new(hook: F) -> Self + where + F: Fn(OptimizePublication) -> Fut + Send + Sync + 'static, + Fut: std::future::Future> + Send + 'static, + { + Self(Arc::new(move |publication| Box::pin(hook(publication)))) + } +} + +impl std::fmt::Debug for OptimizeBeforePublish { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str("OptimizeBeforePublish(..)") + } +} + +/// What one optimize run is asked to do. The default is the direct run as it +/// has always been: every table on main, no bound, and both concurrencies from +/// `OMNIGRAPH_MAINTENANCE_CONCURRENCY`. +#[derive(Debug, Clone, Default)] +#[non_exhaustive] +pub struct OptimizeRequest { + /// Only these table keys (`node:`, `edge:`), in this order. + /// A key that names no declared type is refused before any effect. + pub tables: Option>, + /// For a full-catalog run: start at this table key, or at the first key + /// after it when no table has it, and wrap around, so the run visits every + /// table at most once. + pub start_at: Option, + pub limits: OptimizeLimits, + /// `None` reads [`OptimizeConcurrency::from_env`]. + pub concurrency: Option, + pub before_publish: Option, + pub cancellation: Option, +} + +impl OptimizeRequest { + pub fn with_tables(mut self, tables: I) -> Self + where + I: IntoIterator, + S: Into, + { + self.tables = Some(tables.into_iter().map(Into::into).collect()); + self + } + + pub fn with_start_at(mut self, table_key: impl Into) -> Self { + self.start_at = Some(table_key.into()); + self + } + + pub fn with_limits(mut self, limits: OptimizeLimits) -> Self { + self.limits = limits; + self + } + + pub fn with_concurrency(mut self, concurrency: OptimizeConcurrency) -> Self { + self.concurrency = Some(concurrency); + self + } + + pub fn with_before_publish(mut self, hook: OptimizeBeforePublish) -> Self { + self.before_publish = Some(hook); + self + } + + pub fn with_cancellation(mut self, cancellation: OptimizeCancellation) -> Self { + self.cancellation = Some(cancellation); + self + } + + /// The tables this request visits, in order, out of the catalog's sorted + /// keys. Refuses a malformed request before any effect. + fn visit_order(&self, table_keys: &[String]) -> Result> { + if let Some(concurrency) = self.concurrency + && (concurrency.planning == 0 || concurrency.rewrite == 0) + { + return Err(OmniError::manifest( + "optimize concurrency must be at least 1 for planning and rewriting", + )); + } + if let (Some(table), Some(budget)) = + (self.limits.table.source_bytes, self.limits.run_source_bytes) + && table > budget + { + return Err(OmniError::manifest(format!( + "optimize table byte ceiling {table} exceeds the run byte budget {budget}" + ))); + } + let Some(tables) = &self.tables else { + let start = self + .start_at + .as_ref() + .map_or(0, |start| table_keys.partition_point(|key| key < start)); + let mut order = table_keys[start..].to_vec(); + order.extend_from_slice(&table_keys[..start]); + return Ok(order); + }; + if self.start_at.is_some() { + return Err(OmniError::manifest( + "optimize takes a start position only for a full-catalog run, not with a table list", + )); + } + if tables.is_empty() { + return Err(OmniError::manifest("optimize table list is empty")); } + let mut seen = std::collections::BTreeSet::new(); + for table in tables { + if table_keys.binary_search(table).is_err() { + return Err(OmniError::manifest(format!( + "optimize table '{table}' names no node or edge type" + ))); + } + if !seen.insert(table.as_str()) { + return Err(OmniError::manifest(format!( + "optimize table '{table}' is listed twice" + ))); + } + } + Ok(tables.clone()) } } @@ -177,6 +804,9 @@ struct PreparedOptimizeTable { initial_snapshot: crate::storage_layer::SnapshotHandle, witness: crate::table_store::StagingWitness, last_linear_version: Option, + /// The plan's source bytes, measured when a byte ceiling or the run's + /// byte budget is set. + source_bytes: Option, } enum OptimizePreparation { @@ -208,13 +838,30 @@ decide_seam! { pub static OPTIMIZE_POST_AUTHORITY_CAPTURE_PRE_GATES = ("optimize.post_authority_capture_pre_gates", Unreachable, [Fail]); } -/// Run Lance maintenance across every node + edge dataset on `main` under one -/// graph visibility envelope. Physical dataset work remains bounded-parallel, -/// but every productive dataset stages detached and the batch publishes once -/// with an exact pin CAS, so one public Optimize produces at most one graph -/// commit. The publisher owns main's `__manifest` fragment layout; maintenance -/// only removes obsolete automatic-cleanup configuration from that dataset. -pub async fn optimize_all_datasets(db: &Omnigraph) -> Result> { +/// Run Lance maintenance across the node + edge datasets on `main` that +/// `request` selects, under one graph visibility envelope. Tables are visited +/// in order: up to `concurrency.planning` are planned ahead of the ones being +/// rewritten, rewrites start in visit order, and every productive dataset +/// stages detached. The batch publishes once with an exact pin CAS, so one +/// public Optimize produces at most one graph commit. Once the run has held +/// the writer gates for `max_gate_hold`, it starts no further table and +/// reports the rest deferred. The publisher owns main's `__manifest` fragment +/// layout; maintenance only removes obsolete automatic-cleanup configuration +/// from that dataset, under the gates, and refreshes the adjacency artifact +/// after releasing them. +pub async fn optimize_all_datasets( + db: &Omnigraph, + actor: Option<&str>, + request: OptimizeRequest, +) -> Result { + db.enforce( + omnigraph_policy::PolicyAction::Maintain, + &omnigraph_policy::ResourceScope::Branch("main".to_string()), + actor, + )?; + let concurrency = request + .concurrency + .unwrap_or_else(OptimizeConcurrency::from_env); let _export_exclusion = db.reserve_export_destructive_control()?; db.ensure_schema_state_valid().await?; @@ -231,26 +878,28 @@ pub async fn optimize_all_datasets(db: &Omnigraph) -> Result>(); let table_guards = db.write_queue().acquire_many(&queue_keys).await; + let gates_acquired = Instant::now(); let snapshot = db.revalidate_write_txn(&authority_txn).await?; - - // Whether this run advanced any edge table — consumed by the graph-index - // artifact gate at the tail (a no-work optimize skips the rebuild+PUT). - let mut edge_tables_committed = false; + let input_commit = authority_txn.authority.graph_head.clone(); + let input_manifest_version = authority_txn.base.graph_manifest_version(); let witness = authority_txn.authority.staging_witness()?; - let table_tasks = table_keys + let table_tasks = visit_order .into_iter() .filter_map(|table_key| { let entry = snapshot.dataset(&table_key)?; @@ -265,103 +914,367 @@ pub async fn optimize_all_datasets(db: &Omnigraph) -> Result>(); - let concurrency = maint_concurrency().min(table_tasks.len()).max(1); + let deadline = request + .limits + .max_gate_hold + .map(|hold| gates_acquired + hold); + let cancellation = request.cancellation.as_ref(); + let cancel_requested = || cancellation.is_some_and(OptimizeCancellation::is_cancelled); + let visited = run_table_pipeline( + db, + &catalog, + table_tasks, + concurrency, + deadline, + request.limits, + cancellation, + ) + .await?; + // The last cancellation point: once past it, the run publishes. + let cancelled = visited.cancelled || cancel_requested(); + + let mut updates = Vec::new(); + let mut expected_versions = crate::db::manifest::ExpectedTableVersions::new(); + let mut tables = Vec::with_capacity(visited.tables.len()); + for (mut stat, effect) in visited.tables { + if cancelled && effect.is_some() { + let duration = stat.duration; + stat = DatasetOptimizeStats::deferred(stat.type_key, DeferReason::Cancelled); + stat.duration = duration; + } else if let Some(effect) = effect { + expected_versions.insert( + effect.update.identity, + crate::db::manifest::TableVersionExpectation { + table_key: effect.update.type_key.clone(), + table_version: effect.expected_version, + native_ref: crate::db::manifest::NativeRefPin::Exact(None), + }, + ); + updates.push(effect.update); + } + tables.push(stat); + } + let edge_tables_committed = updates + .iter() + .any(|update| update.type_key.starts_with("edge:")); + let mut graph_commit_id = None; + if !updates.is_empty() { + fail(&OPTIMIZE_POST_PHASE_B_PRE_MANIFEST_COMMIT).map_err(OmniError::unpublished)?; + let lineage = db + .new_lineage_intent_for_branch(None, actor, HistoryReleaseBytes::PRODUCTION) + .await + .map_err(OmniError::unpublished)?; + let before_publish = request.before_publish.as_ref().map(|hook| { + let hook = hook.0.clone(); + let tables = tables.clone(); + move |intent: super::table_ops::PublicationIntent| { + hook(OptimizePublication { + nonce: intent.nonce, + input_commit: intent.input_commit, + input_manifest_version: intent.input_manifest_version, + tables: tables.clone(), + }) + } + }); + let commit = super::table_ops::commit_updates_on_branch_with_expected( + db, + None, + &updates, + &expected_versions, + actor, + &authority_txn, + lineage, + super::table_ops::Publication::Publish, + before_publish + .as_ref() + .map(|hook| hook as super::table_ops::BeforePublishHook<'_>), + ) + .await?; + graph_commit_id = Some(commit.graph_commit_id); + db.runtime_cache.invalidate_all().await; + if edge_tables_committed { + db.invalidate_graph_index().await; + } + } - let preparations: Vec> = futures::stream::iter(table_tasks) - .map(|task| { - let catalog = std::sync::Arc::clone(&catalog); - async move { prepare_optimize_table(db, catalog.as_ref(), task).await } - }) - .buffer_unordered(concurrency) - .collect() - .await; + // The graph commit is durable from here: nothing below fails the run. A + // cancelled run published nothing and leaves the configuration as is. + let configuration_clear = if cancelled { + ConfigurationClear::NotRun + } else { + clear_manifest_auto_cleanup(db).await + }; + let gate_hold = gates_acquired.elapsed(); + drop(table_guards); + drop(main_branch_guard); + drop(schema_permit); - let mut prepared = Vec::new(); - let mut stats = Vec::new(); - for preparation in preparations { - match preparation? { - OptimizePreparation::Work(work) => prepared.push(work), - OptimizePreparation::Stat(stat) => stats.push(stat), + let cancellation = match (cancelled, cancel_requested()) { + (true, _) => RunCancellation::Cancelled, + (false, true) => RunCancellation::ArrivedLate, + (false, false) => RunCancellation::NotRequested, + }; + let artifact = if cancellation != RunCancellation::NotRequested { + ArtifactRefresh::Skipped(ArtifactSkip::Cancelled) + } else { + match persist_graph_index_artifact( + db, + &catalog, + edge_tables_committed, + request.limits.artifact_bytes, + ) + .await + { + Ok(artifact) => artifact, + Err(error) => { + tracing::warn!( + target: "omnigraph::optimize", + error = %error, + "graph index artifact persist failed; traversals keep building in memory" + ); + ArtifactRefresh::Failed(error.to_string()) + } } - } - prepared.sort_by(|left, right| left.table_key.cmp(&right.table_key)); + }; - if !prepared.is_empty() { - let effect_concurrency = maint_concurrency().min(prepared.len()).max(1); - let effect_results: Vec> = futures::stream::iter(prepared) - .map(|work| { - let catalog = std::sync::Arc::clone(&catalog); - async move { apply_optimize_table_effects(db, catalog.as_ref(), work).await } - }) - .buffer_unordered(effect_concurrency) - .collect() - .await; - let mut outcomes = Vec::new(); - for result in effect_results { - outcomes.push(result?); + Ok(OptimizeReport { + input_commit, + input_manifest_version, + graph_commit_id, + tables, + // A cancelled run moved nothing, so the next run starts where this one did. + resume_at: visited.resume_at.filter(|_| full_catalog && !cancelled), + gate_hold, + configuration_clear, + artifact, + cancellation, + }) +} + +/// The tables a run visited, in visit order, each with its staged effect if +/// it has one, the first table the gate hold deferred, and whether a +/// cancellation stopped the pipeline. +struct VisitedTables { + tables: Vec<(DatasetOptimizeStats, Option)>, + resume_at: Option, + cancelled: bool, +} + +/// Why the pipeline stopped starting tables. +#[derive(Clone, Copy)] +enum PipelineStop { + /// A run-wide bound (the gate hold or the byte budget) stopped the run at + /// this table's turn; rewrites already started finish. + Bound(usize, DeferReason), + /// A cancellation: every table in progress is dropped where it stands. + Cancelled, +} + +type PlannedTable = (usize, Instant, Result); + +/// Plan and stage `tasks` in order under the held gates. Plans run up to +/// `concurrency.planning` tables ahead; a table's rewrite starts only after +/// every earlier table's has, with at most `concurrency.rewrite` at once. +/// When a table's turn comes after `deadline`, that table and every later +/// one are deferred and their plans, which only read, are dropped; the first +/// table always starts. A cancellation, checked before every plan and every +/// rewrite and awaited meanwhile, drops every table in progress. A table's +/// planning failure is `BeforeEffect` and its staging failure `Unpublished`: +/// either fails only that table. A failure whose evidence stays uncertain +/// fails the run. +async fn run_table_pipeline( + db: &Omnigraph, + catalog: &std::sync::Arc, + tasks: Vec, + concurrency: OptimizeConcurrency, + deadline: Option, + limits: OptimizeLimits, + cancellation: Option<&OptimizeCancellation>, +) -> Result { + let keys = tasks + .iter() + .map(|task| task.table_key.clone()) + .collect::>(); + let mut visited: Vec)>> = + (0..keys.len()).map(|_| None).collect(); + let plan = |index: usize, task: OptimizeTableTask| { + let catalog = std::sync::Arc::clone(catalog); + async move { + let started = Instant::now(); + let result = prepare_optimize_table(db, catalog.as_ref(), task, &limits).await; + (index, started, result) } - fail(&OPTIMIZE_POST_PHASE_B_PRE_MANIFEST_COMMIT)?; - let mut updates = Vec::new(); - let mut expected_versions = crate::db::manifest::ExpectedTableVersions::new(); - for outcome in &mut outcomes { - if let Some(effect) = outcome.effect.take() { - expected_versions.insert( - effect.update.identity, - crate::db::manifest::TableVersionExpectation { - table_key: effect.update.type_key.clone(), - table_version: effect.expected_version, - native_ref: crate::db::manifest::NativeRefPin::Exact(None), - }, - ); - updates.push(effect.update); + }; + let stage = |index: usize, started: Instant, work: PreparedOptimizeTable| { + let catalog = std::sync::Arc::clone(catalog); + async move { + let result = apply_optimize_table_effects(db, catalog.as_ref(), work).await; + (index, started, result) + } + }; + + let mut unplanned = tasks.into_iter().enumerate(); + let mut planning = FuturesOrdered::new(); + let mut ready: VecDeque = VecDeque::new(); + let mut staging = FuturesUnordered::new(); + let mut turns = 0usize; + let mut spent = 0_u64; + let mut stop = None; + let cancel_requested = || cancellation.is_some_and(OptimizeCancellation::is_cancelled); + + loop { + if stop.is_none() { + while planning.len() + ready.len() < concurrency.planning { + if cancel_requested() { + stop = Some(PipelineStop::Cancelled); + break; + } + let Some((index, task)) = unplanned.next() else { + break; + }; + planning.push_back(plan(index, task)); + } + while stop.is_none() && staging.len() < concurrency.rewrite { + let Some((index, started, result)) = ready.pop_front() else { + break; + }; + if cancel_requested() { + stop = Some(PipelineStop::Cancelled); + break; + } + if turns > 0 && deadline.is_some_and(|deadline| Instant::now() >= deadline) { + stop = Some(PipelineStop::Bound(index, DeferReason::GateHoldLimit)); + break; + } + // A table's bytes never pass the budget alone, so the first + // table with work always fits. + if let (Some(budget), Ok(OptimizePreparation::Work(work))) = + (limits.run_source_bytes, &result) + && spent.saturating_add(work.source_bytes.unwrap_or(0)) > budget + { + stop = Some(PipelineStop::Bound(index, DeferReason::ByteBudget)); + break; + } + turns += 1; + match result { + Ok(OptimizePreparation::Stat(mut stat)) => { + stat.duration = started.elapsed(); + visited[index] = Some((stat, None)); + } + Ok(OptimizePreparation::Work(work)) => { + spent = spent.saturating_add(work.source_bytes.unwrap_or(0)); + staging.push(stage(index, started, work)); + } + // Planning only reads. + Err(error) => { + let failure = table_failure(&keys[index], error.before_effect())?; + visited[index] = Some(( + DatasetOptimizeStats::failed( + keys[index].clone(), + failure, + started.elapsed(), + ), + None, + )); + } + } + } + match stop { + Some(PipelineStop::Bound(..)) => { + planning.clear(); + ready.clear(); + } + Some(PipelineStop::Cancelled) => { + planning.clear(); + ready.clear(); + staging.clear(); + } + None => {} } } - let any_committed = !updates.is_empty(); - let edge_committed = updates - .iter() - .any(|update| update.type_key.starts_with("edge:")); - if any_committed { - let lineage = db - .new_lineage_intent_for_branch(None, None, HistoryReleaseBytes::PRODUCTION) - .await?; - super::table_ops::commit_updates_on_branch_with_expected( - db, - None, - &updates, - &expected_versions, - None, - &authority_txn, - lineage, - super::table_ops::Publication::Publish, - ) - .await?; + // Filling planning above leaves it non-empty while any table is + // unplanned, so an empty pipeline has visited every table or stopped. + if planning.is_empty() && ready.is_empty() && staging.is_empty() { + break; } - edge_tables_committed = edge_committed; - stats.extend(outcomes.into_iter().map(|outcome| outcome.stat)); - if any_committed { - db.runtime_cache.invalidate_all().await; - if edge_committed { - db.invalidate_graph_index().await; + // The deadline stops only a table whose turn has not come. + let timer = deadline.filter(|_| turns > 0 && turns < keys.len() && stop.is_none()); + let awaiting_cancel = + cancellation.filter(|_| !matches!(stop, Some(PipelineStop::Cancelled))); + tokio::select! { + biased; + Some((index, started, result)) = staging.next(), if !staging.is_empty() => { + visited[index] = Some(match result { + Ok(mut outcome) => { + outcome.stat.duration = started.elapsed(); + (outcome.stat, outcome.effect) + } + // Staging writes only detached links no manifest names. + Err(error) => { + let error = error.with_completion_evidence(CompletionEvidence::Unpublished); + let failure = table_failure(&keys[index], error)?; + ( + DatasetOptimizeStats::failed(keys[index].clone(), failure, started.elapsed()), + None, + ) + } + }); + } + Some(planned) = planning.next(), if !planning.is_empty() && stop.is_none() => { + ready.push_back(planned); + } + () = tokio::time::sleep_until(timer.unwrap_or_else(Instant::now)), if timer.is_some() => { + stop = Some(PipelineStop::Bound(turns, DeferReason::GateHoldLimit)); + planning.clear(); + ready.clear(); + } + () = async { + if let Some(cancellation) = awaiting_cancel { + cancellation.cancelled().await; + } + }, if awaiting_cancel.is_some() => { + stop = Some(PipelineStop::Cancelled); + planning.clear(); + ready.clear(); + staging.clear(); } } } - stats.sort_by(|left, right| left.type_key.cmp(&right.type_key)); - - drop(table_guards); - drop(schema_permit); - let mut all = stats.into_iter().map(Ok).collect::>>(); - all.push(clear_manifest_auto_cleanup(db).await); + let (resume_at, unvisited) = match stop { + Some(PipelineStop::Bound(index, reason)) => (Some(keys[index].clone()), reason), + Some(PipelineStop::Cancelled) => (None, DeferReason::Cancelled), + None => (None, DeferReason::GateHoldLimit), + }; + let tables = visited + .into_iter() + .zip(keys) + .map(|(entry, key)| { + entry.unwrap_or_else(|| (DatasetOptimizeStats::deferred(key, unvisited), None)) + }) + .collect(); + Ok(VisitedTables { + tables, + resume_at, + cancelled: matches!(stop, Some(PipelineStop::Cancelled)), + }) +} - if let Err(error) = persist_graph_index_artifact(db, &catalog, edge_tables_committed).await { - tracing::warn!( - target: "omnigraph::optimize", - error = %error, - "graph index artifact persist failed; traversals keep building in memory" - ); +/// Keep a table's failure with its table when its evidence is definite; any +/// other failure, an uncertain one above all, fails the run. +fn table_failure(table_key: &str, error: OmniError) -> Result> { + match error.completion_evidence() { + Some(CompletionEvidence::BeforeEffect | CompletionEvidence::Unpublished) => { + tracing::warn!( + target: "omnigraph::optimize", + table = table_key, + error = %error, + "optimize left a table unchanged after its own failure" + ); + Ok(Arc::new(error)) + } + _ => Err(error), } - - all.into_iter().collect() } /// Build the full-catalog graph index from a fresh main snapshot and write it @@ -379,14 +1292,16 @@ pub async fn optimize_all_datasets(db: &Omnigraph) -> Result Result<()> { + limit: Option, +) -> Result { if catalog.edge_types.is_empty() { - return Ok(()); + return Ok(ArtifactRefresh::Skipped(ArtifactSkip::NoEdgeTypes)); } let uri = crate::graph_index::persist::artifact_uri(db.root_uri()); if !edge_tables_committed && db.storage_adapter().exists(&uri).await? { @@ -394,7 +1309,7 @@ async fn persist_graph_index_artifact( target: "omnigraph::optimize", "graph index artifact refresh skipped: no edge table advanced and an artifact exists" ); - return Ok(()); + return Ok(ArtifactRefresh::Unchanged); } let edge_types: std::collections::HashMap = catalog .edge_types @@ -412,27 +1327,80 @@ async fn persist_graph_index_artifact( target: "omnigraph::optimize", "graph index artifact skipped: not every declared edge table is materialized yet" ); - return Ok(()); + return Ok(ArtifactRefresh::Skipped(ArtifactSkip::IncompleteEdgeTables)); } - let index = - crate::graph_index::GraphIndex::build(&snapshot, &edge_types, db.catalog().system_columns) - .await?; - let written = - crate::graph_index::persist::save(&snapshot, db.storage_adapter(), &edge_types, &index) - .await?; - if let Some(uri) = written { - // A fresh artifact may serve scope keys whose earlier load attempt - // failed; those negative verdicts are stamp-keyed, and when this - // optimize advanced no table version the stamps (hence keys) are - // unchanged — drop them so the new object gets loaded. - db.runtime_cache.note_artifact_replaced().await; - tracing::debug!( - target: "omnigraph::optimize", - uri = %uri, - "graph index artifact persisted" - ); + let mut count = None; + if let Some(limit) = limit { + let Some((shortest, widest)) = + crate::graph_index::persist::fixed_bytes(&snapshot, &edge_types)? + else { + return Ok(ArtifactRefresh::Skipped(ArtifactSkip::IncompleteEdgeTables)); + }; + // A lower bound: every edge is two `u32` targets, and the live row + // count is the number of rows the scan returns. + let mut lower = shortest; + for edge in edge_types.keys() { + let rows = snapshot + .open_lance_dataset(&format!("edge:{edge}")) + .await? + .count_rows(None) + .await + .map_err(OmniError::storage)?; + lower = lower.saturating_add((rows as u64).saturating_mul(8)); + } + if lower > limit { + return Ok(ArtifactRefresh::Skipped(ArtifactSkip::SizeLimit { + counted: lower, + limit, + })); + } + count = Some(crate::graph_index::persist::ArtifactCount::new( + widest, + limit, + &edge_types, + )); } - Ok(()) + let Some(index) = crate::graph_index::GraphIndex::build_counted( + &snapshot, + &edge_types, + db.catalog().system_columns, + count.as_mut(), + ) + .await? + else { + let counted = count.as_ref().map_or(0, |count| count.counted()); + return Ok(ArtifactRefresh::Skipped(ArtifactSkip::SizeLimit { + counted, + limit: limit.unwrap_or(0), + })); + }; + let Some((uri, body)) = + crate::graph_index::persist::artifact_body(&snapshot, &edge_types, &index)? + else { + return Ok(ArtifactRefresh::Skipped(ArtifactSkip::IncompleteEdgeTables)); + }; + drop(index); + // The artifact key is mutable, so a write dropped mid-request would leave + // its outcome unknown: once issued, the write runs to its end even when + // this run is dropped. + let adapter = db.storage_adapter_handle(); + let write_uri = uri.clone(); + tokio::spawn(async move { adapter.write_bytes(&write_uri, &body).await }) + .await + .map_err(|error| { + OmniError::manifest_internal(format!("graph index artifact write task: {error}")) + })??; + // A fresh artifact may serve scope keys whose earlier load attempt + // failed; those negative verdicts are stamp-keyed, and when this + // optimize advanced no table version the stamps (hence keys) are + // unchanged — drop them so the new object gets loaded. + db.runtime_cache.note_artifact_replaced().await; + tracing::debug!( + target: "omnigraph::optimize", + uri = %uri, + "graph index artifact persisted" + ); + Ok(ArtifactRefresh::Refreshed) } /// Pure planning: classify drift/no-work/productive state without advancing @@ -441,6 +1409,7 @@ async fn prepare_optimize_table( db: &Omnigraph, catalog: &omnigraph_compiler::catalog::Catalog, task: OptimizeTableTask, + limits: &OptimizeLimits, ) -> Result { let snapshot = db .open_pinned_for_write(&task.full_path, &task.entry) @@ -450,10 +1419,8 @@ async fn prepare_optimize_table( // dropped property's values is compaction work even when Lance's planner // would leave the table alone. let options = CompactionOptions::default(); - let will_compact = TableStore::plan_table_compaction(snapshot.dataset(), &options) - .await? - .num_tasks() - > 0; + let plan = TableStore::plan_table_compaction(snapshot.dataset(), &options).await?; + let will_compact = plan.num_tasks() > 0; let needs_reindex = TableStore::has_foldable_unindexed_fragments(snapshot.dataset()).await?; let index_work = super::table_ops::index_work_status_on_dataset_for_catalog( db, @@ -464,7 +1431,7 @@ async fn prepare_optimize_table( .await?; if !will_compact && !needs_reindex && !index_work.needs_commit { let mut stat = - DatasetOptimizeStats::compacted(task.table_key, &CompactionMetrics::default(), false); + DatasetOptimizeStats::planned(task.table_key, &CompactionMetrics::default(), false); stat.pending_indexes = index_work.pending; super::table_ops::append_deferred_full_text_indexes( &snapshot, @@ -477,6 +1444,22 @@ async fn prepare_optimize_table( return Ok(OptimizePreparation::Stat(stat)); } + let source = plan + .tasks + .iter() + .flat_map(|task| task.fragments.iter().cloned()) + .collect::>(); + let index_work = needs_reindex || index_work.needs_commit; + let source_bytes = match measure_plan(snapshot.dataset(), &source, index_work, limits).await? { + Ok(source_bytes) => source_bytes, + Err(exceeded) => { + return Ok(OptimizePreparation::Stat(DatasetOptimizeStats::deferred( + task.table_key, + DeferReason::ResourceLimit(exceeded), + ))); + } + }; + Ok(OptimizePreparation::Work(PreparedOptimizeTable { identity: task.identity, table_key: task.table_key, @@ -485,9 +1468,123 @@ async fn prepare_optimize_table( initial_snapshot: snapshot, witness: task.witness, last_linear_version: task.entry.version_metadata.last_linear_version(), + source_bytes, })) } +/// Measure a table's plan against its ceilings, in order, and stop at the +/// first one it passes; the fragment count comes first, so it bounds the size +/// requests. Only a set ceiling is measured. On success, the source bytes when +/// a byte ceiling or the run's byte budget needs them. +async fn measure_plan( + ds: &lance::Dataset, + source: &[lance_table::format::Fragment], + index_work: bool, + limits: &OptimizeLimits, +) -> Result, ResourceExceeded>> { + let ceilings = &limits.table; + let over = |measure, measured, ceiling| { + Ok(Err(ResourceExceeded { + measure, + measured, + ceiling, + })) + }; + if let Some(ceiling) = ceilings.source_fragments + && source.len() as u64 > ceiling + { + return over( + ResourceMeasure::SourceFragments, + Some(source.len() as u64), + ceiling, + ); + } + if let Some(ceiling) = ceilings.source_rows { + let mut rows = 0_u64; + for fragment in source { + let Some(physical) = fragment.physical_rows else { + return over(ResourceMeasure::SourceRows, None, ceiling); + }; + rows = rows.saturating_add(physical as u64); + if rows > ceiling { + return over(ResourceMeasure::SourceRows, Some(rows), ceiling); + } + } + } + let mut source_bytes = None; + if let Some(ceiling) = limits.table_source_bytes() { + match source_file_bytes(ds, source, ceiling).await { + Some(bytes) if bytes > ceiling => { + return over(ResourceMeasure::SourceBytes, Some(bytes), ceiling); + } + Some(bytes) => source_bytes = Some(bytes), + None => return over(ResourceMeasure::SourceBytes, None, ceiling), + } + } + if let Some(ceiling) = ceilings.blob_bytes + && !source.is_empty() + { + match TableStore::managed_blob_bytes(ds, source.to_vec(), ceiling).await? { + Some(bytes) if bytes > ceiling => { + return over(ResourceMeasure::BlobBytes, Some(bytes), ceiling); + } + Some(_) => {} + None => return over(ResourceMeasure::BlobBytes, None, ceiling), + } + } + if let Some(ceiling) = ceilings.index_rows + && index_work + { + match ds.count_rows(None).await { + Ok(rows) if rows as u64 > ceiling => { + return over(ResourceMeasure::IndexRows, Some(rows as u64), ceiling); + } + Ok(_) => {} + Err(_) => return over(ResourceMeasure::IndexRows, None, ceiling), + } + } + Ok(Ok(source_bytes)) +} + +/// The bytes of `source`'s data, overlay and deletion files: a data file's +/// recorded size, or a HEAD of the file when none is recorded, and a HEAD of +/// each deletion file, which records no size in Lance 13. Each file resolves +/// through the collector's base rule. Counting stops once it passes +/// `ceiling`. `None` when a size cannot be read. +async fn source_file_bytes( + ds: &lance::Dataset, + source: &[lance_table::format::Fragment], + ceiling: u64, +) -> Option { + let mut total = 0_u64; + for fragment in source { + for file in fragment.referenced_lance_files() { + let size = match file.file_size_bytes.get() { + Some(size) => size.get(), + None => { + let (store, path) = + super::collector::data_file_location(ds, file).await.ok()?; + store.size(&path).await.ok()? + } + }; + total = total.saturating_add(size); + if total > ceiling { + return Some(total); + } + } + if let Some(deletion) = &fragment.deletion_file { + let (store, path) = super::collector::deletion_file_location(ds, fragment.id, deletion) + .await + .ok()?; + total = total.saturating_add(store.size(&path).await.ok()?); + if total > ceiling { + return Some(total); + } + } + } + Some(total) +} + decide_seam! { /// After one table's detached rewrite or index link committed, before the /// next link or table (RFC 0067). Nothing is published yet. @@ -513,6 +1610,7 @@ async fn apply_optimize_table_effects( let full_path = work.full_path; let base = work.initial_snapshot; let witness = work.witness; + let source_bytes = work.source_bytes; let declared_full_text = super::table_ops::declared_full_text_columns(catalog, &table_key); fail(&OPTIMIZE_BEFORE_COMPACT)?; let options = CompactionOptions::default(); @@ -583,7 +1681,8 @@ async fn apply_optimize_table_effects( fail(&OPTIMIZE_POST_TABLE_EFFECT)?; } let Some(identity) = tip_identity else { - let mut stat = DatasetOptimizeStats::compacted(table_key, &metrics, false); + let mut stat = DatasetOptimizeStats::planned(table_key, &metrics, false); + stat.source_bytes = source_bytes; stat.pending_indexes = index_work.pending; super::table_ops::append_deferred_full_text_indexes( &tip, @@ -595,7 +1694,8 @@ async fn apply_optimize_table_effects( .await?; return Ok(OptimizeEffectOutcome { stat, effect: None }); }; - let mut stat = DatasetOptimizeStats::compacted(table_key.clone(), &metrics, true); + let mut stat = DatasetOptimizeStats::planned(table_key.clone(), &metrics, true); + stat.source_bytes = source_bytes; stat.pending_indexes = index_work.pending; super::table_ops::append_deferred_full_text_indexes( &tip, @@ -660,35 +1760,72 @@ async fn clear_stale_auto_cleanup_config( Ok(true) } -/// Remove stored automatic GC configuration without changing manifest fragments. -async fn clear_manifest_auto_cleanup(db: &Omnigraph) -> Result { +decide_seam! { + /// After the run's publication, before the configuration clear reads + /// `__manifest`. A failure here must leave the published run reported, + /// with the clear `failed`. + pub static OPTIMIZE_CONFIGURATION_CLEAR = ("optimize.configuration_clear", Unreachable, [Fail]); +} + +/// Remove stored automatic GC configuration without changing manifest +/// fragments, and report what happened. A failed removal reads `__manifest` +/// again: keys still present are `Failed`, keys gone are `Cleared`, and a +/// read that fails too is `Unknown`. Nothing here fails the run. +async fn clear_manifest_auto_cleanup(db: &Omnigraph) -> ConfigurationClear { let uri = crate::db::manifest::manifest_uri(db.root_uri()); + let mut last_error = None; for attempt in 0..INTERNAL_CONFIG_RETRY_BUDGET { - let handle = db.storage().open_dataset_head(&uri, None).await?; - let mut ds = handle.into_dataset(); + if let Err(error) = fail(&OPTIMIZE_CONFIGURATION_CLEAR) { + return ConfigurationClear::Failed(error.to_string()); + } + let mut ds = match db.storage().open_dataset_head(&uri, None).await { + Ok(handle) => handle.into_dataset(), + Err(error) => return ConfigurationClear::Failed(error.to_string()), + }; match clear_stale_auto_cleanup_config(&mut ds).await { - Ok(cleared) => { - if cleared { - db.coordinator.write().await.refresh().await?; + Ok(false) => return ConfigurationClear::NothingToClear, + Ok(true) => { + if let Err(error) = db.coordinator.write().await.refresh().await { + tracing::warn!( + target: "omnigraph::optimize", + error = %error, + "coordinator refresh after the configuration clear failed" + ); } - return Ok(DatasetOptimizeStats::compacted( - "__manifest".to_string(), - &CompactionMetrics::default(), - false, - )); + return ConfigurationClear::Cleared; } Err(error) if attempt + 1 < INTERNAL_CONFIG_RETRY_BUDGET && is_retryable_lance_conflict(&error) => { + last_error = Some(error); continue; } - Err(error) => return Err(OmniError::storage(error)), + Err(error) => { + last_error = Some(error); + break; + } + } + } + let message = last_error.map_or_else( + || format!("removal exhausted {INTERNAL_CONFIG_RETRY_BUDGET} retries"), + |error| error.to_string(), + ); + match db.storage().open_dataset_head(&uri, None).await { + Ok(handle) => { + let present = handle + .dataset() + .config() + .keys() + .any(|key| key.starts_with("lance.auto_cleanup.")); + if present { + ConfigurationClear::Failed(message) + } else { + ConfigurationClear::Cleared + } } + Err(error) => ConfigurationClear::Unknown(format!("{message}; read-back failed: {error}")), } - Err(OmniError::manifest_conflict(format!( - "manifest auto-cleanup configuration removal exhausted {INTERNAL_CONFIG_RETRY_BUDGET} retries" - ))) } decide_seam! { @@ -717,7 +1854,13 @@ decide_seam! { pub async fn cleanup_all_datasets( db: &Omnigraph, options: CleanupPolicyOptions, + actor: Option<&str>, ) -> Result> { + db.enforce( + omnigraph_policy::PolicyAction::Reclaim, + &omnigraph_policy::ResourceScope::Graph, + actor, + )?; if options.keep_versions.is_none() && options.older_than.is_none() { return Err(OmniError::manifest( "cleanup requires at least one of keep_versions or older_than", @@ -1650,10 +2793,13 @@ mod tests { let bytes = store.inner.get(&path).await.unwrap().bytes().await.unwrap(); store.delete(&path).await.unwrap(); let rows = db - .cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + .cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); assert!( @@ -1667,10 +2813,13 @@ mod tests { ); store.inner.put(&path, bytes.into()).await.unwrap(); let rows = db - .cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + .cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); diff --git a/crates/omnigraph/src/db/omnigraph/repair.rs b/crates/omnigraph/src/db/omnigraph/repair.rs index ff0ac8ee8..d2d06fca2 100644 --- a/crates/omnigraph/src/db/omnigraph/repair.rs +++ b/crates/omnigraph/src/db/omnigraph/repair.rs @@ -1,27 +1,17 @@ -//! Explicit repair for uncovered published-dataset/Lance-HEAD drift. +//! The repair report: uncovered published-dataset/Lance-HEAD drift. //! //! Pending pins converge automatically: the next writer or `cleanup` promotes //! them. This module is for the different case: a dataset's Lance HEAD is //! ahead of the version recorded in `__manifest` and no published pin explains -//! the movement. `repair` classifies that uncovered drift from Lance -//! transactions and only auto-publishes maintenance-only drift when the -//! operator confirms. +//! the movement. The report classifies that drift from Lance transactions. It +//! publishes nothing: no read or write resolves a table's linear HEAD, so +//! drift never needs adopting. use lance::Dataset; use lance::dataset::transaction::Operation; use super::*; -/// Options for [`Omnigraph::repair`]. -#[derive(Debug, Clone, Copy, Default)] -pub struct RepairOptions { - /// Preview by default. With `confirm`, verified maintenance drift is - /// published to `__manifest`. - pub confirm: bool, - /// Also publish suspicious/unverifiable drift. Requires `confirm`. - pub force: bool, -} - /// Classification of a dataset's graph-manifest/Lance-HEAD state. #[derive(Debug, Clone, Copy, PartialEq, Eq)] #[non_exhaustive] @@ -65,20 +55,15 @@ impl std::fmt::Display for RepairClassification { } } -/// What repair did for a backing dataset. +/// What the repair report says about a backing dataset. #[derive(Debug, Clone, Copy, PartialEq, Eq)] #[non_exhaustive] pub enum RepairAction { - /// Nothing to do. + /// Nothing to report: no drift, or foreign drift no read or write resolves. NoOp, - /// Drift was reported but not published because this was a preview. + /// Drift the report classified and left in place. Preview, - /// Verified maintenance drift was published to `__manifest`. - Healed, - /// Suspicious/unverifiable drift was published because `force` was set. - Forced, - /// Drift was left untouched because it was not safe to publish without - /// `force`. + /// The table could not be judged; its `error` says why. Refused, } @@ -88,8 +73,6 @@ impl RepairAction { match self { Self::NoOp => "no_op", Self::Preview => "preview", - Self::Healed => "healed", - Self::Forced => "forced", Self::Refused => "refused", } } @@ -101,13 +84,13 @@ impl std::fmt::Display for RepairAction { } } -/// Per-dataset repair outcome. +/// One dataset's row of the repair report. #[derive(Debug, Clone)] #[non_exhaustive] pub struct DatasetRepairStats { /// Qualified graph type key, or `__manifest` for the system dataset. pub type_key: String, - /// Published dataset version before repair. + /// The published dataset version the registration pins. pub published_dataset_version: u64, pub lance_head_version: u64, pub classification: RepairClassification, @@ -116,13 +99,11 @@ pub struct DatasetRepairStats { pub error: Option, } -/// Whole-graph repair outcome. +/// The repair report for the whole graph. #[derive(Debug, Clone)] #[non_exhaustive] pub struct RepairStats { pub datasets: Vec, - /// New graph-manifest version if repair published any dataset pins. - pub graph_manifest_version: Option, } struct ClassificationResult { @@ -132,31 +113,27 @@ struct ClassificationResult { } struct RepairTableTask { - identity: crate::db::manifest::TableIdentity, table_key: String, full_path: String, pinned_data_version: u64, - pinned_native_ref: Option, entry: crate::db::manifest::DatasetEntry, } -pub async fn repair_all_datasets(db: &Omnigraph, options: RepairOptions) -> Result { - if options.force && !options.confirm { - return Err(OmniError::manifest("repair --force requires --confirm")); - } - +/// The repair report on `main`, under `maintain` on `main`. +pub(super) async fn repair_report(db: &Omnigraph, actor: Option<&str>) -> Result { + db.enforce( + omnigraph_policy::PolicyAction::Maintain, + &omnigraph_policy::ResourceScope::Branch("main".to_string()), + actor, + )?; db.ensure_schema_state_valid().await?; - // Repair may adopt physical HEADs into graph authority. Bind the entire - // attempt to one accepted view and revalidate after schema -> main -> table - // gates so concurrent graph movement refuses before publication. + // Bind the whole report to one accepted view, revalidated after the + // schema -> main -> table gates. The accepted catalog, the identity/path + // pairs and every raw Lance read stay under those gates, so a concurrent + // schema apply cannot pair an old dataset path with a replacement's alias, + // and a cleanup cannot remove a version the report is reading. let authority_txn = db.open_write_txn(None).await?; - - // Repair publishes manifest authority, so it joins the canonical writer - // envelope. The accepted catalog, identity/path pairs, raw Lance reads, and - // final publish all remain under schema -> main -> sorted-table gates. This - // prevents a concurrent drop/re-add from pairing the old dataset path with - // the replacement's same public alias and new identity. let _schema_permit = db.write_queue().acquire_schema_shared().await; db.refresh_coordinator_only().await?; let catalog = db.load_accepted_catalog_with_schema_gate_held().await?; @@ -175,41 +152,26 @@ pub async fn repair_all_datasets(db: &Omnigraph, options: RepairOptions) -> Resu .filter_map(|table_key| { let entry = snapshot.dataset(&table_key)?; Some(RepairTableTask { - identity: entry.identity, table_key, full_path: format!("{}/{}", db.root_uri, entry.dataset_path), pinned_data_version: entry.published_dataset_version, - pinned_native_ref: entry.native_dataset_branch.clone(), entry: entry.clone(), }) }) .collect::>(); - if table_tasks.is_empty() { - return Ok(RepairStats { - datasets: Vec::new(), - graph_manifest_version: None, - }); - } - let mut tables = Vec::with_capacity(table_tasks.len()); - let mut updates = Vec::new(); - let mut expected = crate::db::manifest::ExpectedTableVersions::new(); - let mut any_forced = false; - for task in table_tasks { let RepairTableTask { - identity, table_key, full_path, pinned_data_version, - pinned_native_ref, entry, } = task; // `classify_drift` inspects raw Lance transaction history // (`read_transaction_by_version`), a Lance-only maintenance read the // staged-write trait does not surface. The raw borrow is an enumerated, - // read-only escape; repair never takes ownership or moves Lance HEAD. + // read-only escape; the report never takes ownership or moves Lance HEAD. let handle = db.storage().open_dataset_head(&full_path, None).await?; if let Some(row) = judge_against_last_linear_version(&table_key, &entry, &handle)? { tables.push(row); @@ -262,90 +224,18 @@ pub async fn repair_all_datasets(db: &Omnigraph, options: RepairOptions) -> Resu } let classification = classify_drift(ds, pinned_data_version, lance_head_version).await; - let action = match ( - options.confirm, - options.force, - classification.classification, - ) { - (false, _, _) => RepairAction::Preview, - (true, _, RepairClassification::VerifiedMaintenance) => RepairAction::Healed, - (true, true, RepairClassification::Suspicious | RepairClassification::Unverifiable) => { - any_forced = true; - RepairAction::Forced - } - (true, _, RepairClassification::Suspicious | RepairClassification::Unverifiable) => { - RepairAction::Refused - } - (true, _, RepairClassification::NoDrift) => RepairAction::NoOp, - // Reported above, before drift is classified; never adopted. - (true, _, RepairClassification::BlockedPromotion) => RepairAction::Refused, - (true, _, RepairClassification::ForeignDrift) => RepairAction::NoOp, - }; - - if matches!(action, RepairAction::Healed | RepairAction::Forced) { - let state = db.storage().table_state(&full_path, &handle).await?; - updates.push(crate::db::DatasetUpdate { - identity, - type_key: table_key.clone(), - published_dataset_version: state.version, - native_dataset_branch: None, - entity_count: state.row_count, - version_metadata: state - .version_metadata - .with_last_linear_version(entry.version_metadata.last_linear_version()), - }); - expected.insert( - identity, - crate::db::manifest::TableVersionExpectation { - table_key: table_key.clone(), - table_version: pinned_data_version, - native_ref: crate::db::manifest::NativeRefPin::Exact(pinned_native_ref), - }, - ); - } - tables.push(DatasetRepairStats { type_key: table_key, published_dataset_version: pinned_data_version, lance_head_version, classification: classification.classification, - action, + action: RepairAction::Preview, operations: classification.operations, error: classification.error, }); } - let manifest_version = if updates.is_empty() { - None - } else { - let actor = if any_forced { - Some("omnigraph:repair:force") - } else { - Some("omnigraph:repair") - }; - let PublishedSnapshot { - graph_manifest_version: manifest_version, - .. - } = db - .coordinator - .write() - .await - .commit_updates_with_actor_with_expected(&updates, &expected, actor) - .await?; - db.runtime_cache.invalidate_all().await; - if updates - .iter() - .any(|update| update.type_key.starts_with("edge:")) - { - db.invalidate_graph_index().await; - } - Some(manifest_version) - }; - - Ok(RepairStats { - datasets: tables, - graph_manifest_version: manifest_version, - }) + Ok(RepairStats { datasets: tables }) } /// The detached-only rule: the recorded last linear version is the baseline diff --git a/crates/omnigraph/src/db/omnigraph/table_ops.rs b/crates/omnigraph/src/db/omnigraph/table_ops.rs index 2f8ff3055..a0a5f3bf7 100644 --- a/crates/omnigraph/src/db/omnigraph/table_ops.rs +++ b/crates/omnigraph/src/db/omnigraph/table_ops.rs @@ -36,14 +36,17 @@ pub(super) async fn graph_index_for_resolved( .await } -pub(super) async fn ensure_indices(db: &Omnigraph) -> Result { +pub(super) async fn ensure_indices_as( + db: &Omnigraph, + actor: Option<&str>, +) -> Result { let current_branch = db .coordinator .read() .await .current_branch() .map(str::to_string); - ensure_indices_on_as(db, current_branch.as_deref().unwrap_or("main"), None).await + ensure_indices_on_as(db, current_branch.as_deref().unwrap_or("main"), actor).await } pub(super) async fn ensure_indices_on_as( @@ -54,7 +57,7 @@ pub(super) async fn ensure_indices_on_as( let branch = normalize_branch_name(branch)?; let public_branch = branch.as_deref().unwrap_or("main"); db.enforce( - omnigraph_policy::PolicyAction::Change, + omnigraph_policy::PolicyAction::Maintain, &omnigraph_policy::ResourceScope::Branch(public_branch.to_string()), actor, )?; @@ -168,8 +171,13 @@ pub(super) async fn rebuild_full_text_indices_on_as( ) -> Result { let branch = normalize_branch_name(branch)?; let public_branch = branch.as_deref().unwrap_or("main"); - db.enforce( - omnigraph_policy::PolicyAction::Change, + // `maintain` is the rebuild's grant; `change`, its grant before + // `maintain` existed, still admits it so existing policies keep working. + db.enforce_any( + &[ + omnigraph_policy::PolicyAction::Maintain, + omnigraph_policy::PolicyAction::Change, + ], &omnigraph_policy::ResourceScope::Branch(public_branch.to_string()), actor, )?; @@ -617,6 +625,7 @@ async fn maintain_indices_for_branch( &txn, lineage, Publication::Publish, + None, ) .await?; Some(published.graph_commit_id) @@ -1578,10 +1587,28 @@ async fn publish_prepared( } } +/// What a publication's before-publish hook learns: the lineage intent's +/// nonce, which every attempt's history id carries +/// (`omnigraph_core::graph_commit_id::commit_id_answers` matches one to it), +/// and the head every attempt must read. +#[derive(Debug, Clone)] +pub(crate) struct PublicationIntent { + pub(crate) nonce: String, + pub(crate) input_commit: Option, + pub(crate) input_manifest_version: u64, +} + +/// A hook the shared publication helper calls once before publishing. +pub(crate) type BeforePublishHook<'a> = &'a ( + dyn Fn(PublicationIntent) -> futures::future::BoxFuture<'static, Result<()>> + Send + Sync + ); + /// Commit updates with a publisher-level OCC fence. The /// `expected_table_versions` map asserts the manifest's pre-write per-table /// versions; mismatches surface as `ManifestConflictDetails::PublishedDatasetVersionMismatch`. /// `publication` chooses the head-advancing publish or an unpublished commit. +/// `before_publish`, when given, runs once after the updates are prepared and +/// before the publisher's first attempt; its failure publishes nothing. pub(super) async fn commit_updates_on_branch_with_expected( db: &Omnigraph, branch: Option<&str>, @@ -1591,6 +1618,7 @@ pub(super) async fn commit_updates_on_branch_with_expected( txn: &crate::db::WriteTxn, lineage_intent: crate::db::manifest::LineageIntent, publication: Publication, + before_publish: Option>, ) -> Result { if branch != txn.branch.as_deref() { return Err(OmniError::manifest_internal( @@ -1604,6 +1632,25 @@ pub(super) async fn commit_updates_on_branch_with_expected( .await .map_err(OmniError::unpublished)?; + // Once, not per attempt: every attempt reads the same head and carries + // the same nonce, so another call would record no new fact. The hook runs + // before the coordinator lock, and the publisher has not been called, so + // whatever the hook's own error says, the graph did not move. + if let Some(before_publish) = before_publish { + before_publish(PublicationIntent { + nonce: lineage_intent.graph_commit_id.clone(), + input_commit: txn.authority.graph_head.clone(), + input_manifest_version: txn.base.graph_manifest_version(), + }) + .await + .map_err(|error| { + error + .into_completion_evidence() + .0 + .with_completion_evidence(crate::error::CompletionEvidence::Unpublished) + })?; + } + debug_assert_eq!(lineage_intent.actor_id.as_deref(), actor_id); let changes = prepared .iter() diff --git a/crates/omnigraph/src/db/upgrade/tests.rs b/crates/omnigraph/src/db/upgrade/tests.rs index 39ab4cb9d..30fa83bb8 100644 --- a/crates/omnigraph/src/db/upgrade/tests.rs +++ b/crates/omnigraph/src/db/upgrade/tests.rs @@ -2752,7 +2752,7 @@ async fn cleanup_after_upgrade_with_keep_four_and_older_than() { .collect(); assert_eq!(links.len(), 1, "{links:?}"); for (round, options) in [week, keep(4)].into_iter().enumerate() { - let stats = graph.cleanup(options).await.unwrap(); + let stats = graph.cleanup_as(options, None).await.unwrap(); let reclaimed = |table_key: &str| { if round == 0 && table_key == PERSON { links.len() as u64 @@ -2800,7 +2800,7 @@ async fn merge_with_legacy_base_pins_and_collector_keeps_it() { "the merge base of feature and main pins Person version {}: {roots:?}", source.a_root ); - let stats = graph.cleanup(keep(1)).await.unwrap(); + let stats = graph.cleanup_as(keep(1), None).await.unwrap(); assert!(stats.iter().all(|row| row.error.is_none()), "{stats:?}"); session(&graph) @@ -2897,7 +2897,7 @@ async fn leftover_merge_input_tag_on_bookkeeping_version_resolves() { roots.is_superset(&[source.b_root, source.c_root].into()), "{roots:?}" ); - let stats = graph.cleanup(keep(1)).await.unwrap(); + let stats = graph.cleanup_as(keep(1), None).await.unwrap(); assert!(stats.iter().all(|row| row.error.is_none()), "{stats:?}"); assert_eq!( open(root, None) diff --git a/crates/omnigraph/src/engine/plan_source.rs b/crates/omnigraph/src/engine/plan_source.rs index fcd88b8a7..f4bb53522 100644 --- a/crates/omnigraph/src/engine/plan_source.rs +++ b/crates/omnigraph/src/engine/plan_source.rs @@ -926,7 +926,7 @@ query people() { match { $p: Person } return { count($p) as n } } ) .await .unwrap(); - session.ensure_indices().await.unwrap(); + session.ensure_indices_as(None).await.unwrap(); let settings = SessionSettings::default(); let (view, catalog) = db .capture_read_view(ReadTarget::branch("main")) diff --git a/crates/omnigraph/src/graph_index/mod.rs b/crates/omnigraph/src/graph_index/mod.rs index 5971116b5..92b1c7969 100644 --- a/crates/omnigraph/src/graph_index/mod.rs +++ b/crates/omnigraph/src/graph_index/mod.rs @@ -183,6 +183,22 @@ impl GraphIndex { edge_types: &HashMap, // edge_name → (from_type, to_type) system_columns: SystemColumns, ) -> Result { + Self::build_counted(snapshot, edge_types, system_columns, None) + .await? + .ok_or_else(|| OmniError::manifest_internal("an uncounted graph index build stopped")) + } + + /// Build as [`Self::build`] does, reading each edge table one scan batch + /// at a time. With `count`, each batch's growth of the persisted artifact + /// is added to it, and once the count passes its limit the build drops + /// what it holds and returns `None`, before it keeps another batch and + /// before the adjacency arrays are allocated. + pub(crate) async fn build_counted( + snapshot: &Snapshot, + edge_types: &HashMap, + system_columns: SystemColumns, + mut count: Option<&mut persist::ArtifactCount>, + ) -> Result> { // Counted here — not at the cache-miss site — so the probe counts // actual edge-table scan builds, never persisted-artifact loads. crate::instrumentation::record_graph_build(edge_types.len()); @@ -209,15 +225,12 @@ impl GraphIndex { let ds = snapshot.open_lance_dataset(&table_key).await?; - let batches: Vec = ds + let mut batches = ds .scan() .project(&[system_columns.src, system_columns.dst]) .map_err(OmniError::storage)? .try_into_stream() .await - .map_err(OmniError::storage)? - .try_collect() - .await .map_err(OmniError::storage)?; type_indices @@ -228,9 +241,11 @@ impl GraphIndex { .or_insert_with(TypeIndex::new); let mut edges: Vec<(u32, u32)> = Vec::new(); - for batch in &batches { - let srcs = string_column(batch, system_columns.src)?; - let dsts = string_column(batch, system_columns.dst)?; + while let Some(batch) = batches.try_next().await.map_err(OmniError::storage)? { + let from_before = type_indices[from_type].len(); + let to_before = type_indices[to_type].len(); + let srcs = string_column(&batch, system_columns.src)?; + let dsts = string_column(&batch, system_columns.dst)?; for i in 0..batch.num_rows() { let src_dense = type_indices @@ -243,6 +258,17 @@ impl GraphIndex { .get_or_insert(dsts.value(i)); edges.push((src_dense, dst_dense)); } + if let Some(count) = count.as_deref_mut() { + count.add_ids(from_type, &type_indices[from_type].ids()[from_before..]); + // A type on both ends of the edge has one dictionary. + if to_type != from_type { + count.add_ids(to_type, &type_indices[to_type].ids()[to_before..]); + } + count.add_edges(batch.num_rows() as u64); + if count.passed() { + return Ok(None); + } + } } edge_pairs.insert(edge_name.clone(), edges); } @@ -264,11 +290,11 @@ impl GraphIndex { csc.insert(edge_name.clone(), CsrIndex::build(dst_count, &reversed)); } - Ok(Self { + Ok(Some(Self { type_indices, csr, csc, - }) + })) } /// Load the persisted adjacency artifact when one matches this snapshot's diff --git a/crates/omnigraph/src/graph_index/persist.rs b/crates/omnigraph/src/graph_index/persist.rs index 18890cd0c..89ef34ab4 100644 --- a/crates/omnigraph/src/graph_index/persist.rs +++ b/crates/omnigraph/src/graph_index/persist.rs @@ -638,31 +638,167 @@ pub(crate) async fn load( } } -/// Persist `index` (covering `edge_types`) for this snapshot. Called by -/// `optimize` only (see the module contract). `Ok(None)` when any covered -/// edge table is absent from the snapshot — a partially-materialized store -/// has no complete identity to stamp. Errors are the caller's to log; a -/// failed write costs nothing but the next cold build. -pub(crate) async fn save( +/// One stamp per covered edge type, sorted; `None` when any covered edge +/// table is absent from the snapshot, which has then no complete identity to +/// stamp. +fn snapshot_stamps( snapshot: &Snapshot, - adapter: &dyn StorageAdapter, edge_types: &HashMap, - index: &GraphIndex, -) -> Result> { +) -> Option> { let mut tables = Vec::with_capacity(edge_types.len()); for (edge_name, (from_type, to_type)) in edge_types { - match snapshot_stamp(snapshot, edge_name, from_type, to_type) { - Some(stamp) => tables.push(stamp), - None => return Ok(None), - } + tables.push(snapshot_stamp(snapshot, edge_name, from_type, to_type)?); } tables.sort_by(|a, b| a.edge_name.cmp(&b.edge_name)); - let uri = artifact_uri(snapshot.root_uri()); - let body = serialize(index, tables)?; + Some(tables) +} + +/// The artifact's URI and bytes for `index` (covering `edge_types`) at this +/// snapshot; `Ok(None)` when any covered edge table is absent from the +/// snapshot. +pub(crate) fn artifact_body( + snapshot: &Snapshot, + edge_types: &HashMap, + index: &GraphIndex, +) -> Result)>> { + let Some(tables) = snapshot_stamps(snapshot, edge_types) else { + return Ok(None); + }; + Ok(Some(( + artifact_uri(snapshot.root_uri()), + serialize(index, tables)?, + ))) +} + +/// Persist `index` (covering `edge_types`) for this snapshot in one call. +/// `Ok(None)` when any covered edge table is absent from the snapshot — a +/// partially-materialized store has no complete identity to stamp. Tests +/// only: `optimize`, the one writer (see the module contract), writes +/// [`artifact_body`] from a task it awaits, so a dropped run cannot drop the +/// write. +#[cfg(test)] +pub(crate) async fn save( + snapshot: &Snapshot, + adapter: &dyn StorageAdapter, + edge_types: &HashMap, + index: &GraphIndex, +) -> Result> { + let Some((uri, body)) = artifact_body(snapshot, edge_types, index)? else { + return Ok(None); + }; adapter.write_bytes(&uri, &body).await?; Ok(Some(uri)) } +/// The bytes `serialize` writes for `edge_types` at this snapshot before any +/// edge: the 20-byte prefix, every section prelude, one extra `u32` per offset +/// array, and the header. The first value has the header's count and length +/// fields at their shortest, a lower bound on the object; the second at their +/// widest, a safe start for [`ArtifactCount`]. `None` when any covered edge +/// table is absent. +pub(crate) fn fixed_bytes( + snapshot: &Snapshot, + edge_types: &HashMap, +) -> Result> { + let Some(tables) = snapshot_stamps(snapshot, edge_types) else { + return Ok(None); + }; + fixed_bytes_for(&tables).map(Some) +} + +fn fixed_bytes_for(tables: &[TableStamp]) -> Result<(u64, u64)> { + let mut type_names = tables + .iter() + .flat_map(|table| [table.from_type.as_str(), table.to_type.as_str()]) + .collect::>(); + type_names.sort_unstable(); + type_names.dedup(); + let mut fixed = (MAGIC.len() + 4 + 8) as u64; + for name in &type_names { + fixed += 1 + 4 + name.len() as u64; + } + for table in tables { + let stamp_json = serde_json::to_vec(table).map_err(|e| { + OmniError::manifest(format!("graph index artifact stamp serialize: {e}")) + })?; + // The prelude, then the extra offset entry of the CSR and the CSC. + fixed += 1 + 4 + table.edge_name.len() as u64 + 4 + stamp_json.len() as u64 + 2 * 4; + } + let header = |field: u64| -> Result { + serde_json::to_vec(&Header { + tables: tables.to_vec(), + payload_sha256_b64: base64::engine::general_purpose::STANDARD.encode([0u8; 32]), + dicts: type_names + .iter() + .map(|name| DictMeta { + type_name: (*name).to_string(), + count: field, + len: field, + }) + .collect(), + adjacencies: tables + .iter() + .map(|table| AdjMeta { + edge_name: table.edge_name.clone(), + lens: [field; 4], + }) + .collect(), + }) + .map(|header| header.len() as u64) + .map_err(|e| OmniError::manifest(format!("graph index artifact header serialize: {e}"))) + }; + Ok((fixed + header(0)?, fixed + header(u64::MAX)?)) +} + +/// A running upper bound on the artifact a counting build would write. It +/// starts at the fixed bytes with the widest header and only grows: each new +/// id of a node type adds its dictionary entry and one `u32` in every CSR +/// offset array whose source type it is and every CSC offset array whose +/// destination type it is; each edge adds its two targets. +pub(crate) struct ArtifactCount { + counted: u64, + limit: u64, + per_id: HashMap, +} + +impl ArtifactCount { + pub(crate) fn new( + fixed_widest: u64, + limit: u64, + edge_types: &HashMap, + ) -> Self { + let mut per_id = HashMap::new(); + for (from_type, to_type) in edge_types.values() { + *per_id.entry(from_type.clone()).or_insert(0) += 4; + *per_id.entry(to_type.clone()).or_insert(0) += 4; + } + Self { + counted: fixed_widest, + limit, + per_id, + } + } + + pub(crate) fn add_ids(&mut self, type_name: &str, ids: &[String]) { + let widen = self.per_id.get(type_name).copied().unwrap_or(0); + for id in ids { + self.counted = self.counted.saturating_add(4 + id.len() as u64 + widen); + } + } + + pub(crate) fn add_edges(&mut self, edges: u64) { + self.counted = self.counted.saturating_add(edges.saturating_mul(8)); + } + + pub(crate) fn passed(&self) -> bool { + self.counted > self.limit + } + + pub(crate) fn counted(&self) -> u64 { + self.counted + } +} + #[cfg(test)] mod tests { use super::*; @@ -713,6 +849,37 @@ mod tests { } } + /// The pre-check's lower bound and the counting build's bound bracket the + /// bytes `serialize` writes. + #[test] + fn size_bounds_bracket_the_serialized_artifact() { + let index = sample_index(); + let tables = vec![sample_stamp()]; + let actual = serialize(&index, tables.clone()).unwrap().len() as u64; + let (shortest, widest) = fixed_bytes_for(&tables).unwrap(); + let edges = 3; + assert!( + shortest + 8 * edges <= actual, + "{shortest} + edges vs {actual}" + ); + let mut edge_types = HashMap::new(); + edge_types.insert( + "Knows".to_string(), + ("Person".to_string(), "Person".to_string()), + ); + let mut count = ArtifactCount::new(widest, u64::MAX, &edge_types); + count.add_ids("Person", index.parts().0["Person"].ids()); + count.add_edges(edges); + assert!(actual <= count.counted(), "{actual} vs {}", count.counted()); + // Exact apart from the header's count and length digits. + let digits = 19 * (2 + 4); + assert!( + count.counted() - actual <= digits, + "{}", + count.counted() - actual + ); + } + /// Header/payload halves round-trip. Stamp verification against a live /// snapshot rides the integration tests in `tests/integration/traversal_adaptive.rs`. #[test] diff --git a/crates/omnigraph/src/runtime_cache.rs b/crates/omnigraph/src/runtime_cache.rs index be6267f34..b17466183 100644 --- a/crates/omnigraph/src/runtime_cache.rs +++ b/crates/omnigraph/src/runtime_cache.rs @@ -648,7 +648,9 @@ edge Likes: Person -> Person {} db.load_jsonl(DATA, crate::loader::LoadMode::Overwrite) .await .unwrap(); - db.optimize().await.unwrap(); + db.optimize_as(None, crate::db::OptimizeRequest::default()) + .await + .unwrap(); // Fresh handle: cold in-memory caches, the artifact stays on the store. let db = crate::db::Omnigraph::open(dir.path().to_str().unwrap()) .await diff --git a/crates/omnigraph/src/seams/catalog.rs b/crates/omnigraph/src/seams/catalog.rs index df637aa8f..50a621b12 100644 --- a/crates/omnigraph/src/seams/catalog.rs +++ b/crates/omnigraph/src/seams/catalog.rs @@ -50,6 +50,7 @@ omnigraph_seams::catalog! { crate::db::omnigraph::optimize::CLEANUP_POST_UNATTACHED_RELEASE, crate::db::omnigraph::optimize::CLEANUP_TABLE_GC, crate::db::omnigraph::optimize::OPTIMIZE_BEFORE_COMPACT, + crate::db::omnigraph::optimize::OPTIMIZE_CONFIGURATION_CLEAR, crate::db::omnigraph::optimize::OPTIMIZE_POST_AUTHORITY_CAPTURE_PRE_GATES, crate::db::omnigraph::optimize::OPTIMIZE_POST_TABLE_EFFECT, crate::db::omnigraph::optimize::OPTIMIZE_POST_PHASE_B_PRE_MANIFEST_COMMIT, diff --git a/crates/omnigraph/src/table_store.rs b/crates/omnigraph/src/table_store.rs index 1acf1d746..5e72959e2 100644 --- a/crates/omnigraph/src/table_store.rs +++ b/crates/omnigraph/src/table_store.rs @@ -4499,6 +4499,41 @@ impl TableStore { /// nothing); after an early stop, a lower bound that already derives one-row batches. /// `None` when a nested or non-Blob-v2 Blob field cannot be sized. async fn max_row_blob_bytes(ds: &Dataset, fragments: Vec) -> Result> { + let mut max = 0_u64; + let sized = Self::visit_row_blob_bytes(ds, fragments, |rows| { + max = rows.iter().copied().fold(max, u64::max); + compaction_blob_batch_rows(max) != 1 + }) + .await?; + Ok(sized.then_some(max)) + } + + /// Managed Blob bytes in `fragments`, from descriptors (externals count nothing and + /// are never fetched); the count stops once it passes `ceiling`, so a value above + /// `ceiling` is a lower bound. `None` when a nested or non-Blob-v2 Blob field cannot + /// be sized. + pub(crate) async fn managed_blob_bytes( + ds: &Dataset, + fragments: Vec, + ceiling: u64, + ) -> Result> { + let mut total = 0_u64; + let sized = Self::visit_row_blob_bytes(ds, fragments, |rows| { + total = rows.iter().fold(total, |sum, row| sum.saturating_add(*row)); + total <= ceiling + }) + .await?; + Ok(sized.then_some(total)) + } + + /// Scan the Blob descriptors of `fragments` and hand each batch's per-row managed + /// bytes to `visit`, which returns false to stop. Returns false, scanning nothing, + /// when a nested or non-Blob-v2 Blob field cannot be sized. + async fn visit_row_blob_bytes( + ds: &Dataset, + fragments: Vec, + mut visit: impl FnMut(&[u64]) -> bool, + ) -> Result { if nested_blob_field(ds.schema()).is_some() || ds .schema() @@ -4506,7 +4541,7 @@ impl TableStore { .iter() .any(|field| field.is_blob() && !field.is_blob_v2()) { - return Ok(None); + return Ok(false); } let blob_columns = ds .schema() @@ -4515,13 +4550,15 @@ impl TableStore { .filter(|field| field.is_blob()) .map(|field| field.name.as_str()) .collect::>(); + if blob_columns.is_empty() { + return Ok(true); + } let mut stream = Self::scan_stream_with(ds, Some(&blob_columns), None, None, false, |scanner| { scanner.with_fragments(fragments); Ok(()) }) .await?; - let mut max = 0_u64; while let Some(batch) = stream.try_next().await.map_err(OmniError::storage)? { let mut row_bytes = vec![0_u64; batch.num_rows()]; for (field, column) in batch.schema().fields().iter().zip(batch.columns()) { @@ -4542,12 +4579,11 @@ impl TableStore { } } } - max = row_bytes.into_iter().fold(max, u64::max); - if compaction_blob_batch_rows(max) == 1 { + if !visit(&row_bytes) { break; } } - Ok(Some(max)) + Ok(true) } /// Stage creation of a new dataset without publishing its first manifest. diff --git a/crates/omnigraph/tests/bound_storage.rs b/crates/omnigraph/tests/bound_storage.rs index 4641a513d..7b2bdefe3 100644 --- a/crates/omnigraph/tests/bound_storage.rs +++ b/crates/omnigraph/tests/bound_storage.rs @@ -303,11 +303,16 @@ async fn bound_handle_workload_child() { .contains("Feature") ); - db.optimize().await.unwrap(); - db.cleanup(CleanupPolicyOptions { - keep_versions: Some(2), - older_than: None, - }) + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(2), + older_than: None, + }, + None, + ) .await .unwrap(); let reopened = session(Omnigraph::open_with_storage(&uri, storage()).await.unwrap()); diff --git a/crates/omnigraph/tests/detached_commit_matrix.rs b/crates/omnigraph/tests/detached_commit_matrix.rs index d0a5c2792..166414040 100644 --- a/crates/omnigraph/tests/detached_commit_matrix.rs +++ b/crates/omnigraph/tests/detached_commit_matrix.rs @@ -414,15 +414,18 @@ fn rfc0067_matrix_child_process() { .block_on(async move { let db = helpers::session(Omnigraph::open(&uri).await.unwrap()); let outcome: omnigraph::error::Result<()> = match op.as_str() { - "cleanup" => db.cleanup(reclaim_everything()).await.map(|_| ()), + "cleanup" => db.cleanup_as(reclaim_everything(), None).await.map(|_| ()), "cleanup_keep" => db - .cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + .cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .map(|_| ()), - "ensure_indices" => db.ensure_indices().await.map(|_| ()), + "ensure_indices" => db.ensure_indices_as(None).await.map(|_| ()), "merge" => db .branch_merge(&format!("src_{name}"), "main") .await @@ -437,9 +440,15 @@ fn rfc0067_matrix_child_process() { .unwrap(); db.apply_prepared_schema_as(&intent, None).await.map(|_| ()) } - "optimize" => db.optimize().await.map(|_| ()), + "optimize" => db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .map(|_| ()), "load" => load_two(&db, &name).await, - "fts_rebuild" => db.rebuild_full_text_indices_on("main").await.map(|_| ()), + "fts_rebuild" => db + .rebuild_full_text_indices_on_as("main", None) + .await + .map(|_| ()), "system_column_upgrade" => db .upgrade_system_columns(upgrade_execute()) .await @@ -696,7 +705,7 @@ async fn run_cell( ) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); } // A clear cell clears a photo a put stored first. if writer == Writer::BlobClear { @@ -736,8 +745,10 @@ async fn run_cell( let outcome: omnigraph::error::Result<()> = match writer { Writer::Insert => insert(&db, &write_name).await, Writer::MultiTable => insert_and_friend(&db, &write_name).await, - Writer::Cleanup => Box::pin(db.cleanup(reclaim_everything())).await.map(|_| ()), - Writer::EnsureIndices => db.ensure_indices().await.map(|_| ()), + Writer::Cleanup => Box::pin(db.cleanup_as(reclaim_everything(), None)) + .await + .map(|_| ()), + Writer::EnsureIndices => db.ensure_indices_as(None).await.map(|_| ()), Writer::Merge => db .branch_merge(&format!("src_{write_name}"), "main") .await @@ -747,9 +758,15 @@ async fn run_cell( .apply_prepared_schema_as(prepared.as_ref().unwrap(), None) .await .map(|_| ()), - Writer::Optimize => db.optimize().await.map(|_| ()), + Writer::Optimize => db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .map(|_| ()), Writer::Load => load_two(&db, &write_name).await, - Writer::FtsRebuild => db.rebuild_full_text_indices_on("main").await.map(|_| ()), + Writer::FtsRebuild => db + .rebuild_full_text_indices_on_as("main", None) + .await + .map(|_| ()), Writer::SystemColumnUpgrade => db .upgrade_system_columns(upgrade_execute()) .await @@ -758,6 +775,14 @@ async fn run_cell( Writer::BlobClear => clear_photo(&db).await, }; acknowledged = outcome.is_ok(); + // A failure after one table's detached effect fails only that + // table, so the run itself succeeds. + if writer == Writer::Optimize && matches!(window, Window::PostDetached(_)) { + assert!( + acknowledged, + "{cell}: a table-local failure does not fail the run: {outcome:?}" + ); + } if let Err(error) = outcome { // A data writer that fails after a detached effect and before // its publication never reports an outcome it cannot @@ -879,7 +904,7 @@ async fn run_cell( } Recovery::Cleanup => { let fresh = helpers::session(Omnigraph::open(&root).await.unwrap()); - Box::pin(fresh.cleanup(reclaim_everything())) + Box::pin(fresh.cleanup_as(reclaim_everything(), None)) .await .unwrap_or_else(|error| panic!("{cell}: recovery cleanup failed: {error}")); } @@ -974,9 +999,12 @@ async fn run_cell( if writer == Writer::Optimize && recovery != Recovery::ReadOnly { let fresh = helpers::session(Omnigraph::open(&root).await.unwrap()); for run in 1..=2 { - fresh.optimize().await.unwrap_or_else(|error| { - panic!("{cell}: optimize run {run} after recovery failed: {error}") - }); + fresh + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap_or_else(|error| { + panic!("{cell}: optimize run {run} after recovery failed: {error}") + }); assert_pin_detached_and_head_unmoved(&cell, &fresh, &person_uri, head_before).await; } drop(fresh); @@ -984,12 +1012,12 @@ async fn run_cell( if writer == Writer::EnsureIndices && recovery != Recovery::ReadOnly { let fresh = helpers::session(Omnigraph::open(&root).await.unwrap()); fresh - .ensure_indices() + .ensure_indices_as(None) .await .unwrap_or_else(|error| panic!("{cell}: index pass after recovery failed: {error}")); assert_pin_detached_and_head_unmoved(&cell, &fresh, &person_uri, head_before).await; fresh - .ensure_indices() + .ensure_indices_as(None) .await .unwrap_or_else(|error| panic!("{cell}: second index pass failed: {error}")); drop(fresh); @@ -998,7 +1026,7 @@ async fn run_cell( let fresh = helpers::session(Omnigraph::open(&root).await.unwrap()); for run in 1..=2 { let rebuilt = fresh - .rebuild_full_text_indices_on("main") + .rebuild_full_text_indices_on_as("main", None) .await .unwrap_or_else(|error| { panic!("{cell}: full-text rebuild run {run} after recovery failed: {error}") diff --git a/crates/omnigraph/tests/failpoints.rs b/crates/omnigraph/tests/failpoints.rs index bd385825e..945ac4a2e 100644 --- a/crates/omnigraph/tests/failpoints.rs +++ b/crates/omnigraph/tests/failpoints.rs @@ -322,7 +322,7 @@ fn rfc023_external_writer_process() { } "collector_advance_merge_source" => { db.load("source", &payload, mode).await.unwrap(); - let rows = db.cleanup(keep_one()).await.unwrap(); + let rows = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); } "collector_retired" => { @@ -514,7 +514,7 @@ async fn branch_start_refuses_a_commit_an_interrupted_cleanup_emptied() { { let _fp = catalog::CLEANUP_SWEEP_PRE_MANIFEST_DELETE.fire_always(); - let _ = db.cleanup(keep_one()).await; + let _ = db.cleanup_as(keep_one(), None).await; } assert!( detached_versions(&person_uri).await.contains(&eve_person), @@ -546,7 +546,7 @@ async fn branch_start_refuses_a_commit_an_interrupted_cleanup_emptied() { assert_eq!(db.branch_list().await.unwrap(), vec!["main".to_string()]); assert_eq!(version_main(&db).await.unwrap(), main_version); - let stats = db.cleanup(keep_one()).await.unwrap(); + let stats = db.cleanup_as(keep_one(), None).await.unwrap(); let person = stats .iter() .find(|row| row.type_key == "node:Person") @@ -623,7 +623,7 @@ async fn branch_delete_cleanup_failure_converges_on_retry() { ); { let _fp = catalog::CLEANUP_TABLE_GC.fire_always(); - main.cleanup(keep_one()).await.unwrap(); + main.cleanup_as(keep_one(), None).await.unwrap(); } assert!( detached_versions(&person_uri) @@ -631,7 +631,7 @@ async fn branch_delete_cleanup_failure_converges_on_retry() { .contains(&former_staging), "failed cleanup preserves the deleted branch's staging" ); - main.cleanup(keep_one()).await.unwrap(); + main.cleanup_as(keep_one(), None).await.unwrap(); assert!( !detached_versions(&person_uri) .await @@ -723,7 +723,7 @@ async fn recreate_over_a_deleted_branch_writes_without_cleanup() { "writes do not collect the former staging" ); assert!(remaining.contains(&second_staging)); - main.cleanup(keep_one()).await.unwrap(); + main.cleanup_as(keep_one(), None).await.unwrap(); let remaining = detached_versions(&person_uri).await; assert!(!remaining.contains(&first_staging)); assert!( @@ -780,7 +780,7 @@ async fn branch_delete_acknowledges_with_staging_awaiting_cleanup() { .contains(&former_staging), "delete and reopen defer collection of the branch's staging" ); - main.cleanup(keep_one()).await.unwrap(); + main.cleanup_as(keep_one(), None).await.unwrap(); assert!( !detached_versions(&person_uri) .await @@ -850,7 +850,7 @@ async fn branch_recreate_completes_while_old_staging_awaits_cleanup() { let remaining = detached_versions(&person_uri).await; assert!(remaining.contains(&first_staging)); assert!(remaining.contains(&second_staging)); - main.cleanup(keep_one()).await.unwrap(); + main.cleanup_as(keep_one(), None).await.unwrap(); let remaining = detached_versions(&person_uri).await; assert!(!remaining.contains(&first_staging)); assert!(remaining.contains(&second_staging)); @@ -923,10 +923,13 @@ async fn branch_write_ignores_unavailable_cleanup_classifier() { helpers::count_rows_branch(&db, "feature", "node:Person").await, 5 ); - db.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); let branches = lance::Dataset::open(&person_uri) @@ -970,10 +973,13 @@ async fn cleanup_isolates_single_table_failure() { // One table's version GC fails once; the sweep must isolate it. let _fp = catalog::CLEANUP_TABLE_GC.fire_once_at(1); let stats = db - .cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + .cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .expect("a single table's GC failure must not abort cleanup"); @@ -1024,10 +1030,13 @@ async fn cleanup_isolates_reconcile_failure() { // not abort. { let _fp = catalog::CLEANUP_RECONCILE_FORK.fire_once_at(1); - db.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .expect("a reconcile force-delete failure must not abort cleanup"); } @@ -1044,10 +1053,13 @@ async fn cleanup_isolates_reconcile_failure() { ); } // A second cleanup with no injected failure converges. - db.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); { @@ -1096,10 +1108,13 @@ async fn reconcile_skips_fork_when_fresh_recheck_is_unavailable_then_converges() // branch is live but unreadable) → cleanup must SKIP it, not delete. { let _fp = catalog::CLASSIFY_FRESH_READ.fire_always(); - db.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); let ds = lance::Dataset::open(&person_uri).await.unwrap(); @@ -1114,10 +1129,13 @@ async fn reconcile_skips_fork_when_fresh_recheck_is_unavailable_then_converges() } // Read succeeds now → cleanup confirms the orphan and reclaims it (converges). - db.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); { @@ -2666,7 +2684,7 @@ async fn merge_target_delete_waits_and_source_delete_does_not_issue_643() { .unwrap() .unwrap_or_else(|error| panic!("deleting the target after the merge: {error}")); } else { - let rows = db.cleanup(keep_one()).await.unwrap(); + let rows = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); assert_eq!( helpers::count_rows_branch(&db, "t", "node:Person").await, @@ -3059,7 +3077,7 @@ async fn cancelled_merge_leaves_the_target_whole_issue_643() { ) .await .unwrap_or_else(|error| panic!("{seam}: a later write on the target: {error}")); - let rows = db.cleanup(keep_one()).await.unwrap(); + let rows = db.cleanup_as(keep_one(), None).await.unwrap(); assert!( rows.iter().all(|row| row.error.is_none()), "{seam}: {rows:?}" @@ -3705,7 +3723,7 @@ async fn ensure_indices_stage_btree_failure_leaves_existing_tables_writable() { &catalog::ENSURE_INDICES_POST_STAGE_PRE_COMMIT_BTREE, ); let writer_a_db = std::sync::Arc::clone(&db); - let writer_a = tokio::spawn(async move { writer_a_db.ensure_indices().await }); + let writer_a = tokio::spawn(async move { writer_a_db.ensure_indices_as(None).await }); rendezvous.wait_until_reached().await; db.load( @@ -4043,15 +4061,9 @@ async fn schema_apply_retry_preserves_the_original_empty_create() { assert_eq!(entry.published_dataset_version, if changed { 2 } else { 1 }); assert_eq!(entry.version_metadata.staged_version().is_some(), changed); assert_eq!(entry.version_metadata.last_linear_version(), Some(1)); - recovered - .repair(omnigraph::db::RepairOptions { - confirm: true, - force: false, - }) - .await - .unwrap(); - recovered.ensure_indices().await.unwrap(); - let cleaned = recovered.cleanup(keep_one()).await.unwrap(); + recovered.repair_report_as(None).await.unwrap(); + recovered.ensure_indices_as(None).await.unwrap(); + let cleaned = recovered.cleanup_as(keep_one(), None).await.unwrap(); assert!(cleaned.iter().all(|row| row.error.is_none()), "{cleaned:?}"); drop(recovered); let reopened = helpers::session(Omnigraph::open(&uri).await.unwrap()); @@ -4475,7 +4487,10 @@ async fn optimize_pre_publish_failure_leaves_no_residue() { .version; { let _failpoint = catalog::OPTIMIZE_POST_PHASE_B_PRE_MANIFEST_COMMIT.fire_always(); - let err = db.optimize().await.unwrap_err(); + let err = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap_err(); assert!( err.to_string().contains( "injected failpoint triggered: optimize.post_phase_b_pre_manifest_commit" @@ -4502,11 +4517,12 @@ async fn optimize_pre_publish_failure_leaves_no_residue() { assert_eq!(helpers::count_rows(&db, "node:Person").await, 4); assert_eq!(helpers::count_rows(&db, "node:Company").await, 4); let stats = db - .optimize() + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) .await - .expect("the retry re-plans from the pins"); + .expect("the retry re-plans from the pins") + .tables; assert!( - stats.iter().any(|stat| stat.committed), + stats.iter().any(|stat| stat.committed()), "the retry compacts what the failed attempt staged again" ); assert_person_pin_detached(&db, "main", person_head).await; @@ -4519,6 +4535,50 @@ async fn optimize_pre_publish_failure_leaves_no_residue() { .expect("schema apply after optimize must succeed"); } +/// Once Optimize's graph commit is durable, a failed configuration clear is +/// reported beside it and fails nothing: the run returns its commit, and +/// the writer gates are released for the next writer. +#[tokio::test] +#[serial(optimize)] +async fn optimize_reports_a_failed_configuration_clear_after_its_publication() { + let _scenario = FailScenario::setup(); + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap().to_string(); + seed_two_productive_optimize_tables(&uri).await; + let db = helpers::session(Omnigraph::open(&uri).await.unwrap()); + let report = { + let _failpoint = catalog::OPTIMIZE_CONFIGURATION_CLEAR.fire_always(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .expect("a post-commit clear failure does not fail the run") + }; + let published = report + .graph_commit_id + .clone() + .expect("the productive run published"); + assert!( + matches!( + report.configuration_clear, + omnigraph::db::ConfigurationClear::Failed(_) + ), + "{:?}", + report.configuration_clear + ); + assert_eq!( + db.list_commits(None).await.unwrap()[0].graph_commit_id, + published, + "the reported commit is main's head" + ); + db.mutate( + "main", + OCC_DISJOINT_MUTATIONS, + "insert_company", + ¶ms(&[("$name", "after-optimize")]), + ) + .await + .expect("the run released its writer gates"); +} + /// RFC 0067: a lost acknowledgement of Optimize's manifest commit reports /// an error, but the detached pins are published: reads serve the compacted /// rows through them and nothing is left to compact. @@ -4533,7 +4593,7 @@ async fn optimize_lost_publish_acknowledgement_leaves_published_detached_pins() let db = helpers::session(Omnigraph::open(&uri).await.unwrap()); let before = person_head_and_published(&db, "main").await; let _failpoint = catalog::GRAPH_PUBLISH_AFTER_MANIFEST_COMMIT.fire_once_at(1); - db.optimize() + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) .await .expect_err("a lost publish acknowledgement surfaces as an error"); before @@ -4549,11 +4609,15 @@ async fn optimize_lost_publish_acknowledgement_leaves_published_detached_pins() before.1 ); assert_person_pin_detached(&recovered, "main", before.0).await; - let stats = recovered.optimize().await.unwrap(); + let stats = recovered + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap() + .tables; assert!( stats .iter() - .all(|stat| stat.type_key == "__manifest" || !stat.committed), + .all(|stat| stat.type_key == "__manifest" || !stat.committed()), "nothing is left to compact after the published compaction: {stats:?}" ); } @@ -4585,7 +4649,7 @@ node Embedding { ) .await .unwrap(); - let pending = db.ensure_indices().await.unwrap(); + let pending = db.ensure_indices_as(None).await.unwrap(); assert!( pending .iter() @@ -4598,7 +4662,9 @@ node Embedding { ) .await .unwrap(); - db.optimize().await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); let snapshot = db.snapshot_of(ReadTarget::branch("main")).await.unwrap(); let embedding = snapshot.open_dataset("node:Embedding").await.unwrap(); assert!(embedding.has_unindexed_fragments().await.unwrap()); @@ -4635,7 +4701,10 @@ node Embedding { .version; { let _failpoint = catalog::OPTIMIZE_POST_PHASE_B_PRE_MANIFEST_COMMIT.fire_once_at(1); - let error = db.optimize().await.unwrap_err(); + let error = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap_err(); assert!( !matches!(error, OmniError::RecoveryRequired { .. }), "{error}" @@ -4663,17 +4732,24 @@ node Embedding { ); drop(db); let db = helpers::session(Omnigraph::open(&uri).await.unwrap()); - let stats = db.optimize().await.unwrap(); + let stats = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap() + .tables; let work = stats .iter() .find(|stat| stat.type_key == "node:Work") .expect("Work stat present"); - assert!(work.committed, "the retry compacts Work"); + assert!(work.committed(), "the retry compacts Work"); let embedding = stats .iter() .find(|stat| stat.type_key == "node:Embedding") .expect("Embedding stat present"); - assert!(!embedding.committed, "pending-only table must stay a no-op"); + assert!( + !embedding.committed(), + "pending-only table must stay a no-op" + ); assert!( embedding .pending_indexes @@ -4692,93 +4768,180 @@ node Embedding { ); } -/// RFC 0067: a failure after one table's detached rewrite and before the -/// batch publishes leaves no residue: no pin moves, no lineage is added, -/// no linear HEAD moves, and the retry compacts every table. +/// A failure after one table's detached rewrite fails only that table: its +/// pin stays, the other table's rewrite publishes in the run's one commit, no +/// linear HEAD moves, and the next run compacts the failed table. #[tokio::test(flavor = "multi_thread", worker_threads = 4)] #[serial(optimize)] -async fn optimize_partial_table_effect_leaves_no_residue() { +async fn optimize_table_failure_fails_only_that_table_and_publishes_the_rest() { let _scenario = FailScenario::setup(); let dir = tempfile::tempdir().unwrap(); let uri = dir.path().to_str().unwrap().to_string(); seed_two_productive_optimize_tables(&uri).await; let db = helpers::session(Omnigraph::open(&uri).await.unwrap()); + let pin = |snapshot: &omnigraph::db::Snapshot, key: &str| { + snapshot.dataset(key).unwrap().published_dataset_version + }; let snapshot_before = db .snapshot_of(omnigraph::db::ReadTarget::branch("main")) .await .unwrap(); - let person_pin = snapshot_before - .dataset("node:Person") - .unwrap() - .published_dataset_version; - let company_pin = snapshot_before - .dataset("node:Company") - .unwrap() - .published_dataset_version; let commits_before = db.list_commits(Some("main")).await.unwrap().len(); - let person_uri = node_table_uri(&db, "Person").await; - let company_uri = node_table_uri(&db, "Company").await; - let person_head = helpers::open_dataset_head_exact(&person_uri, None) - .await - .version() - .version; - let company_head = helpers::open_dataset_head_exact(&company_uri, None) - .await - .version() - .version; - let _failpoint = catalog::OPTIMIZE_POST_TABLE_EFFECT.fire_once_at(1); - let error = db.optimize().await.unwrap_err(); - assert!( - !matches!(error, OmniError::RecoveryRequired { .. }), - "a partial batch before publication needs no recovery: {error}" + let mut heads = Vec::new(); + for type_name in ["Company", "Person"] { + let table_uri = node_table_uri(&db, type_name).await; + let head = helpers::open_dataset_head_exact(&table_uri, None) + .await + .version() + .version; + heads.push((table_uri, head)); + } + + let report = { + let _failpoint = catalog::OPTIMIZE_POST_TABLE_EFFECT.fire_once_at(1); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap() + }; + let failed = report + .tables + .iter() + .filter(|stat| stat.outcome == omnigraph::db::TableOptimizeOutcome::Failed) + .collect::>(); + assert_eq!(failed.len(), 1, "{report:?}"); + let failed = failed[0]; + assert_eq!( + failed.failure.as_ref().unwrap().completion_evidence(), + Some(omnigraph::error::CompletionEvidence::Unpublished), + "{failed:?}" + ); + let published = report + .tables + .iter() + .find(|stat| stat.committed()) + .expect("the other table publishes"); + assert_ne!(published.type_key, failed.type_key); + assert!(report.graph_commit_id.is_some(), "{report:?}"); + assert_eq!( + db.list_commits(Some("main")).await.unwrap().len(), + commits_before + 1, + "the finished table publishes in one commit" ); - assert_no_recovery_sidecars(dir.path()); let snapshot_after = db .snapshot_of(omnigraph::db::ReadTarget::branch("main")) .await .unwrap(); assert_eq!( - snapshot_after - .dataset("node:Person") - .unwrap() - .published_dataset_version, - person_pin + pin(&snapshot_after, &failed.type_key), + pin(&snapshot_before, &failed.type_key), + "the failed table's pin stays" ); - assert_eq!( - snapshot_after - .dataset("node:Company") - .unwrap() - .published_dataset_version, - company_pin - ); - assert_eq!( - db.list_commits(Some("main")).await.unwrap().len(), - commits_before, - "a partial Optimize publishes no lineage" + assert_ne!( + pin(&snapshot_after, &published.type_key), + pin(&snapshot_before, &published.type_key), + "the other table's rewrite is published" ); - for (uri, head_before) in [(&person_uri, person_head), (&company_uri, company_head)] { - let head = lance::Dataset::open(uri).await.unwrap().version().version; + assert_no_recovery_sidecars(dir.path()); + for (table_uri, head_before) in &heads { + let head = lance::Dataset::open(table_uri) + .await + .unwrap() + .version() + .version; assert_eq!( - head, head_before, + head, *head_before, "a detached rewrite never moves the linear HEAD" ); } + drop(db); let recovered = helpers::session(Omnigraph::open(&uri).await.unwrap()); assert_eq!(helpers::count_rows(&recovered, "node:Person").await, 4); assert_eq!(helpers::count_rows(&recovered, "node:Company").await, 4); + let retry = recovered + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); + let compacted = retry + .tables + .iter() + .filter(|stat| stat.committed()) + .map(|stat| stat.type_key.as_str()) + .collect::>(); assert_eq!( - recovered.list_commits(Some("main")).await.unwrap().len(), - commits_before + compacted, + [failed.type_key.as_str()], + "the next run compacts the failed table: {retry:?}" ); - let stats = recovered.optimize().await.unwrap(); +} + +/// A cancellation while a table's staging is under way publishes nothing: +/// every staged and unstarted table is deferred as cancelled, no pin moves, +/// and the next run publishes both tables. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial(optimize)] +async fn optimize_cancelled_mid_run_publishes_nothing_and_the_next_run_does() { + let _scenario = FailScenario::setup(); + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap().to_string(); + seed_two_productive_optimize_tables(&uri).await; + let db = helpers::session(Omnigraph::open(&uri).await.unwrap()); + let commits_before = db.list_commits(Some("main")).await.unwrap().len(); + + let cancellation = omnigraph::db::OptimizeCancellation::new(); + let rendezvous = helpers::failpoint::Rendezvous::park_first(&catalog::OPTIMIZE_BEFORE_COMPACT); + let run = { + let uri = uri.clone(); + let cancellation = cancellation.clone(); + tokio::spawn(async move { + let db = helpers::session(Omnigraph::open(&uri).await.unwrap()); + db.optimize_as( + None, + omnigraph::db::OptimizeRequest::default().with_cancellation(cancellation), + ) + .await + }) + }; + rendezvous.wait_until_reached().await; + cancellation.cancel(); + rendezvous.release(); + let report = tokio::time::timeout(std::time::Duration::from_secs(20), run) + .await + .expect("optimize task hung") + .unwrap() + .unwrap(); assert_eq!( - stats + report.cancellation, + omnigraph::db::RunCancellation::Cancelled + ); + assert_eq!(report.graph_commit_id, None, "{report:?}"); + for key in ["node:Company", "node:Person"] { + let stat = report + .tables .iter() - .filter(|stat| stat.type_key != "__manifest" && stat.committed) - .count(), + .find(|stat| stat.type_key == key) + .unwrap(); + assert_eq!( + stat.outcome, + omnigraph::db::TableOptimizeOutcome::Deferred(omnigraph::db::DeferReason::Cancelled), + "{stat:?}" + ); + } + assert!(report.tables.iter().all(|stat| !stat.committed())); + assert_eq!( + db.list_commits(Some("main")).await.unwrap().len(), + commits_before, + "a cancelled run publishes nothing" + ); + + let next = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); + assert_eq!( + next.tables.iter().filter(|stat| stat.committed()).count(), 2, - "the retry compacts both tables: {stats:?}" + "{next:?}" ); } @@ -4819,7 +4982,11 @@ async fn optimize_refuses_when_graph_authority_moves_before_its_gates() { &catalog::OPTIMIZE_POST_AUTHORITY_CAPTURE_PRE_GATES, ); let optimize_task_db = std::sync::Arc::clone(&optimize_db); - let optimize = tokio::spawn(async move { optimize_task_db.optimize().await }); + let optimize = tokio::spawn(async move { + optimize_task_db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + }); rendezvous.wait_until_reached().await; // Advance the graph head underneath it with an ordinary committed write, so @@ -4915,7 +5082,8 @@ async fn optimize_holds_main_gate_through_disjoint_table_effects() { let uri_opt = uri.clone(); let optimize = tokio::spawn(async move { let db = helpers::session(Omnigraph::open(&uri_opt).await.unwrap()); - db.optimize().await + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await }); rendezvous.wait_until_reached().await; @@ -4959,7 +5127,7 @@ async fn optimize_holds_main_gate_through_disjoint_table_effects() { 1, "queued table-disjoint Company insert must persist", ); - db.optimize() + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) .await .expect("graph must remain healthy / re-optimizable"); } @@ -4999,7 +5167,8 @@ async fn optimize_serializes_concurrent_delete_across_handles() { let uri_opt = uri.clone(); let optimize = tokio::spawn(async move { let db = helpers::session(Omnigraph::open(&uri_opt).await.unwrap()); - db.optimize().await + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await }); rendezvous.wait_until_reached().await; @@ -5059,7 +5228,7 @@ async fn optimize_serializes_concurrent_delete_across_handles() { 3, "the concurrent delete must persist (alice removed)", ); - db.optimize() + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) .await .expect("graph must remain healthy / re-optimizable"); } @@ -5413,7 +5582,7 @@ async fn ensure_indices_phase_b_failure_does_not_leak_sidecar_when_no_work_neede ) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); } // Setup: trigger the failpoint. Steady-state ensure_indices @@ -5423,7 +5592,7 @@ async fn ensure_indices_phase_b_failure_does_not_leak_sidecar_when_no_work_neede { let db = helpers::session(Omnigraph::open(&uri).await.unwrap()); let _failpoint = catalog::ENSURE_INDICES_POST_PHASE_B_PRE_MANIFEST_COMMIT.fire_always(); - let err = db.ensure_indices().await.unwrap_err(); + let err = db.ensure_indices_as(None).await.unwrap_err(); assert!( err.to_string().contains( "injected failpoint triggered: ensure_indices.post_phase_b_pre_manifest_commit" @@ -7340,7 +7509,7 @@ async fn ensure_indices_failure_before_publish_leaves_no_residue() { let version = helpers::version_main(&db).await.unwrap(); { let _failpoint = catalog::ENSURE_INDICES_POST_TABLE_EFFECT.fire_always(); - let err = db.ensure_indices().await.unwrap_err(); + let err = db.ensure_indices_as(None).await.unwrap_err(); assert!( err.to_string() .contains("injected failpoint triggered: ensure_indices.post_table_effect"), @@ -7351,7 +7520,7 @@ async fn ensure_indices_failure_before_publish_leaves_no_residue() { assert_eq!(person_head_and_published(&db, "main").await, before); assert_eq!(helpers::version_main(&db).await.unwrap(), version); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); assert_person_pin_detached(&db, "main", before.0).await; assert_eq!( person_head_and_published(&db, "main").await.1, @@ -7412,7 +7581,7 @@ async fn full_text_rebuild_on_a_branch_stages_on_the_inherited_table() { db.load_jsonl(SEARCH_DATA, LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); db.branch_create("rebuild").await.unwrap(); let main_before = doc_head_and_published(&db, "main").await; let main_entry = helpers::snapshot_main(&db) @@ -7424,7 +7593,7 @@ async fn full_text_rebuild_on_a_branch_stages_on_the_inherited_table() { let main_pin = pinned_version(&db, "main", "node:Doc").await; let branch_version = helpers::version_branch(&db, "rebuild").await.unwrap(); - db.rebuild_full_text_indices_on("rebuild") + db.rebuild_full_text_indices_on_as("rebuild", None) .await .expect("a branch rebuild stages on the inherited table and never forks"); assert!(sidecar_operation_ids(dir.path()).is_empty()); @@ -7521,7 +7690,7 @@ async fn assert_optimize_detached_pin_survives_next_write(uri: &str) { { let _failpoint = catalog::GRAPH_PUBLISH_AFTER_MANIFEST_COMMIT.fire_once_at(1); let error = db - .optimize() + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) .await .expect_err("compaction published before its acknowledgement was lost"); assert!( @@ -8607,10 +8776,13 @@ async fn rfc_0067_cleanup_reaps_aged_surplus_detached_manifests() { } } for _ in 0..2 { - db.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: Some(std::time::Duration::ZERO), - }) + db.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: Some(std::time::Duration::ZERO), + }, + None, + ) .await .unwrap(); } @@ -8675,10 +8847,13 @@ async fn rfc_0067_zero_age_cleanup_keeps_a_fresh_unpublished_detached_manifest() "the interrupted write must strand at least one detached manifest" ); - db.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: Some(std::time::Duration::ZERO), - }) + db.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: Some(std::time::Duration::ZERO), + }, + None, + ) .await .unwrap(); let after = detached_files(); @@ -8694,10 +8869,13 @@ async fn rfc_0067_zero_age_cleanup_keeps_a_fresh_unpublished_detached_manifest() .set_times(std::fs::FileTimes::new().set_modified(std::time::SystemTime::UNIX_EPOCH)) .unwrap(); } - db.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: Some(std::time::Duration::ZERO), - }) + db.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: Some(std::time::Duration::ZERO), + }, + None, + ) .await .unwrap(); let aged = detached_files(); @@ -8719,10 +8897,13 @@ fn rfc_0067_external_cleanup_process() { .unwrap() .block_on(async { let db = Omnigraph::open(&uri).await.unwrap(); - db.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: Some(std::time::Duration::ZERO), - }) + db.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: Some(std::time::Duration::ZERO), + }, + None, + ) .await .unwrap(); }); @@ -9333,20 +9514,20 @@ async fn live_handle_writes_after_every_write_family_seam_failure() { db.apply_schema(&render_schema(&schema_props, age_indexed)) .await .expect("ensure-row schema setup"); - db.ensure_indices() + db.ensure_indices_as(None) .await .map(|_| ()) .map_err(|error| error.to_string()) } Driver::Optimize => db - .optimize() + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) .await .map(|_| ()) .map_err(|error| error.to_string()), - Driver::Cleanup => Box::pin(db.cleanup(omnigraph::db::CleanupPolicyOptions { + Driver::Cleanup => Box::pin(db.cleanup_as(omnigraph::db::CleanupPolicyOptions { keep_versions: Some(1), older_than: None, - })) + }, None)) .await .map(|_| ()) .map_err(|error| error.to_string()), @@ -9422,10 +9603,13 @@ async fn rfc_0067_merge_diff_keeps_every_chunk() { after_rows - before_rows ); db.branch_delete("feature").await.unwrap(); - db.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: Some(std::time::Duration::ZERO), - }) + db.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: Some(std::time::Duration::ZERO), + }, + None, + ) .await .unwrap(); let person_uri = node_table_uri(&db, "Person").await; @@ -9465,10 +9649,13 @@ async fn rfc_0067_cleanup_window_keeps_version_gc_on_a_written_table() { for pass in 0..2 { let stats = db - .cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: Some(std::time::Duration::from_secs(7 * 24 * 60 * 60)), - }) + .cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: Some(std::time::Duration::from_secs(7 * 24 * 60 * 60)), + }, + None, + ) .await .unwrap(); let person = stats @@ -9546,7 +9733,7 @@ async fn collector_keeps_live_staging_and_reclaims_it_once_the_head_moves() { ); assert!(staging.bytes > 0); } - let stats = db.cleanup(keep_one()).await.unwrap(); + let stats = db.cleanup_as(keep_one(), None).await.unwrap(); let row = stats .iter() .find(|row| row.type_key == "node:Person") @@ -9573,7 +9760,7 @@ async fn collector_keeps_live_staging_and_reclaims_it_once_the_head_moves() { staging.verdict ); } - let stats = db.cleanup(keep_one()).await.unwrap(); + let stats = db.cleanup_as(keep_one(), None).await.unwrap(); let row = stats .iter() .find(|row| row.type_key == "node:Person") @@ -9665,7 +9852,7 @@ async fn collector_keeps_pins_published_by_a_late_branch_in_another_process() { let expected_rows = count_rows(&db, "node:Person").await + 1; let cleaner = helpers::session(Omnigraph::open(db.uri()).await.unwrap()); let (guard, hold) = catalog::CLEANUP_COLLECTOR_POST_SNAPSHOT.hold(); - let cleanup = tokio::spawn(async move { cleaner.cleanup(keep_one()).await }); + let cleanup = tokio::spawn(async move { cleaner.cleanup_as(keep_one(), None).await }); let reached = hold.clone(); tokio::task::spawn_blocking(move || reached.wait_until_reached()) .await @@ -9708,7 +9895,7 @@ async fn collector_keeps_pins_published_by_a_late_branch_in_another_process() { path.display() ); } - let retry = reader.cleanup(keep_one()).await.unwrap(); + let retry = reader.cleanup_as(keep_one(), None).await.unwrap(); assert!(retry.iter().all(|row| row.error.is_none()), "{retry:?}"); assert_eq!(count_rows(&reader, "node:Person").await, expected_rows - 1); continue; @@ -9755,7 +9942,7 @@ async fn collector_refuses_mixed_branch_captures_before_deleting() { count_rows(&db, "node:Person").await + usize::from(action == "collector_mixed"); let cleaner = helpers::session(Omnigraph::open(db.uri()).await.unwrap()); let (guard, hold) = catalog::CLEANUP_COLLECTOR_MID_BRANCH_READ.hold(); - let cleanup = tokio::spawn(async move { cleaner.cleanup(keep_one()).await }); + let cleanup = tokio::spawn(async move { cleaner.cleanup_as(keep_one(), None).await }); let reached = hold.clone(); tokio::task::spawn_blocking(move || reached.wait_until_reached()) .await @@ -9793,7 +9980,7 @@ async fn collector_refuses_mixed_branch_captures_before_deleting() { let pin = pinned_version(&reader, "main", "node:Person").await; assert!(detached_versions(&person_uri).await.contains(&pin)); assert_eq!(count_rows(&reader, "node:Person").await, expected_rows); - let retry = reader.cleanup(keep_one()).await.unwrap(); + let retry = reader.cleanup_as(keep_one(), None).await.unwrap(); assert!(retry.iter().all(|row| row.error.is_none()), "{retry:?}"); assert!(detached_versions(&person_uri).await.contains(&pin)); } @@ -9811,7 +9998,7 @@ async fn collector_pass_interrupted_mid_sweep_counts_nothing_and_retries_whole() let stats = { let _fp = catalog::CLEANUP_SWEEP_BETWEEN_MANIFESTS.fail_once_at(1); - db.cleanup(keep_one()).await.unwrap() + db.cleanup_as(keep_one(), None).await.unwrap() }; let row = stats .iter() @@ -9827,7 +10014,7 @@ async fn collector_pass_interrupted_mid_sweep_counts_nothing_and_retries_whole() row.manifests_removed, 1, "the pass stopped after its first manifest: {row:?}" ); - let stats = db.cleanup(keep_one()).await.unwrap(); + let stats = db.cleanup_as(keep_one(), None).await.unwrap(); let row = stats .iter() .find(|row| row.type_key == "node:Person") @@ -9894,7 +10081,7 @@ async fn assert_collector_retry_treats_an_index_directory_an_interrupted_pass_re let stats = { let _fp = catalog::CLEANUP_SWEEP_PRE_FILE_DELETE.fire_always(); - db.cleanup(keep_one()).await.unwrap() + db.cleanup_as(keep_one(), None).await.unwrap() }; let row = stats .iter() @@ -9927,7 +10114,7 @@ async fn assert_collector_retry_treats_an_index_directory_an_interrupted_pass_re let stats = { let _fp = catalog::CLEANUP_SWEEP_PRE_MANIFEST_DELETE.fire_always(); - db.cleanup(keep_one()).await.unwrap() + db.cleanup_as(keep_one(), None).await.unwrap() }; let row = stats .iter() @@ -9948,7 +10135,7 @@ async fn assert_collector_retry_treats_an_index_directory_an_interrupted_pass_re ); } - let stats = db.cleanup(keep_one()).await.unwrap(); + let stats = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(stats.iter().all(|row| row.error.is_none()), "{stats:?}"); let row = stats .iter() @@ -10034,7 +10221,7 @@ async fn assert_collector_retry_after_a_pass_stopped_between_manifests_finishes_ let stats = { let _fp = catalog::CLEANUP_SWEEP_BETWEEN_MANIFESTS.fire_always(); - db.cleanup(keep_one()).await.unwrap() + db.cleanup_as(keep_one(), None).await.unwrap() }; let row = stats .iter() @@ -10082,7 +10269,7 @@ async fn assert_collector_retry_after_a_pass_stopped_between_manifests_finishes_ plan.sweep_paths ); - let stats = db.cleanup(keep_one()).await.unwrap(); + let stats = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(stats.iter().all(|row| row.error.is_none()), "{stats:?}"); let row = stats .iter() @@ -10137,7 +10324,7 @@ async fn assert_collector_retry_after_a_pass_stopped_before_the_dead_manifests_r let before = detached_versions(&person_uri).await; { let _fp = catalog::ENSURE_INDICES_POST_PHASE_B_PRE_MANIFEST_COMMIT.fire_always(); - let outcome = db.ensure_indices().await; + let outcome = db.ensure_indices_as(None).await; assert!( outcome.is_err(), "the seam fails the index build before publish" @@ -10175,7 +10362,7 @@ async fn assert_collector_retry_after_a_pass_stopped_before_the_dead_manifests_r let stats = { let _fp = catalog::CLEANUP_SWEEP_PRE_DEAD_MANIFEST_DELETE.fire_always(); - db.cleanup(keep_one()).await.unwrap() + db.cleanup_as(keep_one(), None).await.unwrap() }; let row = stats .iter() @@ -10216,7 +10403,7 @@ async fn assert_collector_retry_after_a_pass_stopped_before_the_dead_manifests_r plan.dead_paths ); - let stats = db.cleanup(keep_one()).await.unwrap(); + let stats = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(stats.iter().all(|row| row.error.is_none()), "{stats:?}"); let row = stats .iter() @@ -10361,7 +10548,7 @@ async fn branch_merge_in_doubt_keeps_input_tags_until_cleanup() { "the acknowledged source and target guards must survive an in-doubt fast-forward" ); - let rows = db.cleanup(keep_one()).await.unwrap(); + let rows = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); assert!( !native @@ -10895,7 +11082,7 @@ async fn unattached_live_chain_pins_are_rooted_and_survive_cleanup() { report.retained_unattached_tags ); - let rows = db.cleanup(keep_one_at_any_age()).await.unwrap(); + let rows = db.cleanup_as(keep_one_at_any_age(), None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); let remaining = detached_versions(&person_uri).await; assert!( @@ -10940,7 +11127,7 @@ async fn unattached_live_chain_pins_are_rooted_and_survive_cleanup() { let plan = main_plan(&report, "node:Person"); assert!(plan.roots.contains(&grandchild_pin)); assert!(plan.roots.contains(&root_pin)); - let rows = db.cleanup(keep_one_at_any_age()).await.unwrap(); + let rows = db.cleanup_as(keep_one_at_any_age(), None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); let remaining = detached_versions(&person_uri).await; assert!(remaining.contains(&grandchild_pin)); @@ -11016,7 +11203,9 @@ async fn cleanup_releases_expired_unattached_tags_descendants_first() { let interrupted = { let _fp = catalog::CLEANUP_POST_UNATTACHED_RELEASE.fail_once_at(1); - db.cleanup(keep_one_at_any_age()).await.unwrap_err() + db.cleanup_as(keep_one_at_any_age(), None) + .await + .unwrap_err() }; assert!( interrupted @@ -11030,7 +11219,7 @@ async fn cleanup_releases_expired_unattached_tags_descendants_first() { "the child's tag went first; the root's stays for the retry" ); - let rows = db.cleanup(keep_one_at_any_age()).await.unwrap(); + let rows = db.cleanup_as(keep_one_at_any_age(), None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); assert!(unattached_tags(&uri).await.is_empty()); let at = |id: &str| ReadTarget::snapshot(SnapshotId::new(id.to_string())); @@ -11048,7 +11237,7 @@ async fn cleanup_releases_expired_unattached_tags_descendants_first() { person_count_at(&db, ReadTarget::branch("main")).await, main_before + 1 ); - let rows = db.cleanup(keep_one_at_any_age()).await.unwrap(); + let rows = db.cleanup_as(keep_one_at_any_age(), None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); } @@ -11112,7 +11301,7 @@ async fn abandoned_merge_input_tags_expire_after_target_advances() { ) .await .unwrap(); - let rows = db.cleanup(keep_one()).await.unwrap(); + let rows = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); assert!( captured_path.is_file(), @@ -11125,7 +11314,7 @@ async fn abandoned_merge_input_tags_expire_after_target_advances() { ) .await .unwrap(); - let rows = db.cleanup(keep_one()).await.unwrap(); + let rows = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); assert!( !captured_path.exists(), @@ -11177,7 +11366,7 @@ async fn cleanup_keeps_a_late_retired_merge_base_provider() { db.branch_create("target").await.unwrap(); let cleaner = helpers::session(Omnigraph::open(&uri).await.unwrap()); let parked = helpers::failpoint::Rendezvous::park_first(&catalog::CLEANUP_TABLE_GC); - let cleanup = tokio::spawn(async move { cleaner.cleanup(keep_one()).await }); + let cleanup = tokio::spawn(async move { cleaner.cleanup_as(keep_one(), None).await }); parked.wait_until_reached().await; let child_uri = uri.clone(); let child = tokio::task::spawn_blocking(move || { @@ -11241,7 +11430,7 @@ async fn planning_source_failures_keep_their_error_class() { ) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let lookup = params(&[("$name", "Planned")]); for seam in [ &catalog::QUERY_INDEX_FACTS_PRE_LOAD, diff --git a/crates/omnigraph/tests/forbidden_apis.rs b/crates/omnigraph/tests/forbidden_apis.rs index 578fe2cf5..814f68d3e 100644 --- a/crates/omnigraph/tests/forbidden_apis.rs +++ b/crates/omnigraph/tests/forbidden_apis.rs @@ -190,7 +190,6 @@ const SENTINEL: &str = "// forbidden-api-allow:"; enum WriteProtocol { Exact(&'static str), Composed(&'static str), - ManifestAdoption, NativeRefControl, PhysicalOnly, EphemeralScratch, @@ -205,7 +204,6 @@ impl WriteProtocol { match self { Self::Exact(name) => format!("exact adapter ({name})"), Self::Composed(name) => format!("composed protocol ({name})"), - Self::ManifestAdoption => "manifest adoption".into(), Self::NativeRefControl => "native ref control".into(), Self::PhysicalOnly => "physical-only maintenance".into(), Self::EphemeralScratch => "ephemeral scratch".into(), @@ -264,13 +262,11 @@ write_surfaces! { "db/omnigraph.rs" => WriteProtocol::Exact("unpublished commit publication: the proposal's recorded changes replayed through the manifest publisher under its captured authority, required to allocate the proposal's own id") => ["publish_commit_as"], "db/omnigraph.rs" => WriteProtocol::Composed("proposal tag removal confirmed by a readback, then merge-input tag release") => ["drop_commit_as"], "db/omnigraph.rs" => INDICES_V9 => [ - "ensure_indices", "ensure_indices_on", "ensure_indices_on_as", - "rebuild_full_text_indices_on", "rebuild_full_text_indices_on_as", + "ensure_indices_as", "ensure_indices_on_as", "rebuild_full_text_indices_on_as", ], "db/omnigraph.rs" => WriteProtocol::TestOnly => ["failpoint_publish_table_head_without_index_rebuild_for_test", "init_with_legacy_system_columns_for_tests"], - "db/omnigraph.rs" => OPTIMIZE_V9 => ["optimize"], - "db/omnigraph.rs" => WriteProtocol::ManifestAdoption => ["repair"], - "db/omnigraph.rs" => WriteProtocol::PhysicalOnly => ["cleanup"], + "db/omnigraph.rs" => OPTIMIZE_V9 => ["optimize_as"], + "db/omnigraph.rs" => WriteProtocol::PhysicalOnly => ["cleanup_as"], "db/omnigraph.rs" => WriteProtocol::NativeRefControl => ["branch_create", "branch_create_as", "branch_create_from", "branch_create_from_as", "branch_delete", "branch_delete_as"], } @@ -296,6 +292,9 @@ const READ_ONLY_SURFACES: &[(&str, &str)] = &[ ("db/omnigraph.rs", "open_read_only"), ("db/omnigraph.rs", "refresh"), ("db/omnigraph.rs", "cleanup_plan"), + // The repair report: it publishes nothing, but `maintain` gates it like + // the maintenance writers (`maintenance_entry_points_take_an_actor`). + ("db/omnigraph.rs", "repair_report_as"), ("db/omnigraph.rs", "cleanup_plan_missing_paths"), ("db/omnigraph.rs", "cleanup_plan_path_snapshot"), ("db/omnigraph.rs", "open_read_only_with_storage"), @@ -908,7 +907,7 @@ gateway_surfaces! { "predicted_materialized_blob_batch_bytes", "materialize_blob_batch_bounded_with_preflight_cache", "managed_blob_payloads", "can_fold_index", "has_foldable_unindexed_fragments", "index_is_vector", - "plan_table_compaction", + "plan_table_compaction", "managed_blob_bytes", ], "table_store.rs" => "TableStore" => GatewayDisposition::StageOnly => [ "stage_create", "stage_keyed_write", "stage_proven_strict_insert", "stage_overwrite", @@ -1014,12 +1013,6 @@ durable_calls! { ("omnigraph-catalog/commit.rs", "InsertBuilder::new(", 1, WriteProtocol::Exact("lowest manifest publisher gateway")), ("omnigraph-catalog/commit.rs", ".execute_uncommitted(", 1, WriteProtocol::Exact("lowest manifest publisher gateway")), ("omnigraph-catalog/commit.rs", "CommitBuilder::new(", 2, WriteProtocol::Exact("lowest manifest publisher gateway: the linear version with zero retries, or a detached candidate version whose only retry is the random version id")), - // The persisted CSR/CSC adjacency artifact (`__graph_index/csr-current.bin`): - // derived, regenerable topology written ONLY from `optimize`'s tail (never - // the query path, which only loads), outside graph visibility — a stale or - // partial object is rejected by its identity stamps + payload digest and - // rebuilt in memory, so this write can never change a query's result. - ("graph_index/persist.rs", ".write_bytes(", 1, WriteProtocol::PhysicalOnly), ("omnigraph-core/instrumentation.rs", ".write_bytes(", 1, WriteProtocol::Composed("instrumented storage forwarding")), ("omnigraph-core/instrumentation.rs", ".write_text(", 1, WriteProtocol::Composed("instrumented storage forwarding")), ("omnigraph-core/instrumentation.rs", ".write_text_if_absent(", 1, WriteProtocol::Composed("instrumented storage forwarding")), @@ -1092,7 +1085,6 @@ durable_calls! { ("db/omnigraph/table_ops.rs", ".commit_changes_with_intent_and_expected(", 1, WriteProtocol::Exact("shared publisher")), ("db/omnigraph/schema_apply.rs", ".commit_changes_with_intent_and_expected(", 1, SCHEMA_V9), ("db/omnigraph/schema_apply/settlement.rs", ".commit_changes_with_intent_and_expected(", 1, WriteProtocol::Exact("schema settlement: strict numeric neutral lineage publication")), - ("db/omnigraph/repair.rs", ".commit_updates_with_actor_with_expected(", 1, WriteProtocol::ManifestAdoption), ("db/graph_coordinator.rs", ".commit_changes_with_intent_and_expected(", 1, WriteProtocol::Exact("publisher gateway")), ("db/graph_coordinator.rs", ".commit_changes_with_lineage_and_precondition(", 1, WriteProtocol::Exact("lowest manifest publisher gateway")), ("omnigraph-catalog/lib.rs", ".publish_with_precondition(", 1, WriteProtocol::Exact("lowest manifest publisher gateway")), @@ -1144,11 +1136,19 @@ durable_calls! { ("db/omnigraph/collector.rs", ".dataset()", 17, WriteProtocol::ReadOnlyAccess), ("db/omnigraph/collector.rs", ".delete(", 1, WriteProtocol::Composed("detached-only sweep through the table's own Lance store: freed files, then the published manifests no retained `__manifest` version pins (links before tips) and the dead stagings; an interrupted pass is re-swept by the next")), ("db/omnigraph/schema_apply.rs", ".dataset()", 1, SCHEMA_V9), - ("db/omnigraph/repair.rs", ".dataset()", 1, WriteProtocol::ManifestAdoption), + // The repair report reads raw transaction history; it publishes nothing. + ("db/omnigraph/repair.rs", ".dataset()", 1, WriteProtocol::ReadOnlyAccess), // The sixth accessor reports deferred FTS coverage from an immutable // snapshot; it only reads index metadata and never stages or publishes. - ("db/omnigraph/optimize.rs", ".dataset()", 9, WriteProtocol::Composed("Optimize planning + read-only coverage and native-fork inventory + physical cleanup, including the release of expired proposal tags and of expired unattached tags descendants first, each confirmed by a readback")), + ("db/omnigraph/optimize.rs", ".dataset()", 11, WriteProtocol::Composed("Optimize planning, including each plan's measure against its table ceilings, + read-only coverage and native-fork inventory + the configuration clear's read-back after a failed removal + physical cleanup, including the release of expired proposal tags and of expired unattached tags descendants first, each confirmed by a readback")), ("db/omnigraph/optimize.rs", ".into_dataset()", 1, WriteProtocol::PhysicalOnly), + // The persisted CSR/CSC adjacency artifact (`__graph_index/csr-current.bin`): + // derived, regenerable topology written ONLY from `optimize`'s tail (never + // the query path, which only loads), outside graph visibility — a stale or + // partial object is rejected by its identity stamps + payload digest and + // rebuilt in memory, so this write can never change a query's result. It + // runs in a task the run awaits, so a dropped run cannot drop it. + ("db/omnigraph/optimize.rs", ".write_bytes(", 1, WriteProtocol::PhysicalOnly), ("exec/merge.rs", "SnapshotHandle::new(", 4, MERGE_V9), } @@ -2139,6 +2139,107 @@ fn graph_write_surfaces_are_registered() { ); } +/// The public surfaces of maintenance: the graph writers registered under +/// these protocols, plus the read-only repair report the same gate covers. +fn is_maintenance_protocol(protocol: WriteProtocol) -> bool { + protocol == INDICES_V9 || protocol == OPTIMIZE_V9 || protocol == WriteProtocol::PhysicalOnly +} + +const READ_ONLY_MAINTENANCE_SURFACES: &[(&str, &str)] = &[("db/omnigraph.rs", "repair_report_as")]; + +/// Maintenance has no actor-less public form (invariant 10). Every public +/// optimize, cleanup, index or repair entry point is an `_as` method taking the +/// `actor: Option<&str>` its policy check judges, so neither an embedded nor a +/// direct caller reaches maintenance around the gate; with a policy installed, +/// a `None` actor is refused. `policy_engine_chassis.rs` owns which action and +/// scope each one checks. +#[test] +fn maintenance_entry_points_take_an_actor() { + let src = engine_src_root(); + let read_only = READ_ONLY_SURFACES.iter().copied().collect::>(); + for surface in READ_ONLY_MAINTENANCE_SURFACES { + assert!( + read_only.contains(surface), + "{surface:?} must stay a registered read-only surface" + ); + } + let surfaces = WRITE_SURFACES + .iter() + .filter(|surface| is_maintenance_protocol(surface.protocol)) + .map(|surface| (surface.file, surface.function)) + .chain(READ_ONLY_MAINTENANCE_SURFACES.iter().copied()) + .collect::>(); + assert_eq!( + surfaces, + BTreeSet::from([ + ("db/omnigraph.rs", "cleanup_as"), + ("db/omnigraph.rs", "ensure_indices_as"), + ("db/omnigraph.rs", "ensure_indices_on_as"), + ("db/omnigraph.rs", "optimize_as"), + ("db/omnigraph.rs", "rebuild_full_text_indices_on_as"), + ("db/omnigraph.rs", "repair_report_as"), + ]), + "a maintenance surface was added or removed; give it an actor and a policy case" + ); + for (file, function) in surfaces { + assert!( + function.ends_with("_as"), + "{file}::{function}: maintenance entry points are `_as` forms" + ); + let path = guarded_path(&src, file); + let contents = std::fs::read_to_string(&path) + .unwrap_or_else(|error| panic!("failed to read {}: {error}", path.display())); + let ast = parse_rust_source(&contents, file); + let signatures = function_signatures(&ast, function); + assert_eq!( + signatures.len(), + 1, + "{file}::{function} must resolve to exactly one function" + ); + let takes_actor = signatures[0].inputs.iter().any(|input| { + let syn::FnArg::Typed(argument) = input else { + return false; + }; + matches!(&*argument.pat, syn::Pat::Ident(name) if name.ident == "actor") + && type_final_ident(&argument.ty).is_some_and(|ident| ident == "Option") + }); + assert!( + takes_actor, + "{file}::{function} must take `actor: Option<&str>` for its policy check" + ); + } +} + +fn function_signatures(ast: &syn::File, function_name: &str) -> Vec { + struct Finder<'a> { + function_name: &'a str, + signatures: Vec, + } + + impl<'ast> Visit<'ast> for Finder<'_> { + fn visit_item_fn(&mut self, node: &'ast syn::ItemFn) { + if node.sig.ident == self.function_name { + self.signatures.push(node.sig.clone()); + } + visit::visit_item_fn(self, node); + } + + fn visit_impl_item_fn(&mut self, node: &'ast syn::ImplItemFn) { + if node.sig.ident == self.function_name { + self.signatures.push(node.sig.clone()); + } + visit::visit_impl_item_fn(self, node); + } + } + + let mut finder = Finder { + function_name, + signatures: Vec::new(), + }; + finder.visit_file(ast); + finder.signatures +} + #[test] fn low_level_coordinator_surfaces_are_registered() { let src = engine_src_root(); @@ -2464,6 +2565,12 @@ const MUTABLE_KEY_REQUESTS: &[(&str, &str, usize, &str)] = &[ 1, "cleanup reclaims a pre-v11 table fork no registration names; fork names carry a commit ULID and are never reused", ), + ( + "db/omnigraph/optimize.rs", + ".write_bytes(", + 1, + "derived topology artifact stamped with its snapshot identity; a late older copy fails the stamp check and the reader builds in memory", + ), ( "db/omnigraph/prepared_create.rs", ".delete(", @@ -2476,12 +2583,6 @@ const MUTABLE_KEY_REQUESTS: &[(&str, &str, usize, &str)] = &[ 2, "reconciles an unpublished graph create: its tables and manifest have no published version, and nothing recreates them under this claim", ), - ( - "graph_index/persist.rs", - ".write_bytes(", - 1, - "derived topology artifact stamped with its snapshot identity; a late older copy fails the stamp check and the reader builds in memory", - ), ( "omnigraph-catalog/retention.rs", ".delete(", diff --git a/crates/omnigraph/tests/helpers/expand_projection.rs b/crates/omnigraph/tests/helpers/expand_projection.rs index 0fe1e877e..8f792dee2 100644 --- a/crates/omnigraph/tests/helpers/expand_projection.rs +++ b/crates/omnigraph/tests/helpers/expand_projection.rs @@ -63,7 +63,7 @@ pub async fn fixture(dir: &tempfile::TempDir, sources: usize, payload_bytes: usi .await .unwrap(); } - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); with_setting(&db, "engine", "v2") } diff --git a/crates/omnigraph/tests/helpers/mod.rs b/crates/omnigraph/tests/helpers/mod.rs index c39350972..af4c39238 100644 --- a/crates/omnigraph/tests/helpers/mod.rs +++ b/crates/omnigraph/tests/helpers/mod.rs @@ -347,7 +347,7 @@ pub async fn init_and_load_at(uri: &str, schema: &str) -> Session { db.load_jsonl(TEST_DATA, LoadMode::Overwrite).await.unwrap(); // Mutation/load publish only exact data effects; physical indexes are // reconciled separately as derived state. - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); db } diff --git a/crates/omnigraph/tests/integration/branching.rs b/crates/omnigraph/tests/integration/branching.rs index 9c12d0378..430d148b5 100644 --- a/crates/omnigraph/tests/integration/branching.rs +++ b/crates/omnigraph/tests/integration/branching.rs @@ -150,7 +150,7 @@ async fn init_search_db(dir: &tempfile::TempDir) -> Session { db.load_jsonl(SEARCH_DATA, LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); db } @@ -2261,10 +2261,13 @@ fn assert_native_version_case( ); let maintenance = Omnigraph::open(uri).await.unwrap(); maintenance - .cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(100), - older_than: None, - }) + .cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(100), + older_than: None, + }, + None, + ) .await .unwrap(); main.mutate( @@ -3602,7 +3605,7 @@ async fn ensure_indices_on_child_branch_keeps_inherited_table_when_no_work_is_ne .unwrap(); // The insert leaves feature's identity index lagging; building it here // leaves the child below nothing to index. - feature.ensure_indices_on("feature").await.unwrap(); + feature.ensure_indices_on_as("feature", None).await.unwrap(); feature .branch_create_from(ReadTarget::branch("feature"), "experiment") .await @@ -3626,7 +3629,10 @@ async fn ensure_indices_on_child_branch_keeps_inherited_table_when_no_work_is_ne "the child branch inherits feature's pin" ); - experiment.ensure_indices_on("experiment").await.unwrap(); + experiment + .ensure_indices_on_as("experiment", None) + .await + .unwrap(); let experiment_snap = snapshot_branch(&experiment, "experiment").await.unwrap(); assert_eq!( @@ -4213,10 +4219,13 @@ async fn branch_delete_defers_pin_collection_and_allows_recreate() { "recreated writes leave the former pin intact" ); assert!(detached.contains(&second_pin)); - main.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + main.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); let detached = helpers::collector::detached_versions(&person_uri).await; @@ -4412,10 +4421,13 @@ async fn branch_delete_retires_legacy_physical_path_parents() { "{refused}" ); let rows = db - .cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + .cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); @@ -4598,10 +4610,13 @@ async fn branch_delete_retires_native_parent_and_cleanup_preserves_live_child() "{invalid}" ); assert!( - main.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + main.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None + ) .await .is_err(), "{invalid}" @@ -4635,10 +4650,13 @@ async fn branch_delete_retires_native_parent_and_cleanup_preserves_live_child() .await .unwrap(); reopened - .cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + .cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); assert!(manifest.branches().get(&native).await.is_err()); @@ -4652,10 +4670,13 @@ async fn branch_delete_retires_native_parent_and_cleanup_preserves_live_child() ); reopened.branch_delete("experiment").await.unwrap(); reopened - .cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + .cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); assert!(manifest.branches().get(&native).await.is_err()); @@ -4684,7 +4705,7 @@ async fn merged_table_preserves_row_version_for_unchanged_rows() { let dir = tempfile::tempdir().unwrap(); let uri = dir.path().to_str().unwrap(); let main = init_and_load(&dir).await; - main.ensure_indices().await.unwrap(); + main.ensure_indices_as(None).await.unwrap(); main.branch_create("feature").await.unwrap(); let feature = helpers::session(Omnigraph::open(uri).await.unwrap()); @@ -4767,7 +4788,7 @@ async fn merged_table_preserves_row_version_for_unchanged_rows() { async fn edge_tables_have_id_btree_after_ensure_indices() { let dir = tempfile::tempdir().unwrap(); let db = init_and_load(&dir).await; - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let snap = snapshot_main(&db).await.unwrap(); let ds = snap.open_dataset("edge:Knows").await.unwrap(); @@ -4789,7 +4810,7 @@ async fn merge_delta_only_bumps_changed_rows() { let dir = tempfile::tempdir().unwrap(); let uri = dir.path().to_str().unwrap(); let main = init_and_load(&dir).await; - main.ensure_indices().await.unwrap(); + main.ensure_indices_as(None).await.unwrap(); main.branch_create("feature").await.unwrap(); let feature = helpers::session(Omnigraph::open(uri).await.unwrap()); diff --git a/crates/omnigraph/tests/integration/changes.rs b/crates/omnigraph/tests/integration/changes.rs index fc782044f..23964392a 100644 --- a/crates/omnigraph/tests/integration/changes.rs +++ b/crates/omnigraph/tests/integration/changes.rs @@ -17,10 +17,13 @@ use crate::helpers::*; /// name). async fn reclaim_person_history(db: &Session) { let stats = db - .cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + .cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); let person = stats @@ -550,7 +553,7 @@ async fn diff_after_merge_reports_actual_changes() { let dir = tempfile::tempdir().unwrap(); let uri = dir.path().to_str().unwrap(); let main = init_and_load(&dir).await; - main.ensure_indices().await.unwrap(); + main.ensure_indices_as(None).await.unwrap(); let v_before_branch = snapshot_id(&main, "main").await.unwrap(); main.branch_create("feature").await.unwrap(); @@ -2056,11 +2059,15 @@ node Document { // touching logical rows. The moved descriptors force the payload // tie-break, which must classify every row as unchanged: the maintenance // commit is an empty block, not a phantom full-table update. - let stats = db.optimize().await.unwrap(); + let stats = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap() + .tables; assert!( stats .iter() - .any(|table| table.fragments_removed > 0 && table.committed), + .any(|table| table.fragments_removed > 0 && table.committed()), "the fixture must actually compact so descriptors move" ); let optimize_commit = head_commit_id(uri, None).await; @@ -2158,11 +2165,15 @@ async fn physical_only_commit_over_a_large_fragment_reads_as_empty_issue_705() { ) .await .unwrap(); - let stats = main.optimize().await.unwrap(); + let stats = main + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap() + .tables; assert!( stats .iter() - .any(|table| table.fragments_removed > 0 && table.committed), + .any(|table| table.fragments_removed > 0 && table.committed()), "the fixture must actually compact: {stats:?}" ); let optimize_commit = head_commit_id(dir.path().to_str().unwrap(), None).await; @@ -3186,7 +3197,7 @@ async fn change_feed_merge_commit_is_first_parent_relative_with_merged_parent_on let dir = tempfile::tempdir().unwrap(); let uri = dir.path().to_str().unwrap(); let main = init_and_load(&dir).await; - main.ensure_indices().await.unwrap(); + main.ensure_indices_as(None).await.unwrap(); let pre_merge_head = snapshot_id(&main, "main").await.unwrap(); main.branch_create("side").await.unwrap(); diff --git a/crates/omnigraph/tests/integration/changes_cost.rs b/crates/omnigraph/tests/integration/changes_cost.rs index c35e9a104..2f82e1716 100644 --- a/crates/omnigraph/tests/integration/changes_cost.rs +++ b/crates/omnigraph/tests/integration/changes_cost.rs @@ -26,14 +26,10 @@ use crate::helpers; -use std::sync::Arc; - use crate::helpers::cost::{IoCounts, assert_flat, assert_grows, cost_harness, measure}; -use lance::Dataset; -use lance::dataset::UpdateBuilder; use omnigraph::IndexCoverage; use omnigraph::changes::ChangeFeedScope; -use omnigraph::db::{Omnigraph, ReadTarget, RepairOptions}; +use omnigraph::db::{Omnigraph, ReadTarget}; use omnigraph::loader::LoadMode; use omnigraph_compiler::ir::ParamMap; @@ -93,7 +89,7 @@ node Company { // Build full coverage, then append a new parent fragment. // The measured update therefore runs with a normal partial // / stale index state. The false arm never builds an index. - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); db.load_with_receipt( "main", r#"{"type":"Person","data":{"name":"stale-index-tail","age":1}}"#, @@ -303,123 +299,6 @@ node Document { .await; } -/// Multi-version graph intervals are intentionally outside the candidate -/// contract. The fixture advances one physical dataset twice, then adopts both -/// logical updates in one forced repair graph commit. Two stateless -/// `max_changes=1` pages must each fall back before reading either transaction; -/// the page count cannot multiply a transaction-history walk. -#[tokio::test] -#[ignore = "the multi-version interval is built by `repair`, which promotes at head; phase 4 of RFC \"Detached-only tables\" restates it over a detached chain"] -async fn changes_page_size_one_skips_transaction_history_for_multi_version_intervals() { - cost_harness(async { - let dir = tempfile::tempdir().unwrap(); - let db = Omnigraph::init( - dir.path().to_str().unwrap(), - "node Person {\n name: String @key\n age: I32?\n}", - ) - .await - .unwrap(); - let db = helpers::session(db); - db.load_with_receipt( - "main", - concat!( - r#"{"type":"Person","data":{"name":"alice","age":1}}"#, - "\n", - r#"{"type":"Person","data":{"name":"bob","age":1}}"#, - ), - LoadMode::Merge, - ) - .await - .unwrap(); - let snapshot = db.snapshot_of(ReadTarget::branch("main")).await.unwrap(); - let entry = snapshot.dataset("node:Person").unwrap(); - let before = entry.published_dataset_version; - let dataset_uri = format!( - "{}/{}", - db.uri().trim_end_matches('/'), - entry.dataset_path.trim_start_matches('/') - ); - - let dataset = Dataset::open(&dataset_uri).await.unwrap(); - let dataset = UpdateBuilder::new(Arc::new(dataset)) - .update_where("name = 'alice'") - .unwrap() - .set("age", "2") - .unwrap() - .build() - .unwrap() - .execute() - .await - .unwrap() - .new_dataset; - let dataset = UpdateBuilder::new(dataset) - .update_where("name = 'bob'") - .unwrap() - .set("age", "3") - .unwrap() - .build() - .unwrap() - .execute() - .await - .unwrap() - .new_dataset; - assert_eq!(dataset.version().version, before + 2); - - let repair = db - .repair(RepairOptions { - confirm: true, - force: true, - }) - .await - .unwrap(); - assert!(repair.graph_manifest_version.is_some()); - let repair_commit = db - .list_commits(None) - .await - .unwrap() - .first() - .expect("repair publishes a graph lineage commit") - .graph_commit_id - .clone(); - - let (first, first_io) = measure(db.commit_changes_page( - &repair_commit, - &ChangeFeedScope::default(), - None, - Some(1), - None, - )) - .await; - let first = first.unwrap(); - assert_eq!(first.block.changes.len(), 1); - let token = first.next_page_token.expect("two updates need two pages"); - assert_eq!( - first_io.candidate_transaction_reads, 0, - "first multi-version page must fall back before transaction-history I/O" - ); - - let (second, second_io) = measure(db.commit_changes_page( - &repair_commit, - &ChangeFeedScope::default(), - Some(&token), - Some(1), - None, - )) - .await; - let second = second.unwrap(); - assert_eq!(second.block.changes.len(), 1); - assert!(second.next_page_token.is_none()); - assert_ne!(first.block.changes[0].id, second.block.changes[0].id); - for io in [first_io, second_io] { - assert_eq!( - io.candidate_transaction_reads, 0, - "every stateless multi-version page must skip transaction-history I/O" - ); - } - }) - .await; -} - /// A delete's commit path proves it from the recorded ids, so its page data /// reads stay flat as the table grows. #[tokio::test] diff --git a/crates/omnigraph/tests/integration/composite_flow.rs b/crates/omnigraph/tests/integration/composite_flow.rs index a9dfac8fd..cefa9c820 100644 --- a/crates/omnigraph/tests/integration/composite_flow.rs +++ b/crates/omnigraph/tests/integration/composite_flow.rs @@ -309,7 +309,11 @@ async fn composite_flow_canonical_lifecycle() { // version, and this flow exercises exactly that previously-failing // write below. // ───────────────────────────────────────────────────────────────── - let optimize_stats = db.optimize().await.unwrap(); + let optimize_stats = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap() + .tables; assert!( !optimize_stats.is_empty(), "optimize must return per-table stats" @@ -362,10 +366,13 @@ async fn composite_flow_canonical_lifecycle() { use omnigraph::db::CleanupPolicyOptions; use std::time::Duration; let _cleanup_stats = db - .cleanup(CleanupPolicyOptions { - keep_versions: Some(10), - older_than: Some(Duration::from_secs(3600)), - }) + .cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(10), + older_than: Some(Duration::from_secs(3600)), + }, + None, + ) .await .unwrap(); diff --git a/crates/omnigraph/tests/integration/consistency.rs b/crates/omnigraph/tests/integration/consistency.rs index c7e3cd8fa..ab37588e4 100644 --- a/crates/omnigraph/tests/integration/consistency.rs +++ b/crates/omnigraph/tests/integration/consistency.rs @@ -1369,7 +1369,7 @@ node Thing { db.load_jsonl(&seed, LoadMode::Overwrite).await.unwrap(); // Build the key index explicitly before exercising the merge loads. - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let mut delta = String::new(); for i in (1..=25).chain(51..=55) { @@ -1382,7 +1382,7 @@ node Thing { db.load_jsonl(&delta, LoadMode::Merge) .await .expect("first merge after ensure_indices must succeed"); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); db.load_jsonl(&delta, LoadMode::Merge) .await .expect("second merge after ensure_indices must succeed"); diff --git a/crates/omnigraph/tests/integration/end_to_end.rs b/crates/omnigraph/tests/integration/end_to_end.rs index 02e8e5f45..41a72159d 100644 --- a/crates/omnigraph/tests/integration/end_to_end.rs +++ b/crates/omnigraph/tests/integration/end_to_end.rs @@ -2989,9 +2989,9 @@ async fn ensure_indices_does_not_error_on_repeated_call() { // load commits now enforce required indices; repeated ensure_indices calls // should be a no-op at the manifest level. - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let version_after_first = version_main(&db).await.unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let version_after_second = version_main(&db).await.unwrap(); assert_eq!(version_after_first, version_after_load); @@ -3093,9 +3093,13 @@ async fn full_flow_optimize_then_query_update_and_reopen() { .unwrap(); } - let stats = db.optimize().await.unwrap(); + let stats = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap() + .tables; assert!( - stats.iter().any(|s| s.committed), + stats.iter().any(|s| s.committed()), "a multi-fragment table should have compacted in this flow" ); diff --git a/crates/omnigraph/tests/integration/engine_v2.rs b/crates/omnigraph/tests/integration/engine_v2.rs index 078c32321..1f1ab045a 100644 --- a/crates/omnigraph/tests/integration/engine_v2.rs +++ b/crates/omnigraph/tests/integration/engine_v2.rs @@ -413,7 +413,9 @@ async fn indexed_single_hop_switches_to_the_csr_between_input_batches() { db.load_jsonl(&lines.join("\n"), LoadMode::Overwrite) .await .unwrap(); - db.optimize().await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); let v2 = with_setting(&db, "engine", "v2"); let pinned = first_column_sorted( @@ -1187,7 +1189,7 @@ edge Likes: Person -> Doc ) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); for (source, name, params) in [ (ADULTS, "adults", int_params(&[("min", 18)])), (RANKED, "ranked", params(&[("q", "graph")])), diff --git a/crates/omnigraph/tests/integration/engine_v2_memory.rs b/crates/omnigraph/tests/integration/engine_v2_memory.rs index 02660da69..a9c4347b0 100644 --- a/crates/omnigraph/tests/integration/engine_v2_memory.rs +++ b/crates/omnigraph/tests/integration/engine_v2_memory.rs @@ -520,7 +520,7 @@ node Doc { seed.push('\n'); } db.load_jsonl(&seed, LoadMode::Overwrite).await.unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let v2 = with_setting(&db, "engine", "v2"); let source = r#"query fused($q: String) { match { $d: Doc } @@ -608,7 +608,7 @@ edge Knows: Doc -> Doc db.load_jsonl(&lines.join("\n"), LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let v2 = with_setting( &with_setting(&db, "engine", "v2"), "rrf_plan", @@ -724,7 +724,7 @@ edge Knows: Person -> Person ) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let v2 = with_setting(&db, "engine", "v2"); let queries = r#" query fused($q: String) { @@ -846,7 +846,9 @@ async fn hash_join_output_is_the_refusing_reservation() { db.load_jsonl(&lines.join("\n"), LoadMode::Overwrite) .await .unwrap(); - db.optimize().await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); let v2 = with_setting(&db, "engine", "v2"); let limit = 8 * MIB; @@ -929,7 +931,9 @@ async fn underestimated_hash_build_retries_id_lookup() { db.load_jsonl(&lines.join("\n"), LoadMode::Overwrite) .await .unwrap(); - db.optimize().await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); let v2 = with_setting(&db, "engine", "v2"); let limit = 16 * MIB; let explain = with_query_memory_limit( diff --git a/crates/omnigraph/tests/integration/engine_v2_plan_replay.rs b/crates/omnigraph/tests/integration/engine_v2_plan_replay.rs index 9aa138233..5f63ee006 100644 --- a/crates/omnigraph/tests/integration/engine_v2_plan_replay.rs +++ b/crates/omnigraph/tests/integration/engine_v2_plan_replay.rs @@ -65,7 +65,7 @@ async fn incomplete_rank_fuse_row_tiebreak_refuses_replay_issue_659() { .collect::>() .join("\n"); db.load_jsonl(&edges, LoadMode::Append).await.unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let query = r#"query selected_page() { match { $p: Person $p $e:(knows | likes) $q } return { $e.@type as edge_type, $e.@id as edge_id, $q.name as target } @@ -305,7 +305,9 @@ async fn selected_cold_indexed_route_uses_persisted_members_without_building_csr ) .await .unwrap(); - db.optimize().await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); drop(db); let db = with_traversal( &session(Omnigraph::open(dir.path().to_str().unwrap()).await.unwrap()), @@ -2249,7 +2251,7 @@ async fn a_replay_rechecks_recorded_full_text_coverage() { ) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); db.load_jsonl( r#"{"type":"Doc","data":{"slug":"t1","text":"graph","year":2022}} {"type":"Doc","data":{"slug":"t2","text":"graph graph graph graph","year":1990}}"#, diff --git a/crates/omnigraph/tests/integration/literal_filters.rs b/crates/omnigraph/tests/integration/literal_filters.rs index e493037db..41588b1db 100644 --- a/crates/omnigraph/tests/integration/literal_filters.rs +++ b/crates/omnigraph/tests/integration/literal_filters.rs @@ -579,7 +579,7 @@ async fn standalone_string_predicate_is_hoisted_into_scan() { async fn string_predicates_compose_with_search_and_negation() { let dir = tempfile::tempdir().unwrap(); let mut db = metric_db(&dir).await; - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let q = r#" query prefix_and_search() { match { diff --git a/crates/omnigraph/tests/integration/maintenance.rs b/crates/omnigraph/tests/integration/maintenance.rs index 8c740fe1e..3ac180334 100644 --- a/crates/omnigraph/tests/integration/maintenance.rs +++ b/crates/omnigraph/tests/integration/maintenance.rs @@ -15,11 +15,16 @@ use lance::Dataset; use lance::dataset::optimize::{CompactionOptions, compact_files}; use lance_core::datatypes::BlobHandling; use omnigraph::IndexCoverage; +use omnigraph::db::{ + ArtifactRefresh, ArtifactSkip, ConfigurationClear, DeferReason, OptimizeBeforePublish, + OptimizeCancellation, OptimizeLimits, OptimizePublication, OptimizeRequest, ResourceMeasure, + RunCancellation, TableCeilings, TableOptimizeOutcome, +}; use omnigraph::db::{ CleanupPolicyOptions, MergeOutcome, Omnigraph, ReadTarget, RepairAction, RepairClassification, - RepairOptions, SnapshotId, + SnapshotId, }; -use omnigraph::error::{BranchStartRefusal, OmniError}; +use omnigraph::error::{BranchStartRefusal, CompletionEvidence, OmniError}; use omnigraph::loader::LoadMode; use omnigraph::{ExternalBlobBase, ExternalBlobExecutionScope, ExternalBlobPolicy}; @@ -130,22 +135,627 @@ async fn optimize_on_empty_graph_returns_stats_per_table_with_no_changes() { let uri = dir.path().to_str().unwrap(); let db = Omnigraph::init(uri, TEST_SCHEMA).await.unwrap(); - let stats = db.optimize().await.unwrap(); + let head = db.list_commits(None).await.unwrap()[0] + .graph_commit_id + .clone(); + let report = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); - assert_eq!(stats.len(), 5); - for s in &stats { + // One outcome per node and edge table, in type-key order; `__manifest` + // is not a table the run optimizes. + let keys = report + .tables + .iter() + .map(|s| s.type_key.as_str()) + .collect::>(); + let mut sorted = keys.clone(); + sorted.sort(); + assert_eq!( + keys, sorted, + "a full-catalog run visits tables in key order" + ); + assert_eq!(keys.len(), 4); + for s in &report.tables { + assert_eq!(s.outcome, TableOptimizeOutcome::Unchanged, "{s:?}"); assert_eq!(s.fragments_removed, 0, "{} should not remove", s.type_key); assert_eq!(s.fragments_added, 0, "{} should not add", s.type_key); } - // `__manifest` is present and reported as a no-op on an empty graph. - let s = stats + assert_eq!(report.input_commit.as_deref(), Some(head.as_str())); + assert_eq!(report.graph_commit_id, None, "nothing to publish"); + assert_eq!(report.resume_at, None, "no bound stopped the run"); + assert_eq!( + report.configuration_clear, + ConfigurationClear::NothingToClear + ); + assert!( + !matches!(report.artifact, ArtifactRefresh::Failed(_)), + "{:?}", + report.artifact + ); +} + +/// A run whose gate hold is used up when its second table's turn comes runs +/// only the first, defers the rest, publishes once, and names the first +/// deferred table; a full-catalog run that starts there visits it first and +/// wraps, so successive runs reach every table. +#[tokio::test] +async fn optimize_stops_at_the_gate_hold_and_resumes_at_the_first_deferred_table() { + let dir = tempfile::tempdir().unwrap(); + let db = init_and_load(&dir).await; + let bounded = || { + omnigraph::db::OptimizeRequest::default() + .with_limits(OptimizeLimits::default().with_max_gate_hold(Duration::ZERO)) + }; + + let first = db.optimize_as(None, bounded()).await.unwrap(); + let keys = first + .tables .iter() - .find(|s| s.type_key == "__manifest") - .expect("optimize stats missing internal table __manifest"); + .map(|s| s.type_key.clone()) + .collect::>(); + assert_eq!(keys.len(), 4, "every table is reported: {first:?}"); assert!( - !s.committed, - "__manifest should be a no-op on an empty graph" + !matches!(first.tables[0].outcome, TableOptimizeOutcome::Deferred(_)), + "the first table always runs: {first:?}" + ); + for stat in &first.tables[1..] { + assert_eq!( + stat.outcome, + TableOptimizeOutcome::Deferred(DeferReason::GateHoldLimit), + "{stat:?}" + ); + assert_eq!( + stat.duration, + Duration::ZERO, + "a deferred table never started" + ); + } + assert_eq!(first.resume_at.as_deref(), Some(keys[1].as_str())); + + let second = db + .optimize_as(None, bounded().with_start_at(keys[1].clone())) + .await + .unwrap(); + let visited = second + .tables + .iter() + .map(|s| s.type_key.as_str()) + .collect::>(); + assert_eq!( + visited, + [&keys[1], &keys[2], &keys[3], &keys[0]].map(String::as_str), + "the run starts at the resume position and wraps" + ); + assert_eq!(second.resume_at.as_deref(), Some(keys[2].as_str())); + assert_eq!( + second.input_commit, + first.graph_commit_id.clone().or(first.input_commit.clone()), + "each run builds on the head the previous one left" ); + + // A position after every key wraps to the first, and an unbounded run + // finishes the catalog and clears the position. + let wrapped = db + .optimize_as( + None, + omnigraph::db::OptimizeRequest::default().with_start_at("~"), + ) + .await + .unwrap(); + assert_eq!(wrapped.tables[0].type_key, keys[0]); + assert_eq!(wrapped.resume_at, None); +} + +/// A table list runs only those tables, in its order, and neither reads nor +/// sets a resume position; a malformed request is refused before any effect. +#[tokio::test] +async fn optimize_table_list_runs_only_those_tables_and_refuses_bad_requests() { + let dir = tempfile::tempdir().unwrap(); + let db = init_and_load(&dir).await; + let all = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); + let keys = all + .tables + .iter() + .map(|s| s.type_key.clone()) + .collect::>(); + + let listed = db + .optimize_as( + None, + omnigraph::db::OptimizeRequest::default().with_tables([&keys[2], &keys[0]]), + ) + .await + .unwrap(); + let visited = listed + .tables + .iter() + .map(|s| s.type_key.as_str()) + .collect::>(); + assert_eq!(visited, [keys[2].as_str(), keys[0].as_str()]); + assert_eq!(listed.resume_at, None); + + // A bounded list defers its tail but reports no resume position: a list + // neither reads nor moves it. + let bounded = db + .optimize_as( + None, + omnigraph::db::OptimizeRequest::default() + .with_tables([&keys[2], &keys[0]]) + .with_limits(OptimizeLimits::default().with_max_gate_hold(Duration::ZERO)), + ) + .await + .unwrap(); + assert_eq!( + bounded.tables[1].outcome, + TableOptimizeOutcome::Deferred(DeferReason::GateHoldLimit), + "{bounded:?}" + ); + assert_eq!(bounded.resume_at, None); + + let head = db.list_commits(None).await.unwrap()[0] + .graph_commit_id + .clone(); + for request in [ + omnigraph::db::OptimizeRequest::default().with_tables(["node:Missing"]), + omnigraph::db::OptimizeRequest::default().with_tables([&keys[0], &keys[0]]), + omnigraph::db::OptimizeRequest::default().with_tables(Vec::::new()), + omnigraph::db::OptimizeRequest::default() + .with_tables([&keys[0]]) + .with_start_at(keys[1].clone()), + omnigraph::db::OptimizeRequest::default() + .with_concurrency(omnigraph::db::OptimizeConcurrency::new(0, 1)), + omnigraph::db::OptimizeRequest::default().with_limits( + OptimizeLimits::default() + .with_table_ceilings(TableCeilings::default().with_source_bytes(2)) + .with_run_source_bytes(1), + ), + ] { + let error = db.optimize_as(None, request.clone()).await.unwrap_err(); + assert_eq!( + error.completion_evidence(), + Some(CompletionEvidence::BeforeEffect), + "{request:?}: {error}" + ); + } + assert_eq!( + db.list_commits(None).await.unwrap()[0].graph_commit_id, + head, + "a refused request publishes nothing" + ); +} + +fn recording_hook() -> ( + OptimizeBeforePublish, + std::sync::Arc>>, +) { + let seen = std::sync::Arc::new(std::sync::Mutex::new(Vec::new())); + let recorded = std::sync::Arc::clone(&seen); + let hook = OptimizeBeforePublish::new(move |publication| { + recorded.lock().unwrap().push(publication); + async { Ok(()) } + }); + (hook, seen) +} + +#[tokio::test] +async fn optimize_calls_before_publish_once_with_the_nonce_its_commit_answers() { + let dir = tempfile::tempdir().unwrap(); + let db = init_and_load(&dir).await; + add_person_fragments(&db).await; + let head = db.list_commits(None).await.unwrap()[0] + .graph_commit_id + .clone(); + + let (hook, seen) = recording_hook(); + let report = db + .optimize_as(None, OptimizeRequest::default().with_before_publish(hook)) + .await + .unwrap(); + let commit = report.graph_commit_id.clone().expect("the run published"); + let seen = std::mem::take(&mut *seen.lock().unwrap()); + assert_eq!(seen.len(), 1, "the hook runs once: {seen:?}"); + let publication = &seen[0]; + assert!( + omnigraph_core::graph_commit_id::commit_id_answers(&commit, &publication.nonce).unwrap(), + "{commit} answers {}", + publication.nonce + ); + assert_eq!(publication.input_commit.as_deref(), Some(head.as_str())); + assert_eq!(publication.input_commit, report.input_commit); + assert_eq!( + publication.input_manifest_version, + report.input_manifest_version + ); + let outcomes = |tables: &[omnigraph::db::DatasetOptimizeStats]| { + tables + .iter() + .map(|stat| (stat.type_key.clone(), stat.outcome)) + .collect::>() + }; + assert_eq!(outcomes(&publication.tables), outcomes(&report.tables)); + + // A run that stages no table publishes nothing and never calls the hook. + let (hook, seen) = recording_hook(); + let idle = db + .optimize_as(None, OptimizeRequest::default().with_before_publish(hook)) + .await + .unwrap(); + assert_eq!(idle.graph_commit_id, None, "{idle:?}"); + assert!(seen.lock().unwrap().is_empty()); +} + +#[tokio::test] +async fn optimize_publishes_nothing_when_before_publish_fails() { + let dir = tempfile::tempdir().unwrap(); + let db = init_and_load(&dir).await; + add_person_fragments(&db).await; + let head = db.list_commits(None).await.unwrap()[0] + .graph_commit_id + .clone(); + + // The hook's own evidence claims no effect; the staged rewrites exist, so + // the run must report them unpublished instead. + let hook = OptimizeBeforePublish::new(|_| async { + Err(OmniError::manifest("the operation record was not written").before_effect()) + }); + let error = db + .optimize_as(None, OptimizeRequest::default().with_before_publish(hook)) + .await + .unwrap_err(); + assert_eq!( + error.completion_evidence(), + Some(CompletionEvidence::Unpublished), + "{error}" + ); + assert_eq!( + db.list_commits(None).await.unwrap()[0].graph_commit_id, + head, + "a failed hook publishes nothing" + ); + + // The staged rewrites are unreachable; the next run re-plans and publishes. + let retry = db + .optimize_as(None, OptimizeRequest::default()) + .await + .unwrap(); + assert!(retry.graph_commit_id.is_some(), "{retry:?}"); +} + +#[tokio::test] +async fn optimize_cancelled_before_it_starts_publishes_nothing() { + let dir = tempfile::tempdir().unwrap(); + let db = init_and_load(&dir).await; + add_person_fragments(&db).await; + let head = db.list_commits(None).await.unwrap()[0] + .graph_commit_id + .clone(); + + let cancellation = OptimizeCancellation::new(); + cancellation.cancel(); + let (hook, seen) = recording_hook(); + let report = db + .optimize_as( + None, + OptimizeRequest::default() + .with_cancellation(cancellation) + .with_before_publish(hook), + ) + .await + .unwrap(); + assert_eq!(report.cancellation, RunCancellation::Cancelled); + assert!( + report + .tables + .iter() + .all(|stat| stat.outcome == TableOptimizeOutcome::Deferred(DeferReason::Cancelled)), + "{report:?}" + ); + assert_eq!(report.graph_commit_id, None); + assert_eq!(report.resume_at, None); + assert_eq!(report.configuration_clear, ConfigurationClear::NotRun); + assert_eq!( + report.artifact, + ArtifactRefresh::Skipped(ArtifactSkip::Cancelled) + ); + assert!( + seen.lock().unwrap().is_empty(), + "a cancelled run never calls the hook" + ); + assert_eq!( + db.list_commits(None).await.unwrap()[0].graph_commit_id, + head + ); +} + +#[tokio::test] +async fn optimize_finishes_a_cancellation_that_arrives_after_its_last_check() { + let dir = tempfile::tempdir().unwrap(); + let db = init_and_load(&dir).await; + add_person_fragments(&db).await; + + // The hook runs after the last check, inside the publication helper. + let cancellation = OptimizeCancellation::new(); + let late = cancellation.clone(); + let hook = OptimizeBeforePublish::new(move |_| { + late.cancel(); + async { Ok(()) } + }); + let report = db + .optimize_as( + None, + OptimizeRequest::default() + .with_cancellation(cancellation) + .with_before_publish(hook), + ) + .await + .unwrap(); + assert_eq!(report.cancellation, RunCancellation::ArrivedLate); + assert!(report.graph_commit_id.is_some(), "{report:?}"); + assert!( + report + .tables + .iter() + .any(|stat| stat.type_key == "node:Person" && stat.committed()), + "{report:?}" + ); + assert_eq!( + report.artifact, + ArtifactRefresh::Skipped(ArtifactSkip::Cancelled), + "the edge scan starts only without a cancellation" + ); +} + +async fn person_pin(db: &Omnigraph) -> u64 { + db.snapshot_of(ReadTarget::branch("main")) + .await + .unwrap() + .dataset("node:Person") + .unwrap() + .published_dataset_version +} + +/// Each per-table ceiling defers a table over it with nothing written; the +/// plan holds a deletion file, which records no size, so the byte measure +/// reads it from storage. +#[tokio::test] +async fn optimize_defers_a_table_over_a_ceiling_and_writes_nothing_for_it() { + let dir = tempfile::tempdir().unwrap(); + let db = init_and_load(&dir).await; + add_person_fragments(&db).await; + mutate_main( + &db, + MUTATION_QUERIES, + "remove_person", + &helpers::params(&[("$name", "Eve")]), + ) + .await + .unwrap(); + let pin = person_pin(&db).await; + + for (ceilings, measure) in [ + ( + TableCeilings::default().with_source_fragments(1), + ResourceMeasure::SourceFragments, + ), + ( + TableCeilings::default().with_source_rows(1), + ResourceMeasure::SourceRows, + ), + ( + TableCeilings::default().with_source_bytes(1), + ResourceMeasure::SourceBytes, + ), + ( + TableCeilings::default().with_index_rows(1), + ResourceMeasure::IndexRows, + ), + ] { + let report = db + .optimize_as( + None, + OptimizeRequest::default() + .with_limits(OptimizeLimits::default().with_table_ceilings(ceilings)), + ) + .await + .unwrap(); + let person = report + .tables + .iter() + .find(|stat| stat.type_key == "node:Person") + .unwrap(); + let TableOptimizeOutcome::Deferred(DeferReason::ResourceLimit(exceeded)) = person.outcome + else { + panic!("{measure:?}: {person:?}"); + }; + assert_eq!(exceeded.measure, measure); + assert_eq!(exceeded.ceiling, 1); + assert!( + exceeded.measured.is_some_and(|measured| measured > 1), + "{exceeded:?}" + ); + assert_eq!( + report.resume_at, None, + "a resource limit never moves the resume position" + ); + assert_eq!(person_pin(&db).await, pin, "{measure:?} wrote nothing"); + } + + // Measured whole, including the deletion file's size, the plan fits. + let report = db + .optimize_as( + None, + OptimizeRequest::default() + .with_limits(OptimizeLimits::default().with_run_source_bytes(u64::MAX)), + ) + .await + .unwrap(); + let person = report + .tables + .iter() + .find(|stat| stat.type_key == "node:Person") + .unwrap(); + assert!(person.committed(), "{person:?}"); + assert!( + person.source_bytes.is_some_and(|bytes| bytes > 0), + "{person:?}" + ); +} + +/// The run budget stops the run at the first table that would pass it: +/// that table and every later one are deferred, and the next run resumes +/// there. +#[tokio::test] +async fn optimize_byte_budget_defers_the_tables_after_it_and_resumes_there() { + let dir = tempfile::tempdir().unwrap(); + let db = init_and_load(&dir).await; + add_person_fragments(&db).await; + for name in ["Acme", "Beta", "Cygnus", "Delta"] { + db.load_jsonl( + &serde_json::json!({"type": "Company", "data": {"name": name}}).to_string(), + LoadMode::Merge, + ) + .await + .unwrap(); + } + + // Measure every table's source bytes in a run whose hook refuses to + // publish it. + let seen = std::sync::Arc::new(std::sync::Mutex::new(Vec::new())); + let recorded = std::sync::Arc::clone(&seen); + let probe = OptimizeBeforePublish::new(move |publication| { + recorded.lock().unwrap().push(publication); + async { Err(OmniError::manifest("measuring only")) } + }); + db.optimize_as( + None, + OptimizeRequest::default() + .with_limits(OptimizeLimits::default().with_run_source_bytes(u64::MAX)) + .with_before_publish(probe), + ) + .await + .unwrap_err(); + let measured = seen.lock().unwrap()[0] + .tables + .iter() + .filter_map(|stat| Some((stat.type_key.clone(), stat.source_bytes?))) + .collect::>(); + assert_eq!( + measured + .iter() + .map(|(key, _)| key.as_str()) + .collect::>(), + ["node:Company", "node:Person"], + "{measured:?}" + ); + // Each table fits alone, both together do not. + let budget = measured[0].1.max(measured[1].1); + let limits = OptimizeLimits::default().with_run_source_bytes(budget); + + let report = db + .optimize_as(None, OptimizeRequest::default().with_limits(limits)) + .await + .unwrap(); + let outcome = |report: &omnigraph::db::OptimizeReport, key: &str| { + report + .tables + .iter() + .find(|stat| stat.type_key == key) + .unwrap() + .outcome + }; + assert!( + matches!( + outcome(&report, "node:Company"), + TableOptimizeOutcome::Optimized + ), + "{report:?}" + ); + assert_eq!( + outcome(&report, "node:Person"), + TableOptimizeOutcome::Deferred(DeferReason::ByteBudget) + ); + assert_eq!(report.resume_at.as_deref(), Some("node:Person")); + + let next = db + .optimize_as( + None, + OptimizeRequest::default() + .with_limits(limits) + .with_start_at("node:Person"), + ) + .await + .unwrap(); + assert!( + matches!( + outcome(&next, "node:Person"), + TableOptimizeOutcome::Optimized + ), + "{next:?}" + ); + assert_eq!(next.resume_at, None); +} + +/// The artifact limit holds before the scan (a lower bound) and during it +/// (the counting build), writes nothing when it stops, and admits the whole +/// object once there is room for it. +#[tokio::test] +async fn optimize_skips_an_artifact_over_its_size_limit() { + use std::sync::atomic::{AtomicU64, Ordering}; + let dir = tempfile::tempdir().unwrap(); + let db = init_and_load(&dir).await; + let artifact = dir.path().join("__graph_index/csr-current.bin"); + let report = db + .optimize_as(None, OptimizeRequest::default()) + .await + .unwrap(); + assert_eq!(report.artifact, ArtifactRefresh::Refreshed); + let size = std::fs::metadata(&artifact).unwrap().len(); + std::fs::remove_file(&artifact).unwrap(); + + let bounded = |limit: u64| { + let builds = std::sync::Arc::new(AtomicU64::new(0)); + let probes = omnigraph::instrumentation::QueryIoProbes { + graph_build_count: std::sync::Arc::clone(&builds), + ..Default::default() + }; + let run = omnigraph::instrumentation::with_query_io_probes( + probes, + db.optimize_as( + None, + OptimizeRequest::default() + .with_limits(OptimizeLimits::default().with_artifact_bytes(limit)), + ), + ); + async move { (run.await.unwrap().artifact, builds.load(Ordering::Relaxed)) } + }; + + let (outcome, builds) = bounded(1).await; + assert!( + matches!(outcome, ArtifactRefresh::Skipped(ArtifactSkip::SizeLimit { counted, limit: 1 }) if counted > 1), + "{outcome:?}" + ); + assert_eq!(builds, 0, "a lower bound over the limit starts no scan"); + assert!(!artifact.exists()); + + let (outcome, builds) = bounded(size - 1).await; + let ArtifactRefresh::Skipped(ArtifactSkip::SizeLimit { counted, limit }) = outcome else { + panic!("{outcome:?}"); + }; + assert_eq!(limit, size - 1); + assert!(counted > limit, "{counted} vs {limit}"); + assert_eq!(builds, 1, "the counting build started and stopped"); + assert!(!artifact.exists(), "a stopped build writes nothing"); + + // The counted header is the widest, so admitting the whole object needs + // room for its count and length digits. + let (outcome, _) = bounded(size + 4096).await; + assert_eq!(outcome, ArtifactRefresh::Refreshed); + assert_eq!(std::fs::metadata(&artifact).unwrap().len(), size); } #[tokio::test] @@ -154,7 +764,10 @@ async fn optimize_after_load_then_again_is_idempotent() { let db = init_and_load(&dir).await; // First pass may compact (load wrote real fragments). - let _first = db.optimize().await.unwrap(); + let _first = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); let commits_before = db.list_commits(None).await.unwrap(); let head_before = commits_before @@ -165,7 +778,11 @@ async fn optimize_after_load_then_again_is_idempotent() { // Second pass should be a no-op: already-compacted graph produces no // fragments_removed / fragments_added. - let second = db.optimize().await.unwrap(); + let second = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap() + .tables; for s in &second { assert_eq!( s.fragments_removed, 0, @@ -178,7 +795,7 @@ async fn optimize_after_load_then_again_is_idempotent() { s.type_key ); assert!( - !s.committed, + !s.committed(), "{} re-optimize should not commit a new version", s.type_key ); @@ -229,7 +846,11 @@ async fn optimize_preserves_manifest_registry_fragments() { !cold_files.is_empty(), "the fixture must release more registry rows than the hot bound" ); - let stats = db.optimize().await.unwrap(); + let report = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); + let stats = &report.tables; let after = Dataset::open(&manifest_uri).await.unwrap(); let after_files: BTreeSet<_> = after .fragments() @@ -241,13 +862,14 @@ async fn optimize_preserves_manifest_registry_fragments() { "optimize rewrote cold registry files" ); - let manifest_stats = stats - .iter() - .find(|s| s.type_key == "__manifest") - .expect("optimize stats missing internal table __manifest"); assert!( - !manifest_stats.committed && manifest_stats.fragments_removed == 0, - "the manifest publisher owns hot and cold fragments, so optimize must leave them intact: {manifest_stats:?}" + stats.iter().all(|s| s.type_key != "__manifest"), + "the manifest publisher owns hot and cold fragments, so optimize must not compact them" + ); + assert_eq!( + report.configuration_clear, + ConfigurationClear::NothingToClear, + "a graph with no stale auto-cleanup keys has nothing to clear" ); assert!( @@ -323,7 +945,9 @@ async fn optimize_clears_stale_auto_cleanup_and_preserves_versions() { .unwrap() .len(); - db.optimize().await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); let ds = Dataset::open(&manifest_uri).await.unwrap(); // (a) the stale auto_cleanup config was cleared (non-destructive by construction). @@ -363,10 +987,9 @@ async fn optimize_preserves_versions_under_stale_auto_cleanup_config_on_data_tab add_person_fragments(&db).await; // multiple fragments → will_compact // Simulate an upgraded graph: set an aggressive stored auto_cleanup config on - // the Person table. This is an out-of-band Lance commit (an `UpdateConfig` that - // advances HEAD past the manifest), so realign the manifest with a forced repair - // first — otherwise optimize skips the table as uncovered drift and never - // reaches the scrub. (Forced because UpdateConfig is not verified maintenance.) + // the Person table. This is an out-of-band Lance commit (an `UpdateConfig` + // above the table's last linear version); optimize stages on the pin, never + // the linear HEAD, so it still compacts the table. let (_, _, person_full) = person_pin_and_head(&db, &root).await; { let mut ds = Dataset::open(&person_full).await.unwrap(); @@ -377,12 +1000,6 @@ async fn optimize_preserves_versions_under_stale_auto_cleanup_config_on_data_tab .await .unwrap(); } - db.repair(RepairOptions { - confirm: true, - force: true, - }) - .await - .unwrap(); let versions_before = Dataset::open(&person_full) .await @@ -393,7 +1010,9 @@ async fn optimize_preserves_versions_under_stale_auto_cleanup_config_on_data_tab .len(); let rows_before = count_rows(&db, "node:Person").await; - db.optimize().await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); let ds = Dataset::open(&person_full).await.unwrap(); assert!( @@ -440,7 +1059,7 @@ node Doc { ) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); // A second load with NEW keys appends a fragment the existing BTREEs do not // cover (the existence gate skips re-building an index that already exists). @@ -462,7 +1081,11 @@ node Doc { ); } - let stats = db.optimize().await.unwrap(); + let stats = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap() + .tables; // Postcondition: optimize_indices folded the scalar tail in, but preserves // the existing FTS artifact instead of merging uncertified postings. @@ -491,12 +1114,16 @@ node Doc { ); let commits_before = db.list_commits(None).await.unwrap().len(); for _ in 0..2 { - let repeated = db.optimize().await.unwrap(); + let repeated = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap() + .tables; let doc = repeated .iter() .find(|stat| stat.type_key == "node:Doc") .unwrap(); - assert!(!doc.committed, "deferred FTS alone is not optimize work"); + assert!(!doc.committed(), "deferred FTS alone is not optimize work"); assert_eq!(doc.pending_indexes.len(), 1); assert_eq!(doc.pending_indexes[0].property, "slug"); assert_same_dataset_entry( @@ -510,7 +1137,10 @@ node Doc { assert_eq!(recovery_sidecar_count(&dir), 0); } assert_eq!(db.list_commits(None).await.unwrap().len(), commits_before); - let rebuilt = db.rebuild_full_text_indices_on("main").await.unwrap(); + let rebuilt = db + .rebuild_full_text_indices_on_as("main", None) + .await + .unwrap(); assert_eq!(rebuilt.rebuilt_indexes.len(), 1); assert_eq!(rebuilt.rebuilt_indexes[0].property, "slug"); let snap = snapshot_main(&db).await.unwrap(); @@ -648,9 +1278,12 @@ node Tag {\n slug: String @key\n}\n"; .clone(); let probes = omnigraph::instrumentation::MergeWriteProbes::default(); - let stats = omnigraph::instrumentation::with_merge_write_probes(probes.clone(), db.optimize()) - .await - .expect("optimize must not crash on a graph with a Blob table"); + let stats = omnigraph::instrumentation::with_merge_write_probes( + probes.clone(), + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()), + ) + .await + .expect("optimize must not crash on a graph with a Blob table"); assert_eq!( ( probes.compaction_blob_batch_calls(), @@ -661,16 +1294,18 @@ node Tag {\n slug: String @key\n}\n"; ); let doc = stats + .tables .iter() .find(|s| s.type_key == "node:Doc") .expect("Doc stat present"); let tag = stats + .tables .iter() .find(|s| s.type_key == "node:Tag") .expect("Tag stat present"); // Lance 10's null/empty-safe blob-v2 compaction uses the ordinary path. - assert_eq!(doc.skipped, None, "blob table must no longer be skipped"); - assert!(doc.committed, "blob table compaction must be published"); + assert_eq!(doc.skipped(), None, "blob table must no longer be skipped"); + assert!(doc.committed(), "blob table compaction must be published"); assert!( doc.fragments_removed >= 2 && doc.fragments_added >= 1, "expected a real rewrite of the multi-fragment blob table, got \ @@ -678,8 +1313,8 @@ node Tag {\n slug: String @key\n}\n"; doc.fragments_removed, doc.fragments_added ); - assert_eq!(tag.skipped, None, "non-blob table must not be skipped"); - assert!(tag.committed, "plain table compaction must be published"); + assert_eq!(tag.skipped(), None, "non-blob table must not be skipped"); + assert!(tag.committed(), "plain table compaction must be published"); // Both productive tables share one graph visibility point. Physical // __manifest compaction below may add Lance versions, so lineage is the @@ -782,9 +1417,12 @@ async fn optimize_compacts_large_blob_rows_in_bounded_batches() { let commits_before = db.list_commits(None).await.unwrap().len(); let probes = omnigraph::instrumentation::MergeWriteProbes::default(); - let stats = omnigraph::instrumentation::with_merge_write_probes(probes.clone(), db.optimize()) - .await - .unwrap(); + let stats = omnigraph::instrumentation::with_merge_write_probes( + probes.clone(), + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()), + ) + .await + .unwrap(); assert_eq!( probes.compaction_blob_batch_calls(), 1, @@ -796,11 +1434,15 @@ async fn optimize_compacts_large_blob_rows_in_bounded_batches() { "the task runs with the batch derived from its largest row" ); let doc = stats + .tables .iter() .find(|stat| stat.type_key == "node:Doc") .expect("Doc stat present"); - assert_eq!(doc.skipped, None); - assert!(doc.committed, "the Blob table compaction must be published"); + assert_eq!(doc.skipped(), None); + assert!( + doc.committed(), + "the Blob table compaction must be published" + ); assert_eq!(doc.fragments_removed, fragments_before); assert_eq!( db.list_commits(None).await.unwrap().len(), @@ -937,9 +1579,12 @@ async fn optimize_sizes_a_blob_batch_by_the_sum_of_a_rows_blob_columns() { } let probes = omnigraph::instrumentation::MergeWriteProbes::default(); - let stats = omnigraph::instrumentation::with_merge_write_probes(probes.clone(), db.optimize()) - .await - .unwrap(); + let stats = omnigraph::instrumentation::with_merge_write_probes( + probes.clone(), + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()), + ) + .await + .unwrap(); assert_eq!(probes.compaction_blob_batch_calls(), 1); assert_eq!( probes.compaction_blob_batch_rows(), @@ -947,10 +1592,14 @@ async fn optimize_sizes_a_blob_batch_by_the_sum_of_a_rows_blob_columns() { "the batch is derived from the row's 2 MiB, not from its largest 1 MiB column" ); let doc = stats + .tables .iter() .find(|stat| stat.type_key == "node:Doc") .expect("Doc stat present"); - assert!(doc.committed, "the Blob table compaction must be published"); + assert!( + doc.committed(), + "the Blob table compaction must be published" + ); assert_eq!(doc.fragments_removed, 2); let expected = rows .into_iter() @@ -1026,12 +1675,19 @@ async fn optimize_compacts_blob_fragments_with_deleted_rows() { "test precondition: both fragments carry a deletion vector" ); - let stats = db.optimize().await.unwrap(); + let stats = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap() + .tables; let doc = stats .iter() .find(|stat| stat.type_key == "node:Doc") .expect("Doc stat present"); - assert!(doc.committed, "the Blob table compaction must be published"); + assert!( + doc.committed(), + "the Blob table compaction must be published" + ); assert_eq!(doc.fragments_removed, 2); let compacted = helpers::open_pinned_dataset_for_test(&db, "main", "node:Doc").await; let fragments = compacted.get_fragments(); @@ -1080,14 +1736,17 @@ async fn optimize_sizes_each_compaction_task_from_its_own_fragments() { }; db.load_jsonl(&line, mode).await.unwrap(); if load == 1 { - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); } } let probes = omnigraph::instrumentation::MergeWriteProbes::default(); - let stats = omnigraph::instrumentation::with_merge_write_probes(probes.clone(), db.optimize()) - .await - .unwrap(); + let stats = omnigraph::instrumentation::with_merge_write_probes( + probes.clone(), + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()), + ) + .await + .unwrap(); assert_eq!( probes.compaction_blob_batch_calls(), 2, @@ -1099,10 +1758,14 @@ async fn optimize_sizes_each_compaction_task_from_its_own_fragments() { "the second task is sized by its own 64 KiB row, not the first task's 1 MiB row" ); let doc = stats + .tables .iter() .find(|stat| stat.type_key == "node:Doc") .expect("Doc stat present"); - assert!(doc.committed, "the Blob table compaction must be published"); + assert!( + doc.committed(), + "the Blob table compaction must be published" + ); let expected = rows .into_iter() .map(|(slug, content)| (slug.to_string(), vec![Some(content)])) @@ -1162,10 +1825,40 @@ async fn optimize_does_not_size_a_blob_batch_by_external_references() { "test precondition: the load keeps the external reference as a descriptor" ); - let probes = omnigraph::instrumentation::MergeWriteProbes::default(); - let stats = omnigraph::instrumentation::with_merge_write_probes(probes.clone(), db.optimize()) + // Nor toward the Blob ceiling: only the 3 managed bytes count. + let bounded = db + .optimize_as( + None, + omnigraph::db::OptimizeRequest::default().with_limits( + OptimizeLimits::default() + .with_table_ceilings(TableCeilings::default().with_blob_bytes(2)), + ), + ) .await .unwrap(); + let doc = bounded + .tables + .iter() + .find(|stat| stat.type_key == "node:Doc") + .unwrap(); + assert_eq!( + doc.outcome, + TableOptimizeOutcome::Deferred(DeferReason::ResourceLimit( + omnigraph::db::ResourceExceeded { + measure: ResourceMeasure::BlobBytes, + measured: Some(3), + ceiling: 2, + } + )) + ); + + let probes = omnigraph::instrumentation::MergeWriteProbes::default(); + let stats = omnigraph::instrumentation::with_merge_write_probes( + probes.clone(), + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()), + ) + .await + .unwrap(); assert_eq!(probes.compaction_blob_batch_calls(), 1); assert_eq!( probes.compaction_blob_batch_rows(), @@ -1173,10 +1866,14 @@ async fn optimize_does_not_size_a_blob_batch_by_external_references() { "a 1 MiB external reference must not lower the batch to 32 rows" ); let doc = stats + .tables .iter() .find(|stat| stat.type_key == "node:Doc") .expect("Doc stat present"); - assert!(doc.committed, "the Blob table compaction must be published"); + assert!( + doc.committed(), + "the Blob table compaction must be published" + ); assert_eq!(doc.fragments_removed, 2); } @@ -1282,9 +1979,16 @@ async fn optimize_then_cleanup_erases_dropped_property_values() { .await .unwrap(); // Settle index work, then prove Lance's planner leaves both tables alone. - db.optimize().await.unwrap(); - for stat in db.optimize().await.unwrap() { - assert!(!stat.committed, "test precondition: {stat:?}"); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); + for stat in db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap() + .tables + { + assert!(!stat.committed(), "test precondition: {stat:?}"); } assert_eq!( helpers::open_pinned_dataset_for_test(&db, "main", "node:Doc") @@ -1329,17 +2033,26 @@ async fn optimize_then_cleanup_erases_dropped_property_values() { let commits_before = db.list_commits(None).await.unwrap().len(); let probes = omnigraph::instrumentation::MergeWriteProbes::default(); - let stats = omnigraph::instrumentation::with_merge_write_probes(probes.clone(), db.optimize()) - .await - .unwrap(); - let stat = |key: &str| stats.iter().find(|stat| stat.type_key == key).unwrap(); + let stats = omnigraph::instrumentation::with_merge_write_probes( + probes.clone(), + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()), + ) + .await + .unwrap(); + let stat = |key: &str| { + stats + .tables + .iter() + .find(|stat| stat.type_key == key) + .unwrap() + }; let doc = stat("node:Doc"); assert!( - doc.committed, + doc.committed(), "the fragment holding dropped values is rewritten" ); assert_eq!((doc.fragments_removed, doc.fragments_added), (1, 1)); - assert!(!stat("node:Tag").committed, "Tag has nothing to rewrite"); + assert!(!stat("node:Tag").committed(), "Tag has nothing to rewrite"); assert_eq!( helpers::pinned_version(&db, "main", "node:Tag").await, tag_pin @@ -1379,9 +2092,14 @@ async fn optimize_then_cleanup_erases_dropped_property_values() { doc_blob_values(&db, &["content"]).await == expected, "the remaining properties survive the rewrite" ); - for stat in db.optimize().await.unwrap() { + for stat in db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap() + .tables + { assert!( - !stat.committed, + !stat.committed(), "a second optimize has nothing left: {stat:?}" ); } @@ -1403,8 +2121,10 @@ async fn optimize_then_cleanup_erases_dropped_property_values() { db.branch_merge("retained", "main").await.unwrap().outcome, MergeOutcome::Merged ); - db.optimize().await.unwrap(); - let stats = db.cleanup(keep_one()).await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); + let stats = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(stats.iter().all(|row| row.error.is_none()), "{stats:?}"); for needle in &needles { assert!( @@ -1413,7 +2133,7 @@ async fn optimize_then_cleanup_erases_dropped_property_values() { ); } db.branch_delete("retained").await.unwrap(); - let stats = db.cleanup(keep_one()).await.unwrap(); + let stats = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(stats.iter().all(|row| row.error.is_none()), "{stats:?}"); for needle in &needles { let found = files_containing(dir.path(), needle); @@ -1467,13 +2187,17 @@ async fn optimize_publishes_compaction_to_manifest_so_schema_apply_succeeds() { .get_fragments() .len(); - let stats = db.optimize().await.unwrap(); + let stats = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap() + .tables; let person = stats .iter() .find(|s| s.type_key == "node:Person") .expect("Person stat present"); assert!( - person.committed, + person.committed(), "Person is multi-fragment, so optimize must have compacted it" ); @@ -1527,13 +2251,20 @@ async fn optimize_compacts_the_pin_under_foreign_head_drift() { let (pin_before, head_before, _) = forge_person_compaction_drift(&db, &root).await; let rows_before = count_rows(&db, "node:Person").await; - let stats = db.optimize().await.unwrap(); + let stats = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap() + .tables; let person = stats .iter() .find(|s| s.type_key == "node:Person") .expect("Person stat present"); - assert_eq!(person.skipped, None); - assert!(person.committed, "optimize compacts the multi-fragment pin"); + assert_eq!(person.skipped(), None); + assert!( + person.committed(), + "optimize compacts the multi-fragment pin" + ); let (pin_after, head_after, _) = person_pin_and_head(&db, &root).await; assert_ne!(pin_after, pin_before, "optimize publishes a new pin"); @@ -1545,10 +2276,11 @@ async fn optimize_compacts_the_pin_under_foreign_head_drift() { assert_eq!(count_rows(&db, "node:Person").await, rows_before); } -/// Repair preview names linear commits above the last linear version as -/// `foreign_drift` and publishes nothing. +/// The repair report names linear commits above the last linear version as +/// `foreign_drift` and publishes nothing, and writers stage on the pin +/// regardless: a strict schema apply still succeeds. #[tokio::test] -async fn repair_preview_reports_foreign_drift_without_adopting_it() { +async fn repair_report_names_foreign_drift_without_adopting_it() { let dir = tempfile::tempdir().unwrap(); let root = dir .path() @@ -1559,14 +2291,13 @@ async fn repair_preview_reports_foreign_drift_without_adopting_it() { let db = init_and_load(&dir).await; let (pin_before, head_before, _) = forge_person_compaction_drift(&db, &root).await; - let stats = db - .repair(RepairOptions { - confirm: false, - force: false, - }) - .await - .unwrap(); - assert_eq!(stats.graph_manifest_version, None); + let commits_before = db.list_commits(None).await.unwrap(); + let stats = db.repair_report_as(None).await.unwrap(); + assert_eq!( + db.list_commits(None).await.unwrap(), + commits_before, + "the repair report publishes nothing" + ); let person = stats .datasets .iter() @@ -1584,44 +2315,6 @@ async fn repair_preview_reports_foreign_drift_without_adopting_it() { let (pin_after, head_after, _) = person_pin_and_head(&db, &root).await; assert_eq!(pin_after, pin_before); assert_eq!(head_after, head_before); -} - -/// A confirmed repair never adopts foreign drift, and writers stage on the -/// pin regardless: the strict schema apply still succeeds. -#[tokio::test] -async fn repair_confirm_never_adopts_foreign_drift() { - let dir = tempfile::tempdir().unwrap(); - let root = dir - .path() - .to_str() - .unwrap() - .trim_end_matches('/') - .to_string(); - let db = init_and_load(&dir).await; - let (pin_before, head_before, _) = forge_person_compaction_drift(&db, &root).await; - - let stats = db - .repair(RepairOptions { - confirm: true, - force: false, - }) - .await - .unwrap(); - assert_eq!( - stats.graph_manifest_version, None, - "confirmed repair publishes nothing for foreign drift" - ); - let person = stats - .datasets - .iter() - .find(|s| s.type_key == "node:Person") - .expect("Person repair stat present"); - assert_eq!(person.classification, RepairClassification::ForeignDrift); - assert_eq!(person.action, RepairAction::NoOp); - - let (pin_after, head_after, _) = person_pin_and_head(&db, &root).await; - assert_eq!(pin_after, pin_before); - assert_eq!(head_after, head_before); let desired = TEST_SCHEMA.replace( " age: I32?\n}", @@ -1634,10 +2327,10 @@ async fn repair_confirm_never_adopts_foreign_drift() { assert!(result.applied); } -/// Force does not change the judgement: a foreign raw delete is reported as -/// `foreign_drift`, never adopted, and reads still resolve the pin. +/// A foreign raw delete is reported as `foreign_drift`, never adopted, and +/// reads still resolve the pin. #[tokio::test] -async fn repair_force_never_adopts_a_foreign_delete() { +async fn repair_report_never_adopts_a_foreign_delete() { let dir = tempfile::tempdir().unwrap(); let root = dir .path() @@ -1648,14 +2341,7 @@ async fn repair_force_never_adopts_a_foreign_delete() { let db = init_and_load(&dir).await; let (pin_before, head_before, _) = forge_person_delete_drift(&db, &root).await; - let stats = db - .repair(RepairOptions { - confirm: true, - force: true, - }) - .await - .unwrap(); - assert_eq!(stats.graph_manifest_version, None); + let stats = db.repair_report_as(None).await.unwrap(); let person = stats .datasets .iter() @@ -1721,17 +2407,7 @@ async fn write_on_a_never_written_table_succeeds_over_foreign_drift_and_repair_r "the write neither moves nor adopts the foreign HEAD" ); - let stats = db - .repair(RepairOptions { - confirm: true, - force: false, - }) - .await - .unwrap(); - assert_eq!( - stats.graph_manifest_version, None, - "repair publishes nothing for foreign drift" - ); + let stats = db.repair_report_as(None).await.unwrap(); let company = stats .datasets .iter() @@ -1779,7 +2455,7 @@ async fn full_text_rebuild_replaces_all_columns_and_segments_in_one_publication( db.load_jsonl(r#"{"type":"Person","data":{"name":"Alice","age":30,"biography":"organism university running"}}"#, LoadMode::Merge, ) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); helpers::forge_linear_head_from_pin(&db, "main", "node:Person", 0).await; let person_uri = node_table_uri(&db, "Person").await; @@ -1923,7 +2599,10 @@ async fn full_text_rebuild_replaces_all_columns_and_segments_in_one_publication( assert_eq!(recovery_sidecar_count(&dir), 0); // A force rebuild remains a real operation after coverage is complete. - let repeated = db.rebuild_full_text_indices_on("main").await.unwrap(); + let repeated = db + .rebuild_full_text_indices_on_as("main", None) + .await + .unwrap(); assert_eq!(repeated.rebuilt_indexes, expected); assert_ne!(repeated.graph_commit_id, result.graph_commit_id); assert_eq!( @@ -2092,7 +2771,7 @@ async fn full_text_rebuild_on_a_branch_stages_on_the_inherited_location_without_ let db = init_and_load(&dir).await; db.branch_create("feature").await.unwrap(); let inherited = db.snapshot_of(ReadTarget::branch("feature")).await.unwrap(); - db.ensure_indices_on("feature").await.unwrap(); + db.ensure_indices_on_as("feature", None).await.unwrap(); assert_same_dataset_entry( inherited.dataset("node:Person").unwrap(), db.snapshot_of(ReadTarget::branch("feature")) @@ -2106,7 +2785,10 @@ async fn full_text_rebuild_on_a_branch_stages_on_the_inherited_location_without_ let main_commits_before = db.list_commits(None).await.unwrap(); let feature_commits_before = db.list_commits(Some("feature")).await.unwrap(); - let result = db.rebuild_full_text_indices_on("feature").await.unwrap(); + let result = db + .rebuild_full_text_indices_on_as("feature", None) + .await + .unwrap(); assert_eq!(result.branch, "feature"); assert_eq!(result.rebuilt_indexes.len(), 2); let feature = db.snapshot_of(ReadTarget::branch("feature")).await.unwrap(); @@ -2192,7 +2874,10 @@ async fn full_text_rebuild_reports_empty_builds_but_not_no_work() { .await .unwrap(); let before = snapshot_main(&db).await.unwrap(); - let result = db.rebuild_full_text_indices_on("main").await.unwrap(); + let result = db + .rebuild_full_text_indices_on_as("main", None) + .await + .unwrap(); assert_eq!(result.branch, "main"); assert_eq!(result.graph_commit_id, None); assert!(result.rebuilt_indexes.is_empty()); @@ -2207,7 +2892,10 @@ async fn full_text_rebuild_reports_empty_builds_but_not_no_work() { .await .unwrap(); let before = snapshot_main(&db).await.unwrap(); - let result = db.rebuild_full_text_indices_on("main").await.unwrap(); + let result = db + .rebuild_full_text_indices_on_as("main", None) + .await + .unwrap(); assert_eq!(result.rebuilt_indexes.len(), 2); assert!(result.graph_commit_id.is_some()); let after = snapshot_main(&db).await.unwrap(); @@ -2294,7 +2982,10 @@ async fn full_text_rebuild_refuses_unsupported_physical_inventory_before_effects .version, ); } - let error = db.rebuild_full_text_indices_on("main").await.unwrap_err(); + let error = db + .rebuild_full_text_indices_on_as("main", None) + .await + .unwrap_err(); assert!( error.to_string().contains("unsupported_internal_id_fts"), "{error}" @@ -2337,7 +3028,7 @@ async fn cleanup_without_any_policy_option_errors() { let db = init_and_load(&dir).await; let err = db - .cleanup(CleanupPolicyOptions::default()) + .cleanup_as(CleanupPolicyOptions::default(), None) .await .expect_err("cleanup with no policy options must error"); @@ -2367,10 +3058,13 @@ async fn cleanup_keep_one_preserves_head_and_table_remains_readable() { ); let _stats = db - .cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + .cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); @@ -2403,7 +3097,7 @@ async fn cleanup_keep_exceeding_history_preserves_every_available_version() { ); let pins = pins_in_history(&db, "main", "node:Person").await; - db.cleanup(keep_ten).await.unwrap(); + db.cleanup_as(keep_ten, None).await.unwrap(); // Every version a commit pins stays; only a chain's links below its // pinned tip (the first load's effect beneath its full-text declaration) @@ -2425,10 +3119,13 @@ async fn cleanup_older_than_zero_preserves_head() { // Lance must still preserve the head manifest, so the table is openable // afterwards and a subsequent load still works. let _stats = db - .cleanup(CleanupPolicyOptions { - keep_versions: None, - older_than: Some(Duration::from_secs(0)), - }) + .cleanup_as( + CleanupPolicyOptions { + keep_versions: None, + older_than: Some(Duration::from_secs(0)), + }, + None, + ) .await .unwrap(); @@ -2467,10 +3164,13 @@ async fn cleanup_preserves_main_version_pinned_by_live_lazy_branch() { "precondition: lazy-branch pin must fall outside keep=1 retention" ); - db.cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); @@ -2484,10 +3184,13 @@ async fn cleanup_preserves_main_version_pinned_by_live_lazy_branch() { // timestamp and version predicates with AND, so injecting the branch pin // as `before_version` keeps the inherited version even when every old // manifest satisfies the timestamp cutoff. - db.cleanup(CleanupPolicyOptions { - keep_versions: None, - older_than: Some(Duration::from_secs(0)), - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: None, + older_than: Some(Duration::from_secs(0)), + }, + None, + ) .await .unwrap(); assert_eq!( @@ -2525,10 +3228,13 @@ async fn cleanup_retains_the_commit_a_branch_starts_at() { db.branch_create_from(at(&loaded), "at-load").await.unwrap(); let people_at_load = count_rows_branch(&db, "at-load", "node:Person").await; - db.cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); assert_eq!( @@ -2584,10 +3290,13 @@ async fn cleanup_uses_oldest_pin_across_multiple_live_lazy_branches() { .await .unwrap(); - db.cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); @@ -2627,10 +3336,13 @@ async fn cleanup_fails_closed_when_live_lazy_branch_pin_is_unopenable() { let person_before = detached_versions(&person_uri).await; let stats = db - .cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + .cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); let row = |type_key: &str| { @@ -2673,7 +3385,7 @@ async fn cleanup_reports_foreign_head_drift_and_keeps_it() { .expect("every registration records its last linear version"); let foreign: Vec = (last_linear + 1..=head_version).collect(); - let stats = db.cleanup(keep_one()).await.unwrap(); + let stats = db.cleanup_as(keep_one(), None).await.unwrap(); let person = stats .iter() .find(|row| row.type_key == "node:Person") @@ -2713,13 +3425,18 @@ async fn cleanup_then_optimize_preserves_rows_and_table_remains_writable() { "fixture should seed both Person and Company rows" ); - db.cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); - db.optimize().await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); // Head is preserved through both ops. assert_eq!(count_rows(&db, "node:Person").await, people_before); @@ -2799,10 +3516,13 @@ async fn cleanup_reconciles_orphaned_branch_forks() { ); } - db.cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); @@ -2851,10 +3571,13 @@ async fn cleanup_reconciles_orphaned_branch_forks() { ); // Idempotent: a second cleanup with the orphan already gone is a no-op. - db.cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); assert!( @@ -2930,10 +3653,13 @@ async fn cleanup_reconciles_live_branch_orphan_fork_and_keeps_the_branch_write() let main_people = count_rows(&db, "node:Person").await; let main_companies = count_rows(&db, "node:Company").await; - db.cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); @@ -3033,10 +3759,13 @@ async fn cleanup_age_window_preserves_a_recent_snapshot_across_a_pointer_switch( .await .unwrap(); assert_eq!(before.num_rows(), base_count + 1); - db.cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: Some(Duration::from_secs(30 * 24 * 60 * 60)), - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: Some(Duration::from_secs(30 * 24 * 60 * 60)), + }, + None, + ) .await .unwrap(); assert!( @@ -3054,10 +3783,13 @@ async fn cleanup_age_window_preserves_a_recent_snapshot_across_a_pointer_switch( .await .unwrap(); assert_eq!(after.num_rows(), base_count + 1); - db.cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: Some(Duration::ZERO), - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: Some(Duration::ZERO), + }, + None, + ) .await .unwrap(); assert!( @@ -3087,10 +3819,13 @@ async fn cleanup_age_window_preserves_recent_ref_absent_descendant() { .set_times(std::fs::FileTimes::new().set_modified(std::time::SystemTime::UNIX_EPOCH)) .unwrap(); std::fs::write(&recent_file, b"recent interrupted residue").unwrap(); - db.cleanup(CleanupPolicyOptions { - keep_versions: None, - older_than: Some(Duration::from_secs(30 * 24 * 60 * 60)), - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: None, + older_than: Some(Duration::from_secs(30 * 24 * 60 * 60)), + }, + None, + ) .await .unwrap(); assert!( @@ -3101,10 +3836,13 @@ async fn cleanup_age_window_preserves_recent_ref_absent_descendant() { recent_file.exists(), "a missing ref is not permission to discard a recent object" ); - db.cleanup(CleanupPolicyOptions { - keep_versions: None, - older_than: Some(Duration::ZERO), - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: None, + older_than: Some(Duration::ZERO), + }, + None, + ) .await .unwrap(); assert!( @@ -3174,10 +3912,13 @@ async fn cleanup_reclaims_retirement_archive_after_the_age_window() { > std::time::SystemTime::UNIX_EPOCH, "retirement writes a recent archive even when the tree is old" ); - db.cleanup(CleanupPolicyOptions { - keep_versions: None, - older_than: Some(Duration::from_secs(30 * 24 * 60 * 60)), - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: None, + older_than: Some(Duration::from_secs(30 * 24 * 60 * 60)), + }, + None, + ) .await .unwrap(); assert!( @@ -3188,10 +3929,13 @@ async fn cleanup_reclaims_retirement_archive_after_the_age_window() { tree.exists(), "the recent retirement metadata must protect the aged physical tree" ); - db.cleanup(CleanupPolicyOptions { - keep_versions: None, - older_than: Some(Duration::ZERO), - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: None, + older_than: Some(Duration::ZERO), + }, + None, + ) .await .unwrap(); assert!( @@ -3250,10 +3994,13 @@ async fn cleanup_age_window_preserves_recent_native_mutation_and_aged_parent() { let changed = child.delete("name = 'Alice'").await.unwrap(); assert_eq!(changed.num_deleted_rows, 1); let latest = changed.new_dataset.version().version; - db.cleanup(CleanupPolicyOptions { - keep_versions: None, - older_than: Some(Duration::from_secs(30 * 24 * 60 * 60)), - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: None, + older_than: Some(Duration::from_secs(30 * 24 * 60 * 60)), + }, + None, + ) .await .unwrap(); let branches = Dataset::open(&person_uri) @@ -3285,10 +4032,13 @@ async fn cleanup_age_window_preserves_recent_native_mutation_and_aged_parent() { retained.count_rows(None).await.unwrap(), count_rows(&db, "node:Person").await - 1 ); - db.cleanup(CleanupPolicyOptions { - keep_versions: None, - older_than: Some(Duration::ZERO), - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: None, + older_than: Some(Duration::ZERO), + }, + None, + ) .await .unwrap(); let branches = Dataset::open(&person_uri) @@ -3384,10 +4134,13 @@ async fn cleanup_preserves_the_pin_a_lazy_child_reads_after_its_parent_retires() .clone(); db.branch_delete("feature").await.unwrap(); db.branch_create("feature").await.unwrap(); - db.cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); @@ -3446,10 +4199,13 @@ async fn cleanup_reclaims_dead_incarnation_fork_of_live_branch() { ds.create_branch(&dead_native, base, None).await.unwrap(); } - db.cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); @@ -3515,10 +4271,13 @@ async fn cleanup_reclaims_dead_incarnation_fork_of_live_branch() { db.apply_schema(TEST_SCHEMA).await.unwrap(); let replacement_uri = node_table_uri(&db, "Company").await; assert_ne!(replacement_uri, company_uri); - db.cleanup(CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); assert!( @@ -3577,7 +4336,7 @@ async fn index_build_tolerates_null_vector_rows() { // Must not abort: the untrainable vector column is deferred, while id, // slug (FTS), and n (BTREE) are built together in one table transaction. let pending = db - .ensure_indices() + .ensure_indices_as(None) .await .expect("ensure_indices must not abort when a vector column has no trainable vectors yet"); assert_eq!(pending.len(), 1, "only the null vector index is pending"); @@ -3644,7 +4403,9 @@ async fn optimize_materializes_index_declared_but_unbuilt() { ); } - db.optimize().await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); // Postcondition: optimize's reconciler materialized the declared index. let snap = snapshot_main(&db).await.unwrap(); @@ -3700,7 +4461,9 @@ async fn optimize_materializes_index_after_type_rename() { ); } - db.optimize().await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); let snap = snapshot_main(&db).await.unwrap(); let ds = snap.open_dataset("node:Item").await.unwrap(); @@ -3732,7 +4495,7 @@ async fn collector_retains_malformed_witnesses_after_retirement_history_is_gone( let identity = manifest.branches().get(&native).await.unwrap().identifier; assert_ne!(identity, BranchIdentifier::main()); db.branch_delete("retired").await.unwrap(); - let rows = db.cleanup(keep_one()).await.unwrap(); + let rows = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); assert!(!dir.path().join("__manifest/tree").join(&native).exists()); let identity = serde_json::to_string(&identity).unwrap(); @@ -3809,7 +4572,7 @@ async fn collector_retains_malformed_witnesses_after_retirement_history_is_gone( ); } } - let rows = db.cleanup(keep_one()).await.unwrap(); + let rows = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); let remaining = detached_versions(&table_uri).await; for (label, version) in versions { @@ -3863,7 +4626,7 @@ async fn collector_sweeps_the_pin_that_only_a_pruned_manifest_version_names() { ); assert!(plan.sweep_paths.is_disjoint(&plan.marked_paths)); - let stats = db.cleanup(keep_one()).await.unwrap(); + let stats = db.cleanup_as(keep_one(), None).await.unwrap(); let row = stats .iter() .find(|row| row.type_key == "node:Person") @@ -4077,7 +4840,7 @@ async fn collector_repeat_run_discovers_what_an_earlier_pass_left() { async fn collector_finds_nothing_to_sweep_after_todays_cleanup() { let dir = tempfile::tempdir().unwrap(); let db = init_and_load(&dir).await; - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); insert_person(&db, "main", "Eve").await; mutate_main( &db, @@ -4087,7 +4850,9 @@ async fn collector_finds_nothing_to_sweep_after_todays_cleanup() { ) .await .unwrap(); - db.optimize().await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); db.branch_create("feature").await.unwrap(); insert_person(&db, "feature", "Frank").await; db.branch_merge("feature", "main").await.unwrap(); @@ -4097,7 +4862,7 @@ async fn collector_finds_nothing_to_sweep_after_todays_cleanup() { ) .await .unwrap(); - let stats = db.cleanup(keep_one()).await.unwrap(); + let stats = db.cleanup_as(keep_one(), None).await.unwrap(); for row in &stats { assert!(row.error.is_none(), "every table's version GC ran: {row:?}"); } @@ -4146,10 +4911,13 @@ async fn cleanup_on_a_detached_only_graph_counts_manifest_versions_for_keep() { let third = staged_since(&person_uri, &after_second).await; let stats = db - .cleanup(CleanupPolicyOptions { - keep_versions: Some(2), - older_than: None, - }) + .cleanup_as( + CleanupPolicyOptions { + keep_versions: Some(2), + older_than: None, + }, + None, + ) .await .unwrap(); let person = |stats: &[omnigraph::db::DatasetCleanupStats]| { @@ -4174,7 +4942,7 @@ async fn cleanup_on_a_detached_only_graph_counts_manifest_versions_for_keep() { assert!(!remaining.contains(&first), "{remaining:?}"); assert!(remaining.contains(&second) && remaining.contains(&third)); - let row = person(&db.cleanup(keep_one()).await.unwrap()); + let row = person(&db.cleanup_as(keep_one(), None).await.unwrap()); assert_eq!( row.manifests_removed, 1, "keep 1 prunes the publication that named the second pin: {row:?}" @@ -4183,7 +4951,7 @@ async fn cleanup_on_a_detached_only_graph_counts_manifest_versions_for_keep() { assert!(!remaining.contains(&second), "{remaining:?}"); assert!(remaining.contains(&third)); - let row = person(&db.cleanup(keep_one()).await.unwrap()); + let row = person(&db.cleanup_as(keep_one(), None).await.unwrap()); assert_eq!(row.manifests_removed, 0, "a rerun finds nothing: {row:?}"); let report = db.cleanup_plan(keep_one()).await.unwrap(); for plan in &report.tables { @@ -4378,7 +5146,7 @@ async fn collector_reclaims_aged_orphans_without_deleting_recent_blob_sidecars() .unwrap(); } let rows_before = count_rows(&db, "node:Person").await; - let stats = db.cleanup(keep_one()).await.unwrap(); + let stats = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(stats.iter().all(|row| row.error.is_none()), "{stats:?}"); for relative in recent { assert!( @@ -4446,7 +5214,7 @@ async fn collector_reclaims_blob_sidecars_after_the_last_retained_pin() { db.load_jsonl(&row(¤t_payload), LoadMode::Overwrite) .await .unwrap(); - let stats = db.cleanup(keep_one()).await.unwrap(); + let stats = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(stats.iter().all(|row| row.error.is_none()), "{stats:?}"); for path in &old_sidecars { assert!( @@ -4467,7 +5235,7 @@ async fn collector_reclaims_blob_sidecars_after_the_last_retained_pin() { let report = db.cleanup_plan(keep_one()).await.unwrap(); let plan = main_plan(&report, "node:Doc"); assert!(plan.errors.is_empty(), "{:?}", plan.errors); - let stats = db.cleanup(keep_one()).await.unwrap(); + let stats = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(stats.iter().all(|row| row.error.is_none()), "{stats:?}"); for path in &old_sidecars { assert!( diff --git a/crates/omnigraph/tests/integration/merge_cleanup.rs b/crates/omnigraph/tests/integration/merge_cleanup.rs index 471842586..bad2124f8 100644 --- a/crates/omnigraph/tests/integration/merge_cleanup.rs +++ b/crates/omnigraph/tests/integration/merge_cleanup.rs @@ -36,7 +36,7 @@ async fn repeated_merge_keeps_its_selected_base_through_cleanup() { for age in 1..=3 { set_age(&db, "source", "p", age).await; set_age(&db, "target", "q", age).await; - let rows = db.cleanup(keep_one()).await.unwrap(); + let rows = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); let reopened = helpers::session(Omnigraph::open(uri).await.unwrap()); assert_eq!( @@ -106,7 +106,7 @@ async fn imported_merge_base_survives_owner_retirement_and_recreation() { .starts_with("zz-unrelated.") }) .expect("unrelated retirement archive exists before cleanup"); - let rows = db.cleanup(keep_one()).await.unwrap(); + let rows = db.cleanup_as(keep_one(), None).await.unwrap(); assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); assert!( !unrelated.exists(), diff --git a/crates/omnigraph/tests/integration/point_in_time.rs b/crates/omnigraph/tests/integration/point_in_time.rs index 6f088bd5c..4cb87dc97 100644 --- a/crates/omnigraph/tests/integration/point_in_time.rs +++ b/crates/omnigraph/tests/integration/point_in_time.rs @@ -1184,7 +1184,10 @@ async fn historical_read_of_reclaimed_version_is_typed() { .await .unwrap(); - let stats = db.cleanup(helpers::collector::keep_one()).await.unwrap(); + let stats = db + .cleanup_as(helpers::collector::keep_one(), None) + .await + .unwrap(); let person = stats .iter() .find(|row| row.type_key == "node:Person") diff --git a/crates/omnigraph/tests/integration/recovery.rs b/crates/omnigraph/tests/integration/recovery.rs index 0a3608558..f0cd513b5 100644 --- a/crates/omnigraph/tests/integration/recovery.rs +++ b/crates/omnigraph/tests/integration/recovery.rs @@ -105,12 +105,12 @@ async fn recovery_ensure_indices_steady_state_no_sidecar() { {"type":"Company","data":{"name":"acme"}} "#; db.load_jsonl(test_data, LoadMode::Append).await.unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); drop(db); let db = Omnigraph::open(uri).await.unwrap(); let before = snapshot_main(&db).await.unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let after = snapshot_main(&db).await.unwrap(); assert_eq!( after.graph_manifest_version(), @@ -153,7 +153,7 @@ async fn recovery_ensure_indices_handles_empty_tables() { let db = Omnigraph::init(uri, TEST_SCHEMA).await.unwrap(); // Don't load any data — every table is empty. let before = snapshot_main(&db).await.unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let after = snapshot_main(&db).await.unwrap(); assert_eq!( after.graph_manifest_version(), @@ -173,7 +173,7 @@ async fn recovery_ensure_indices_handles_empty_tables() { drop(db); let db = Omnigraph::open(uri).await.unwrap(); let before = snapshot_main(&db).await.unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let after = snapshot_main(&db).await.unwrap(); assert_eq!( after.graph_manifest_version(), diff --git a/crates/omnigraph/tests/integration/repro_issue_563.rs b/crates/omnigraph/tests/integration/repro_issue_563.rs index 409ce09ac..733ca35c8 100644 --- a/crates/omnigraph/tests/integration/repro_issue_563.rs +++ b/crates/omnigraph/tests/integration/repro_issue_563.rs @@ -141,7 +141,7 @@ async fn ranked_read_with_join_returns_top_limit_issue_563() { db.load_jsonl(&edges, LoadMode::Append).await.unwrap(); } - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); // The ranked read without the join stays under the i32 offset ceiling. let no_join = query_main( diff --git a/crates/omnigraph/tests/integration/rrf_prefilter_gate.rs b/crates/omnigraph/tests/integration/rrf_prefilter_gate.rs index db0cdf997..121cef7f6 100644 --- a/crates/omnigraph/tests/integration/rrf_prefilter_gate.rs +++ b/crates/omnigraph/tests/integration/rrf_prefilter_gate.rs @@ -265,7 +265,7 @@ async fn init_gate_db(dir: &tempfile::TempDir) -> Session { db.load_jsonl(&gate_seed_data(), LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); db } @@ -457,7 +457,7 @@ async fn init_stats_db(dir: &tempfile::TempDir) -> Session { db.load_jsonl(&stats_seed_data(), LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); db } @@ -614,7 +614,7 @@ async fn natural_gate_prefilters_at_ratio_boundary() { db.load_jsonl(&rows.join("\n"), LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let db = db; let text_params = params(&[("$q1", "needle"), ("$q2", "sharp")]); @@ -767,7 +767,7 @@ async fn empty_eligible_set_falls_back() { ) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let db = db; let text_params = params(&[("$q1", "needle"), ("$q2", "sharp")]); @@ -807,7 +807,7 @@ async fn natural_gate_prefilters_selective_fixture() { db.load_jsonl(&rows.join("\n"), LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let db = db; let text_params = params(&[("$q1", "needle"), ("$q2", "sharp")]); @@ -880,7 +880,7 @@ async fn decoy_flood_winner_holds_under_both_plans() { db.load_jsonl(&rows.join("\n"), LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let db = db; let text_params = params(&[("$q1", "alpha"), ("$q2", "beta")]); @@ -926,7 +926,7 @@ async fn natural_gate_falls_back_on_broad_fixture() { db.load_jsonl(&rows.join("\n"), LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let db = db; let text_params = params(&[("$q1", "needle"), ("$q2", "sharp")]); diff --git a/crates/omnigraph/tests/integration/scalar_indexes.rs b/crates/omnigraph/tests/integration/scalar_indexes.rs index a87b5be4a..29443468b 100644 --- a/crates/omnigraph/tests/integration/scalar_indexes.rs +++ b/crates/omnigraph/tests/integration/scalar_indexes.rs @@ -42,7 +42,7 @@ async fn node_scalar_and_enum_index_columns_get_btree() { let uri = dir.path().to_str().unwrap(); let db = session(Omnigraph::init(uri, SCHEMA).await.unwrap()); db.load_jsonl(DATA, LoadMode::Overwrite).await.unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let snap = snapshot_main(&db).await.unwrap(); let ds = snap.open_dataset("node:Item").await.unwrap(); @@ -74,7 +74,7 @@ async fn node_scalar_and_enum_index_columns_get_btree() { // Idempotence: a second run finds every declared index present and // publishes nothing (dual indexes must not re-plan as missing forever). let version_before = ds.published_dataset_version(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let snap2 = snapshot_main(&db).await.unwrap(); let ds2 = snap2.open_dataset("node:Item").await.unwrap(); assert_eq!( diff --git a/crates/omnigraph/tests/integration/traversal_adaptive.rs b/crates/omnigraph/tests/integration/traversal_adaptive.rs index dc3f70ce3..5d1764d07 100644 --- a/crates/omnigraph/tests/integration/traversal_adaptive.rs +++ b/crates/omnigraph/tests/integration/traversal_adaptive.rs @@ -83,7 +83,9 @@ async fn clique_db(dir: &tempfile::TempDir) -> Session { .unwrap(); // The endpoint BTREEs must exist, or dispatch prices a degraded scan and // takes CSR outright — the switch under test would never arm. - db.optimize().await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); db } @@ -189,7 +191,9 @@ query far($name: String) { jsonl.push('\n'); let db = session(Omnigraph::init(uri, TEST_SCHEMA).await.unwrap()); db.load_jsonl(&jsonl, LoadMode::Overwrite).await.unwrap(); - db.optimize().await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); let p = params(&[("$name", "p0")]); let csr_db = with_traversal(&db, Traversal::Csr); @@ -322,7 +326,9 @@ async fn corrupt_artifact_falls_back_to_the_scan_build() { async fn stale_artifact_is_rejected_after_an_edge_write() { let dir = tempfile::tempdir().unwrap(); let db = init_and_load(&dir).await; - db.optimize().await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); let artifact = dir.path().join("__graph_index/csr-current.bin"); assert!(artifact.is_file()); diff --git a/crates/omnigraph/tests/integration/warm_read_cost.rs b/crates/omnigraph/tests/integration/warm_read_cost.rs index d9e05e316..172c2d8b4 100644 --- a/crates/omnigraph/tests/integration/warm_read_cost.rs +++ b/crates/omnigraph/tests/integration/warm_read_cost.rs @@ -956,10 +956,13 @@ async fn assert_cached_borrower_survives_branch_delete( "{branch} payload changed" ); } - db.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); assert!( @@ -995,10 +998,13 @@ async fn assert_cached_borrower_survives_branch_delete( .unwrap(); let recreated_pin = helpers::pinned_version(writer, "feature", "node:Person").await; assert_ne!(recreated_pin, source_pin); - db.cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + db.cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); let pins = helpers::collector::detached_versions(&table_uri).await; diff --git a/crates/omnigraph/tests/integration/write_cost.rs b/crates/omnigraph/tests/integration/write_cost.rs index d33fd8053..6b155a215 100644 --- a/crates/omnigraph/tests/integration/write_cost.rs +++ b/crates/omnigraph/tests/integration/write_cost.rs @@ -38,7 +38,7 @@ use crate::helpers::{ // `_graph_commits`/`_graph_commit_actors` tables are retired), so the manifest scan // also covers the lineage and actor rows the authenticated write path appends. // Without internal-table compaction these scans are O(fragments) and grow forever; -// step 2 brings the internal tables into `db.optimize()`, so after compaction the +// step 2 brings the internal tables into `db.optimize_as(None, omnigraph::db::OptimizeRequest::default())`, so after compaction the // per-write scan is flat. The test runs the **authenticated (actorful) write path** // — every commit carries an actor — and compacts at each depth checkpoint before // measuring, pinning the production invariant "a periodically-compacted graph's @@ -62,7 +62,9 @@ async fn internal_table_scans_are_flat_in_history() { } // Step 2: compaction folds all three internal tables' O(depth) fragments back // to a small constant, so the following write's scan of them is flat. - db.optimize().await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); let io = measure_insert_as(&db, &format!("lock_{d}"), ACTOR).await; current += 1; // the measured write advanced depth by one eprintln!( @@ -87,7 +89,9 @@ async fn compacted_graph_at_depth(depth: u64) -> (u64, tempfile::TempDir, helper let dir = tempfile::tempdir().unwrap(); let db = local_graph(&dir).await; commit_many(&db, depth as usize).await; - db.optimize().await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); (depth, dir, db) } @@ -106,7 +110,7 @@ async fn ensure_indices_manifest_reads_are_flat_in_history() { for (depth, _dir, db) in [graphs.0, graphs.1] { let indexed_schema = helpers::TEST_SCHEMA.replace("age: I32?", "age: I32? @index"); db.apply_schema(&indexed_schema).await.unwrap(); - let (result, io) = measure(db.ensure_indices()).await; + let (result, io) = measure(db.ensure_indices_as(None)).await; result.unwrap(); eprintln!( "ensure_indices depth~{depth}: __manifest={} data={}", @@ -147,7 +151,7 @@ async fn ensure_indices_writes_no_control_object() { let before_write_text = counts.write_text(); let before_delete = counts.delete(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); assert_eq!( counts.write_text() - before_write_text, 0, @@ -274,11 +278,15 @@ async fn optimize_writes_no_control_object() { let before_write_text = counts.write_text(); let before_delete = counts.delete(); - let stats = db.optimize().await.unwrap(); + let stats = db + .optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap() + .tables; assert!( stats .iter() - .any(|stat| stat.type_key == "node:Person" && stat.committed), + .any(|stat| stat.type_key == "node:Person" && stat.committed()), "Person must compact: {stats:?}" ); assert_eq!( @@ -310,7 +318,8 @@ async fn optimize_manifest_reads_are_flat_in_history() { commit_many(&db, 3).await; let commits_before = db.list_commits(None).await.unwrap().len(); - let (result, io) = measure(db.optimize()).await; + let (result, io) = + measure(db.optimize_as(None, omnigraph::db::OptimizeRequest::default())).await; result.unwrap(); assert_eq!( db.list_commits(None).await.unwrap().len(), @@ -357,7 +366,7 @@ async fn internal_table_scans_are_flat_without_compaction() { commit_many_as(&db, (d - current) as usize, ACTOR).await; current = d; } - // NO `db.optimize()` here — that omission is the whole point. The flat gate + // NO `db.optimize_as(None, omnigraph::db::OptimizeRequest::default())` here — that omission is the whole point. The flat gate // above compacts before measuring and so never exercises this served regime. let io = measure_insert_as(&db, &format!("served_{d}"), ACTOR).await; current += 1; // the measured write advanced depth by one diff --git a/crates/omnigraph/tests/policy_engine_chassis.rs b/crates/omnigraph/tests/policy_engine_chassis.rs index 8982d42dc..dd3f2017e 100644 --- a/crates/omnigraph/tests/policy_engine_chassis.rs +++ b/crates/omnigraph/tests/policy_engine_chassis.rs @@ -8,7 +8,10 @@ //! PR #2 wired `apply_schema_as`. PR #3 fans the same `enforce()` call //! out to the remaining six writers — `mutate_as`, `load_as`, //! `ingest_as`, `branch_create_as` / `branch_create_from_as`, -//! `branch_delete_as`, `branch_merge_as`. Each writer pair below +//! `branch_delete_as`, `branch_merge_as`. The maintenance entry points — +//! `optimize_as`, `ensure_indices_as` / `ensure_indices_on_as`, +//! `rebuild_full_text_indices_on_as`, `repair_report_as` (`maintain`) and +//! `cleanup_as` (`reclaim`) — are covered the same way. Each writer pair below //! covers allow + deny via the engine-side gate; the allow case proves //! the enforce call is correctly scoped (i.e. doesn't reject a legit //! actor), the deny case proves it actually denies an unauthorized @@ -111,6 +114,31 @@ async fn init_with_policy_and_feature_branch( /// One JSONL record for `load_as` / `ingest_as` exercises. const ONE_PERSON_JSONL: &str = r#"{"type": "Person", "data": {"name": "Eve"}}"#; +/// Every directory and file under `root`, with file bytes: what a refused +/// call must leave as it found it. +fn physical_tree(root: &Path) -> std::collections::BTreeMap>> { + fn visit( + root: &Path, + path: &Path, + entries: &mut std::collections::BTreeMap>>, + ) { + for entry in fs::read_dir(path).unwrap() { + let entry = entry.unwrap(); + let path = entry.path(); + let relative = path.strip_prefix(root).unwrap().to_path_buf(); + if entry.file_type().unwrap().is_dir() { + entries.insert(relative, None); + visit(root, &path, entries); + } else { + entries.insert(relative, Some(fs::read(path).unwrap())); + } + } + } + let mut entries = std::collections::BTreeMap::new(); + visit(root, root, &mut entries); + entries +} + fn assert_denied(result: Result, what: &str) { match result { Err(OmniError::Policy(msg)) => { @@ -801,99 +829,91 @@ async fn non_canonical_branch_names_cannot_reach_a_protected_branch() { } #[tokio::test] -async fn index_maintenance_enforces_selected_branch_before_effects_and_records_actor_issue_840() { - fn physical_tree( - root: &Path, - ) -> std::collections::BTreeMap>> { - fn visit( - root: &Path, - path: &Path, - entries: &mut std::collections::BTreeMap>>, - ) { - for entry in fs::read_dir(path).unwrap() { - let entry = entry.unwrap(); - let path = entry.path(); - let relative = path.strip_prefix(root).unwrap().to_path_buf(); - if entry.file_type().unwrap().is_dir() { - entries.insert(relative, None); - visit(root, &path, entries); - } else { - entries.insert(relative, Some(fs::read(path).unwrap())); - } - } - } - let mut entries = std::collections::BTreeMap::new(); - visit(root, root, &mut entries); - entries - } - +async fn full_text_rebuild_enforces_change_or_maintain_on_the_selected_branch_and_records_actor() { let dir = tempfile::tempdir().unwrap(); let db = init_and_load(&dir).await; db.branch_create("feature").await.unwrap(); - // A row on feature leaves its Person identity index lagging, so an - // index build there has real work to refuse or publish. - db.load("feature", ONE_PERSON_JSONL, LoadMode::Merge) - .await - .unwrap(); drop(db); - // This actor can change unprotected feature, but not protected main. - // A hardcoded main scope or a different maintenance action must fail. - let policy = POLICY_YAML.replace(" branch_scope: any", " branch_scope: unprotected"); + // `act-allowed` can change unprotected feature, `act-maintainer` can + // maintain it, and neither reaches protected main. A hardcoded main scope + // or an action other than `change` or `maintain` must fail. + let policy = format!( + "{} - id: maintainers\n allow:\n actors: {{ group: maintainers }}\n actions: [maintain]\n branch_scope: unprotected\n", + POLICY_YAML + .replace(" branch_scope: any", " branch_scope: unprotected") + .replace( + " readers: [act-denied]\n", + " readers: [act-denied]\n maintainers: [act-maintainer]\n", + ) + ); let db = Omnigraph::open(dir.path().to_str().unwrap()).await.unwrap(); let (db, _engine) = install_policy_source(db, dir.path(), &policy); let main_before = db.list_commits(None).await.unwrap(); let feature_before = db.list_commits(Some("feature")).await.unwrap(); let files_before = physical_tree(dir.path()); - for (branch, actor) in [("feature", "act-denied"), ("main", "act-allowed")] { - assert_denied( - db.rebuild_full_text_indices_on_as(branch, Some(actor)) - .await, - "rebuild_full_text_indices_on_as", - ); - assert_denied( - db.ensure_indices_on_as(branch, Some(actor)).await, - "ensure_indices_on_as", + for (branch, actor) in [ + ("feature", "act-denied"), + ("main", "act-allowed"), + ("main", "act-maintainer"), + ] { + let result = db + .rebuild_full_text_indices_on_as(branch, Some(actor)) + .await; + // Either grant would admit the rebuild, so the refusal names both. + assert!( + matches!(&result, Err(OmniError::Policy(message)) + if message.contains("'maintain'") && message.contains("'change'")), + "{actor} on {branch}: {result:?}" ); + assert_denied(result, "rebuild_full_text_indices_on_as"); assert_eq!( physical_tree(dir.path()), files_before, "policy refusal must not write index artifacts, native refs, sidecars, or publications" ); } - for error in [ - db.rebuild_full_text_indices_on("feature") - .await - .unwrap_err(), - db.ensure_indices_on("feature").await.unwrap_err(), - ] { - assert!( - matches!(&error, OmniError::Policy(message) if message.contains("no actor")), - "{error}" - ); - } + let error = db + .rebuild_full_text_indices_on_as("feature", None) + .await + .unwrap_err(); + assert!( + matches!(&error, OmniError::Policy(message) if message.contains("no actor")), + "{error}" + ); assert_eq!( physical_tree(dir.path()), files_before, - "the no-actor wrappers must not bypass policy" + "a missing actor must not bypass policy" ); - let result = db - .rebuild_full_text_indices_on_as("feature", Some("act-allowed")) - .await - .expect("Change on the selected unprotected branch must permit a real rebuild"); - assert_eq!(result.branch, "feature"); - assert_eq!(result.rebuilt_indexes.len(), 2); - let feature_after = db.list_commits(Some("feature")).await.unwrap(); - assert_eq!(feature_after.len(), feature_before.len() + 1); - assert_eq!( - result.graph_commit_id.as_deref(), - Some(feature_after[0].graph_commit_id.as_str()) - ); - assert_eq!(feature_after[0].actor_id.as_deref(), Some("act-allowed")); + let mut parent = feature_before[0].graph_commit_id.clone(); + for (actor, grant) in [("act-allowed", "change"), ("act-maintainer", "maintain")] { + let result = db + .rebuild_full_text_indices_on_as("feature", Some(actor)) + .await + .unwrap_or_else(|error| { + panic!( + "{grant} on the selected unprotected branch must permit a real rebuild: {error}" + ) + }); + assert_eq!(result.branch, "feature"); + assert_eq!(result.rebuilt_indexes.len(), 2); + let feature_after = db.list_commits(Some("feature")).await.unwrap(); + assert_eq!( + result.graph_commit_id.as_deref(), + Some(feature_after[0].graph_commit_id.as_str()) + ); + assert_eq!(feature_after[0].actor_id.as_deref(), Some(actor)); + assert_eq!( + feature_after[0].parent_commit_id.as_deref(), + Some(parent.as_str()) + ); + parent = feature_after[0].graph_commit_id.clone(); + } assert_eq!( - feature_after[0].parent_commit_id.as_deref(), - Some(feature_before[0].graph_commit_id.as_str()) + db.list_commits(Some("feature")).await.unwrap().len(), + feature_before.len() + 2 ); let main_after = db.list_commits(None).await.unwrap(); assert_eq!(main_after.len(), main_before.len()); @@ -901,11 +921,131 @@ async fn index_maintenance_enforces_selected_branch_before_effects_and_records_a main_after[0].graph_commit_id, main_before[0].graph_commit_id ); +} + +/// Maintenance grants, apart from data writes: `act-maintainer` may maintain +/// any branch and `act-branch-maintainer` only unprotected ones; +/// `act-reclaimer` may reclaim; `act-writer` holds `change` everywhere and no +/// maintenance grant; `act-denied` holds nothing. +const MAINTENANCE_POLICY_YAML: &str = r#" +version: 1 +groups: + maintainers: [act-maintainer] + branch-maintainers: [act-branch-maintainer] + reclaimers: [act-reclaimer] + writers: [act-writer] + readers: [act-denied] +protected_branches: [main] +rules: + - id: maintainers + allow: + actors: { group: maintainers } + actions: [maintain] + branch_scope: any + - id: branch-maintainers + allow: + actors: { group: branch-maintainers } + actions: [maintain] + branch_scope: unprotected + - id: reclaimers + allow: + actors: { group: reclaimers } + actions: [reclaim] + - id: writers + allow: + actors: { group: writers } + actions: [change] + branch_scope: any +"#; + +/// A graph with a `feature` branch whose Person identity index lags one +/// loaded row, so an index build there has work to publish, and two extra +/// Person fragments on main, so optimize has a compaction to publish, under +/// [`MAINTENANCE_POLICY_YAML`]. +async fn init_with_maintenance_policy(dir: &tempfile::TempDir) -> Session { + let db = init_and_load(dir).await; + db.branch_create("feature").await.unwrap(); + db.load("feature", ONE_PERSON_JSONL, LoadMode::Merge) + .await + .unwrap(); + for (name, age) in [("Eve", 40), ("Frank", 41)] { + let params = mixed_params(&[("$name", name)], &[("$age", age)]); + db.mutate("main", MUTATION_QUERIES, "insert_person", ¶ms) + .await + .unwrap(); + } + drop(db); + let db = Omnigraph::open(dir.path().to_str().unwrap()).await.unwrap(); + install_policy_source(db, dir.path(), MAINTENANCE_POLICY_YAML).0 +} + +fn assert_no_actor(result: Result, what: &str) { + match result { + Err(OmniError::Policy(message)) if message.contains("no actor") => {} + other => panic!("{what}: expected the no-actor refusal, got {other:?}"), + } +} + +/// Optimize, index builds and the repair report require `maintain` on the +/// branch they maintain, before any effect: `change` does not imply it, a +/// grant on unprotected branches does not reach main, and a missing actor is +/// refused. An allowed index build or optimize records its actor on the +/// commit it publishes, and an index build on a branch leaves main alone. +#[tokio::test] +async fn maintenance_entry_points_enforce_maintain_on_their_branch_before_effects_issue_840() { + let dir = tempfile::tempdir().unwrap(); + let db = init_with_maintenance_policy(&dir).await; + let main_before = db.list_commits(None).await.unwrap(); + let files_before = physical_tree(dir.path()); + + for actor in [ + "act-denied", + "act-writer", + "act-reclaimer", + "act-branch-maintainer", + ] { + let actor = Some(actor); + assert_denied( + db.optimize_as(actor, omnigraph::db::OptimizeRequest::default()) + .await, + "optimize_as", + ); + assert_denied(db.ensure_indices_as(actor).await, "ensure_indices_as"); + assert_denied( + db.ensure_indices_on_as("main", actor).await, + "ensure_indices_on_as(main)", + ); + assert_denied(db.repair_report_as(actor).await, "repair_report_as"); + } + for actor in ["act-denied", "act-writer", "act-reclaimer"] { + assert_denied( + db.ensure_indices_on_as("feature", Some(actor)).await, + "ensure_indices_on_as(feature)", + ); + } + assert_no_actor( + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await, + "optimize_as", + ); + assert_no_actor(db.ensure_indices_as(None).await, "ensure_indices_as"); + assert_no_actor( + db.ensure_indices_on_as("feature", None).await, + "ensure_indices_on_as", + ); + assert_no_actor(db.repair_report_as(None).await, "repair_report_as"); + assert_eq!( + physical_tree(dir.path()), + files_before, + "a refused maintenance call must not write index artifacts, compactions or publications" + ); + assert_eq!(db.list_commits(None).await.unwrap(), main_before); + let feature_before = db.list_commits(Some("feature")).await.unwrap(); let built = db - .ensure_indices_on_as("feature", Some("act-allowed")) + .ensure_indices_on_as("feature", Some("act-branch-maintainer")) .await - .expect("Change on the selected unprotected branch must permit an index build"); + .expect("maintain on an unprotected branch admits its index builds"); assert_eq!(built.branch, "feature"); assert!( built @@ -915,16 +1055,81 @@ async fn index_maintenance_enforces_selected_branch_before_effects_and_records_a && index.action == omnigraph::db::BuiltIndexAction::Refreshed), "{built:?}" ); - let feature_built = db.list_commits(Some("feature")).await.unwrap(); - assert_eq!(feature_built.len(), feature_after.len() + 1); + let feature_after = db.list_commits(Some("feature")).await.unwrap(); + assert_eq!(feature_after.len(), feature_before.len() + 1); assert_eq!( built.graph_commit_id.as_deref(), - Some(feature_built[0].graph_commit_id.as_str()) + Some(feature_after[0].graph_commit_id.as_str()) ); - assert_eq!(feature_built[0].actor_id.as_deref(), Some("act-allowed")); assert_eq!( - db.list_commits(None).await.unwrap()[0].graph_commit_id, - main_before[0].graph_commit_id + feature_after[0].actor_id.as_deref(), + Some("act-branch-maintainer") + ); + assert_eq!(db.list_commits(None).await.unwrap(), main_before); + let report = db + .repair_report_as(Some("act-maintainer")) + .await + .expect("maintain on main admits the repair report"); + assert!(!report.datasets.is_empty()); + db.ensure_indices_as(Some("act-maintainer")) + .await + .expect("maintain on main admits index builds"); + + let head_before_optimize = db.list_commits(None).await.unwrap()[0] + .graph_commit_id + .clone(); + let stats = db + .optimize_as( + Some("act-maintainer"), + omnigraph::db::OptimizeRequest::default(), + ) + .await + .expect("maintain on main admits optimize") + .tables; + assert!( + stats + .iter() + .any(|stat| stat.type_key == "node:Person" && stat.committed()), + "{stats:?}" + ); + let main_after = db.list_commits(None).await.unwrap(); + assert_eq!( + main_after[0].parent_commit_id.as_deref(), + Some(head_before_optimize.as_str()) + ); + assert_eq!(main_after[0].actor_id.as_deref(), Some("act-maintainer")); +} + +/// Cleanup requires `reclaim`, which no other grant implies, before any +/// effect. +#[tokio::test] +async fn cleanup_as_enforces_reclaim_before_effects() { + let dir = tempfile::tempdir().unwrap(); + let db = init_with_maintenance_policy(&dir).await; + let keep_one = || omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }; + let files_before = physical_tree(dir.path()); + for actor in ["act-denied", "act-writer", "act-maintainer"] { + assert_denied(db.cleanup_as(keep_one(), Some(actor)).await, "cleanup_as"); + } + assert_no_actor(db.cleanup_as(keep_one(), None).await, "cleanup_as"); + assert_eq!( + physical_tree(dir.path()), + files_before, + "a refused cleanup must not delete anything" + ); + + let rows = db + .cleanup_as(keep_one(), Some("act-reclaimer")) + .await + .expect("reclaim admits cleanup"); + assert!(rows.iter().all(|row| row.error.is_none()), "{rows:?}"); + assert_ne!( + physical_tree(dir.path()), + files_before, + "the admitted cleanup reclaims the history keep 1 does not retain" ); } diff --git a/crates/omnigraph/tests/schema_apply.rs b/crates/omnigraph/tests/schema_apply.rs index 9ac4ee162..e461967d4 100644 --- a/crates/omnigraph/tests/schema_apply.rs +++ b/crates/omnigraph/tests/schema_apply.rs @@ -308,7 +308,7 @@ query insert_project($name: String) { ); let detached_before_indices = helpers::collector::detached_versions(&project_uri).await; stale_handle - .ensure_indices_on("source") + .ensure_indices_on_as("source", None) .await .expect("index planning must use the same token-bound post-apply catalog"); let project_after_indices = project_published( @@ -372,7 +372,7 @@ async fn stale_handle_branch_delete_gates_tables_added_by_schema_apply() { let index_handle = Arc::clone(&index_reconciler); let mut index_task = - tokio::spawn(async move { index_handle.ensure_indices_on("target").await }); + tokio::spawn(async move { index_handle.ensure_indices_on_as("target", None).await }); let index_blocked = tokio::time::timeout(std::time::Duration::from_millis(250), &mut index_task) .await @@ -1821,7 +1821,7 @@ node Tag { .collect::>() .join("\n"); db.load_jsonl(&rows, LoadMode::Overwrite).await.unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let before = db.snapshot_of(ReadTarget::branch("main")).await.unwrap(); let before_snapshot = db.resolve_snapshot("main").await.unwrap(); @@ -1937,7 +1937,7 @@ node Tag { } // Reconciliation builds from the desired declarations, so nothing indexes // a dropped column again. - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); for (table, dropped_id) in dropped_ids { assert!( coverage(&db, table) @@ -2124,7 +2124,9 @@ query strong($min: I32) { "an update carries the untouched Blob of an evolved row" ); - db.optimize().await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); assert_eq!( read_managed_blob_bytes( &db, @@ -2479,7 +2481,7 @@ async fn apply_schema_pure_type_rename_preserves_identity_path_and_version() { ); let db = helpers::session(Omnigraph::init(uri, &initial).await.unwrap()); db.load_jsonl(&data, LoadMode::Overwrite).await.unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let before_snapshot_id = db.resolve_snapshot("main").await.unwrap(); let before_snapshot = db.snapshot_of(ReadTarget::branch("main")).await.unwrap(); let before_version = before_snapshot.graph_manifest_version(); @@ -2675,10 +2677,13 @@ edge WorksAt: Human -> Company /// prunes the `__manifest` version naming it. async fn reclaim_dropped_history(db: &Session) { let stats = db - .cleanup(omnigraph::db::CleanupPolicyOptions { - keep_versions: Some(1), - older_than: None, - }) + .cleanup_as( + omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }, + None, + ) .await .unwrap(); for row in &stats { @@ -3110,7 +3115,7 @@ async fn apply_schema_defers_vector_index_on_empty_table() { db.load_jsonl(r#"{"type":"Doc","data":{"slug":"d1","body":"hello","embedding":[0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8]}}"#, LoadMode::Merge, ) .await .expect("loading a Doc with an embedding must succeed"); - db.ensure_indices() + db.ensure_indices_as(None) .await .expect("the deferred vector index must build once the table has a trainable vector"); } @@ -3261,7 +3266,7 @@ async fn full_text_analyzer_survives_schema_evolution_issue_904() { assert_eq!(search("summary").await, ["d3"]); // The reconciler builds the postings over the declarations. - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); assert_eq!(segments("text").await, [Some(2)]); assert_eq!(search("text").await, ["d1", "d2"]); assert_eq!(search("summary").await, ["d3"]); diff --git a/crates/omnigraph/tests/search.rs b/crates/omnigraph/tests/search.rs index 9468bfbe9..2abc8381d 100644 --- a/crates/omnigraph/tests/search.rs +++ b/crates/omnigraph/tests/search.rs @@ -207,7 +207,7 @@ async fn init_search_db(dir: &tempfile::TempDir) -> Session { db.load_jsonl(SEARCH_DATA, LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); db } @@ -226,7 +226,7 @@ async fn init_mock_embedding_search_db(dir: &tempfile::TempDir) -> Session { db.load_jsonl(&mock_embedding_seed_data(), LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); db } @@ -236,7 +236,7 @@ async fn init_model_recorded_search_db(dir: &tempfile::TempDir) -> Session { db.load_jsonl(&mock_embedding_seed_data(), LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); db } @@ -561,7 +561,7 @@ query delete_middle() { "#; let db = session(Omnigraph::init(uri, schema).await.unwrap()); db.load_jsonl(&seed, LoadMode::Overwrite).await.unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); helpers::forge_linear_head_from_pin(&db, "main", "node:Doc", 0).await; { use lance::index::DatasetIndexExt; @@ -1237,7 +1237,7 @@ query nearest_all($q: Vector(4)) { let uri = dir.path().to_str().unwrap(); let db = session(Omnigraph::init(uri, schema).await.unwrap()); db.load_jsonl(&rows, LoadMode::Overwrite).await.unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let db = with_setting(&db, "ann_nprobes", "100000"); use omnigraph::instrumentation::{QueryIoProbes, with_query_io_probes}; @@ -1318,7 +1318,7 @@ query nearest_all($q: Vector(4)) { let uri = dir.path().to_str().unwrap(); let db = session(Omnigraph::init(uri, schema).await.unwrap()); db.load_jsonl(&rows, LoadMode::Overwrite).await.unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let probes = QueryIoProbes::default(); let result = with_query_io_probes(probes.clone(), async { @@ -1425,7 +1425,7 @@ async fn issue_567_line_docs(uri: &str, rows: usize, edge_every: Option) db.load_jsonl(&lines.join("\n"), LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); db } @@ -2081,7 +2081,7 @@ async fn injected_embedding_config_is_used_instead_of_env() { db.load_jsonl(&mock_embedding_seed_data(), LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let result = query_main( &db, @@ -2172,7 +2172,7 @@ async fn bm25_ordered_aggregate_counts_all_matches_not_the_capped_scan() { db.load_jsonl(&underfill_seed_data(), LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let db = db; use omnigraph::instrumentation::{QueryIoProbes, with_query_io_probes}; @@ -2221,7 +2221,7 @@ async fn rrf_arms_scan_uncapped_in_one_pass() { db.load_jsonl(&underfill_seed_data(), LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); use omnigraph::instrumentation::{QueryIoProbes, with_query_io_probes}; // This test pins the POSTFILTER plan's invariants (uncapped one-pass @@ -2293,7 +2293,7 @@ query uncapped_all($q: String) { db.load_jsonl(&underfill_seed_data(), LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); // Appended AFTER the index build: their fragment is uncovered, and their // term frequency (40 > the seed's max 20) puts them at the top of any // correct ranking. @@ -2438,7 +2438,9 @@ async fn mutation_with_deferred_index_coverage_remains_searchable() { // Ordinary optimize must preserve certified postings, not silently replace // them with an uncertified incremental fold. Both old and tail rows remain // searchable after data compaction and unrelated index maintenance. - db.optimize().await.unwrap(); + db.optimize_as(None, omnigraph::db::OptimizeRequest::default()) + .await + .unwrap(); for (term, slug) in [("Quasar", "quasar-notes"), ("Learning", "ml-intro")] { let result = query_main(&db, SEARCH_QUERIES, "text_search", ¶ms(&[("$q", term)])) .await @@ -2599,7 +2601,10 @@ async fn uncertified_full_text_refuses_all_search_routes_but_not_ordinary_reads( Err(OmniError::FullTextIndexRebuildRequired { .. }) )); - let rebuilt = db.rebuild_full_text_indices_on("main").await.unwrap(); + let rebuilt = db + .rebuild_full_text_indices_on_as("main", None) + .await + .unwrap(); assert!(!rebuilt.rebuilt_indexes.is_empty()); assert!( query_main( @@ -2702,7 +2707,7 @@ node Doc { 0, "load publishes exact data effects and leaves physical indexes pending" ); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); let ds = snapshot_main(&db) .await @@ -2800,7 +2805,7 @@ async fn full_text_analyzer_holds_after_first_rows_and_overwrites_issue_904() { .unwrap(); assert_eq!(hits("main", "docs").await, ["d1", "d2"], "first rows"); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); db.load("main", &rows("Doc", "d"), LoadMode::Overwrite) .await .unwrap(); @@ -2892,7 +2897,7 @@ async fn a_full_text_index_with_no_segment_refuses_the_call_issue_747() { .unwrap_err(), ); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); assert_eq!( first_column_sorted( &query_branch(&db, "main", ANALYZER_QUERIES, "docs", &deep) diff --git a/crates/omnigraph/tests/system_column_upgrade.rs b/crates/omnigraph/tests/system_column_upgrade.rs index cfaf0e354..ca5a2dc84 100644 --- a/crates/omnigraph/tests/system_column_upgrade.rs +++ b/crates/omnigraph/tests/system_column_upgrade.rs @@ -71,7 +71,7 @@ async fn legacy_graph_with_data(dir: &tempfile::TempDir) -> Session { db.load_jsonl(LEGACY_DATA, LoadMode::Overwrite) .await .unwrap(); - db.ensure_indices().await.unwrap(); + db.ensure_indices_as(None).await.unwrap(); db } @@ -226,7 +226,7 @@ async fn assert_upgraded(db: &Session, dir: &tempfile::TempDir, expected_export: .await .expect_err("the respelled @unique(@src, @dst) still refuses a duplicate endpoint pair"); assert_eq!(count_rows(db, "edge:WorksAt").await, 3); - db.ensure_indices() + db.ensure_indices_as(None) .await .expect("index reconciliation after the rename is a no-op by field id"); } @@ -407,7 +407,7 @@ async fn system_column_upgrade_history_survives_unrelated_reclaimed_tables() { main.retained.contains(&company_moved) && main.would_prune.contains(&version_before), "the run keeps the Company write's `__manifest` version and drops the pre-write one: {main:?}" ); - let stats = db.cleanup(keep).await.unwrap(); + let stats = db.cleanup_as(keep, None).await.unwrap(); let row = stats.iter().find(|row| row.type_key == first).unwrap(); assert!(row.error.is_none(), "{row:?}"); assert!( diff --git a/docs/dev/control-plane.md b/docs/dev/control-plane.md index 81502ae40..360665e71 100644 --- a/docs/dev/control-plane.md +++ b/docs/dev/control-plane.md @@ -618,6 +618,7 @@ server. Public actor construction cannot grant signed-token permissions. on `AuthenticatedActor` exposes only verified claims. Policy embedders must handle `PolicyAction::ConfigManage`, +`PolicyAction::Maintain`, `PolicyAction::Reclaim`, `PolicyResourceKind::Cluster`, and `PolicyEngineKind::Cluster`. These public enums are exhaustive, so an external exhaustive match must add the relevant arm when upgrading. Preserving existing constructors and direct storage-holder diff --git a/docs/dev/writes.md b/docs/dev/writes.md index f9971b22f..4f4d44f7e 100644 --- a/docs/dev/writes.md +++ b/docs/dev/writes.md @@ -266,6 +266,41 @@ unpublished staging the collector reclaims once it is dead. A strict mutation prepared before Optimize's publication reports the same read-set conflict as it would after any other writer. +A run with `max_gate_hold` defers every table whose turn comes after the +deadline, measured from gate acquisition, and publishes the tables it finished +(`run_table_pipeline`). A table whose own planning or staging fails with +definite evidence is reported `failed` and the others still publish; a +failure whose evidence stays uncertain fails the run. A cancellation +(`OptimizeRequest::with_cancellation`) is checked before every table's plan +and rewrite and before the publication: it drops the tables in progress, +defers every table as `cancelled` and publishes nothing; one that arrives +after the last check lets the run finish and skips only the artifact scan. +Per-table ceilings (`TableCeilings`) are measured from the table's compaction +plan before its first write, fragments first, and only when set: rows from +fragment metadata, bytes from recorded data-file sizes or a HEAD resolved by +the collector's base rule (`collector::data_file_location`, +`deletion_file_location`; Lance 13 records no deletion-file size), managed Blob +bytes from the descriptor scan compaction sizing uses, and the table's rows +when an index would be rebuilt or built. A table over one, or one whose size +cannot be read, is `deferred(resource_limit)` with nothing written. The run +byte budget (`run_source_bytes`, also the default table byte ceiling) defers +the first table that would pass it, and every later one, as `byte_budget`. +With `artifact_bytes`, the adjacency artifact is bounded twice: before the +edge scan by a lower bound (`persist::fixed_bytes` at the shortest header plus +8 bytes per live edge row), and during it by `GraphIndex::build_counted`, which +reads one batch at a time, adds each batch's growth of the object to +`persist::ArtifactCount` (from the widest header), and stops before keeping +another batch or allocating the adjacency arrays. Either stop is +`skipped(size_limit)` and writes nothing. The artifact write runs in a spawned +task the run awaits, so a dropped run cannot drop an issued write on the +mutable key. A caller's before-publish hook +(`OptimizeRequest::with_before_publish`) belongs to the shared publication +helper, `table_ops::commit_updates_on_branch_with_expected`: it runs once, +after the updates are prepared and before the coordinator lock and the +publisher's first attempt, with the lineage intent's nonce and the captured +head. Its failure publishes nothing and is `Unpublished`. The helper's other +callers pass no hook. + The index writer (`ensure_indices`, which `omnigraph index build` runs on any branch, and the explicit full-text rebuild) follows the same protocol: it opens each productive table at its pin, stages the complete BTREE/FTS/vector @@ -558,7 +593,7 @@ carried stored external reference the policy refuses fails as | Retryable authority movement before effects on a replay-safe adapter | Discard the complete attempt and reprepare boundedly | | Strict read-set movement | `ReadSetChanged` | | Exact duplicate on strict insert | `KeyConflict` | -| Mutation, load, Blob write or merge fails before publication, after detached effects | Typed error; no graph movement; the detached staging is unpublished and the collector reclaims it once its recorded authority is gone. The table store marks an opaque error (storage, I/O, internal) of a staged write or a detached commit `CompletionEvidence::Unpublished` at the source, and the mutation, load, Blob and merge writers mark the steps between, so a server does not treat that logical error as uncertain; native storage settlement is checked independently, and a typed refusal keeps its shape | +| Mutation, load, Blob write, merge or optimize fails before publication, after detached effects | Typed error; no graph movement; the detached staging is unpublished and the collector reclaims it once its recorded authority is gone. The table store marks an opaque error (storage, I/O, internal) of a staged write or a detached commit `CompletionEvidence::Unpublished` at the source, and the mutation, load, Blob, merge and optimize writers mark the steps between (an optimize table's own failure, `BeforeEffect` in planning and `Unpublished` in staging, fails only that table), so a server does not treat that logical error as uncertain; native storage settlement is checked independently, and a typed refusal keeps its shape | | A manifest request no re-issue settles | The write is reported in doubt and nothing records it. The version is created only if absent, so the request lands before the branch's next publication or never: that publication either loads it as the head and builds on it, or takes the version first, after which the late request fails its precondition (a race between the two is the ordinary version-CAS retry). The attempt's detached effects stay unreachable unless it lands, and the collector keeps them while their recorded publication authority is current | | The publisher fails | Every publisher failure carries its evidence. Its reads, preparation and history archiving, and a manifest request that was never sent (a failed transaction-file put precedes it) or that the store refused outright, are `Unpublished`. A manifest request with an ambiguous answer is re-issued unchanged by the dataset's commit handler until the store answers it; Lance reads a held version back by its transaction. Lance deletes the attempt's transaction file only on a missing version the commit's own request record calls definitive, so a request that may still land keeps the file its manifest would name. A version that may hold the commit after that is read back, and only an outcome the read-back cannot settle is reported in doubt | | Schema apply or the system-column upgrade reports an error after proven publication | `RecoveryRequired` can name the committed outcome; its tables and contract are durable, and refresh/reopen rebuilds only the in-memory view | @@ -587,6 +622,15 @@ callers must pass one when a policy checker is installed. Actor attribution travels with the pre-minted graph lineage and is published with the same manifest CAS. +Maintenance has no actor-less public form. `optimize_as`, `ensure_indices_as`, +`ensure_indices_on_as` and `rebuild_full_text_indices_on_as` check `maintain` +on the branch they maintain, `main` for optimize; the full-text rebuild also +admits `change`, its grant before `maintain` existed (`Omnigraph::enforce_any`, +`PolicyChecker::check_any`). `cleanup_as` checks `reclaim` with no branch, and +the read-only `repair_report_as` checks `maintain` on `main`. +`forbidden_apis.rs::maintenance_entry_points_take_an_actor` keeps every +registered maintenance surface an `_as` form with an `actor` parameter. + Bearer authentication (`401 unauthorized`), the HTTP contract check (`400 api_contract_mismatch`) and policy (`403 forbidden`) refuse a served write only before its first durable effect. The server runs diff --git a/docs/user/operations/maintenance.md b/docs/user/operations/maintenance.md index c79812b03..31cfbb47c 100644 --- a/docs/user/operations/maintenance.md +++ b/docs/user/operations/maintenance.md @@ -22,6 +22,14 @@ targets accept local files only; an inline [S3 cluster root](s3-cluster-roots.md is the one cloud-direct exception. See [operator configuration](../cli/targets.md#transport-and-capability-boundaries). +Each command acts as the operator configuration's `operator.actor`, and the +commit it publishes records that actor; `index build` and `index rebuild` also +accept `--as` in its place. Direct CLI access, including `--target`, does not +load the server's Cedar policy; storage permissions are its trust boundary. An +embedded host that installs a policy checker enforces the +[policy actions](policy.md#actions) before any effect: `maintain` for optimize, +index builds, the full-text rebuild and repair, and `reclaim` for cleanup. + ## Optimize ```bash @@ -38,58 +46,56 @@ dropped-value rewrite and the traversal artifact stay with optimize. Optimize does not delete old versions or collect unused table forks. Use `cleanup` for storage reclamation. -Dropping a property removes it from the schema without rewriting its table -(see [schema changes](../schema/index.md#schema-changes)), so its values stay -in the table's data files. Every optimize rewrites each fragment that still -holds them, whatever its size, copying only the remaining properties; the -dropped values are not read. A `cleanup` then deletes the old files, Blob -values included, once nothing it retains references them: no commit its -retention keeps, no live branch created from a commit before that optimize, -and no tag naming such a commit. Merging a branch does not release its files; -deleting it does. A table holding no dropped values and nothing else to -compact is left alone. +Dropping a property removes it from the schema without rewriting its table (see +[schema changes](../schema/index.md#schema-changes)), so its values stay in the +table's data files. Every optimize rewrites each fragment that still holds them, +whatever its size, copying only the remaining properties; the dropped values are +not read. A `cleanup` then deletes the old files, Blob values included, once +nothing it retains references them: no commit its retention keeps, no live +branch created from a commit before that optimize, and no tag naming such a +commit. Merging a branch does not release its files; deleting it does. A table +holding no dropped values and nothing else to compact is left alone. Each table's work is staged as detached Lance versions of the table's current -pin and published in one graph commit, like any other write. A run -that fails before that commit leaves the graph unchanged and no recovery -state; after it the published pins are the tables' versions, with nothing -left to finish. Optimize runs beside -live writers: a write that lands on a table while its compaction is staged -fails the run with a read-set conflict, and the next run re-plans from the -new state. An update or delete prepared before an optimize published reports -the same conflict and is retried by its caller. Lance compacts neighbouring -fragments only when the same indexes cover them, so a table whose index -coverage was uneven before a run may coalesce fully only on the next run, +pin and published in one graph commit, like any other write. A run that fails +before that commit leaves the graph unchanged and no recovery state; after it +the published pins are the tables' versions, with nothing left to finish. +Optimize runs beside live writers: a write that lands on a table while its +compaction is staged fails the run with a read-set conflict, and the next run +re-plans from the new state. An update or delete prepared before an optimize +published reports the same conflict and is retried by its caller. Lance compacts +neighbouring fragments only when the same indexes cover them, so a table whose +index coverage was uneven before a run may coalesce fully only on the next run, after the rebuilt coverage is in place. Compacting a table with Blob properties reads each managed Blob value into -memory to rewrite it. Optimize sizes the read batches of each compaction task -(a group of neighbouring fragments compacted together) from the task's largest -row, summing that row's managed Blob values, so one batch materializes at most -32 MiB of managed Blob payload whatever the fragment sizes. External Blob -references are carried without reading the referenced object and count -nothing. A task holding a row over 16 MiB compacts one row per batch, and a -single row over 32 MiB is materialized whole. This bounds the payload of a -batch, not the process heap: Lance's writer copies each inline payload into -the arrays it prepares while it still holds the batch, so memory use exceeds -the payload. Every task of a Blob table gets this batch size explicitly, 1 to -8192 rows, so it takes precedence over `LANCE_DEFAULT_BATCH_SIZE` there, -including when the variable is smaller. When a fragment being compacted holds -a Blob descriptor the engine's decoder rejects, sizing can refuse the -compaction with a Blob integrity error before that table is rewritten, and -nothing is committed; a table with nothing to compact is not scanned. -Optimize works on up to -`OMNIGRAPH_MAINTENANCE_CONCURRENCY` tables at once (default 8), so budget for -that many Blob tables compacting together. +memory. Optimize sizes each compaction batch from its largest row, whatever +`LANCE_DEFAULT_BATCH_SIZE` says, so one batch holds at most 32 MiB of managed +Blob payload; a larger row is read whole, external references are not read, +and the writer's own copies come on top. A Blob descriptor the decoder rejects +refuses the compaction before that table is rewritten. Optimize plans up to +`OMNIGRAPH_MAINTENANCE_CONCURRENCY` tables ahead (default 8) and rewrites up to +that many at once, so budget for that many Blob tables compacting together. + +### Bounding a run + +Writes to `main` wait while optimize holds its writer gates. A run given +`--max-gate-hold 30s` starts no table after that long except the first: the +rest are `deferred`, finished tables publish, and a writer waits at most the +bound plus one rewrite. Tables run in type-key order; `resume_at` names the +first deferred one, and `--tables` runs only the listed tables, in order. Each +table's `outcome` is `optimized`, `unchanged`, `deferred`, `skipped` or +`failed` (with its `error`; the other tables still publish). The report adds +`input_commit`, `graph_commit_id`, `gate_hold_ms` and the `configuration_clear` +and `artifact` outcomes, which never fail a published run. Optimize also persists the traversal-adjacency artifact (`__graph_index/csr-current.bin`), which cold traversal builds load instead of -scanning every edge dataset. The artifact is derived and regenerable: optimize -is its only writer, every load is verified and falls back to the in-memory -build, and a failed write is a warning, not an optimize error. The artifact is -rewritten by optimize runs that advance an edge dataset; on a store with no -edge work it is left as-is, and a stale copy costs only the artifact's speedup, -never correctness. +scanning every edge dataset. It is derived: every load is verified and falls +back to the in-memory build, so a stale copy or a failed write costs only its +speedup, and a failed write is reported as the `artifact` outcome. Optimize +builds it after releasing the writer gates, and only when the run advanced an +edge dataset or no artifact exists. Existing full-text indexes are preserved rather than incrementally merged. Unindexed rows remain searchable, but a growing uncovered tail can cost more to @@ -115,15 +121,15 @@ BTREE or vector index that leaves rows uncovered. Every table publishes in one graph commit on that branch; other branches and snapshots keep their indexes, and a branch with nothing to do publishes nothing. The default branch is `main`. As with optimize, a write that lands on a table while its batch is staged fails -the run with a read-set conflict; run the command again. A vector property with no vectors yet and full-text coverage stay pending: only -`index rebuild --kind full-text` refreshes a full-text index's coverage, and -queries match uncovered rows with the index's analyzer meanwhile. +the run with a read-set conflict; run the command again. A vector property with +no vectors yet and full-text coverage stay pending: only `index rebuild --kind +full-text` refreshes a full-text index's coverage, and queries match uncovered +rows with the index's analyzer meanwhile. JSON reports `branch`, `graph_commit_id` (null for a no-op), `indexes` with each `type_key`, `property` (the indexed column, `__id` for the identity index), `kind` (`btree`, `full_text` or `vector`) and `action` (`built` or `refreshed`), -and `pending_indexes` with each `type_key`, `property` and `reason`. `--as` and -the trust boundary are as for the full-text rebuild below. +and `pending_indexes` with each `type_key`, `property` and `reason`. ## Rebuild full-text indexes @@ -134,7 +140,8 @@ omnigraph index rebuild --kind full-text --target ./cluster-root \ ``` `--kind` is required. The deprecated spelling `rebuild-full-text-indexes` still -works for one release. +works for one release, and so does admitting the rebuild under `change` on the +selected branch instead of `maintain`. The command fully replaces text-search indexes from current entities using the engine's default English analyzer, including already indexed entities. It builds @@ -149,11 +156,6 @@ Success means the selected branch's planned rebuild was published, or an explici no-op with an empty index list, null commit, and empty `warnings` array. The default branch is `main`. -`--as` supplies actor attribution for this command. Direct CLI access, including -`--target`, does not load the server's Cedar policy; storage permissions are its -trust boundary. An embedded host that installs a policy checker also enforces -the Change permission on the selected branch before building or rebuilding. - Other branches and historical snapshots are not rewritten. Rebuild every live branch that needs full-text search; restoring an older snapshot may require rebuilding again. The operation does not regenerate embeddings or alter entity @@ -194,18 +196,17 @@ export the graph and load it into a new one; see ## Cleanup -Cleanup is a tracing collector over the graph's table storage. Every table -write is a detached Lance commit that a graph commit pins, so what a table -keeps is decided by which graph commits the run retains. Per live branch -the run computes the graph commits its policy keeps, takes the table -versions those commits pin as roots, marks every file the root manifests -reference (base and overlay data files, deletion files, index directories, -transaction files), and deletes unretained table versions and the files only -they reference. Selected merge bases and native tagged snapshots are retained -in addition to the versions selected by the policy. -Historical table lifetimes participate even after a type is dropped or its -name is re-added under a new identity. Blob sidecars follow their parent data -file's reachability. +Cleanup is a tracing collector over the graph's table storage. Every table write +is a detached Lance commit that a graph commit pins, so what a table keeps is +decided by which graph commits the run retains. Per live branch the run computes +the graph commits its policy keeps, takes the table versions those commits pin +as roots, marks every file the root manifests reference (base and overlay data +files, deletion files, index directories, transaction files), and deletes +unretained table versions and the files only they reference. Selected merge +bases and native tagged snapshots are retained in addition to the versions +selected by the policy. Historical table lifetimes participate even after a type +is dropped or its name is re-added under a new identity. Blob sidecars follow +their parent data file's reachability. `omnigraph cleanup` is the only cleanup to run against a graph root. Lance's own cleanup (`cleanup_old_versions`, and every vacuum or retention tool built diff --git a/docs/user/operations/policy.md b/docs/user/operations/policy.md index 350020a52..eab41948c 100644 --- a/docs/user/operations/policy.md +++ b/docs/user/operations/policy.md @@ -21,6 +21,8 @@ Graph-scoped actions: | `branch_merge` | Branch merge | | `commit_publish` | Publication of an unpublished commit onto its recorded target branch | | `commit_drop` | Drop of an unpublished commit | +| `maintain` | Content-preserving maintenance: optimize, index builds, the full-text rebuild and the repair report | +| `reclaim` | Cleanup, which permanently removes table versions and files no retained commit needs | | `invoke_query` | Entry to a stored query | | `admin` | Reserved graph administration | @@ -42,6 +44,17 @@ The reserved graph `admin` action does not grant cluster management. A stored mutation requires both `invoke_query` and `change`. A stored read requires `invoke_query` and `read`. +`maintain` and `reclaim` are separate from `change`, so an automation account +can compact a graph without the right to write data, and can hold `maintain` +without the right to delete history. `maintain` is judged on the branch the +operation maintains (`main` for optimize and the repair report) and takes +`branch_scope`; `reclaim` covers every branch's history and takes no branch +scope. The full-text rebuild accepts `change` or `maintain` on its branch; a +later release accepts `maintain` only, so grant `maintain` to the accounts that +run it. The engine checks these actions whenever a policy is installed on the +graph handle; the direct maintenance commands load no policy (see +[maintenance](maintenance.md)). + ## Bind a policy ```yaml @@ -124,7 +137,7 @@ empty, has leading or trailing whitespace, or contains a control character fails policy validation, and a request naming such a branch is refused as a bad request before any rule is evaluated (see [branch names](../branching/index.md#branch-names)). Server actions and graph-wide -`invoke_query` rules do not take branch scopes. `commit_publish` and +`invoke_query` and `reclaim` rules do not take branch scopes. `commit_publish` and `commit_drop` take `target_branch_scope`: the scope is the branch the unpublished commit recorded as its target. Writing an unpublished commit needs no new action: a mutation `as commit` is a `change` on its branch and a @@ -214,7 +227,8 @@ authentication allows every graph action and reports `actor_id: null`. The MCP server or managed target ([what you may do at a target](../cli/targets.md#what-you-may-do-at-a-target)). It reports policy actions, not actor views, so a view can still refuse a -`change` the policy allows. +`change` the policy allows. It does not report `maintain` or `reclaim`, since +no server route runs maintenance. ## Actor identity diff --git a/skills/omnigraph/references/server-policy.md b/skills/omnigraph/references/server-policy.md index 5eccf49d8..9486333b6 100644 --- a/skills/omnigraph/references/server-policy.md +++ b/skills/omnigraph/references/server-policy.md @@ -120,6 +120,8 @@ Graph-scoped actions are: | `schema_apply` | Schema changes: `cluster apply --target` needs it on each existing graph whose schema changes | | `branch_create`, `branch_delete`, `branch_merge` | Corresponding branch operation | | `commit_publish`, `commit_drop` | Publish or drop an unpublished commit; drop an unattached commit | +| `maintain` | Optimize, index builds, the full-text rebuild (which also accepts `change` for now) and the repair report | +| `reclaim` | Cleanup, which permanently removes unretained history | | `invoke_query` | Entry to a stored query | | `admin` | Reserved; no current public operation | @@ -165,12 +167,12 @@ rules: actions: [invoke_query] ``` -`branch_scope` is valid only with `read`, `export`, and `change`; +`branch_scope` is valid only with `read`, `export`, `change`, and `maintain`; `target_branch_scope` only with `branch_create`, `branch_delete`, `branch_merge`, `schema_apply`, `commit_publish`, and `commit_drop`. Any other pairing, or both scopes on one rule, is refused. Values are `any`, `protected`, or `unprotected`, where protected branches are the bundle's top-level `protected_branches: [main, …]` -list. `invoke_query`, `graph_list` and `config_manage` take no branch scope. +list. `invoke_query`, `reclaim`, `graph_list` and `config_manage` take no branch scope. ```bash omnigraph policy validate --target . --graph knowledge