Repository navigation
test(cleanup): qualify sweep retries across storage and DST scheduling - #1036
Merged
aaltshuler merged 23 commits intoOct 18, 2026
Merged
Conversation
…x directory The collector deletes a swept version's files before its manifest, so a pass that stops in between leaves the manifest for the next pass. On a local store that next pass failed for the table: `delete_paths` called `remove_dir_all` again on the `_indices/<uuid>` directory the stopped pass had removed, and Lance's local provider reports an absent directory as `NotFound`. The table kept the error and deleted nothing on every later run. Object stores list the prefix empty and succeed, so S3, Azure and the DST memory store converged already. An absent index directory is now already deleted, as an absent file already was. The files-before-manifest order is unchanged, and the deletion set still holds only paths no retained root marks. The failpoints test stops a pass at `cleanup.sweep_pre_manifest_delete` after it removed an index directory only the pruned pin names, then proves the next pass removes that manifest, reports no error, and leaves every retained root's paths in place.
Reword the maintenance guide's sentence on an interrupted cleanup so the deletion order and the next run's outcome read as two steps, and point the release note at the guide's Cleanup section.
…staging manifests Cover the directory-touching windows of the collector's sweep on a local store, where Lance reports removing an absent directory as NotFound: - a pass stopped between two manifests of the published sweep, with an index directory among the files it already deleted; - a pass stopped after the dead stagings' files and before their manifests, through the new seam cleanup.sweep_pre_dead_manifest_delete. Each retry reports no error, removes the remaining manifests and keeps every retained root's paths.
…anup-sweep-crash-windows # Conflicts: # crates/omnigraph/tests/failpoints.rs
The three collector-retry fixtures in failpoints.rs become helpers over a graph uri. The file tests keep their names and call them on a temporary directory; one RustFS/S3 test and one Azure/Azurite test run all three, each on a fresh graph root, and both CI jobs require them by name. The first fixture now also stops a pass at cleanup.sweep_pre_file_delete, the one sweep window no failpoint test reached: the pass deletes nothing and the next pass plans the same sweep. Its retry asserts what the other two already did, no error on any table and nothing left to remove. Index-directory presence goes through the storage adapter's exists rather than std::fs, so the same assertion holds on an object store. The RFC's crash-window gate is satisfied and leaves blocked_on; the evidence bullet names the owners per backend.
Register cleanup.sweep_pre_dead_manifest_delete in the crash-window catalog (appended, 53 windows) and give it a milestone recipe: insert, index build, insert, cleanup. The reach census's setup crash stops the index build before its manifest commit, stranding a detached staging; the second insert moves main's head past the head the staging recorded, so the recipe's cleanup judges it dead and crosses the window. A window inside cleanup's per-table pass does not fail the op: cleanup records the table's error and returns Ok. The harness treated that as a plain success, so no retry was judged and the report kept no trace of the window. crash_op now returns CleanupDeferred when the op's deferrals name the scheduled window; the universe records deferred:<window>@op<i> and runs the obligation pass a crashed cleanup gets (a clean rerun that defers nothing, the collector oracle, the world read). This covers cleanup.table_gc and the three earlier sweep windows as well. The pin dst_dead_staging_sweep_window_is_reached_and_retried probes the window and stops the recipe's cleanup at it; the sensitivity proof gains a deferred-cleanup case. Both go red with the routing reverted.
cleanup.sweep_pre_dead_manifest_delete needs an earlier crash's aftermath: an index build stopped before its manifest commit. Only the reach census added that crash, by spending the universe's one crash_on_match slot on it, so the hunt and the fleet scheduled the window on a cleanup with no dead staging and it never fired there. Scenario::setup_crash is a second slot, matched before crash_on_match and always a real crash, even under probe_only; crash_on_match starts matching after it. harness::setup_crash_for replaces the test-only census_setup and keeps only its live entry; window_needs_wide and window_needs_schema_ops include the setup's needs. The reach census, the hunt and the fleet's window arm take the setup from it, and the hunt counts a hit from the window's own verdict instead of crashes > 0, which a setup crash alone would satisfy. The pin now stops the cleanup through the fleet arm's scenario instead of a pinned op index.
…t every release (ModernRelay#1031) * feat(release): publish daily, pinnable and attested edge builds Edge releases now build once a day from the newest audited commit on main and publish each build as its own prerelease, edge-<commit date>-<short commit>, which never changes once published; the rolling edge release copies it and only moves forward. Both release workflows share one archive build that stamps the commit and channel into the binaries, runs each archive on its platform and verifies the exact set, and both attest build provenance. Each edge release states its storage format and HTTP contract next to the latest stable release's, then the notes recorded since that release. * fix(release): attest only the commit a release run is for The provenance predicate takes its source commit from the run's OIDC sha claim, not from the checkout. An edge run now builds only its own commit, waiting up to 90 minutes for that commit's audit instead of walking back to an older one, and both publishers refuse to attest a build of any other commit; a stable backfill from v0.14.0 is dispatched at its tag. * fix(release): keep edge runs from overlapping A newer edge run could move the rolling tag after an older run published but before its installer checks ran, failing a release that succeeded. Whole runs now share one concurrency group. * feat(release): stamp and attest the container image The image's bookworm-built binaries now carry the same commit and stable channel stamp as the release archives, checked inside the image before it is pushed, and the pushed GHCR digest carries a build provenance attestation under the same rule: only for the run's own commit, and from v0.14.0 a backfill is dispatched at its tag. * fix(release): address review of the edge release rework - Refuse an edge build while the package version is not above the latest stable release: X.Y.Z-edge+<commit> sorts before X.Y.Z, so the ordering claim held only after main advanced its version. - Require the commit's CI Gate as well as its vocabulary audit, waiting up to 180 minutes, since the build becomes a pinnable, attested release; a run cancelled because main moved on now ends the edge run with a notice. - Keep the newest edge build of every storage format beyond the retained 30, read from a marker line in the release text, so the build that can export an abandoned development format survives. - Report one build identity from omnigraph-api-types: the CLI, /healthz and the MCP server info share the stamped version and its compile-time checks. - Compose the edge release text from the real release-note preview in CI, and hold the two copies of the v0.14.0 attestation cutoff to one value.
Review of
|
| Probe | fbba88d1 (before) |
781f7152 (after) |
|---|---|---|
Fleet arm, seeds 10016 and 40052: is a deferred:cleanup.sweep_pre_dead_manifest_delete@op<i> verdict present with Deferred? |
FAILED: crashes=0 verdicts=[] for both seeds |
passed: @op4 (10016), @op8 (40052), each after crash:ensure_indices.post_phase_b_pre_manifest_commit |
| Hunt cells, seeds 7-12 × skips 0-3: does any cell give a verdict at the window? | FAILED: all 24 cells crashes=0 verdicts=[] |
passed: seed 9 skip 0, deferred:…@op11 |
Both commits fail or pass for the predicted reason. At the head, hunt seed 8 (skips 0-3) gives crashes=1 with only the setup's verdict. The old crashes > 0 rule would have counted those cells as hits, so window_hit is necessary. Seed 40052 defers at op 8, which is the op the old pin hard-coded.
Full hunt (cargo test --locked --test scenarios dst_hunt_crash_window_sweep -- --exact --ignored --nocapture):
- before: 570 universes; 32 hit, 7 absorbed, 8 never reached, 6 unschedulable. The dead-staging window was never reached.
cleanup.table_gcand the threecleanup.sweep_*windows were labelled absorbed. - after: 451 universes; 37 hit, 3 absorbed, 7 never reached, 6 unschedulable. This is the same as the PR body. Every hit from before is still a hit.
Other checks:
- Crate suite at head (
cargo test --locked --no-fail-fast): lib 53 passed; integration 2 passed, 3 ignored; scenarios 52 passed, 28 ignored, includingdst_dead_staging_sweep_window_is_reached_and_retried. - The base form of that pin passes at
fbba88d1, because it forces op 8 withcrash_at. So the pin is a reach guard, not a before/after test for this defect. The temporary probe provides that test. - CI at the exact head
781f7152: theCIandDSTworkflows succeeded, including the DST pinned suite, Lint (clippy), Format and Test Workspace.
Limits
- I did not run the reach census. The PR says it stops on the old floor of 48, as it does on test(dst): reach the dead-staging sweep window #1034; test(dst): name unreachable crash windows and fix index-build reach #1035 fixes that.
- I did not run the full
dst_fleettest. I ran only the window arm's scenario directly. - I did not test option (a) of finding 3.
- I make no performance claim.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
* fix(cli): accept --as on commit publish and drop The plane guard classified every `commit` subcommand as a read and refused `--as`, so `commit publish` and `commit drop` against a direct graph could only attribute `operator.actor`, although both dispatch through the write resolver and the engine's `_as` entry points. The guard now matches the `commit` subcommands exhaustively: publish and drop take `--as` on a direct target, while list, show and changes still refuse it. A served publish or drop still refuses `--as` before any request. Guard errors name the subcommand (`commit list`, ...). The published commit keeps the actor of the write that created it; the branching guide now says so. * test(cli): pin created_at and fold the served refusal into its owner The publish regression now also checks that the published commit keeps the proposal's `created_at`, the second half of the branching guide's claim. The served `--as` refusal for `commit publish` and `commit drop` moves into `as_on_a_served_write_is_rejected` as two more argument rows instead of a parallel fixture. Rewrap the edited guide paragraph. * docs(changelog): say a published commit keeps its timestamp too The branching guide and the regression test state both the actor and the timestamp; the release note now says the same.
…lay#1047) * refactor(cli): capture operator config during client setup * refactor(cli): reuse config for aliases and served clusters * refactor(cli): isolate invocation parsing * refactor(cli): make scope resolution pure * refactor(cli): separate application completion from process exit * refactor(cli): extract query family and typed preparation * refactor(cli): capture presentation and preserve delivery outcomes * test(cli): qualify captured presentation across execution * refactor(cli): isolate graph history execution and output * fix(cli): track dedicated commit write outcomes * docs(cli): describe history delivery and outcome boundaries * refactor(cli): isolate transfer execution and delivery * refactor(cli): isolate schema lifecycle and lint execution * refactor(cli): isolate maintenance execution and delivery * refactor(cli): isolate cluster control execution and delivery * refactor(cli): complete invocation config and I/O boundaries * fix(cli): preserve failures and recovery across output errors * docs(cli): consolidate server credential guidance
* feat(commits): page a branch's commit list
GET /commits returned a branch's whole lineage, and building it read all of
__history. It now takes `limit` (1 to 1,000) and an opaque `page_token`;
a paged answer names `head_commit_id`, the head its first page resolved,
and a `next_page_token` while older commits remain. Later pages answer
for that head's first-parent chain even after the branch moves, and the
token carries a digest of the branch, never its name. Without either
parameter the answer is unchanged.
A page walks first parents by falling generation from the head or the
token's next commit: the head and buffer cost nothing, and an older commit
is read by id with the rest of its __history extent, so a page costs the
extents its own commits occupy and never the history below them.
`omnigraph commit list --limit/--page-token` and the MCP `commits_list`
tool page the same way.
* test(commits): register the paged commit list as a read-only surface
The public and coordinator API registries in forbidden_apis classify every
async entry point; Omnigraph::list_commits_page and
GraphCoordinator::first_parent_page read and publish nothing.
* test(commits): track the reworded commit list description in the vocabulary inventory
The listCommits description still names the graph-manifest version in its
ordering sentence; its prose digest changed with the paging text.
* test(commits): pin commit-list page storage calls in the DST cost golden
The newest and the oldest page of main's commit list on a reopened handle,
after 12 and 36 publishes under a 2048-byte history release. Both rows
match at the two depths: a page reads the extents of its own commits, not
the history above or below them.
* fix(commits): prove a commit-list continuation for a branch-scoped caller
A commit-list page token is checksummed, not signed, so the commit a
continuation starts at is the caller's claim. A caller allowed to read
main only could re-sign a main token to list a secret branch's commits,
which GET /commits?branch=secret and GET /commits/{id} both refuse it.
Under a policy, a continuation by a caller without the graph-wide read gate
(the gate of GET /commits/{commit_id}) now goes through
list_commits_page_on_branch: the token's commit must be the first commit
below its generation on the branch's first-parent chain from the head, or
the page is refused with 400. That proof walks the chain, so such a
continuation costs what the whole listing costs; a caller with graph-wide
read keeps the page cost. The list_commits_page doc says what the token
does not prove.
Pages through both doors concatenate to the listing on an upgraded root,
whose converted commits carry pre-upgrade ids.
* fix(commits): check a scoped commit-list continuation cheaply, through a read check
list_commits_page_on_branch becomes list_commits_page_readable, which takes
the same ReadCheck as diff_page_readable. A continuation's commit is on the
branch's history when the branch's __manifest holds it (its head or buffer),
when a branch the caller may read published it, or else when a walk down
the branch's first-parent chain finds it. So a branch-scoped caller paging
its own branch no longer walks the history on every page; only a commit
neither cheap rule covers costs the walk. The page is read once and the
check uses its first commit. An unknown commit is refused exactly as an
unreadable one is.
A missing commit met inside the walk is an integrity error, not a 404.
The server shares one read decision between the diff and commit-list
doors. The API documents that a continuation's head_commit_id is the one
its token carries, and openapi.json describes the scoped caller's cost.
…odernRelay#1060) * feat(release): publish an edge container image from each edge build The daily edge run now also publishes a container image of the same build. It packages the published build release's Linux x86_64 archive, checked against its checksum and its attestation for this workflow and commit, on Debian trixie, since the Ubuntu-built archives need glibc 2.39 or newer. The root Dockerfile takes its base as BASE_IMAGE, bookworm by default, so the stable image is unchanged. The image is tagged edge-<commit date>-<short commit> on GHCR and Docker Hub, checked inside the image for its commit and channel, attested on both registries, and tagged edge only when the rolling edge release moved. It never moves latest, and a rerun reuses the image first pushed for the build. * fix(release): match the image's server features and follow edge as it is now - Build every release archive with the server's aws feature, as the stable image already is, so the edge image packaged from the Linux archive accepts OMNIGRAPH_SERVER_BEARER_TOKENS_AWS_SECRET like the image it replaces. Lance's S3 support already builds the AWS SDK base; the feature adds one crate. The gate checker requires both release builds to enable it. - Move the edge image tags, and check the edge installer channel, only while the rolling edge release still names this build, so a rerun of an older run's failed job cannot move them behind a newer build.
…ModernRelay#1061) GitHub refuses the workflow token a new tag on a commit whose workflow files differ from main's current ones ("refusing to allow a GitHub App to create or update workflow"), and the token can never hold the workflows permission. An edge build of 02026b5 failed this way after a workflow change merged during its build. The edge run now checks that the build's workflow files still match main's before building and again before tagging, and otherwise publishes nothing, with a notice; the image and installer jobs run only for a published build.
… runner (ModernRelay#1063) `ubuntu-latest` moves to Ubuntu 26.04 between 2026-10-19 and 2026-11-19 (actions/runner-images#14748), and a run may land on either image during the rollout. A Linux archive needs the glibc it was linked against, so the x86_64 archive would start requiring Ubuntu 26.04's glibc, breaking Ubuntu 24.04 and Debian 13 users and the trixie edge image. Pin it to ubuntu-24.04, as the arm64 archive already is (ubuntu-24.04-arm), document the archives' glibc 2.39 floor, and have the gate checker refuse ubuntu-latest for archive builds.
…Relay#1065) The first edge image build failed twice pulling debian:trixie-slim from ECR Public: "429 Too Many Requests: Data limit exceeded". GitHub-hosted runners pull it anonymously from shared addresses, whose data limit is often spent, and the stable image pulls its bookworm base the same way. Both image workflows now log in to Docker Hub when the DOCKERHUB_* secrets exist and build on the Docker Official Image; the stable workflow's rust:1-bookworm builder pull is then logged in too. Without the secrets, the base still comes from ECR Public.
…only it (ModernRelay#1062) A property-level @index, @unique or @key is also stored as a type-level constraint naming the property by stable id. Dropping the property made the planner see that constraint disappear and refuse the whole change, because the constraint comparison did not know which properties the plan drops. An indexed property could therefore never be removed. The property planner now returns the stable ids of the properties it drops, and the constraint comparison no longer refuses a removed constraint when every field it names is one of them. A @key, a constraint that also names a field that stays (a remaining property or @src/@dst), and a constraint removed from a property that stays are still refused. Apply needs no change: the drop's Operation::Project already removes the index on the dropped column, and the evolved table declares full-text analyzers from the desired catalog. ConstraintIR::fields replaces two copies of the same match in schema_ir.rs and serves the new rule. Fixes ModernRelay#1014
…nRelay#1066) * query: refuse full-text calls without a full-text index (T27, FullTextIndexRequired) A full-text call (`search`, `fuzzy`, `match_text`, `bm25`) reads its property's full-text index, whose analyzer folds case and stems; without one Lance scans with a bare, case-sensitive tokenizer, so the same call answered differently on an indexed and an unindexed property. - T27 at type checking: a full-text call on a String node property with no one-column full-text `@index`/`@key` is refused, from the schema alone. `PropType::index_kind` moves to the compiler and index maintenance uses it, so the type checker and the index builder cannot drift. An enum's fix points at `=`; a non-String field keeps its operand error. - FullTextIndexRequired at planning (HTTP 409, `full_text_index_required` in the error body): the planner reads the coverage of every property a full-text call names, filters included, and refuses `absent`. An untrained segment, which every write now declares, is `partial` and answers; a table with no fragment, or a type the snapshot does not hold, counts as covered. Acceptance refuses a plan that records no coverage, or `absent`, for a call, and replay re-derives every recorded coverage from the pinned dataset. Regressions: cases/v2/issue_747_full_text_calls_need_a_full_text_index.gqt and search.rs::a_full_text_index_with_no_segment_refuses_the_call_issue_747, both red with their check removed. * docs: rescope RFC 0047 step 3 and document the full-text index requirement RFC 0047 records step 3 as refusing only an index with no segment, now that every write declares a declared column's analyzer, with a decision-log entry, and its rollout table records steps 2 and 2a as merged. The search guide, troubleshooting page and execution guide describe T27 and the 409; the release notes call T27 breaking and tell operators to validate stored queries before upgrading.
* feat(query): the ranked binding roots its component A nearest(), bm25() or rrf() order on a binding a traversal reaches was refused at planning unless that binding was declared first, and a ranked binding no declaration introduced could not be ranked at all. Lowering now roots the top-level component holding the bindings the leading order key ranks at that binding; a ranked binding only a traversal introduces gets a NodeScan of its checked type, and every other binding of the component is reached by a forward or reverse expand. An rrf() whose arms rank two bindings one traversal connects is the type error T59. The planner codes P001 and P005 are withdrawn; the planner keeps both checks as internal errors. * docs(search): rank a traversal-reached binding
# Conflicts: # docs/rfcs/2026-09-21-detached-only-tables.md
aaltshuler
changed the title
This was referenced Oct 18, 2026
Review of
|
| Commit | Result |
|---|---|
7e48be51 plus the new test only |
FAILED. crashes=1, verdict crash:cleanup.pre_gates@op2 / Applied / query. The setup seam did not fire. |
f25d256f |
passed. crashes=0. Strict replay passed. |
Other local runs at f25d256f
dst_dead_staging_sweep_window_is_reached_and_retried: passed. Fleet: setup crash at op 4,deferred:...@op8. Hunt: setup crash at op 4,deferred:...@op11. Both replays passed.collector_retry_after_a_pass_stopped_between_manifests_finishes_the_sweep: passed (direct file and scoped file). The temporary scheme log above was the second run. The log line was removed.
CI on this exact head
- CI
39135847273: success, including Test Workspace, RustFS failpoints, and Azurite. - DST
39135847308: pinned suite success. - GQ Logic Tests
39135847214: success.
Limits
- I did not run the full hunt or
dst_fleet. - I did not run the S3 or Azure fixtures in this environment. CI on this head did.
- I did not pin a legal rejection or a deferred cleanup as the setup attempt.
- I make no performance claim.
| let db = helpers::init_and_load_at(uri, helpers::TEST_SCHEMA).await; | ||
| // Serving scopes file I/O through Lance's file-object-store provider, | ||
| // whose list/delete path also removes the directory and can return NotFound. | ||
| let db = if scoped { |
There was a problem hiding this comment.
Optional. This row reopens through open_with_io_scope, but the assertions below are the same convergence checks as the direct file row. Those checks also pass if this reopen stops using file-object-store.
On this head the row does use that store. A temporary log in delete_paths (removed after the run) printed scheme=file-object-store and direct_local=false for this iteration, and scheme=file / direct_local=true for the other.
Please assert, when scoped is true, that the dataset store cleanup opens has scheme file-object-store and has_direct_local_paths() == false. Lance 11's file-object-store remove_dir_all lists the prefix and then returns NotFound for an absent directory. The direct store returns NotFound immediately. delete_paths already accepts both. The missing lock is the scheme, not the retry result.
This was referenced Oct 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Cleanup retries need qualification at each physical sweep boundary, including when a preceding crash creates the dead staging being collected. This consolidates #1027, #1033, #1034, and the original #1036 into one reviewable branch on current main, preserving their commits.
The collector's deletion policy is unchanged. Warning-only native-fork reconciliation is outside the new deferral attribution. The pins use clean, deterministic fixtures; this change does not redesign crash attribution when storage weather is composed with a setup crash. The independent ignored census-floor adjustment in #1035 is not required by these regressions.
Backing issue / RFC
Completes cleanup qualification for the accepted Detached-only tables RFC.
Checklist
Local verification
The setup-miss regression failed against the original merged harness: it fired
cleanup.pre_gatesat op2 without the setup crash. Temporarily removing setup from the shared hunt constructor also made the seed9 pin fail; restoring it passed.cargo test -p omnigraph-engine -p omnigraph-server -p omnigraph-cluster \ -p omnigraph-core -p omnigraph-storage \ --features omnigraph-engine/failpoints,omnigraph-cluster/failpoints \ --locked --test failpoints collector_retry -- --nocapturecrates/omnigraph-dst:cargo test --locked --test scenarios -- --exact --nocapture \ dst_dead_staging_sweep_window_is_reached_and_retried \ dst_setup_crash_waits_for_a_crossed_window \ dst_maintenance_obligations_bite_and_replay \ dst_sensitivity_maintenance_rerun_failure_is_redDeferredcleanup verdict at op11; the fleet pin records it at op8.DST_FLEET_SEEDS=1 DST_FLEET_SEED_BASE=10016 cargo test --locked --test scenarios dst_fleet -- --exact --ignored --nocapture— eight universes, zero failures. The actual fleet's selected window records the targetDeferredat op4 after setup at op0.cargo test -p omnigraph-seams --locked --test failpoint_names_guard— eight passed.Emulators do not qualify live Azure lease loss or delayed accepted writes. Full workspace and pinned DST CI remain the broad checks for this updated head.
The three updated branches (#1070, #1071, #1036) also merge cleanly. Their combined code passed five file/scoped-file/S3/Azure collector owners without skips, the real server failure journey, range unit owners, both setup/dead-staging DST pins with replay, and all four range-filter GQT cases. Combined documentation (341 files), formatting, links, required CI names, and merge-group checks passed. Local validation is retained on
codex/release-014-integrated-validation; the final additional merge changes only the clarified testing-guide wording.