Repository navigation
test(core): cover publication recovery under version cleanup - #408
Merged
Merged
Conversation
45 of 52 tasks
beinan
added a commit
that referenced
this pull request
Oct 20, 2026
## Summary A catalog plan must not release its source dependencies during publisher takeover or replica recovery. Add signed catalog observations and source-quorum pins for each selected table prefix. Sources verify the exact selected metadata against their own committed records, preserve independent group positions, and return the original pin index on an exact retry. Catalog replicas issue Ed25519 attestations only after safe ReadIndex completion. Source-only replicas hold the public key; private signing keys are neither serialized nor included in snapshots. Signatures cover publication metadata, while Arrow payloads remain opaque. Pin counts and command/image sizes are bounded, and captured views share immutable pin metadata. The optional application/checkpoint format advances to v4 and rejects older identities and images. Pins and payloads remain retained. Source release, external Lance commit/outcome coordination, key provisioning and transport/caller authentication remain separate work; an attestation does not authorize a Lance commit or payload deletion. Refs #378. ## Testing The complete repository CI helper passed for this source increment: 605 workspace Rust tests, Python 229 passed / 2 skipped / 1 expected failure, formatting, Clippy, Ruff and Pyright. All-feature WAL Clippy, server build and workspace packaging also passed. The final branch also includes the independently qualified publication-GC regressions from #408. The source increment is unchanged, and final-head CI covers the combined branch. All 72 focused Rust tests passed, including disjoint catalog/source replica sets, signature tampering, incomplete prefixes, retries at capacity, source snapshot transfer, independent leader loss, publisher takeover and full process restart. Recovered replicas return the original pin index and exact source payloads.
beinan
added a commit
that referenced
this pull request
Oct 20, 2026
## Summary Persist one immutable set of materialized Lance fragment descriptors for each pending publication plan. Exact retries return the original preparation index, changed bytes are rejected, and publisher takeover or snapshot recovery retains the chosen files and fixed transaction. Descriptors use bounded shared bytes with a format identifier and SHA-256 digest; WAL replicas do not deserialize Lance fragments or Arrow bodies. The optional application/checkpoint format advances to v5. Recording descriptors neither proves source pins nor authorizes external commits or source retirement. Add real Lance integration tests that write Arrow data through source WAL processes, materialize immutable files, restart all six catalog/source processes, recover the recorded descriptors, and verify exact payloads and transaction identity. An occupied target must not rebase the recovered plan. Refs #378. ## Testing The complete repository CI helper passed for this source increment: 605 workspace Rust tests, Python 229 passed / 2 skipped / 1 expected failure, formatting, Clippy, Ruff and Pyright. All-feature WAL Clippy, server build and workspace packaging also passed. All 86 focused Rust tests passed, including the real Lance process tests. The seven process tests also passed ten consecutive repetitions. Immutable source hashes matched before and after qualification. The final branch also includes the independently qualified publication-GC regressions from #408. The qualified increment is unchanged after rebasing; final-head CI validates the combined branch.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Publication recovery must not treat a missing or recreated Lance target as proof that WAL payloads can be retired. Add real Lance 13 integration tests for exact transaction-proof cleanup, a numeric cleanup floor that retains the pending base/target, and delayed fixed-target commits after collection.
The concurrent cases pause before the object-store write, let other transactions publish the target and successor, collect the target, then release the old commit. Conditional PUT and atomic rename recreate the old target while the latest version excludes its payload; retained-target controls reject the delayed commit. Every case verifies transaction identities and complete row payloads after reopening. The accompanying design note records the required publication/cleanup coordination. These are local storage characterization tests; they do not enable a distributed cleaner or claim cloud throughput.
Refs #378.
Testing
The complete repository CI helper passed: 609 Rust tests, Python 229 passed / 2 skipped / 1 expected failure, formatting, Clippy, Ruff and Pyright. All four focused Lance tests, the server build and workspace packaging also passed.