Repository navigation
feat(wal): attest catalog selections and retain source pins - #409
Merged
Merged
Conversation
46 of 53 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A catalog plan must not release its source dependencies during publisher takeover or replica recovery. Add signed catalog observations and source-quorum pins for each selected table prefix. Sources verify the exact selected metadata against their own committed records, preserve independent group positions, and return the original pin index on an exact retry.
Catalog replicas issue Ed25519 attestations only after safe ReadIndex completion. Source-only replicas hold the public key; private signing keys are neither serialized nor included in snapshots. Signatures cover publication metadata, while Arrow payloads remain opaque. Pin counts and command/image sizes are bounded, and captured views share immutable pin metadata.
The optional application/checkpoint format advances to v4 and rejects older identities and images. Pins and payloads remain retained. Source release, external Lance commit/outcome coordination, key provisioning and transport/caller authentication remain separate work; an attestation does not authorize a Lance commit or payload deletion.
Refs #378.
Testing
The complete repository CI helper passed for this source increment: 605 workspace Rust tests, Python 229 passed / 2 skipped / 1 expected failure, formatting, Clippy, Ruff and Pyright. All-feature WAL Clippy, server build and workspace packaging also passed.
The final branch also includes the independently qualified publication-GC regressions from #408. The source increment is unchanged, and final-head CI covers the combined branch.
All 72 focused Rust tests passed, including disjoint catalog/source replica sets, signature tampering, incomplete prefixes, retries at capacity, source snapshot transfer, independent leader loss, publisher takeover and full process restart. Recovered replicas return the original pin index and exact source payloads.