Skip to content

feat: add shared WAL record envelopes and payload validation - #381

Merged
beinan merged 1 commit into
lance-format:mainfrom
beinan:codex/wal-record-envelope
Oct 19, 2026
Merged

beinan merged 1 commit into
lance-format:mainfrom
beinan:codex/wal-record-envelope

Conversation

@beinan

@beinan beinan commented Oct 19, 2026

Copy link
Copy Markdown
Collaborator

Summary

Replicas need to validate opaque ingestion payloads without decoding Arrow or copying the payload allocation. Add an independent record feature with shared Batch ownership, a separately serializable BatchMeta, caller-bounded length checks and SHA-256 validation. Callers retain responsibility for routing, producer sequencing, control commands and durable commit.

Fixed binary fixtures cover the record domain tags and metadata layout. CI enables the new feature; production ingestion is unchanged.

Refs #378.

Testing

  • Five record tests cover shared ownership, complete payload validation, length bounds, reserved control producers and metadata encoding compatibility.
  • Optional-feature release Clippy and record tests passed.
  • Full repository Rust/Python checks passed.

@beinan
beinan merged commit f9dd637 into lance-format:main Oct 19, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant