Skip to content

feat: share validated Arrow IPC decoding for WAL materialization - #380

Merged
beinan merged 2 commits into
lance-format:mainfrom
beinan:codex/wal-ipc-decoder
Oct 19, 2026
Merged

beinan merged 2 commits into
lance-format:mainfrom
beinan:codex/wal-ipc-decoder

Conversation

@beinan

@beinan beinan commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Materializers need to decode replicated Arrow IPC without copying large binary payloads or bypassing format validation. Add an optional ipc::decode(Bytes, expected_schema) module with shared input ownership, dictionary and multi-batch support, schema/block/array-buffer checks and row/count limits. Compressed IPC and dictionary deltas remain rejected until their expansion is budgeted. Reject foreign endianness and contain Arrow metadata panics in panic-unwind builds; panic-abort builds cannot recover from dependency panics.

The component benchmark now calls this same decoder, including validation costs. Replication does not require the feature; production ingestion is not connected by this PR. Callers still reserve decoding workspace, and numeric alignment repair can allocate.

Refs #378.

Testing

  • Twelve optional-module tests passed, including aligned/unaligned payload ownership, dictionary/null round trips, truncated/malformed ranges, schema mismatch, corrupt dictionary types, foreign endianness and expansion limits.
  • Optional-feature release Clippy, codec correctness/allocation smoke, formatting, changed-file spelling and whitespace checks passed.
  • Full repository checks passed: workspace Rust formatting, Clippy and tests; Python pytest, Ruff and Pyright.

@beinan
beinan marked this pull request as ready for review October 19, 2026 05:13
@beinan
beinan merged commit c478b47 into lance-format:main Oct 19, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant