Skip to content

feat: add bounded vectored WAL frame writes - #393

Merged
beinan merged 1 commit into
lance-format:mainfrom
beinan:codex/wal-vectored-io
Oct 19, 2026
Merged

beinan merged 1 commit into
lance-format:mainfrom
beinan:codex/wal-vectored-io

Conversation

@beinan

@beinan beinan commented Oct 19, 2026

Copy link
Copy Markdown
Collaborator

Refs #378

Summary

WAL framing can write the length, header and each shared payload with separate socket calls. Add an opt-in vectored-io helper that sends the same bytes using at most 64 borrowed IoSlice descriptors, without concatenating payloads. Short writes, scalar fallback, errors and cancellation preserve exact body-byte accounting; prepared exchanges can send bodies separately after a grant.

This is a transport building block for the ingestion work tracked in #378. It does not enable distributed ingestion in the production writer or claim a throughput improvement. Incomplete exchanges still require discarding the connection, and socket acceptance is not a durability ACK.

Testing

  • Eight focused tests cover partial writes, fragmented payload ownership, empty segments, scalar fallback, interrupted/error/cancelled sends, counter overflow, invalid progress and real TCP framing.
  • Enable the feature tests in the existing WAL CI step; all-feature coverage also includes them.
  • Local repository check script passed: formatting, workspace Clippy/tests (596 passed; existing ignored suites excluded), Python tests (229 passed, 2 skipped, 1 expected failure), Ruff and Pyright. The HTTP tests used a server built from this worktree.
  • cargo test -p lance-context-wal --features vectored-io --lib and feature-enabled Clippy with --all-targets -- -D warnings passed.

@beinan
beinan merged commit 5ce765c into lance-format:main Oct 19, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant