Repository navigation
feat(ingest): publish session-sharded WAL records into Lance - #450
Merged
Merged
Conversation
beinan
force-pushed
the
codex/wal-rollout-ingress
branch
from
October 21, 2026 21:42
3374f9e to
14d76f5
Compare
beinan
force-pushed
the
codex/wal-rollout-materializer
branch
from
October 21, 2026 21:42
09f02cd to
bc4268f
Compare
beinan
force-pushed
the
codex/wal-rollout-ingress
branch
from
October 21, 2026 21:54
14d76f5 to
e79c4a0
Compare
beinan
force-pushed
the
codex/wal-rollout-materializer
branch
from
October 21, 2026 21:54
bc4268f to
15deff2
Compare
beinan
force-pushed
the
codex/wal-rollout-ingress
branch
from
October 21, 2026 23:02
e79c4a0 to
1668847
Compare
beinan
force-pushed
the
codex/wal-rollout-materializer
branch
from
October 21, 2026 23:02
15deff2 to
2130778
Compare
beinan
force-pushed
the
codex/wal-rollout-ingress
branch
from
October 22, 2026 00:41
1668847 to
bc71dd8
Compare
…e Lance publications
beinan
force-pushed
the
codex/wal-rollout-materializer
branch
from
October 22, 2026 02:11
2130778 to
5593db4
Compare
beinan
added a commit
that referenced
this pull request
Oct 22, 2026
## Summary Add an HTTP gateway for durable rollout ingestion and a Rust HTTP producer that journals exact retry state. Gateways accept the original Arrow IPC buffer, reserve memory before buffering, separate data/control admission, and return 202 only after the WAL quorum has persisted the append. Signed producer credentials work across gateways without another producer-allocation RPC on each append. Producer lanes share one bounded group-commit journal and HTTP pool. Original bytes and sequence are durable before transmission; the ACK is durable locally before the body is released or the sequence advances. An uncertain network result preserves the original intent, a different body cannot replace it, and local journal ambiguity fences the lane until reopen. Large digest/byte comparisons run off the async executor. The gateway takes authorized fixed table bindings and must sit behind application access control. These are application library interfaces; deployment, dynamic namespace integration and sustained performance qualification remain follow-up work. Refs #378. Built on merged #450. The feature diff is unchanged when moved to main; all CI must pass on the new commit before merging. ## Testing - Complete repository CI-equivalent script passed: Rust, Python (229 passed, 2 skipped, 1 xfailed), formatting, Clippy, Ruff and Pyright. - HTTP and HTTP-client/materializer feature checks passed, along with all five integration tests. - Real HTTP to TLS replicas to Lance: inject an intermediary error after a durable ACK, reopen the client journal/gateway/all original replica directories, recover the exact ACK, reuse prepared Lance files after another restart, and verify every row/column and subsequent producer sequences. - Data-lane saturation keeps control requests available; draining joins a disconnected partial-body reader without advancing its sequence. Producer/receipt domain separation and tampering checks remain required. - Native client-journal recovery regressions passed (10 tests, with two child fixtures invoked by their parent tests). The new application restart tests use joined shutdown/reopen, not application SIGKILL or throughput measurements. APT wheel setup now uses only signed official distribution repositories in an isolated metadata directory. This avoids unrelated PPA outages; bounded network retries, HTTPS and archive signature verification remain enabled. Bash/ShellCheck and a local Debian metadata/install simulation passed; the Ubuntu cross-build installation is validated by remote CI. CI runs the existing workspace, WAL, Lance, and etcd suites in separate jobs so cold feature builds do not exhaust one shared 60-minute timeout. Every original check command and the coverage jobs are preserved. Checkpoint reopen cases are serialized around subprocess creation to prevent inherited file locks; corruption cases must report the expected validation error. Minimal-feature and all-feature checkpoint tests and Clippy pass; the workflow passes actionlint. Final-commit CI is required before merging.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Durable rollout ACKs need a recoverable path into queryable Lance files. The materializer selects bounded prefixes across session-sharded WAL groups, fetches their Arrow IPC, and writes Lance fragments with bounded parallelism. It records prepared descriptors before publication, reconciles uncertain outcomes, and retires source payloads with catalog-issued closure proofs. Takeover reuses prepared files or completes retirement without republishing rows.
Refs #378. Built on merged #449; the feature diff is unchanged when moved to main.
Testing
These checks establish correctness; they do not establish sustained throughput.