Repository navigation
feat(ingest): add durable rollout ingestion over WAL RPCs - #449
Merged
Merged
Conversation
This was referenced Oct 21, 2026
beinan
force-pushed
the
codex/wal-client-rpc
branch
from
October 21, 2026 21:42
dbd75bf to
d047d2c
Compare
beinan
force-pushed
the
codex/wal-rollout-ingress
branch
from
October 21, 2026 21:42
3374f9e to
14d76f5
Compare
beinan
force-pushed
the
codex/wal-client-rpc
branch
from
October 21, 2026 21:53
d047d2c to
68b84d5
Compare
beinan
force-pushed
the
codex/wal-rollout-ingress
branch
from
October 21, 2026 21:54
14d76f5 to
e79c4a0
Compare
beinan
force-pushed
the
codex/wal-client-rpc
branch
2 times, most recently
from
October 21, 2026 23:02
3be873b to
4cf154c
Compare
beinan
force-pushed
the
codex/wal-rollout-ingress
branch
from
October 21, 2026 23:02
e79c4a0 to
1668847
Compare
beinan
force-pushed
the
codex/wal-rollout-ingress
branch
from
October 22, 2026 00:41
1668847 to
bc71dd8
Compare
beinan
added a commit
that referenced
this pull request
Oct 22, 2026
## Summary Durable rollout ACKs need a recoverable path into queryable Lance files. The materializer selects bounded prefixes across session-sharded WAL groups, fetches their Arrow IPC, and writes Lance fragments with bounded parallelism. It records prepared descriptors before publication, reconciles uncertain outcomes, and retires source payloads with catalog-issued closure proofs. Takeover reuses prepared files or completes retirement without republishing rows. Refs #378. Built on merged #449; the feature diff is unchanged when moved to main. ## Testing - Real TLS replicas and complete rollout schemas: restart original directories after preparation and publication, verify descriptors and every row/column, reject stale owners, and preserve exact append retries. - Bounded manifest history, multi-round publication, and client/catalog/Lance reader regressions. - Previous feature revision passed repository Rust and Python checks (229 passed, 2 skipped, 1 xfailed), formatting, Clippy, Ruff and Pyright. CI must pass again for the new commit before merge. These checks establish correctness; they do not establish sustained throughput.
beinan
added a commit
that referenced
this pull request
Oct 22, 2026
## Summary Add `context-wal-node`, an independently deployable Rust WAL replica service. Validated, persisted configuration starts durable replication, mutually authenticated peer and snapshot transport, separate client data/control listeners, automatic maintenance, and joined shutdown. Local diagnostics expose runtime and disk state; the executable does not depend on Lance. Refs #378. Built on merged #449; the feature diff is unchanged when moved to main. ## Testing - Six-process service test with two disjoint three-voter groups: checkpointing, compacted-follower snapshot catch-up, leader SIGKILL, complete restart, payload verification, and exact retries. - Library/process regressions, service-feature Clippy, and standalone build/dependency validation. - Previous feature revision passed repository Rust and Python checks (229 passed, 2 skipped, 1 xfailed), formatting, Clippy, Ruff and Pyright. CI must pass again for the new commit before merge. These checks establish correctness; they do not establish sustained throughput.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add an independent rollout-ingestion adapter using the durable WAL's TLS client interface. It verifies the authorized table generation and rollout contract at quorum, reserves shared memory before buffering, validates Arrow IPC and session-bucket boundaries, and retains the original bytes for exact retries. Durable ACKs produce authenticated receipts that can be restored after restart and checked against compact publication visibility.
The adapter lives in the unpublished
lance-context-ingestcrate so the existing publishable core does not acquire an unpublished WAL dependency. Applications must persist producer-open and append retry state before sending; HTTP routing, that journal wiring and the materializer loop remain separate integration work.Refs #378.
Testing
Built on merged #448. The change is now based on main with an identical feature diff; all checks must pass for the new commit before merging.
CI correction: registry packaging excludes the private ingress adapter; its Rust feature tests remain required. Linux wheel setup uses bounded downloads from the official AlmaLinux repositories, with TLS and package signature checks preserved.
APT wheel setup now uses only signed official distribution repositories in an isolated metadata directory. This avoids unrelated PPA outages; bounded network retries, HTTPS and archive signature verification remain enabled. Bash/ShellCheck and a local Debian metadata/install simulation passed; the Ubuntu cross-build installation is validated by remote CI.
CI runs the existing workspace, WAL, Lance, and etcd suites in separate jobs so cold feature builds do not exhaust one shared 60-minute timeout. Every original check command and the coverage jobs are preserved. Checkpoint reopen cases are serialized around subprocess creation to prevent inherited file locks; corruption cases must report the expected validation error. Minimal-feature and all-feature checkpoint tests and Clippy pass; the workflow passes actionlint. Final-commit CI is required before merging.