Repository navigation
feat(wal): add bounded TLS RPCs for ingress and materializers - #448
Merged
Merged
Conversation
This was referenced Oct 21, 2026
beinan
force-pushed
the
codex/wal-client-rpc
branch
3 times, most recently
from
October 21, 2026 22:10
68b84d5 to
3be873b
Compare
Retain all feature checks in separate CI jobs and serialize checkpoint fork/reopen cases. Refs #378
beinan
force-pushed
the
codex/wal-client-rpc
branch
from
October 21, 2026 23:02
3be873b to
4cf154c
Compare
beinan
added a commit
that referenced
this pull request
Oct 22, 2026
## Summary Add an independent rollout-ingestion adapter using the durable WAL's TLS client interface. It verifies the authorized table generation and rollout contract at quorum, reserves shared memory before buffering, validates Arrow IPC and session-bucket boundaries, and retains the original bytes for exact retries. Durable ACKs produce authenticated receipts that can be restored after restart and checked against compact publication visibility. The adapter lives in the unpublished `lance-context-ingest` crate so the existing publishable core does not acquire an unpublished WAL dependency. Applications must persist producer-open and append retry state before sending; HTTP routing, that journal wiring and the materializer loop remain separate integration work. Refs #378. ## Testing - Full repository CI-equivalent checks, including Rust, Python, Ruff and Pyright. - Real TLS three-replica test with a real Lance genesis manifest, all original replica directories restarted, exact retries, and complete metadata/payload reads. - IPC validation, shared-memory pressure, virtual buckets sharing a Raft group, receipt tampering/key rotation/scope, producer fencing, and compact quorum visibility bounds. Built on merged #448. The change is now based on main with an identical feature diff; all checks must pass for the new commit before merging. CI correction: registry packaging excludes the private ingress adapter; its Rust feature tests remain required. Linux wheel setup uses bounded downloads from the official AlmaLinux repositories, with TLS and package signature checks preserved. APT wheel setup now uses only signed official distribution repositories in an isolated metadata directory. This avoids unrelated PPA outages; bounded network retries, HTTPS and archive signature verification remain enabled. Bash/ShellCheck and a local Debian metadata/install simulation passed; the Ubuntu cross-build installation is validated by remote CI. CI runs the existing workspace, WAL, Lance, and etcd suites in separate jobs so cold feature builds do not exhaust one shared 60-minute timeout. Every original check command and the coverage jobs are preserved. Checkpoint reopen cases are serialized around subprocess creation to prevent inherited file locks; corruption cases must report the expected validation error. Minimal-feature and all-feature checkpoint tests and Clippy pass; the workflow passes actionlint. Final-commit CI is required before merging.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ingestion and materializers need to reach durable replicas without embedding a WAL node. This adds authenticated TLS client RPCs with separate data/control listeners, bounded shared connection pools, leader hints, and source-payload reads. Append sends metadata and the existing payload buffer in one exchange; success is returned only after durable quorum completion. An uncertain result preserves the exact producer/sequence/body retry contract.
This supplies the network interface; HTTP ingestion and an independently deployable service follow separately.
Refs #378.
Testing
Depends on #447. This PR is stacked on that branch so its diff can be reviewed independently; it will be rebased onto main after its dependency merges, with CI required again before merging.
CI correction: Linux wheel setup uses bounded downloads from the official AlmaLinux repositories, with TLS and package signature checks preserved.
APT wheel setup now uses only signed official distribution repositories in an isolated metadata directory. This avoids unrelated PPA outages; bounded network retries, HTTPS and archive signature verification remain enabled. Bash/ShellCheck and a local Debian metadata/install simulation passed; the Ubuntu cross-build installation is validated by remote CI.
CI runs the existing workspace, WAL, Lance, and etcd suites in separate jobs so cold feature builds do not exhaust one shared 60-minute timeout. Every original check command and the coverage jobs are preserved. Checkpoint reopen cases are serialized around subprocess creation to prevent inherited file locks; corruption cases must report the expected validation error. Minimal-feature and all-feature checkpoint tests and Clippy pass; the workflow passes actionlint. Final-commit CI is required before merging.