Skip to content

feat(ingest): add durable rollout ingestion over WAL RPCs - #449

Merged
beinan merged 1 commit into
mainfrom
codex/wal-rollout-ingress
Oct 22, 2026
Merged

beinan merged 1 commit into
mainfrom
codex/wal-rollout-ingress

Conversation

@beinan

@beinan beinan commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Add an independent rollout-ingestion adapter using the durable WAL's TLS client interface. It verifies the authorized table generation and rollout contract at quorum, reserves shared memory before buffering, validates Arrow IPC and session-bucket boundaries, and retains the original bytes for exact retries. Durable ACKs produce authenticated receipts that can be restored after restart and checked against compact publication visibility.

The adapter lives in the unpublished lance-context-ingest crate so the existing publishable core does not acquire an unpublished WAL dependency. Applications must persist producer-open and append retry state before sending; HTTP routing, that journal wiring and the materializer loop remain separate integration work.

Refs #378.

Testing

  • Full repository CI-equivalent checks, including Rust, Python, Ruff and Pyright.
  • Real TLS three-replica test with a real Lance genesis manifest, all original replica directories restarted, exact retries, and complete metadata/payload reads.
  • IPC validation, shared-memory pressure, virtual buckets sharing a Raft group, receipt tampering/key rotation/scope, producer fencing, and compact quorum visibility bounds.

Built on merged #448. The change is now based on main with an identical feature diff; all checks must pass for the new commit before merging.

CI correction: registry packaging excludes the private ingress adapter; its Rust feature tests remain required. Linux wheel setup uses bounded downloads from the official AlmaLinux repositories, with TLS and package signature checks preserved.

APT wheel setup now uses only signed official distribution repositories in an isolated metadata directory. This avoids unrelated PPA outages; bounded network retries, HTTPS and archive signature verification remain enabled. Bash/ShellCheck and a local Debian metadata/install simulation passed; the Ubuntu cross-build installation is validated by remote CI.

CI runs the existing workspace, WAL, Lance, and etcd suites in separate jobs so cold feature builds do not exhaust one shared 60-minute timeout. Every original check command and the coverage jobs are preserved. Checkpoint reopen cases are serialized around subprocess creation to prevent inherited file locks; corruption cases must report the expected validation error. Minimal-feature and all-feature checkpoint tests and Clippy pass; the workflow passes actionlint. Final-commit CI is required before merging.

@beinan
beinan force-pushed the codex/wal-rollout-ingress branch from 1668847 to bc71dd8 Compare October 22, 2026 00:41
@beinan
beinan changed the base branch from codex/wal-client-rpc to main October 22, 2026 00:41
@beinan
beinan merged commit 1b5ef6b into main Oct 22, 2026
24 of 33 checks passed
beinan added a commit that referenced this pull request Oct 22, 2026
## Summary

Durable rollout ACKs need a recoverable path into queryable Lance files.
The materializer selects bounded prefixes across session-sharded WAL
groups, fetches their Arrow IPC, and writes Lance fragments with bounded
parallelism. It records prepared descriptors before publication,
reconciles uncertain outcomes, and retires source payloads with
catalog-issued closure proofs. Takeover reuses prepared files or
completes retirement without republishing rows.

Refs #378. Built on merged #449; the feature diff is unchanged when
moved to main.

## Testing

- Real TLS replicas and complete rollout schemas: restart original
directories after preparation and publication, verify descriptors and
every row/column, reject stale owners, and preserve exact append
retries.
- Bounded manifest history, multi-round publication, and
client/catalog/Lance reader regressions.
- Previous feature revision passed repository Rust and Python checks
(229 passed, 2 skipped, 1 xfailed), formatting, Clippy, Ruff and
Pyright. CI must pass again for the new commit before merge.

These checks establish correctness; they do not establish sustained
throughput.
beinan added a commit that referenced this pull request Oct 22, 2026
## Summary

Add `context-wal-node`, an independently deployable Rust WAL replica
service. Validated, persisted configuration starts durable replication,
mutually authenticated peer and snapshot transport, separate client
data/control listeners, automatic maintenance, and joined shutdown.
Local diagnostics expose runtime and disk state; the executable does not
depend on Lance.

Refs #378. Built on merged #449; the feature diff is unchanged when
moved to main.

## Testing

- Six-process service test with two disjoint three-voter groups:
checkpointing, compacted-follower snapshot catch-up, leader SIGKILL,
complete restart, payload verification, and exact retries.
- Library/process regressions, service-feature Clippy, and standalone
build/dependency validation.
- Previous feature revision passed repository Rust and Python checks
(229 passed, 2 skipped, 1 xfailed), formatting, Clippy, Ruff and
Pyright. CI must pass again for the new commit before merge.

These checks establish correctness; they do not establish sustained
throughput.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant