Repository navigation
feat(ingest): add HTTP gateway and recoverable Rust producers - #453
Merged
Merged
Conversation
beinan
force-pushed
the
codex/wal-rollout-materializer
branch
from
October 21, 2026 21:54
bc4268f to
15deff2
Compare
beinan
force-pushed
the
codex/wal-ingest-http
branch
from
October 21, 2026 21:54
c4bc35c to
df2070a
Compare
beinan
force-pushed
the
codex/wal-ingest-http
branch
from
October 21, 2026 23:02
df2070a to
e199493
Compare
beinan
force-pushed
the
codex/wal-rollout-materializer
branch
2 times, most recently
from
October 22, 2026 02:11
2130778 to
5593db4
Compare
Retain all feature checks in separate CI jobs and serialize checkpoint fork/reopen cases. Refs #378
beinan
force-pushed
the
codex/wal-ingest-http
branch
from
October 22, 2026 02:54
e199493 to
41822bf
Compare
beinan
added a commit
that referenced
this pull request
Oct 22, 2026
#455) ## Summary Coordinate materializer processes with bounded etcd claims and durable WAL publisher epochs. A worker reads the WAL epoch before validating its claim, so an expired owner cannot steal a successor's newer epoch. Prepared Lance files survive ownership changes and can be reused exactly. Workers share memory and WAL client pools, bound concurrent tables and publication rounds, and join admitted file work and lease renewal on shutdown. Busy-table probes use a local, keys-only hint before attempting a lease, avoiding quorum barriers and replicated grant/revoke writes on occupied-table polls. Ownership still requires the linearizable create-if-absent transaction, and claim validation remains linearizable. Managed readers can open the selected Lance version without acquiring publisher ownership. Assignments are an application-authorized fixed set. Dynamic namespace discovery, application process startup, and sustained throughput qualification are follow-up work. Refs #378. Built on merged #453. The feature diff is unchanged when moved to main; all final-commit CI checks must pass before merging. ## Testing - The complete repository CI-equivalent script passed: Rust, Python (229 passed, 2 skipped, 1 xfailed), formatting, Clippy, Ruff and Pyright. - The final busy-claim optimization was qualified separately with real etcd tests and worker-only / worker plus HTTP-client feature Clippy checks. - Revoke a claim after real Lance preparation, acquire a successor epoch, reject the old claimant, reuse the exact prepared result, and publish through two competing automatic workers with full-column readback. - Abandon a claim and require actual TTL expiry before takeover; all RPC and expiration deadlines remain enforced. These are correctness checks. They do not establish process-kill recovery or sustained throughput for the application. CI runs the existing workspace, WAL, Lance, and etcd suites in separate jobs so cold feature builds do not exhaust one shared 60-minute timeout. Every original check command and the coverage jobs are preserved. Checkpoint reopen cases are serialized around subprocess creation to prevent inherited file locks; corruption cases must report the expected validation error. Minimal-feature and all-feature checkpoint tests and Clippy pass; the workflow passes actionlint. Final-commit CI is required before merging. Independent Raft-log recovery cases exclude sibling subprocess-creation windows, preventing inherited file locks from racing immediate reopen. The two-group concurrent-write case and real SIGKILL, rotation, and write-error assertions remain unchanged. All nine Raft-log cases pass with minimal and all features, and Clippy passes. This adjustment changes only test scheduling. TLS process startup failures include the child stdout/stderr and identify peer/image listener binding errors. The unchanged quorum, snapshot-interruption, SIGKILL and full-restart assertions pass in five complete peer-transport library runs (126 tests per run), plus six process tests and feature Clippy. The intermittent CI startup failure has not been reproduced locally; these diagnostics do not establish its root cause. Final-head CI remains required.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add an HTTP gateway for durable rollout ingestion and a Rust HTTP producer that journals exact retry state. Gateways accept the original Arrow IPC buffer, reserve memory before buffering, separate data/control admission, and return 202 only after the WAL quorum has persisted the append. Signed producer credentials work across gateways without another producer-allocation RPC on each append.
Producer lanes share one bounded group-commit journal and HTTP pool. Original bytes and sequence are durable before transmission; the ACK is durable locally before the body is released or the sequence advances. An uncertain network result preserves the original intent, a different body cannot replace it, and local journal ambiguity fences the lane until reopen. Large digest/byte comparisons run off the async executor.
The gateway takes authorized fixed table bindings and must sit behind application access control. These are application library interfaces; deployment, dynamic namespace integration and sustained performance qualification remain follow-up work.
Refs #378. Built on merged #450. The feature diff is unchanged when moved to main; all CI must pass on the new commit before merging.
Testing
The new application restart tests use joined shutdown/reopen, not application SIGKILL or throughput measurements.
APT wheel setup now uses only signed official distribution repositories in an isolated metadata directory. This avoids unrelated PPA outages; bounded network retries, HTTPS and archive signature verification remain enabled. Bash/ShellCheck and a local Debian metadata/install simulation passed; the Ubuntu cross-build installation is validated by remote CI.
CI runs the existing workspace, WAL, Lance, and etcd suites in separate jobs so cold feature builds do not exhaust one shared 60-minute timeout. Every original check command and the coverage jobs are preserved. Checkpoint reopen cases are serialized around subprocess creation to prevent inherited file locks; corruption cases must report the expected validation error. Minimal-feature and all-feature checkpoint tests and Clippy pass; the workflow passes actionlint. Final-commit CI is required before merging.