Repository navigation
feat(ingest): bootstrap managed rollout tables idempotently - #457
Merged
Merged
Conversation
beinan
force-pushed
the
codex/wal-ingest-bootstrap
branch
2 times, most recently
from
October 22, 2026 00:46
9a85bbc to
b49a4e2
Compare
beinan
force-pushed
the
codex/wal-ingest-worker
branch
from
October 22, 2026 03:45
db4e752 to
9e4064c
Compare
beinan
force-pushed
the
codex/wal-ingest-bootstrap
branch
from
October 22, 2026 04:17
b49a4e2 to
157e593
Compare
beinan
added a commit
that referenced
this pull request
Oct 22, 2026
## Summary Compose the durable ingestion components into a Rust executable with bootstrap, HTTP ingress, materializer worker, and managed-reader inspection roles. A fixed deployment-authorized registry shares WAL connections and memory admission across tables; startup validates existing generations, and shutdown joins admitted work before releasing claims. WAL connections use mutual TLS and pinned endpoints. HTTP requests require a workload token and expect TLS termination at the edge. Configuration parsing rejects unknown fields and bounds file reads. Dynamic namespace discovery, legacy-table migration and sustained scaling remain follow-up work. Refs #378. Built on merged #457. The feature patch is unchanged after moving to main; all final-commit CI checks must pass before merging. ## Testing - Cargo formatting and service/http-client Clippy passed. - All 7 ordinary ingestion feature tests passed, including exact retry after replica restart and full Lance publication recovery. - Built and executed this branch's application binary against real TLS WAL nodes and isolated etcd: repeated bootstrap, SIGKILL/restarted ingress, original-receipt retry, two workers, full Arrow readback and joined shutdown passed. - Both etcd worker ownership/expiry tests passed. - CI runs the executable recovery test and retains process logs on failure. These tests establish the exercised recovery behavior, not sustained throughput.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Initialize a managed rollout generation in Lance and the durable WAL catalog using persisted operation identities. Retrying after a crash reuses the original genesis manifest; initialization rejects an existing foreign table instead of adopting or overwriting it.
The regression tests use real Lance files and replicated WALs, including a restart after the genesis file exists but before catalog selection. Focused ingestion/bootstrap tests, feature Clippy, and the complete repository checks pass (Python: 229 passed, 2 skipped, 1 xfailed).
Refs #378. Built on merged #455. The feature diff is unchanged when moved to main; all final-commit CI checks must pass before merging.
Independent Raft-log recovery cases exclude sibling subprocess-creation windows, preventing inherited file locks from racing immediate reopen. The two-group concurrent-write case and real SIGKILL, rotation, and write-error assertions remain unchanged. All nine Raft-log cases pass with minimal and all features, and Clippy passes. This adjustment changes only test scheduling.
TLS process startup failures include the child stdout/stderr and identify peer/image listener binding errors. The unchanged quorum, snapshot-interruption, SIGKILL and full-restart assertions pass in five complete peer-transport library runs (126 tests per run), plus six process tests and feature Clippy. The intermittent CI startup failure has not been reproduced locally; these diagnostics do not establish its root cause. Final-head CI remains required.