Repository navigation
feat(wal): share node disk admission across logs and snapshots - #447
Merged
Merged
Conversation
beinan
added a commit
that referenced
this pull request
Oct 21, 2026
…452) A legacy merge POST can lose its response while the worker keeps writing. Previously the task could finish as FAILED with no durable execution record, delete its target lock, and allow another task to dispatch the same table. Stopping the current fan-out did not protect the next task or a restarted master. Record a protocol-0 running execution and a persistent target owner atomically with the canonical claim before each legacy POST. Release it only after a valid successful response or a connection failure that proves the request was not admitted. Ambiguous responses retain ownership across task completion and lease loss. Protocol-2 cancellation and storage recovery cannot retire these unfenced executions. An acknowledged terminal record remains releasable after a master crash. This adds protection for newly dispatched legacy work; it neither retrospectively accounts for existing legacy writers nor makes them automatically recoverable. A crash after recording ownership but before sending is conservatively unresolved. Initial production retirement still requires verified termination/storage evidence, and direct clients outside canonical admission are outside this guarantee. Serial fan-out and current progress-based recovery for fenced executions remain in place. Validation: existing HTTP classification tests plus new real loopback-etcd cases cover a live handler after a lost response, persistent target protection after task completion, blocked second POST, refusal of protocol-2 recovery, claim lease loss before HTTP, and successful multi-endpoint cleanup. All 9 targeted tests and 11 existing fenced-execution/progress compatibility tests passed against isolated local etcd. Rebased onto main including #447, which fixes the snapshot leader/transport-accounting assumptions that failed earlier CI; its four snapshot tests also pass on this final tree. Full CI must pass on the final head before merge; this is not deployed. The new path adds durable coordinator operations per legacy worker request. Production rollout must measure private-etcd request load as well as merge progress; it is not a throughput optimization. --------- Co-authored-by: Beinan Wang <beinanwang@microsoft.com>
beinan
added a commit
that referenced
this pull request
Oct 22, 2026
## Summary Ingestion and materializers need to reach durable replicas without embedding a WAL node. This adds authenticated TLS client RPCs with separate data/control listeners, bounded shared connection pools, leader hints, and source-payload reads. Append sends metadata and the existing payload buffer in one exchange; success is returned only after durable quorum completion. An uncertain result preserves the exact producer/sequence/body retry contract. This supplies the network interface; HTTP ingestion and an independently deployable service follow separately. Refs #378. ## Testing - Full repository CI-equivalent script passed, including Rust tests, format/Clippy, Python tests (229 passed, 2 skipped, 1 xfailed), Ruff and Pyright. - Feature tests cover bounded RPC admission, authentication and wrong-policy rejection, disconnected clients, exact retries, and separate control capacity. - Real three-process TLS tests kill the leader, restart all original replica directories, and verify every payload and retry position. Lance qualification registers an existing manifest through the network catalog and reads it before and after all replicas restart. Depends on #447. This PR is stacked on that branch so its diff can be reviewed independently; it will be rebased onto main after its dependency merges, with CI required again before merging. CI correction: Linux wheel setup uses bounded downloads from the official AlmaLinux repositories, with TLS and package signature checks preserved. APT wheel setup now uses only signed official distribution repositories in an isolated metadata directory. This avoids unrelated PPA outages; bounded network retries, HTTPS and archive signature verification remain enabled. Bash/ShellCheck and a local Debian metadata/install simulation passed; the Ubuntu cross-build installation is validated by remote CI. CI runs the existing workspace, WAL, Lance, and etcd suites in separate jobs so cold feature builds do not exhaust one shared 60-minute timeout. Every original check command and the coverage jobs are preserved. Checkpoint reopen cases are serialized around subprocess creation to prevent inherited file locks; corruption cases must report the expected validation error. Minimal-feature and all-feature checkpoint tests and Clippy pass; the workflow passes actionlint. Final-commit CI is required before merging.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Groups sharing a physical WAL node previously admitted log and snapshot work independently. This adds one node-wide logical disk budget, counts retained files on restart, and reserves capacity before append, snapshot receipt and snapshot construction. Reclamation releases accounting only after the corresponding files are removed.
Data admission preserves headroom for replication, control commits and rewrites. This is application accounting, not an OS quota or a distributed replica-credit protocol; automatic maintenance scheduling and application ingestion integration follow separately.
Refs #378.
Validation
The Python wheel setup also uses bounded package-download waits and the official AlmaLinux BaseOS/AppStream repositories. The original jobs stalled or failed in unrelated mirror metadata/package downloads before Rust compilation; package signatures and TLS validation remain enabled.