Repository navigation
fix: join admitted master work during executor drain - #313
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Master shutdown currently drains HTTP but can drop detached scheduler tasks while they are still writing. Add executor-local admission tracking before claim RPCs and keep each reservation until its task finishes. An identity-checked admin drain endpoint stops new local claims, scans and catch-up controller operations; SIGTERM/SIGINT uses the same gate and waits for admitted work before runtime exit. Other masters continue consuming the durable queue.
Regression coverage checks in-flight reservations, late subscribers, repeated drain, wrong process identity, a held worker merge finishing normally, and a second master completing work left queued. Validation: 142 master tests passed with real etcd. After the final HTTP identity/controller additions, all six drain-related regressions passed from a fresh build verified to contain the final branch tests; Clippy also passed. Existing progress watchdogs/fences are unchanged; drain does not impose a writer cancellation timeout.
The operator guide specifies exact-Pod/identity polling and Kubernetes grace-period limits. This protocol only exists in new images: it cannot establish that an old auxiliary executor has drained. Dedicated one-table mode continues to use its supervisor's joined pass lifecycle.
Isolated Azure integration validation also passed on staging candidate
975550fcombining #310–314: wrong executor identity returned 409, local drain held a newly queued task, a second master completed that task, and the drained master exited with code 0. Continuous ingestion/native publishing and compact/index maintenance completed beforehand; all 125 full payloads were verified and cgroup OOM/max events remained zero. This bounded functional test is not a production-size benchmark.