Skip to content

fix: join admitted master work during executor drain - #313

Merged
beinan merged 1 commit into
mainfrom
codex/master-executor-drain
Oct 5, 2026
Merged

beinan merged 1 commit into
mainfrom
codex/master-executor-drain

Conversation

@beinan

@beinan beinan commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Master shutdown currently drains HTTP but can drop detached scheduler tasks while they are still writing. Add executor-local admission tracking before claim RPCs and keep each reservation until its task finishes. An identity-checked admin drain endpoint stops new local claims, scans and catch-up controller operations; SIGTERM/SIGINT uses the same gate and waits for admitted work before runtime exit. Other masters continue consuming the durable queue.

Regression coverage checks in-flight reservations, late subscribers, repeated drain, wrong process identity, a held worker merge finishing normally, and a second master completing work left queued. Validation: 142 master tests passed with real etcd. After the final HTTP identity/controller additions, all six drain-related regressions passed from a fresh build verified to contain the final branch tests; Clippy also passed. Existing progress watchdogs/fences are unchanged; drain does not impose a writer cancellation timeout.

The operator guide specifies exact-Pod/identity polling and Kubernetes grace-period limits. This protocol only exists in new images: it cannot establish that an old auxiliary executor has drained. Dedicated one-table mode continues to use its supervisor's joined pass lifecycle.

Isolated Azure integration validation also passed on staging candidate 975550f combining #310–314: wrong executor identity returned 409, local drain held a newly queued task, a second master completed that task, and the drained master exited with code 0. Continuous ingestion/native publishing and compact/index maintenance completed beforehand; all 125 full payloads were verified and cgroup OOM/max events remained zero. This bounded functional test is not a production-size benchmark.

@beinan
beinan merged commit eeb2bcf into main Oct 5, 2026
16 of 22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant