Skip to content

fix: drain stalled HTTP responses without cancelling accepted writes - #318

Merged
beinan merged 1 commit into
mainfrom
codex/native-runtime-recovery
Oct 6, 2026
Merged

beinan merged 1 commit into
mainfrom
codex/native-runtime-recovery

Conversation

@beinan

@beinan beinan commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

A worker can spend its entire termination grace period waiting for HTTP clients that stopped receiving responses, leaving its WAL merge endpoint unavailable during a rollout. Graceful shutdown now closes admission, joins accepted handlers (even after client disconnect), then retires only response sockets whose writes have stopped making progress. Resident writers close afterward.

HTTP_SHUTDOWN_RESPONSE_IDLE_SECS defaults to 120. Successful socket writes reset the interval; there is no new total merge/query/download deadline. Exact-socket control descriptors avoid descriptor-reuse races, at the cost of one additional descriptor per connection. Response bodies remain cancellable; staged append streams only prepare unreachable immutable files and do not publish manifests.

Validation:

  • All 104 ordinary server tests and all 7 tests against isolated local etcd passed. This includes seven new TCP/admission regressions for disconnected accepted work, rejection during drain, panic/error joining, a stalled download while another handler is active, and continuing or resumed downloads.
  • Reproduced the old-image failure with a 32 MiB blob in isolated Azure staging: still waiting after 6 seconds without a reader; closing that reader allowed normal exit in 0.36 seconds, with no OOM.
  • Local Clippy passed; CI workspace tests and coverage are still running. The merged image will be checked with the same Azure download fixture, WAL recovery and existing native maintenance tests before production deployment.

@beinan
beinan marked this pull request as ready for review October 6, 2026 02:15
@beinan
beinan merged commit 53756ca into main Oct 6, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant