Skip to content

fix(ws): refuse WebSocket upgrades with 503 while shutting down - #6

Draft
snowkide wants to merge 1 commit into
mainfrom
fix/ws-refuse-upgrade-while-draining
Draft

snowkide wants to merge 1 commit into
mainfrom
fix/ws-refuse-upgrade-while-draining

Conversation

@snowkide

Copy link
Copy Markdown

Summary

Once shutdown has begun (draining), a new WebSocket upgrade still completed the handshake and was then closed with 1001 straight away (ws_server.go, check after activeWsConns.Store). To a client that is a successful dial followed by a connection that drops before or during its first request.

That is worse than a refusal for clients that verify a connection before using it: the dial and check pass, the client reports the endpoint as healthy, and its next call (for example eth_subscribe newHeads) fails or waits for its own timeout before the client reconnects. A load balancer can keep sending new connections to a shutting-down instance for a short while (its endpoint update has to propagate), so this happens on ordinary rolling restarts, not only in races.

The docs already describe new WebSocket work as refused from the start of shutdown. This makes the upgrade path match:

  • handleWebSocket refuses the upgrade with 503 before the handshake while draining is set, the same flag that makes the healthcheck return 503.
  • The check after the handshake stays, for a shutdown that begins while a handshake is in progress.
  • Existing connections are unchanged: still closed with 1001 when shutdown begins, in-flight requests still get up to waitBeforeShutdown.
  • docs/pages/operation/websocket.mdx: one sentence on the 503.

No config change, no change to shutdown timing.

Test plan

  • TestWebSocket_ShutdownClosesPromptlyAfterInflightRequests: its last step now expects the late dial to fail the handshake (ErrBadHandshake) with HTTP 503. It fails without the fix (handshake completes with 101) and passes with it (also with -race, 3 runs).
  • go test ./erpc/ -run 'WebSocket|Ws|HealthCheck|Shutdown|Drain' passes (2 runs). TestWebSocket_UnsubscribeIsScopedToConnection is flaky on main as well (it counts process-wide connection gauges left by earlier tests); it failed once in one of 3 batch runs with this change and once when run alone on unchanged main.
  • go vet ./erpc/, gofmt.

Once shutdown has begun, new client connections completed the handshake and
were closed with 1001 straight away. A client sees a successful dial, then a
connection that drops before or during its first request; clients that verify
a connection before subscribing can report it as healthy and then wait for a
subscribe timeout. The docs already describe new WebSocket work as refused
from the start of shutdown.

Refuse the upgrade with 503 before the handshake instead. The check after the
handshake stays for a shutdown that begins mid-handshake.
@snowkide
snowkide force-pushed the fix/ws-refuse-upgrade-while-draining branch from 8c122c7 to 4736c3b Compare November 4, 2026 14:03

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant