Skip to content

fix(model-gateway): terminate downstream streams when passthrough or Grok upstreams close mid-response - #820

Merged
Eigenwise merged 2 commits into
Eigenwise:developfrom
chris7iu:fix/model-gateway-passthrough-stream-abort
Nov 12, 2026
Merged

Eigenwise merged 2 commits into
Eigenwise:developfrom
chris7iu:fix/model-gateway-passthrough-stream-abort

Conversation

@chris7iu

Copy link
Copy Markdown
Contributor

Summary

  • Propagate mid-stream upstream aborts on Path B (upRes.pipe(clientRes)) in request-worker.js:
    • On non-Codex passthrough responses (e.g. claude-sonnet-4-6, claude-opus-4-7), request-worker.js attaches upRes.once('error', () => routeTelemetry?.finish(upRes.statusCode, 'upstream_error')) before piping upRes.pipe(clientRes). Because upRes already has an 'error' listener, Node's http client does not re-emit mid-body chunked framing/socket errors (HPE_INVALID_CHUNK_SIZE, premature socket close) on upReq, and Readable.prototype.pipe() forwards neither 'error' nor 'aborted' to clientRes. When an upstream aborts mid-stream after flushing HTTP 200 headers, clientRes previously hung open indefinitely instead of propagating UND_ERR_SOCKET to serve-shim / Claude Code's withRetry.
    • Attaches terminateDownstream on upRes 'error', 'aborted', and non-complete 'close' (!upRes.complete) so clientRes is destroyed immediately when headers are already sent (or answered with 502 Bad Gateway if headers were not yet sent).
  • Propagate mid-stream upstream aborts in forwardGrok:
    • Attaches terminateGrokDownstream on Grok upRes 'error', 'aborted', and non-complete 'close' so truncated Grok SSE responses after 200 headers also destroy clientRes rather than stalling or finalizing cleanly.
  • Pairs with fix(stream): retry pre-output TLS close_notify EOF and abort thinking-only streams on transient drops raine/claude-code-proxy#210 so mid-thinking transient upstream drops cleanly trigger Claude Code's native withRetry across the two-hop serve-shim -> serve-worker -> upstream path.

Test Plan

  • node --require ./plugins/model-gateway/test/isolate-home.js --test plugins/model-gateway/test/relay-stream-abort.test.js plugins/model-gateway/test/request-worker.test.js plugins/model-gateway/test/grok-backend.test.js
    • Added tests in relay-stream-abort.test.js covering both serve-worker and two-hop serve-shim propagation when a passthrough Anthropic SSE upstream (claude-sonnet-4-6) or a Grok SSE upstream (grok-4.20) closes mid-chunk after flushing 200 headers.
  • node scripts/release/guard.mjs --mode dev --publish-ref origin/main --default-branch main --integration-branch develop ...

@chris7iu
chris7iu force-pushed the fix/model-gateway-passthrough-stream-abort branch from 4fcf54a to 2ea24f8 Compare November 12, 2026 07:50
@chris7iu
chris7iu force-pushed the fix/model-gateway-passthrough-stream-abort branch from 2ea24f8 to 12964f6 Compare November 12, 2026 07:58
@Eigenwise
Eigenwise merged commit 7350b19 into Eigenwise:develop Nov 12, 2026
37 of 39 checks passed
@Eigenwise

Copy link
Copy Markdown
Owner

Merged, thanks. The fix reads right: with an error listener already on the upstream response, nothing told the client side that the body was cut short, so the stream just hung. Destroying the downstream response on error, abort or an incomplete close covers the passthrough and Grok paths, and the two-hop test proves it through the shim too. It goes out with the next model-gateway release.

The red Windows check on the first run was ours, not yours: a gateway lock test that times out on a slow runner. It's filed and the rerun passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants