Skip to content

server: observe resumable stream cancellation during first result wait (#27481) - #27522

Draft
loulanyue wants to merge 1 commit into
ggml-org:masterfrom
loulanyue:fix/27481-resumable-stream-first-result-cancel
Draft

server: observe resumable stream cancellation during first result wait (#27481)#27522
loulanyue wants to merge 1 commit into
ggml-org:masterfrom
loulanyue:fix/27481-resumable-stream-first-result-cancel

Conversation

@loulanyue

Copy link
Copy Markdown

Summary

Fixes #27481.

For requests carrying X-Conversation-Id (resumable streams), when an explicit DELETE /v1/stream?conv_id=... arrived during prompt evaluation / before the first result, handle_completions_impl polled req.should_stop instead of res->should_stop(). Because the peer socket was still connected, the cancellation was not observed before the first token, keeping the slot/GPU processing.

Changes

  • In tools/server/server-context.cpp (handle_completions_impl), poll res->should_stop() when waiting for the first streaming result and verify first_should_stop() in the assertion. This matches the effective_should_stop predicate already used for subsequent chunks in the stream generator.

@loulanyue
loulanyue requested a review from a team as a code owner August 21, 2026 23:00
@ggml-gh-bot

ggml-gh-bot Bot commented Aug 21, 2026

Copy link
Copy Markdown

Hi @loulanyue, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

  • PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

@ggml-gh-bot ggml-gh-bot Bot added the draft PR will be changed to draft by github-actions bot label Aug 21, 2026
@github-actions
github-actions Bot marked this pull request as draft August 22, 2026 00:02
@github-actions github-actions Bot removed the draft PR will be changed to draft by github-actions bot label Aug 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

server: DELETE resumable stream can abort child or fail to cancel before first token

1 participant