Skip to content

server: cancel resumable streams before first token - #27482

Draft
NitayRabi wants to merge 1 commit into
ggml-org:masterfrom
NitayRabi:fix/router-stream-cancel-generation
Draft

server: cancel resumable streams before first token#27482
NitayRabi wants to merge 1 commit into
ggml-org:masterfrom
NitayRabi:fix/router-stream-cancel-generation

Conversation

@NitayRabi

Copy link
Copy Markdown

Summary

  • poll the stream-aware stop predicate while waiting for the first streaming result
  • accept explicit resumable-stream cancellation in the adjacent assertion
  • add a router regression test that stops after session creation but before the first result

Fixes #27481.

Why

Requests with X-Conversation-Id deliberately survive physical socket disconnects. The pre-first-result wait nevertheless used only req.should_stop, so DELETE /v1/stream could not cancel tokenization/prompt processing. Later streaming results already use server_res_spipe::should_stop().

Testing

  • python -m pytest -q tools/server/tests/unit/test_stream.py — 4 passed
  • Release/Vulkan llama-server build
  • live router reproduction with a long prompt: DELETE posts SERVER_TASK_TYPE_CANCEL; no generation follows the prompt batch

@ggml-gh-bot

ggml-gh-bot Bot commented Aug 21, 2026

Copy link
Copy Markdown

Hi @NitayRabi, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

  • PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.

  • AI-generated content: While code is allowed to be generated by AI, please write the PR description and commit messages on your own without the help of AI.


Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

server: DELETE resumable stream can abort child or fail to cancel before first token

1 participant