Skip to content

fix(pflash): drafter resumes after a tool output - #846

Merged
davide221 merged 2 commits into
Luce-Org:mainfrom
Graffioh:fix/pflash-resume-after-tool-output
Oct 15, 2026
Merged

davide221 merged 2 commits into
Luce-Org:mainfrom
Graffioh:fix/pflash-resume-after-tool-output

Conversation

@Graffioh

@Graffioh Graffioh commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

In an agent session, the drafter scores the whole prompt again from token 0 on each request after a compressed tool output. These requests should resume from the drafter's stored session instead.

Cause. A tool session's prompt ends with the tool output. Because of that, the window that represents the latest user question as a history query is never scored, and its rows are never stored. This window is the last token of the header of the question's reply. When the next request arrives, it needs that window from before the resume point and finds no rows, so it re-scores from 0.

Fix. The server passes that window as a keep span (CompressRequest::keep_query_spans). The drafter computes the window's rows from hidden states it already has and stores them with the scoring session. Nothing is scored from the keep span, so selections are unchanged.

The session's stored query windows use the same exact-row sets as the query step's token sets. A window is dropped once a later prompt diverges at or before it, so no request scores with rows from another prompt. The windows the next request needs are stored last, so the 8-entry cap evicts this request's windows first.

Measurements

Setup: one R9700 with the release profile and the unified view. The replay is the agent-paste replay, sessions A and B.

Turn Before After Drafter resume
a8: question after a 57K-char tool output 12.8 s 4.5 s 0 → 98304
a9 11.6 s 3.3 s 0 → 98304
b3 10.2 s 5.5 s 0 → 65536

Compressed ids and selected chunks are identical before and after on all 10 compressed requests.

Tests

  • New test test_pflash_tool_session_keeps_next_history_query. It checks that the keep span of a tool session equals the history span that the next request scores, and that the token prefix is the same.
  • The existing multi-turn test now also checks that a chat whose prompt ends at the reply has no keep span.
  • New test query_windows_are_position_sets: a window that reaches the divergence point is dropped, and one before it is kept.
  • test_server_unit: 1001 passed, 0 failed.

🤖 Generated with Claude Code

Graffioh and others added 2 commits October 14, 2026 18:01
A tool session's prompt ends after the tool output, so the latest user
question's history window (the last token of its reply's header) is
never scored and its rows are never stored. The next request scores that
window as a history query, finds no stored rows before the resume point,
and re-scores the whole prompt from 0.

The server now passes that window as a keep span; the drafter computes
its rows from the hidden states it already has and stores them with the
session. Selections are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scoring session's query windows now use the exact-row sets the query
step already keeps: a window is dropped once the prompt diverges at or
before it, so a later request never scores with rows from another
prompt. The windows the next request needs are stored last, so the
eight-entry cap evicts this call's windows first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Graffioh
Graffioh marked this pull request as ready for review October 14, 2026 18:21

@davide221 davide221 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review

No issues found. Checked for bugs and logic errors in the changed code.

Reviewed d8b0b1ae · 1 candidate issue checked, kept at ≥70% confidence · add the code-review label to review again

@davide221
davide221 merged commit 6f2b179 into Luce-Org:main Oct 15, 2026
10 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants