Repository navigation
Judge: a reply cut off at the 200-token cap can leave a lone quoted score object, which the parser takes as the verdict #1044
Description
Summary
PR #1038 (the fix for #127) makes the judge parser fail closed when a reply holds several score objects. A reply that holds exactly one is still accepted, and that one is not necessarily the judge's own: if the judge is led to quote the judged answer first and its own verdict is cut off by the 200-token reply cap, the only score object left is the quoted one.
What happens
_call_api_judge sends max_tokens: 200 and _call_local_judge generates at most max_new_tokens=200. Neither looks at whether the reply was cut off. The judged answer may be up to 1000 characters (_JUDGE_RESPONSE_MAX_CHARS), so quoting it can use the whole budget.
Offline, with the parser from #1038 (head 1413ffd):
| Judge reply | Result |
|---|---|
a quoted {"score": 10, ...}, a long quote, then My verdict: {"score": 2, "rea (cut off) |
score 10 |
the same, cut off before the brace (My verdict: ) |
score 10 |
| a quoted object, then the judge's complete verdict | None (two objects: fails closed) |
| JSON-mode reply cut off inside the object | None (invalid JSON) |
So a poisoned model that makes the judge quote can still pass the judge gate whenever the quote is long enough to push the verdict past the cap. It is the same attack as #127 with the same precondition, through a path the "exactly one score object" rule cannot see.
Fix
Treat a truncated reply as a failure before it is parsed, with the existing score=None sentinel and a reason that names truncation (log the length, never the content):
- API judge:
choices[0].finish_reason == "length". - Local judge: the number of generated tokens equals
max_new_tokensand the last token is not the EOS token. - A gateway that sends no
finish_reasoncannot be checked; that is a documented limit, not an error.
This is part (2) of CR-A06-032 in #553, where the same cap causes the opposite problem: well-formed but long verdicts counted as invalid. Part (1) there (a configurable reply budget) stays in #553. Deliver the truncation check once, here, and tick both.
Acceptance criteria
- A regression test per judge path that fails on current
developmentand passes after the change: an API reply withfinish_reason: "length"and a lone score object gives None; the same reply withfinish_reason: "stop"is accepted; the local equivalent with injected token ids. No GPU, network access or model downloads. - A reply cut off in the middle of the judge's own object is also None (the table above).
- A
### Fixedentry under## [Unreleased]inCHANGELOG.md. - If the judge manual describes how replies are parsed by then, it mentions truncation, with the Turkish mirror in the same PR.
Context
Found while deciding the open points of #1042. The change is independent of #1038 in code, but its tests are meaningful only with the stricter parser; land it right after #1038. Related: #127, #1038, #1042, #553 (CR-A06-032).
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsDone
Activity
docs(changelog): append the judge entry at the end of the Unreleased …
docs(changelog): file the judge entry under the Unreleased Fixed list
fix(judge): also treat content_filter as incomplete, report the count…
docs(judge): tell the three incomplete-reply cases apart
Merge origin/development into fix/1044-judge-truncated-reply
fix(judge): a reply without text is one judge failure, not a crash
fix(judge): count a reply cut off at the token cap as a judge failure (…