Skip to content

Judge: a reply cut off at the 200-token cap can leave a lone quoted score object, which the parser takes as the verdict #1044

Closed

Description

@cemililik

Summary

PR #1038 (the fix for #127) makes the judge parser fail closed when a reply holds several score objects. A reply that holds exactly one is still accepted, and that one is not necessarily the judge's own: if the judge is led to quote the judged answer first and its own verdict is cut off by the 200-token reply cap, the only score object left is the quoted one.

What happens

_call_api_judge sends max_tokens: 200 and _call_local_judge generates at most max_new_tokens=200. Neither looks at whether the reply was cut off. The judged answer may be up to 1000 characters (_JUDGE_RESPONSE_MAX_CHARS), so quoting it can use the whole budget.

Offline, with the parser from #1038 (head 1413ffd):

Judge reply Result
a quoted {"score": 10, ...}, a long quote, then My verdict: {"score": 2, "rea (cut off) score 10
the same, cut off before the brace (My verdict: ) score 10
a quoted object, then the judge's complete verdict None (two objects: fails closed)
JSON-mode reply cut off inside the object None (invalid JSON)

So a poisoned model that makes the judge quote can still pass the judge gate whenever the quote is long enough to push the verdict past the cap. It is the same attack as #127 with the same precondition, through a path the "exactly one score object" rule cannot see.

Fix

Treat a truncated reply as a failure before it is parsed, with the existing score=None sentinel and a reason that names truncation (log the length, never the content):

  • API judge: choices[0].finish_reason == "length".
  • Local judge: the number of generated tokens equals max_new_tokens and the last token is not the EOS token.
  • A gateway that sends no finish_reason cannot be checked; that is a documented limit, not an error.

This is part (2) of CR-A06-032 in #553, where the same cap causes the opposite problem: well-formed but long verdicts counted as invalid. Part (1) there (a configurable reply budget) stays in #553. Deliver the truncation check once, here, and tick both.

Acceptance criteria

  • A regression test per judge path that fails on current development and passes after the change: an API reply with finish_reason: "length" and a lone score object gives None; the same reply with finish_reason: "stop" is accepted; the local equivalent with injected token ids. No GPU, network access or model downloads.
  • A reply cut off in the middle of the judge's own object is also None (the table above).
  • A ### Fixed entry under ## [Unreleased] in CHANGELOG.md.
  • If the judge manual describes how replies are parsed by then, it mentions truncation, with the Turkish mirror in the same PR.

Context

Found while deciding the open points of #1042. The change is independent of #1038 in code, but its tests are meaningful only with the stricter parser; land it right after #1038. Related: #127, #1038, #1042, #553 (CR-A06-032).

Activity

  1. added
    bugSomething isn't working
    severity: highWrong behaviour or a false claim users rely on
    wave: 1Roadmap wave 1: high severity, small effort
    on Oct 20, 2026
  2. added
    needs-triageNot triaged yet: every new issue starts with it; only a maintainer removes it
    on Oct 20, 2026
  3. removed
    needs-triageNot triaged yet: every new issue starts with it; only a maintainer removes it
    on Oct 20, 2026
  4. self-assigned this
    on Oct 20, 2026
  5. added a commit that references this issue on Oct 20, 2026
    7550f75
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area: eval-gatesCode area: eval-gatesbugSomething isn't workingseverity: highWrong behaviour or a false claim users rely onwave: 1Roadmap wave 1: high severity, small effort

Type

No type

Projects

Relationships

None yet

Development

No branches or pull requests

Issue actions