fix(zcode): retry invalid structured output in-session before failing - #31
fix(zcode): retry invalid structured output in-session before failing#31zhangweijian97 wants to merge 3 commits into
Conversation
A successful turn whose final response contains no JSON object used to terminalize the whole attempt as invalid_structured_output. High-effort models periodically reply with a prose report instead, so the harness now sends a schema-repeating correction message into the same session and waits for the next completion, up to two times, before failing. The retry counter is ephemeral: a restart already abandons the in-flight turn, so no cursor migration is needed.
|
已使用 当前建议：修复以下两项 P2 问题，并让完整检查通过后再合并。 Spec[P2] 纠正轮次没有保留已报告的 token 用量位置： JSON-less completion 进入纠正分支后，其 已通过真实 修复指令：
[P2] JSON 标量和数组会绕过纠正分支位置：
已用上述三种响应分别复现：即使队列里已有下一轮有效角色输出，也只发送一次初始 修复指令：检查提取结果确实为非 null、非数组对象；添加 primitive / array 后接有效角色输出的边界测试，确认发出 correction 并成功完成。对象本身不符合角色 schema 时，仍可保留现有 fail-closed 行为。 Standards / 验证人工规范审查没有额外 finding；工具检查发现一个需要处理的格式错误：
汇总：Standards 人工 finding 0；Spec 2 项 P2。现有测试重跑通过，但两个已复现的行为问题和完整检查失败仍需处理。 |
Extend the in-session structured-output retry to responses whose JSON object violates the role's strict schema, not only JSON-less prose. Live runs of the ZCode backend showed the dominant failure shape is a JSON reply that embeds the prompt's schema metadata as a "\$schema" key or omits required fields, which strict validation rejects; the correction prompt already restates the schema, so both shapes share one retry budget (two corrections total) before the attempt fails through the existing invalid_structured_output path. Implement commit resolution stays outside the retry: a schema-valid draft whose commit cannot be resolved is a workspace problem restating the schema cannot correct.
zhangweijian97
changed the title
The native protocol reports usage per turn, but the correction branch dropped each JSON-less turn's usage before sending the retry, so a corrected attempt reported only the final turn's tokens and a retry-exhausted attempt reported zero. Every successful completion now adds to an attempt-scoped accumulator seeded from the persisted cursor baseline: correction success, correction-send failure, and retry exhaustion all publish the consumed tokens through a usage_delta delivery whose cursor carries the total, and a replay from that cursor never re-counts already-published usage. Also adds boundary tests for JSON scalars and arrays (null, [], string), which earn a correction round under the retry extension instead of bypassing it, and expands the harness.test.ts correction type declaration per Biome format.
|
两项 P2 已在 76ca962 修复并推送（当前 head）。先说明 head 状态：本轮评审针对 0db3d4b，其后分支已前进到 43270f9——那是独立于本轮评审的扩展（真跑 ZCode 后端观察到主导失败形态是内嵌 [P2] 纠正轮次没有保留已报告的 token 用量 按三条修复指令落地：
[P2] JSON 标量和数组会绕过纠正分支
Standards / 验证
|
2 participants
Problem
Under high effort, the model behind the ZCode app-server periodically answers a role turn in a form that fails structured-output extraction, burning the whole attempt as
invalid_structured_outputeven though the session is still healthy. Live runs showed two failure shapes:{"$schema":"https://json-schema.org/draft/2020-12/schema","kind":"review","decision":"accepted",...}— it echoes the prompt's schema metadata as a$schemakey and drops required fields likeremainingGaps, so strict validation fails even though a JSON object is present.The app-server protocol exposes no output-format control (no
responseFormat-style parameter onsession/*), so the constraint cannot be enforced at the protocol layer today.Change
When a successful
turn.completedcarries a response that does not decode into a schema-valid role output — either shape above — the harness sends one in-session correction message (repeating the exact role schema) and waits for the next completion instead of failing:decodeStructuredOutput(role, response) === undefined: no extractable JSON object or a JSON object that violates the strict role schema;invalid_structured_outputfailure path runs unchanged;orphaned_turn), so no cursor migration is needed;structuredOutputRetryPromptinprompts.ts); its wording now covers both shapes ("contained no JSON object matching the required schema");invalid_implementation_commit) that restating the schema cannot correct.This mirrors the tolerance-extraction self-rescue already accepted for the Pi backend (f97b51d): when the protocol cannot guarantee structured output, the backend recovers in-session instead of terminalizing.
Testing
invalid_structured_output;$schema-pollution shape) triggers one correction and the retried reply completes;bun test406 pass / 0 fail,tsc --noEmitclean.