Repository navigation
Conversation
There was a problem hiding this comment.
Pull request overview
该 PR 主要围绕 LLM 调用的可观测性与稳定性:在流式调用失败时也统计 token 费用、增强阅卷失败/重试的调试日志落盘能力,并放宽阅卷结果 JSON 中“题目名称”一致性校验,同时同步 README 徽章展示。
Changes:
- 在多处流式 LLM 消费逻辑中加入异常捕获,失败时也尝试记录 usage/费用,并补充错误日志输出
- 阅卷链路增强:解析失败详情(位置/snippet)、fallback/retry/failed 调试报告落盘,且放宽题目名称校验并以 rubric 为准归一化输出
- README 徽章布局调整并固定 v0.2.0 release workflow 状态展示
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| src/main/services/smartNameMatchService.ts | 流式读取失败时补记 usage(用于失败调用费用统计) |
| src/main/services/settingsService.ts | 连接测试流式读取失败时补记 usage(用于失败调用费用统计) |
| src/main/services/scorePostProcessService.ts | AI 脚本生成流式读取失败时补记 usage(用于失败调用费用统计) |
| src/main/services/llmJsonDebug.ts | 增加通用文本调试产物写入能力并复用 debug 路径生成逻辑 |
| src/main/services/gradingService.ts | 阅卷链路:JSON 解析失败详情、fallback/retry/failed 调试落盘、放宽题目名称校验、流式失败补记 usage |
| src/main/services/answerGeneratorService.ts | 参考答案生成流式失败时输出错误日志并补记 usage |
| README.md | 调整徽章布局,新增 CI / Release v0.2.0 状态徽章 |
Comments suppressed due to low confidence (1)
src/main/services/gradingService.ts:1036
- 这里 lastParsedCandidate 只在 parseJsonObject 成功时更新;如果当前 attempt 的 parseJsonObject 抛错,lastParsedCandidate 会保留上一次 attempt 的值,后续写入的 retry/failed 调试报告会混入不相关的 parsedCandidate。建议在每次 attempt 开始时将 lastParsedCandidate 重置为 null,或在 parseJsonObject 前先置空再赋值。
lastFinishReason = response.finishReason ?? '';
lastParsedCandidate = parseJsonObject(lastRawOutput);
const modelResult = validateModelResult(lastParsedCandidate, input.rubric, input.drawRegions);
logLlmResult(`grading-paper:${input.paper.paperCode}`, {
status: 'success',
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| await this.llmUsage?.recordUsage({ | ||
| source: 'settings-test', | ||
| label: '连接测试', | ||
| model: input.requestPayload.model, | ||
| usage, | ||
| }); |
There was a problem hiding this comment.
流式读取 catch 中 await this.llmUsage?.recordUsage(...) 可能抛错并覆盖原始流式异常,导致 collectConnectionTestResponse 的 isStreamingFallbackCandidate 判断不再成立,从而无法 fallback 到普通请求。建议将 recordUsage 包裹在 try/catch 中,确保仍然抛出原始 error(或至少保留原始 error 作为 cause)。
| await this.llmUsage?.recordUsage({ | |
| source: 'settings-test', | |
| label: '连接测试', | |
| model: input.requestPayload.model, | |
| usage, | |
| }); | |
| try { | |
| await this.llmUsage?.recordUsage({ | |
| source: 'settings-test', | |
| label: '连接测试', | |
| model: input.requestPayload.model, | |
| usage, | |
| }); | |
| } catch { | |
| // Ignore usage recording failures here so the original streaming error | |
| // is preserved for fallback detection by the caller. | |
| } |
| await this.llmUsage.recordUsage({ | ||
| source: 'score-postprocess-ai-script', | ||
| label: 'AI脚本生成', | ||
| model: input.requestPayload.model, | ||
| usage, | ||
| }); |
There was a problem hiding this comment.
这里在流式消费失败时记录用量,如果 recordUsage 抛错会覆盖原始流式错误并改变上层逻辑(例如 fallback/错误分类),也会让实际错误原因丢失。建议对 recordUsage 做 best-effort:catch 记录异常但不影响原始 error 的抛出。
| await this.llmUsage.recordUsage({ | |
| source: 'score-postprocess-ai-script', | |
| label: 'AI脚本生成', | |
| model: input.requestPayload.model, | |
| usage, | |
| }); | |
| try { | |
| await this.llmUsage.recordUsage({ | |
| source: 'score-postprocess-ai-script', | |
| label: 'AI脚本生成', | |
| model: input.requestPayload.model, | |
| usage, | |
| }); | |
| } catch (recordUsageError) { | |
| console.warn( | |
| '[ScorePostProcessService] Failed to record usage after streaming error.', | |
| recordUsageError, | |
| ); | |
| } |
| await this.llmUsage.recordUsage({ | ||
| source: 'grading-paper', | ||
| label: `答卷 ${input.paperCode}`, | ||
| model: input.requestPayload.model, | ||
| usage, | ||
| }); | ||
| throw this.wrapPaperAttemptError(error, { | ||
| rawOutput: rawText, | ||
| reasoningText, | ||
| mode: 'stream', | ||
| finishReason: finishReason ?? undefined, | ||
| }); |
There was a problem hiding this comment.
collectStreamingPaperResponseText 的 catch 里先 await recordUsage 再 throw,如果 recordUsage 因 DB/IO 抛错,会覆盖原始流式异常,导致上层 collectPaperResponseText 无法按 isStreamingFallbackCandidate 进行 fallback,并丢失真实错误。建议将 recordUsage 包裹在 try/catch 中并始终抛出(wrap 之后的)原始 error。
| await this.llmUsage.recordUsage({ | |
| source: 'grading-paper', | |
| label: `答卷 ${input.paperCode}`, | |
| model: input.requestPayload.model, | |
| usage, | |
| }); | |
| throw this.wrapPaperAttemptError(error, { | |
| rawOutput: rawText, | |
| reasoningText, | |
| mode: 'stream', | |
| finishReason: finishReason ?? undefined, | |
| }); | |
| const wrappedError = this.wrapPaperAttemptError(error, { | |
| rawOutput: rawText, | |
| reasoningText, | |
| mode: 'stream', | |
| finishReason: finishReason ?? undefined, | |
| }); | |
| try { | |
| await this.llmUsage.recordUsage({ | |
| source: 'grading-paper', | |
| label: `答卷 ${input.paperCode}`, | |
| model: input.requestPayload.model, | |
| usage, | |
| }); | |
| } catch { | |
| // Preserve the original streaming failure for upstream fallback handling. | |
| } | |
| throw wrappedError; |
| await this.llmUsage.recordUsage({ | ||
| source: 'grading-rubric', | ||
| label: '评分标准整理', | ||
| model: input.requestPayload.model, | ||
| usage, | ||
| }); |
There was a problem hiding this comment.
collectStreamingRubricResponseText 在 catch 中记录 usage:如果 recordUsage 抛错会覆盖流式异常并让上层的 fallback 判断失效。建议把 recordUsage 变成 best-effort(try/catch 忽略失败或仅 warn),然后继续抛出原始 error。
| await this.llmUsage.recordUsage({ | |
| source: 'grading-rubric', | |
| label: '评分标准整理', | |
| model: input.requestPayload.model, | |
| usage, | |
| }); | |
| try { | |
| await this.llmUsage.recordUsage({ | |
| source: 'grading-rubric', | |
| label: '评分标准整理', | |
| model: input.requestPayload.model, | |
| usage, | |
| }); | |
| } catch { | |
| // Best-effort only: do not let usage logging mask the original streaming error. | |
| } |
| mode: debugContext.mode, | ||
| finishReason: debugContext.finishReason, | ||
| fallbackReason: debugContext.fallbackReason, | ||
| fallbackAt: debugContext.fallbackAt, |
There was a problem hiding this comment.
写 retry 调试报告时这里使用 debugContext.mode/finishReason/fallbackReason 等字段,但这些字段只有在 wrapPaperAttemptError 赋值时才会存在;像 JSON 解析/校验失败这类错误并不会附带这些字段,导致报告里 mode/finishReason 等信息丢失(而 lastMode/lastFinishReason 已经在 try 中拿到了)。建议这里改用 lastMode/lastFinishReason/lastFallbackReason/lastFallbackAt(并且与 lastRawOutput/lastReasoningText 保持一致)。
| mode: debugContext.mode, | |
| finishReason: debugContext.finishReason, | |
| fallbackReason: debugContext.fallbackReason, | |
| fallbackAt: debugContext.fallbackAt, | |
| mode: lastMode, | |
| finishReason: lastFinishReason, | |
| fallbackReason: lastFallbackReason, | |
| fallbackAt: lastFallbackAt, |
| await this.llmUsage.recordUsage({ | ||
| source: 'answer-generation', | ||
| label: '参考答案生成', | ||
| model: input.requestPayload.model, | ||
| usage, | ||
| }); |
There was a problem hiding this comment.
这里在 catch 中 await this.llmUsage.recordUsage(...) 后再 throw error;如果 recordUsage 抛错会覆盖原始模型/网络异常,导致最终看到的错误与真实失败原因不一致。建议将 recordUsage 包裹为 best-effort(try/catch),并始终 rethrow 原始 error。
| await this.llmUsage.recordUsage({ | |
| source: 'answer-generation', | |
| label: '参考答案生成', | |
| model: input.requestPayload.model, | |
| usage, | |
| }); | |
| try { | |
| await this.llmUsage.recordUsage({ | |
| source: 'answer-generation', | |
| label: '参考答案生成', | |
| model: input.requestPayload.model, | |
| usage, | |
| }); | |
| } catch { | |
| // Best-effort only: never let usage recording mask the original stream/model error. | |
| } |
| await this.llmUsage.recordUsage({ | ||
| source: 'smart-name-match', | ||
| label: '智能核名', | ||
| model: input.requestPayload.model, | ||
| usage, | ||
| }); |
There was a problem hiding this comment.
这里在流式读取失败时直接 await this.llmUsage.recordUsage(...),如果 recordUsage 因数据库/IO 异常抛错,会把原始流式错误覆盖掉,进而让上层的 isStreamingFallbackCandidate 判断失效(无法回退 non-stream),并且丢失真实失败原因。建议把 recordUsage 做成 best-effort:用 try/catch 吞掉记录失败,或捕获后仍然 rethrow 原始 error。
| await this.llmUsage.recordUsage({ | |
| source: 'smart-name-match', | |
| label: '智能核名', | |
| model: input.requestPayload.model, | |
| usage, | |
| }); | |
| try { | |
| await this.llmUsage.recordUsage({ | |
| source: 'smart-name-match', | |
| label: '智能核名', | |
| model: input.requestPayload.model, | |
| usage, | |
| }); | |
| } catch { | |
| // Best-effort only: do not let usage recording failures mask the original streaming error. | |
| } |
fix: 失败的llm调用也会计入费用中
feat: 调试模式下支持输出试卷重试与失败的详细 llm 调试日志
feat: 完善阅卷重试调试日志并放宽题目名称校验
docs: 同步 README 徽章布局并固定 v0.2.0 发布状态