Skip to content

完善文档,统计失败的llm调用费用,放宽阅卷json校验,添加错误日志输出 - #7

Merged
BobH233 merged 6 commits into
mainfrom
dev
Apr 14, 2026
Merged

BobH233 merged 6 commits into
mainfrom
dev

Conversation

Copilot AI review requested due to automatic review settings April 14, 2026 10:47

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

该 PR 主要围绕 LLM 调用的可观测性与稳定性:在流式调用失败时也统计 token 费用、增强阅卷失败/重试的调试日志落盘能力,并放宽阅卷结果 JSON 中“题目名称”一致性校验,同时同步 README 徽章展示。

Changes:

  • 在多处流式 LLM 消费逻辑中加入异常捕获,失败时也尝试记录 usage/费用,并补充错误日志输出
  • 阅卷链路增强:解析失败详情(位置/snippet)、fallback/retry/failed 调试报告落盘,且放宽题目名称校验并以 rubric 为准归一化输出
  • README 徽章布局调整并固定 v0.2.0 release workflow 状态展示

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
src/main/services/smartNameMatchService.ts 流式读取失败时补记 usage(用于失败调用费用统计)
src/main/services/settingsService.ts 连接测试流式读取失败时补记 usage(用于失败调用费用统计)
src/main/services/scorePostProcessService.ts AI 脚本生成流式读取失败时补记 usage(用于失败调用费用统计)
src/main/services/llmJsonDebug.ts 增加通用文本调试产物写入能力并复用 debug 路径生成逻辑
src/main/services/gradingService.ts 阅卷链路:JSON 解析失败详情、fallback/retry/failed 调试落盘、放宽题目名称校验、流式失败补记 usage
src/main/services/answerGeneratorService.ts 参考答案生成流式失败时输出错误日志并补记 usage
README.md 调整徽章布局,新增 CI / Release v0.2.0 状态徽章
Comments suppressed due to low confidence (1)

src/main/services/gradingService.ts:1036

  • 这里 lastParsedCandidate 只在 parseJsonObject 成功时更新;如果当前 attempt 的 parseJsonObject 抛错,lastParsedCandidate 会保留上一次 attempt 的值,后续写入的 retry/failed 调试报告会混入不相关的 parsedCandidate。建议在每次 attempt 开始时将 lastParsedCandidate 重置为 null,或在 parseJsonObject 前先置空再赋值。
        lastFinishReason = response.finishReason ?? '';
        lastParsedCandidate = parseJsonObject(lastRawOutput);
        const modelResult = validateModelResult(lastParsedCandidate, input.rubric, input.drawRegions);
        logLlmResult(`grading-paper:${input.paper.paperCode}`, {
          status: 'success',

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +276 to +281
await this.llmUsage?.recordUsage({
source: 'settings-test',
label: '连接测试',
model: input.requestPayload.model,
usage,
});

Copilot AI Apr 14, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

流式读取 catch 中 await this.llmUsage?.recordUsage(...) 可能抛错并覆盖原始流式异常,导致 collectConnectionTestResponse 的 isStreamingFallbackCandidate 判断不再成立,从而无法 fallback 到普通请求。建议将 recordUsage 包裹在 try/catch 中,确保仍然抛出原始 error(或至少保留原始 error 作为 cause)。

Suggested change
await this.llmUsage?.recordUsage({
source: 'settings-test',
label: '连接测试',
model: input.requestPayload.model,
usage,
});
try {
await this.llmUsage?.recordUsage({
source: 'settings-test',
label: '连接测试',
model: input.requestPayload.model,
usage,
});
} catch {
// Ignore usage recording failures here so the original streaming error
// is preserved for fallback detection by the caller.
}

Copilot uses AI. Check for mistakes.
Comment on lines +1409 to +1414
await this.llmUsage.recordUsage({
source: 'score-postprocess-ai-script',
label: 'AI脚本生成',
model: input.requestPayload.model,
usage,
});

Copilot AI Apr 14, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里在流式消费失败时记录用量,如果 recordUsage 抛错会覆盖原始流式错误并改变上层逻辑(例如 fallback/错误分类),也会让实际错误原因丢失。建议对 recordUsage 做 best-effort:catch 记录异常但不影响原始 error 的抛出。

Suggested change
await this.llmUsage.recordUsage({
source: 'score-postprocess-ai-script',
label: 'AI脚本生成',
model: input.requestPayload.model,
usage,
});
try {
await this.llmUsage.recordUsage({
source: 'score-postprocess-ai-script',
label: 'AI脚本生成',
model: input.requestPayload.model,
usage,
});
} catch (recordUsageError) {
console.warn(
'[ScorePostProcessService] Failed to record usage after streaming error.',
recordUsageError,
);
}

Copilot uses AI. Check for mistakes.
Comment on lines +1364 to +1375
await this.llmUsage.recordUsage({
source: 'grading-paper',
label: `答卷 ${input.paperCode}`,
model: input.requestPayload.model,
usage,
});
throw this.wrapPaperAttemptError(error, {
rawOutput: rawText,
reasoningText,
mode: 'stream',
finishReason: finishReason ?? undefined,
});

Copilot AI Apr 14, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

collectStreamingPaperResponseText 的 catch 里先 await recordUsage 再 throw,如果 recordUsage 因 DB/IO 抛错,会覆盖原始流式异常,导致上层 collectPaperResponseText 无法按 isStreamingFallbackCandidate 进行 fallback,并丢失真实错误。建议将 recordUsage 包裹在 try/catch 中并始终抛出(wrap 之后的)原始 error。

Suggested change
await this.llmUsage.recordUsage({
source: 'grading-paper',
label: `答卷 ${input.paperCode}`,
model: input.requestPayload.model,
usage,
});
throw this.wrapPaperAttemptError(error, {
rawOutput: rawText,
reasoningText,
mode: 'stream',
finishReason: finishReason ?? undefined,
});
const wrappedError = this.wrapPaperAttemptError(error, {
rawOutput: rawText,
reasoningText,
mode: 'stream',
finishReason: finishReason ?? undefined,
});
try {
await this.llmUsage.recordUsage({
source: 'grading-paper',
label: `答卷 ${input.paperCode}`,
model: input.requestPayload.model,
usage,
});
} catch {
// Preserve the original streaming failure for upstream fallback handling.
}
throw wrappedError;

Copilot uses AI. Check for mistakes.
Comment on lines +1600 to +1605
await this.llmUsage.recordUsage({
source: 'grading-rubric',
label: '评分标准整理',
model: input.requestPayload.model,
usage,
});

Copilot AI Apr 14, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

collectStreamingRubricResponseText 在 catch 中记录 usage:如果 recordUsage 抛错会覆盖流式异常并让上层的 fallback 判断失效。建议把 recordUsage 变成 best-effort(try/catch 忽略失败或仅 warn),然后继续抛出原始 error。

Suggested change
await this.llmUsage.recordUsage({
source: 'grading-rubric',
label: '评分标准整理',
model: input.requestPayload.model,
usage,
});
try {
await this.llmUsage.recordUsage({
source: 'grading-rubric',
label: '评分标准整理',
model: input.requestPayload.model,
usage,
});
} catch {
// Best-effort only: do not let usage logging mask the original streaming error.
}

Copilot uses AI. Check for mistakes.
Comment on lines +1096 to +1099
mode: debugContext.mode,
finishReason: debugContext.finishReason,
fallbackReason: debugContext.fallbackReason,
fallbackAt: debugContext.fallbackAt,

Copilot AI Apr 14, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

写 retry 调试报告时这里使用 debugContext.mode/finishReason/fallbackReason 等字段,但这些字段只有在 wrapPaperAttemptError 赋值时才会存在;像 JSON 解析/校验失败这类错误并不会附带这些字段,导致报告里 mode/finishReason 等信息丢失(而 lastMode/lastFinishReason 已经在 try 中拿到了)。建议这里改用 lastMode/lastFinishReason/lastFallbackReason/lastFallbackAt(并且与 lastRawOutput/lastReasoningText 保持一致)。

Suggested change
mode: debugContext.mode,
finishReason: debugContext.finishReason,
fallbackReason: debugContext.fallbackReason,
fallbackAt: debugContext.fallbackAt,
mode: lastMode,
finishReason: lastFinishReason,
fallbackReason: lastFallbackReason,
fallbackAt: lastFallbackAt,

Copilot uses AI. Check for mistakes.
Comment on lines +1112 to +1117
await this.llmUsage.recordUsage({
source: 'answer-generation',
label: '参考答案生成',
model: input.requestPayload.model,
usage,
});

Copilot AI Apr 14, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里在 catch 中 await this.llmUsage.recordUsage(...) 后再 throw error;如果 recordUsage 抛错会覆盖原始模型/网络异常,导致最终看到的错误与真实失败原因不一致。建议将 recordUsage 包裹为 best-effort(try/catch),并始终 rethrow 原始 error。

Suggested change
await this.llmUsage.recordUsage({
source: 'answer-generation',
label: '参考答案生成',
model: input.requestPayload.model,
usage,
});
try {
await this.llmUsage.recordUsage({
source: 'answer-generation',
label: '参考答案生成',
model: input.requestPayload.model,
usage,
});
} catch {
// Best-effort only: never let usage recording mask the original stream/model error.
}

Copilot uses AI. Check for mistakes.
Comment on lines +541 to +546
await this.llmUsage.recordUsage({
source: 'smart-name-match',
label: '智能核名',
model: input.requestPayload.model,
usage,
});

Copilot AI Apr 14, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里在流式读取失败时直接 await this.llmUsage.recordUsage(...),如果 recordUsage 因数据库/IO 异常抛错,会把原始流式错误覆盖掉,进而让上层的 isStreamingFallbackCandidate 判断失效(无法回退 non-stream),并且丢失真实失败原因。建议把 recordUsage 做成 best-effort:用 try/catch 吞掉记录失败,或捕获后仍然 rethrow 原始 error。

Suggested change
await this.llmUsage.recordUsage({
source: 'smart-name-match',
label: '智能核名',
model: input.requestPayload.model,
usage,
});
try {
await this.llmUsage.recordUsage({
source: 'smart-name-match',
label: '智能核名',
model: input.requestPayload.model,
usage,
});
} catch {
// Best-effort only: do not let usage recording failures mask the original streaming error.
}

Copilot uses AI. Check for mistakes.
@BobH233
BobH233 merged commit aea9992 into main Apr 14, 2026
1 check passed
@BobH233
BobH233 deleted the dev branch April 14, 2026 14:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants