[codelformat]

Summary

• Knowledge base: 16 look-alike models (40 in total). New 《配送与签收政策》 and 《预售与缺货政策》. 12 new FAQ entries. 5 new lexicon synonyms. scripts/check_kb_v2.py checks the additions against the spec appendices, and checks that no existing line changed.
• v2 evaluation set (evals/rag_eval_v2.jsonl, A 40 / B 50 / C 50 / E 100 / D 60):
  • Written from a seeded target list. Reviewed in 2 rounds.
  • Frozen in da67f2b before any v2 retrieval run. run_rag_eval.py --set v2 checks the SHA-256 and compares the result with 6 expectations written before the freeze.
• Evaluation: v2, the old 300-question set and the ch11 hard set, run on the old and the new knowledge base (retrieval only). The ch11 hard set ran from a ch11 worktree. ch11 is not merged.
• Runner fix: the report header now shows Query-rewrite fallbacks, and a run with more than 3 fallbacks exits 1. Before this fix, a DeepSeek balance outage made all 540 rewrites fall back, and the run still exited 0.

Results (evals/reports/ch12_comparison.md)

• v2: hybrid_rerank R@5 0.933, against bm25 0.917, hybrid 0.915 and dense 0.882. 3 of 6 pre-registered expectations met. The target band 0.88–0.92 and "others < 0.90" were not met. The set was not changed after the run.
• On v2, A, B and C have R@5 1.000 for every strategy. All of the difficulty comes from the 100 multi-fact E questions.
• hybrid_rerank leads the runner-up by about 0.02 R@1 on all 3 sets.
• Erratum: E080, E086 and E093 miss one equivalent key. Corrected, the v2 R@5 gap is about +0.010.
• On the old sets, hybrid_rerank stays almost the same on the new knowledge base (base R@5 0.992 → 0.988; hard R@5 1.000).

Open decisions (not done in this PR)

• The new lexicon aliases rewrite unrelated production queries. For example, the rerank query turns 退回快递点 into 拒收点. See docs/backlog/2026-10-11-lexicon-alias-side-effects.md.
• GATE_* were calibrated on the old knowledge base. README and facts.ts numbers were not updated. Faithfulness was not evaluated.

Test plan

• uv run pytest -q: 1347 passed
• uv run python scripts/check_kb_v2.py, evals/v2/check_d_evidence.py, run_rag_eval.py --check, run_rag_eval.py --set v2 --check
• build_kb.py --rebuild and --check: 302 = 302
• Whole-branch code review (Opus): "With fixes". Important findings addressed; the details are in dev-notes/ch12.md.

🤖 Generated with Claude Code