Repository navigation
ch11: hard evaluation set (240 questions) for the four retrieval strategies - #3
Merged
Merged
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oup cap Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…in RAG eval Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
First real synthesis under-labeled general chunks (category table, 7-day policy). Add per-chunk pair checks over structural same-topic keys and give both labeling prompts the model's category and the FAQ default-model rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Strict labeling left H4 at 56/60 after 128 seeds. Add a third H4 slot at the end of the pool and raise attempts per seed from 3 to 5. Labeling and filter rules are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
v3 spot-check: labeler still missed general chunks when a question drifted off the target aspect (e.g. K2 same-style exchange vs 售后手册 申请换货). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…h rule v4 spot-check: pair checks accepted off-topic general chunks (warranty questions labeled with the return-period table or the 'see product page' warranty note). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fifth synthesis run. Spot-check by Claude (user delegated): 19/20 clean, one over-labeled key noted. No retrieval has been run on this set. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ot-check note Review: with --gen-repeat N the 未检索/裁判失败 columns summed N runs and case lines repeated without a run marker. The spot-check note wrongly claimed an over-labeled key cannot favor any strategy; it can, and the note now records the deviation from spec §8 (Claude, not the user, did the spot-check). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Retrieval on the frozen 240-question hard set: hybrid+rerank R@1 0.992, MRR 0.996 (dense 0.975 / 0.987, BM25 0.904 / 0.940). 4 of 6 pre-registered expectations held. Generation failed midway (upstream 402 Insufficient Balance), so hard-set faithfulness is not reported. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
evals/hard_set/andscripts/synth_hard_set.py: a 240-question hard retrieval set (4 buckets × 60: sibling models, colloquial with no shared words, buried key term, model-specific rules). Code picks targets from the knowledge-base structure; DeepSeek writes questions; mechanical filters; labeling = full-list pass → general-chunk recall → two consecutive pair confirmations.19e983dbefore any retrieval ran; 6 expectations were written down before the run and are checked automatically in the report.evals/run_rag_eval.py:--set hard,--gen-repeat N(faithfulness averaged per run; faith_cases from run 1 only). Base set behavior and report format unchanged.Results (retrieval, 240 hard questions)
4 of 6 expectations held. Two did not (buried-term: dense not below BM25; model-specific: RRF tied rerank). The gap to dense stays small (1.7 pts R@1).
Known limitations (also in README and dev-notes)
402 Insufficient Balancemidway (1023/1920 failed). Retrieval is unaffected (all query rewrites finished before the balance ran out).Test plan
uv run pytest -q→ 1201 passeduv run python scripts/synth_hard_set.py --check→ passed🤖 Generated with Claude Code