feat(eval): chunking experiments, corrected nDCG, and a 1000/100 default

[mrsibe]

What does this PR do?

Runs the chunking experiments #78 asked for, adopts the winner, and fixes a metric bug the experiments exposed.

Why?

#78 is a timeboxed spike: compare chunking strategies against the frozen v1.4 baseline and adopt one only if the numbers support it. It is also the prerequisite for #77 — retrieval must be measured on a frozen chunking baseline, or the retrieval experiment is measuring a moving target.

Related issue

Fixes #78
Related to #154, #77

The result

Every variant runs the real harness over the same corpus and 30 questions, with dense retrieval held fixed (docs/eval/chunking-v1.5.md):

Variant	Recall@1	Recall@5	MRR	nDCG@10	chunks
baseline v1.4 (500/50)	0.7667	0.9333	0.8492	0.8899	36
small (250/25)	0.7333	0.9833	0.8372	0.8787	70 (+94%)
large (1000/100)	0.8333	1.0000	0.9278	0.9437	19 (−47%)
large + page spanning (1000/100)	0.8333	1.0000	0.9278	0.9437	19
heading-aware (500/50)	0.7667	0.9000	0.8317	0.8756	52 (+44%)
heading-aware large (1000/100)	0.7667	0.9000	0.8317	0.8756	52 (+44%)

Adopted: large (1000/100). It clears the frozen rule (Recall@5 improves, nDCG@10 does not regress), improves Recall@1 and MRR too, and halves the index. DEFAULT_CHUNK_OPTIONS is now chunkSize=1000, chunkOverlap=100; docs/eval/baseline-v1.5.json is the frozen successor and baseline-v1.4.* is kept untouched as history.

Limitation, stated in the report: the corpus is 13 short documents (19–36 chunks), so Recall@5 saturates near 1.0 and is the least discriminating metric here — Recall@1 and MRR carry the result. This should be re-checked on a larger, multi-format corpus before it is treated as settled.

The metric bug it exposed

ndcgAtK() counted a gain for every rank that covered a ground-truth block, then normalized by an ideal with one gain per block. Several overlapping chunks recovering the same block therefore scored several gains against an ideal of one, and nDCG rose above 1 — the small-chunk variant measured 1.2164, which is not a valid ranking metric. The baseline never showed it because 500-char chunks rarely re-cover the same block.

A rank now gains only when it covers a block no earlier rank covered, and matches beyond the declared ground truth are ignored, so nDCG ≤ 1 by construction. The v1.4 baseline numbers are unchanged (0.8899) — the bug only manifested under the smaller chunks this spike introduced.

What changed

• scripts/eval-chunking.mjs + npm run eval:chunking: runs each variant and writes docs/eval/chunking-v1.5.{md,json}.
• ChunkOptions.respectHeadings (default false): forces a new chunk at each heading (the heading-aware variants).
• The harness takes chunkOptions, reports chunkCount (deterministic → committed JSON) and indexingMs (timing → excluded, as before).
• --eval-baseline= selects the report label, so npm run eval writes baseline-v1.5.*.
• CI's determinism check now diffs baseline-v1.5.json.
• ndcgAtK fix + tests, and the generated chunking files are added to .prettierignore (same reason as the baseline).

How was this tested?

• npm run typecheck — passes.
• npm test — 382 pass (new: nDCG counts a block once, nDCG never exceeds 1).
• npm run check:design — no violations.
• npm run build — passes.
• npm run eval twice is byte-identical (the CI determinism gate).
• electron . --smoke-test — PASS, 26 checks, with the new default.
• The chunking table is harness output, reproducible with npm run eval:chunking.

Checklist

• I have reviewed my own changes.
• npm run typecheck passes.
• npm run build passes.
• I have tested the affected user workflow.
• I have not included unrelated changes.
• I have updated documentation when necessary.

Desktop / build changes

• Not applicable (a default chunk size changed, but no schema or packaging change).