Plan: retriever rewrite in three stages

[adamjohnwright]

Implementation plan for spec 001. No code yet.

The staging, and why

A structural change and a behaviour change are never in the same commit:

stage	what	exit criterion
1	BaseRetriever with vendored RRF; expansion and SelfQuery both stay	retrieval_baseline compare shows zero difference
2	D1 — plain semantic search replaces SelfQuery; retrieval drops 21 → 1 LLM calls	differences confined to the vector side
3	D2/D3 — budget and bundle become arguments	two callers get different budgets; four mypy baseline entries deleted

Stage 1 producing identical output is the point. If it does not, the refactor changed something it should not have, and that is far easier to find before Stage 2 moves the results deliberately.

Each stage ends by answering a real question through the full chain, via both invoke and ainvoke — not by testing the retriever in isolation. That is the failure mode that recurred most this week and is Article I of the constitution.

What is deliberately not generated

No research.md, data-model.md or contracts/.

D1–D4 are settled with measured evidence in the spec, so there is nothing to research. The feature introduces no data model and no external contract. Generating that scaffolding would be exactly the ceremony the constitution says to avoid — Spec Kit is for decisions, not paperwork.

Complexity tracking

Four simplifications considered and rejected, with reasons — notably keeping metadata_info.py. It looks like 339 dead lines after D1, but evaluator.py and bin/retrieval_baseline still construct SelfQuery, so deleting it would break the tool that measures Stage 2.

Verification

Every factual claim was checked against the code rather than written from memory:

userguide is already a plain BaseRetriever          ✓
four collections hold 121,291 documents             ✓ (had written "~121k")
RRF = weight/(rank+60), rank from 1, page_content   ✓ (read from the pinned source)
four mypy baseline entries, not three               ✓
metadata_info still used by 2 files                 ✓

🤖 Generated with Claude Code