[YoniRaviv]

Phase 3 — Verifier + Eval Harness

Adds the bounded LLM verifier and the eval harness that measures it, completing the deterministic-suspects → LLM-verdict pipeline.

What's here

• llm.py — LLMClient: single chokepoint for LLM traffic (litellm wrapper, token accounting, LLMError).
• verification/tools.py — root-sandboxed read_file / grep / list_symbols + OpenAI tool schemas + make_dispatch.
• verification/{prompts,verifier}.py — bounded tool-use loop (read tools + submit_verdict). Every failure path → unverified, never stale (LLM error, malformed args, budget exhausted, prose-only) — false positives are the product's life bar.
• eval/harness.py + docpulse eval — file-fixture cases → precision/recall + per-case table, with a --min-precision regression gate.
• evals/cases/ — 12 hand-labeled seed cases (6 stale / 6 accurate, 4 per language; labels independently verified).
• .github/workflows/ci.yml — deterministic lint + test job on every PR; gated live-eval job enforcing the precision gate.

Baseline

docpulse eval on the 12 seed cases with anthropic/claude-haiku-4-5: precision 1.00 · recall 1.00 (12/12, 65.9k tokens). Exit gate (≥0.9) cleared. Seed cases are clear-cut by design — real robustness accrues as real-world false positives become new accurate cases (the README's golden rule).

Notes

• 96 tests, repo-wide ruff check clean.
• Built subagent-driven (task-by-task spec + quality review + a final integrated review). The review loop caught real bugs in plan-prescribed code: a grep glob sandbox escape + unhandled TypeError in tool dispatch; a non-last submit_verdict leaving the tool-call conversation protocol-invalid (real-provider 400 on retry); and evals/ fixtures breaking repo-wide lint.
• Boundary rules preserved: indexing/ and diffing/ import no LLM code.

🤖 Generated with Claude Code