validation.md

CodeGraph validation methodology

Read on demand when adding a language/framework or changing retrieval quality; this document is not auto-injected into agent context. It preserves the upstream validation methodology and historical cases, and code paths in it are relative to the repository root. Case tool names and results may belong to older versions; check the current entry points when running them, and do not report historical data as this validation.

Validation methodology (REQUIRED for every new language/framework)

For each language × framework, validate on small, medium, and large real repos with ≥3 different flow prompts each:

1. Pick the canonical flow for the framework ("how does X reach Y": state→render, request→handler→view, query→SQL, action→reducer→store…).
2. Deterministic probes (scripts/agent-eval/probe-{node,explore}.mjs against the built dist/): codegraph_explore with the flow's symbol names connects from→to end-to-end with no break (its Flow section shows the path); no node explosion (select count(*) from nodes stable before/after re-index); synthesized-edge precision spot-check (select … where provenance='heuristic').
3. Agent A/B (scripts/agent-eval/run-all.sh <repo> "<Q>"): with vs without codegraph, ≥2 runs/arm (run-to-run variance is large — never conclude from n=1). Record duration, total tool calls, Read, Grep. Optional forced-Read-0 sufficiency proof via the block-read hook (scripts/agent-eval/hook-settings.json).
   • Every run also reports three feedback metrics — residual context occupancy, explore sufficiency (what the agent did NEXT after each explore), and allocation efficiency (share of returned bytes the answer cited) — under each run, plus a side-by-side arm table (compare-arms.mjs). Entry point: docs/benchmarks/agent-eval-feedback-metrics.md. Reading them: Read a file we returned is an allocation miss, Read a file we did NOT return/Grep is recall; allocation efficiency is relative (attribution is by citation) so it is only valid between builds on the same question; occupancy shares are Claude Code / 200k and don't transfer to another host — the arm ratio does.
   • The codegraph CLI is blocked in every arm (no-cli-shim.sh: sanitized PATH + a PreToolUse hook, shared by both harnesses). Without it 14 of 15 without-arm runs in one 7-repo pass reached codegraph through Bash. Check the contamination row before believing any number: CLI calls that RETURNED output > 0 invalidates the run (in a new-vs-baseline A/B it silently drops calls from all three metrics, since a CLI explore is not a tool call).
   • Model policy — every A/B arm runs Claude with --model sonnet --effort high. Always. Never Opus/Fable. All scripts/agent-eval/*.sh default to this (MODEL/EFFORT env override exists — don't raise it without an explicit reason from the maintainer). Two reasons, and the second matters more than cost: (a) Sonnet doesn't burn tokens; (b) Sonnet is the deliberate floor model — codegraph's real users attach it to whatever agent they already run (Cursor Composer, Gemini, etc.), so we validate on a "dumber" model on purpose: a stronger model's tool-use covers up the salience/sufficiency problems a weaker one exposes. An affordance that lands on Sonnet generalizes up to every host; one that only works on Opus/Fable doesn't generalize down to the agents most users actually have. Both arms always use the same model.
   • MCP attach is a startup-latency issue, not a hard block. On a multi-step task the agent dives into Read/grep before codegraph finishes its ~2-3s startup (worse when the eval is itself run nested inside a Claude session, under CPU contention), so it runs with no codegraph. Fix: pre-warm a persistent daemon for the target (CODEGRAPH_DAEMON_IDLE_TIMEOUT_MS high; spawn serve --mcp --path <target> </dev/null &; wait for .codegraph/daemon.sock) and skip the startup re-exec (CODEGRAPH_WASM_RELAUNCHED=1) so claude connects before the agent's first turn. Don't trust claude's init snapshot — it can read status:"pending" / 0 tools even when it then connects; judge by actual codegraph usage in parse-run.mjs's by type. To isolate a change — new-build vs baseline-build, both codegraph-on (vs run-all.sh's with-vs-without) — use scripts/agent-eval/ab-new-vs-baseline.sh <indexed-repo> "<task>" [baseline-ref] (it bakes in the pre-warm).
4. Pass bar: a normal flow question reaches ~0 Read/Grep within the repo's explore-call budget, runs faster than without-codegraph, and shows no regression on a control repo. Record the numbers in docs/design/dynamic-dispatch-coverage-playbook.md (the coverage matrix).

Full playbook + per-mechanism design: docs/design/dynamic-dispatch-coverage-playbook.md and docs/design/callback-edge-synthesis.md.

Worked example — Excalidraw (TS/React, medium, 643 files)

The template to replicate per language/framework. Question: "how does updating an element re-render the canvas on screen?" (the full flow crosses three React boundaries: observer callback, setState→render, and JSX child).

Stage	duration	Read	Grep	codegraph
Without codegraph	115–139s	9–10	10–11	0
Broken (explore-budget regression)	131–139s	5–10	3–5	6–14
Fixed (budget + msgs + synthesis)	64–112s	0–2	2–4	3–10
+ trace-first steering	51–74s	0–2	0–4	3–4

n=4 unhooked runs/stage, same prompt. After steering flow questions to codegraph_trace first: best run 0 Read / 0 Grep / 3 codegraph / 51s; 2 of 4 fully clean (0 Read, 0 Grep). Steering eliminated the over-drill variance — call count tightened from 3–10 to 3–4, trace adoption went 3/4 → 4/4, and the search+callers path-reconstruction floundering dropped to 0. Run-to-run variance is still real; report the range, never a single run. Residual reads/greps are all the nonce data-flow (canvasNonce — a local prop with no graph edges); that's the def-use/data-flow frontier, left deliberately uncovered (tracking every local would explode the graph). Validated: trace(mutateElement, renderStaticScene) connects in 6 hops across all three boundaries (mutateElement → triggerUpdate → [callback] triggerRender → [react-render] render → [jsx] StaticCanvas → renderStaticScene), each hop showing inline source + the wiring site; node count stable at 9,289; 1 callback + 46 react-render + 280 jsx-render synthesized edges (no explosion, precision-checked).

Also see: docs/design/dynamic-dispatch-coverage-playbook.md, docs/design/callback-edge-synthesis.md, docs/benchmarks/call-sequence-analysis.md, docs/benchmarks/agent-eval-feedback-metrics.md.