[Dong1017]

Summary

Refs #255. This is an evidence-only delivery, not acceptance of an Indexer runtime optimization and not closure of the two-goal tracker.

• Add a reusable CPU retrieval evaluation runner, 8 development + 4 independently frozen held-out queries, six contract checks, and per-case/resource-delivery reporting.
• Preserve frozen protocols, raw measurements, integrity records, a lossless compressed evidence archive, and a single readable efficiency/accuracy table.
• Preserve the rejected experimental candidate as a patch and Git bundle for reproduction. Production runtime source is unchanged from the experiment baseline.

Results and decision

Full efficiency/accuracy report

• Initial 20 paired groups: primary median reduction 58.61% (95% CI 52.54–61.35), but CLI search p95 regressed 52.72%, exceeding its frozen 20% guard. v1 remains FAIL.
• Exactly one prospectively frozen 120-pair confirmation plus 40 baseline/baseline controls: CLI p50 +9.9734%, p95 -3.2103%. Point guards pass, but intervals remain wide and do not exclude meaningful regression. No pooling, discarded samples or repeated attempts to obtain PASS.
• Held-out Hit@5 3/4 and MRR@5 0.75 in both arms; no accuracy improvement. The natural-language storage/cache query misses.
• Supplemental post-run mandatory-resource audit passes 58/58 emitted contexts per arm. This does not erase retrieval misses.
• One frozen development label has mixed AFD/KV intent; the original eight-case metric and a separately labelled seven-case source-valid sensitivity subtotal are retained. Held-out means unseen queries/labels, not fully family-disjoint evaluation.

Decision: do not promote the runtime candidate. This PR supplies reusable measurement and honest evidence.

Reproduction and provenance

• Frozen experiment baseline: ab03f3ffb8f2e5bc02c304828c5c41890eef673e; 751-file corpus. Later main 401ee8a256c21e6c489e7d179e62c70ac08723d3 changes corpus/reader metadata behavior; these numbers are not measurements on that newer corpus.
• Experimental candidate: 6da28494891b21cdc89f632d84d763bc997b7d15, retained only as reproducible patch/bundle.
• Reviewed local evidence-only commit: 1a06702788ec834d2fd5eca7ee31877163e9cbb6.
• Connector-published commit: 2a31244e98b3d94b7ceffcde184c6330a5ac5f11; identical tree 94f873d304e708f224ca2215c08aa9677fc70f5c, independently read back by Git fetch.
• Protocol freeze receipts: v1, pre-candidate filter correction, single confirmation study.

Verification and limits

• The 89-pass log predates the final manifest and does not prove that the final tree passed; the 17 affected tests passed at corrective commit 10e6c0cb, while the latest full CI remains to be confirmed.
• Configured lint and evidence inventory validation pass; 33 integrity hashes and six compressed originals verified.
• Independent review recomputed confirmation statistics, checked frozen fixtures, candidate patch/bundle, archives, runtime equality and limits.
• Initial full-suite result is retained: 2504 passed, 9 failed, 10 skipped. Missing installed-package environment and evidence registration caused failures; source-environment affected checks were repaired/rerun, but installed-package smoke remains blocked pending installation permission. This is not a claim of a final full-suite pass.
• Current-main combined-tree tests/CI remain to be verified. No model, judge, GPU/NPU or paid API experiment occurred. No downstream agent-effect claim.

The remaining efficiency acceptance and retrieval accuracy gap stay tracked in #255.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-29T14:32:44.330231Z	10e6c0c	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2a31244e98

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Update the expected verified-hash count for the new bundle

Registering the new campaign adds 33 verified hashes, so validate_inventory(repo) now returns checked_hashes == 199, but the next assertion still expects 166. I reproduced this with PYTHONPATH=src pytest -q tests/test_evidence_inventory.py tests/test_indexer_evaluation_summary.py: test_checked_in_inventory fails despite inventory validation reporting no errors. Update the hash-count expectation alongside the bundle count so the full pytest step in .github/workflows/ci-ready.yml does not fail on this change.

Useful? React with 👍 / 👎.

[Dong1017]

Fixed in 10e6c0c: expected checked_hashes is now 199. The exact committed tree passes the affected tests (17 passed); inventory validation reports no errors and 199 checked hashes. The report now explicitly identifies the retained 89-pass log as preceding final manifest inclusion, rather than proof of the final tree. A separate post-manifest verification receipt avoids a self-referential log/hash cycle. Frozen benchmark data and the v1 FAIL verdict are unchanged. New-head hosted CI is pending; this reply does not claim a full-suite pass.

[Dong1017]

Scoped LGTM for the evidence-only delivery at head 10e6c0cb00323b295468fc1480956dee5b3acf69, against API base / observed main 401ee8a256c21e6c489e7d179e62c70ac08723d3.

The earlier inventory-count finding is cleared: the current assertion expects 199 hashes. I reviewed the runner, fixtures, protocol, supplemental audit, report and changed-path inventory; no production src/ changes are included. No additional blocking finding in this assessed scope.

From the retained observations, I independently recomputed the 20-group primary reduction (58.610526%), CLI-search p95 regression (52.719334%), paired output equality, and the separate 120-pair confirmation point estimates (+9.973407% p50 / -3.210342% p95). They agree with the report. The frozen v1 verdict remains FAIL; v2 does not replace it. The small partial-label held-out result and the post-run delivery audit do not establish an accuracy gain or downstream agent benefit.

All four current-head Ready jobs pass. The inspected Python3.11 log records Ruff success, 2534 passed, 11 skipped and 284 subtests on merge a28cbe8a455214798beb3212bb8df97d6bcb1fe2, combining the stated base and head. The latest per-head policy status succeeds. These changed paths do not select the existing SkillEval workflow; no model execution is implied.

The original measurements remain bound to baseline ab03f3ffb8f2e5bc02c304828c5c41890eef673e and its frozen candidate/corpus, not the newer main corpus. Verification here was source/existing-CI inspection and arithmetic on retained timings; I did not rerun benchmarks, replay compressed case archives, or execute a model/device experiment.

Next for the existing author and #255 acceptance owner: decide whether to merge this evidence/tooling slice; keep runtime promotion and the two tracker objectives separate. This review does not accept the rejected runtime optimization or close #255. COMMENTED is used because the authenticated account is also the PR author; no formal self-approval or merge action.