Add paired answer-quality A/B evaluation

[AronAxe]

Objective

Measure whether Token Terminator preserves answer quality versus off mode when the exact same prompts and evidence are sent to the same Hermes model/provider/reasoning configuration. Answer quality is the primary metric; token savings are secondary and cannot compensate for a quality loss.

Stage 1 experiment

• 12 synthetic, easily degradable task classes with 310,082 characters of evidence
• 3 repetitions per task: 36 matched pairs / 72 answer runs
• randomized adjacent arm order stable under the 3-to-6 repetition extension
• fresh session and isolated workspace per arm; full process-tree timeout cleanup
• strict deterministic grading of factual, numerical, diagnostic, recency, and completeness quality
• 80% minimum control pass rate to prove assay sensitivity
• Stage 1 reports answer-quality results as a checkpoint, not a final victory
• predeclared decision-grade extension remains 72 matched pairs; existing results resume without rerunning Stage 1
• contaminated arms automatically retry; latest clean replacements win in analysis
• provider/model drift, timeout, malformed output, and missing usage receipts are contamination, not hidden failures

Methodological defects fixed

• switched to hermes chat --query-file so reasoning/rule-isolation flags are honoured and Windows prompt limits are avoided
• canonical state.db accounting keyed by emitted session id
• task-clustered bootstrap intervals rather than treating repetitions as independent tasks
• assay-sensitivity floor prevents both arms failing from passing non-inferiority
• corrected PCI-DSS and Envoy regex fixture ambiguity
• stable pair-level randomization across extension
• graceful Hermes run budget ends before the outer process timeout
• documented session-total token-accounting limitations

Verification

• 156 passed, 2 skipped
• Ruff lint and format checks passed
• suite validation: 12 tasks, 36 pairs, 72 answer runs, 72-pair final decision threshold
• Twine and release-containment verification passed on clean Windows staging
• AGY fixture and implementation audits: APPROVE
• external Claude Code reviews identified the methodological defects above; all are now covered by implementation and regression tests

Run cost

No experiment provider calls were made by this PR. Stage 1 uses 72 OpenAI Codex OAuth answer calls with expected /usr/bin/bash marginal API cost while consuming account quota and wall-clock time.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 507769928f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Make usage paths absolute before launching Hermes

With the documented relative --results .quality-ab/results.jsonl, this passes a relative usage path to a subprocess whose cwd is the disposable evidence workspace. Hermes therefore writes—or attempts to write—the receipt beneath that workspace rather than the repository-level .quality-ab/usage directory, and the workspace is deleted before usage_path is read. Every successful trial is consequently marked missing_usage, preventing the experiment from producing eligible pairs; resolve the receipt path before launching the subprocess.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Retry contaminated arms instead of marking them complete

When any arm times out, fails, lacks usage, or reports route drift, its contaminated row is still added to completed. Subsequent run invocations permanently skip that arm, while analyze requires all 72 pairs to be uncontaminated, so one transient failure leaves the documented resumable experiment stuck in collecting unless the user manually edits the JSONL file. Only valid rows should satisfy completion, or contaminated rows must be replaceable on retry.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Reject rows from a different experiment configuration

Eligibility checks only the row-controlled contaminated flag and never verifies expected pair IDs, experiment ID, prompt hash, arm mode, or consistent model/provider/reasoning. If a results file is reused after changing the suite or route, its previously valid rows remain uncontaminated and can be combined into a complete non-inferiority decision attributed to the current suite, despite violating the matched fixed-configuration design. Validate every pair against the current plan and reject mixed run configurations before computing statistics.

Useful? React with 👍 / 👎.