Pipeline results only; fixed-output offline clients do not establish skill efficacy. Discovery is separate. Unknown usage and cost remain unknown.
Generated by the fixed-output fixture with real temporary Git snapshots. These fixture revisions are test data, not campaign defaults.
JSON excerpt · Reproduce the full report and retained evidence
| Case | Repeat | Baseline | Candidate | Outcome |
|---|---|---|---|---|
| fix | 1 | FAIL | PASS | fix |
| regression | 1 | PASS | FAIL | regression |
| error | 1 | PASS | INCONCLUSIVE | inconclusive |
| missing | 1 | missing | missing | missing_pair |
Counts:
{
"fix": 1,
"regression": 1,
"unchanged_correct": 0,
"unchanged_incorrect": 0,
"missing_pair": 1,
"inconclusive": 1
}The regression candidate returned claim_scope=production; the independently specified audit contract rejected it. The error candidate returned malformed JSON and remained INCONCLUSIVE. The observed-token limit stopped new trials before the missing pair; neither missing side is counted as a fix or regression.
JSON includes the recorded/current answer and execution/grader evidence hashes, plus all five frozen controllers. Missing or modified retained answers, execution records, agent traces or grader artifacts invalidate the trial. Reporting performs these checks without rerunning the client or grader.
Each completed record also has a separate SHA-256 receipt. Changing only the regression candidate's recorded verdict/checks produced INCONCLUSIVE (retained_trial_record_hash_mismatch) and no accepted assertions; unchanged grader evidence could not become a positive pair. Restoring the original record regenerated identical reports without model or grader calls.
Client runtime and completion are retained in the verified run status. This injected offline client has unknown executable/version metadata. Editing only the status file produced unverified status with unknown budget fields; restoration reproduced identical reports without client or grader calls.
{
"client_runtime": {
"implementation": "tests.fixtures.skill_eval_compare_support.FixedOutputClient",
"executable": null,
"executable_sha256": null,
"version": null
},
"recorded_trials": 6,
"expected_trials": 8,
"observed_tokens": 120,
"stop_reason": "token_budget",
"execution_state": "stopped"
}Candidate output and grading evidence for the regression:
{
"verdict": "FAIL",
"evaluation_error": null,
"output": "{\"report_verdict\": \"PASS\", \"claim_scope\": \"production\", \"findings\": [], \"latency_reduction_pct\": 20, \"evidence\": [\"baseline.latencies_ms\", \"candidate.latencies_ms\", \"contract\"], \"reason\": \"Group means are 100/200/300 and 50/160/900: medians 200 and 160 give exactly 20 percent.\"}",
"checks": {
"passed": false,
"assertions": {
"report_verdict": true,
"claim_scope": false,
"findings": true,
"reduction": true,
"evidence": true,
"explanation": true
}
},
"output_sha256": "d42de8a3924cc9b0867cfbf510433b510acad688f5a872829b235b8224e0dc2e",
"retained_output_sha256": "d42de8a3924cc9b0867cfbf510433b510acad688f5a872829b235b8224e0dc2e",
"evidence_sha256": {
"execution.json": "9191efb9ae1b040d0eeef520aeadbe91db378b461132d22e705e9c8e479cf0c6",
"agent/stdout.log": "cae0a0049534a8720917a328866e40de12e3a95495ab4f5c07cbeb525eb7ccdf",
"agent/stderr.log": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"verification/result.json": "8a64fcee2c58d57083f195a727ba7a1b35d2f400bdf0699ec8d0eaac72b14f73",
"verification/stdout.log": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"verification/stderr.log": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
},
"retained_evidence_sha256": {
"execution.json": "9191efb9ae1b040d0eeef520aeadbe91db378b461132d22e705e9c8e479cf0c6",
"agent/stdout.log": "cae0a0049534a8720917a328866e40de12e3a95495ab4f5c07cbeb525eb7ccdf",
"agent/stderr.log": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"verification/result.json": "8a64fcee2c58d57083f195a727ba7a1b35d2f400bdf0699ec8d0eaac72b14f73",
"verification/stdout.log": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"verification/stderr.log": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
}
}