fix(cognition): a DoD that can't RUN is infra, not a model miss (systemic hardening) - #2472
Merged
Conversation
… — the right concern
Joel (2026-08-25): 'harden ALL benchmarks, and really the system — these are
issues they'd have in projects. Make sure it's not coded into the wrong
concern.' The astropy-13236 false-failure was the symptom; the disease was
run_dod returning a flat (bool,String) that collapsed 'she failed the tests'
and 'the grader itself could not run' into one ok=false — so a harness fault
(missing workspace, refused/not-found command, wrong cwd) scored as a MODEL
miss AND taught her 'I failed'. That hits EVERY dod-graded activity, real
projects included, not just benchmarks.
Fix at the constraint: run_dod → DodVerdict::{Pass, Fail, InfraError}. A narrow
classifier (dod_output_is_infra: substrate-refused / no-such-file /
command-not-found / could-not-read / permission-denied / exit 127/4/5) marks a
could-not-RUN as infra; a genuine red (pytest '5 failed', AssertionError,
FAILED) stays a real miss. The verify loop short-circuits infra (no re-drive —
nothing to fix), and the grade path routes it to the infra_faults counter →
InfraUnavailable (no phantom score) and SKIPS teaching (a harness fault teaches
nothing). Same protection the gold gate gives, now on the DoD's own execution.
Complements the adapter /Users/joel/Development/continuum fix (#2471): that makes THIS instance's path
resolve; this makes the whole CLASS never mis-score. Battery: dod_infra +
run_dod + eval::tests + ratchets green (32 passed, EXIT=0).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The right-concern fix for the astropy false-failure class: run_dod → DodVerdict{Pass,Fail,InfraError}. A grader that could-not-run (missing workspace, refused/not-found command, infra exit) is marked infra — not re-driven, not scored as a miss, not taught — while a genuine red stays a real miss. Protects every dod-graded activity, projects included. Narrow classifier + test.
🤖 Generated with Claude Code
https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo