Repository navigation
feat(rules): make verification confirm the claim, not the existence of a token - #246
Merged
Merged
Conversation
…f a token Tuned with empirical-prompt-tuning against fixtures where fresh executors reproduced a real miss: accepting a "subsumed" test on the report's word. harness-engineering: the evaluator may read the subject to design a check but judges by what ran; a test suite's behavior is failing on a broken subject; counts/deletions are load-bearing and must be enumerated before/after. workflow: "never claim a fact without proof" widened from completion claims to any stated fact; detectors need a positive completion signal; existing worktrees must be clean before dispatching into them. Removed to stay near the instruction budget: the /goal line, ADR format details (kept the one-line rule), a clause duplicated in workflow, and the 114K-token anecdote. Bullets 159 -> 158. Eval set (fixtures, checklists, per-iteration log) stored beside the skill. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
The fixtures are read by nobody at skill runtime; they exist to re-run the Red/Green check when the rules or the model change. docs/ is outside .chezmoiroot, so they stay in git without being deployed to ~/.claude. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two always-on rules learn to verify the claim rather than the presence of a token, and the evaluator learns what "behavior" means when the artifact is a test suite.
rules/harness-engineering.md/goalline; ADR format details (kept a one-line rule); clause duplicated in workflowrules/workflow.mdBullets 159 → 158. Still above the 150 budget noted in
harness-meta.md; deletion candidates indevelopment-principles.mdare a separate decision.Why these sentences and not others
Ran
empirical-prompt-tuningwith fresh executors on small fixtures reproducing 8 real misses from one long session. Only one reproduced without the new text: when evaluating an implementer's fold of 12 tests into a table, the executor verified the one disputed test by mutation but accepted the other 11 on the report's categorization. That is the sentence that landed ("a summary that adds up is not evidence").Three candidate sentences for
development-principles.md(enumerate branch-exclusive inputs before folding, convention over majority, validate CLI input before destructive use) were not added: fresh executors already did all three without them. Those misses came from a long main session evaluating its own work, not from missing text.Evidence
Judges were separate sonnet agents that saw only the deliverable and the checklist. Residual unclear points (what counts as "one condition"; whether merging alone triggers enumeration) are recorded in the log and accepted as definitional residue.
Eval set:
docs/evals/2026-09-15-verification-claims/(outside.chezmoiroot, so not deployed to~/.claude) (fixtures, frozen checklists, per-iteration log, v1/v2 candidate texts). Fixture names are neutral.Not in this PR
chezmoi applyofsettings.json— the live file has drifted from source (proxy env vs. deny list); apply the two rule files selectively.🤖 Generated with Claude Code