Skip to content

feat(rules): make verification confirm the claim, not the existence of a token - #246

Merged
paveg merged 3 commits into
mainfrom
rules/verification-confirms-the-claim
Sep 15, 2026
Merged

paveg merged 3 commits into
mainfrom
rules/verification-confirms-the-claim

Conversation

@paveg

@paveg paveg commented Sep 15, 2026 •

Copy link
Copy Markdown
Owner

What

Two always-on rules learn to verify the claim rather than the presence of a token, and the evaluator learns what "behavior" means when the artifact is a test suite.

File Added Removed
rules/harness-engineering.md evaluator may read the subject to design a check, verdict from what ran; a test suite's behavior is failing on a broken subject; counts / deletions / "no longer applies" are load-bearing and must be enumerated before/after /goal line; ADR format details (kept a one-line rule); clause duplicated in workflow
rules/workflow.md "never claim a fact without proof" widened from completion claims to any stated fact (state what the output shows, not what you inferred); detectors need a positive completion signal; existing worktrees must be clean before dispatch 114K-token anecdote

Bullets 159 → 158. Still above the 150 budget noted in harness-meta.md; deletion candidates in development-principles.md are a separate decision.

Why these sentences and not others

Ran empirical-prompt-tuning with fresh executors on small fixtures reproducing 8 real misses from one long session. Only one reproduced without the new text: when evaluating an implementer's fold of 12 tests into a table, the executor verified the one disputed test by mutation but accepted the other 11 on the report's categorization. That is the sentence that landed ("a summary that adds up is not evidence").

Three candidate sentences for development-principles.md (enumerate branch-exclusive inputs before folding, convention over majority, validate CLI input before destructive use) were not added: fresh executors already did all three without them. Those misses came from a long main session evaluating its own work, not from missing text.

Evidence

Scenario Red (current rules) iter 1 iter 2 hold-out
A evaluate a fold report ✗ 70% ✓ 100% ✓ 100%
C mutation-detection harness ✓ 90% ✓ 90% ✓ 100%
B cluster names in a PR body (sealed) ✓ 100%

Judges were separate sonnet agents that saw only the deliverable and the checklist. Residual unclear points (what counts as "one condition"; whether merging alone triggers enumeration) are recorded in the log and accepted as definitional residue.

Eval set: docs/evals/2026-09-15-verification-claims/ (outside .chezmoiroot, so not deployed to ~/.claude) (fixtures, frozen checklists, per-iteration log, v1/v2 candidate texts). Fixture names are neutral.

Not in this PR

  • chezmoi apply of settings.json — the live file has drifted from source (proxy env vs. deny list); apply the two rule files selectively.
  • A hook for multi-line rationale comments in code (the existing rot-comment hook only matches English issue/temporal patterns).

🤖 Generated with Claude Code

…f a token

Tuned with empirical-prompt-tuning against fixtures where fresh executors
reproduced a real miss: accepting a "subsumed" test on the report's word.

harness-engineering: the evaluator may read the subject to design a check but
judges by what ran; a test suite's behavior is failing on a broken subject;
counts/deletions are load-bearing and must be enumerated before/after.
workflow: "never claim a fact without proof" widened from completion claims to
any stated fact; detectors need a positive completion signal; existing
worktrees must be clean before dispatching into them.

Removed to stay near the instruction budget: the /goal line, ADR format
details (kept the one-line rule), a clause duplicated in workflow, and the
114K-token anecdote. Bullets 159 -> 158.

Eval set (fixtures, checklists, per-iteration log) stored beside the skill.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@changeset-bot

changeset-bot Bot commented Sep 15, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: fe81dea

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

paveg and others added 2 commits September 15, 2026 18:57
The fixtures are read by nobody at skill runtime; they exist to re-run the
Red/Green check when the rules or the model change. docs/ is outside
.chezmoiroot, so they stay in git without being deployed to ~/.claude.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@paveg
paveg merged commit 207b4d3 into main Sep 15, 2026
12 checks passed
@paveg
paveg deleted the rules/verification-confirms-the-claim branch September 15, 2026 09:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant