Skip to content

Evals: round 6 (use-the-details rule) - logged, rule reverted - #8

Merged
davehallmon merged 2 commits into
mainfrom
skill/fit-use-details
Oct 4, 2026
Merged

davehallmon merged 2 commits into
mainfrom
skill/fit-use-details

Conversation

@davehallmon

Copy link
Copy Markdown
Owner

What this changes

Logs eval round 6, a test of a SKILL.md rule aimed at Fit, and then reverts the rule because it didn't work. Net effect on the Skill: none. SKILL.md matches main.

Commits:

  1. skill: adds "Use the user's details": let a stated number or limit change a check, option, or order, and keep moves doable with the user's resources.
  2. evals: logs round 6 in evals/runs/round-6_draft/ and updates evals/README.md and CHANGELOG.md.
  3. Revert removes the rule.

Round 6 results (same 11 problems, same baselines as round 4)

  • Fit did not move: 4.09 in round 4 and round 6, up on 2 problems, same on 7, down on 2. Plain Claude: 4.73.
  • Overall: 6–1–4, gap +0.8 (round 4: +1.6). This is within judge noise; the same baseline files moved 1.45 points on average.
  • The rule worked when it worked: turnover worked out that 28% is about 11 exits a year, too few to trust exit-interview themes, and the judge called that the deciding insight.
  • It also produced a confident wrong number: onboarding estimated an A/B test at about 4 weeks. The judge put it at 6–7, and noted the run assumed all 200 signups reach step 3.
  • Judges' Fit complaints are mostly about coverage, such as the career run not using "8 years of experience." That isn't what a number rule fixes.
  • Rule checks passed 11 of 11. OODA was chosen for the first time, so all 11 eligible lenses have now been used.

If you want to keep the rule anyway

Drop the revert commit before merging.

Checklist

  • I read CONTRIBUTING.md.
  • I used active voice and short words.
  • I kept SKILL.md under 500 lines (unchanged after the revert).
  • I updated README.md if needed (not needed: no Skill change).
  • I updated CHANGELOG.md.
  • I tested the Skill on real problems (eval round 6: 11 problems).
  • I ran python3 evals/check_rules.py problem-lens/examples/*.md (3 of 3 pass).

🤖 Generated with Claude Code

https://claude.ai/code/session_01X9xscdGXL5xQmmxnwEf9T7


Generated by Claude Code

davehallmon and others added 2 commits October 4, 2026 12:39
Adds 'Use the user's details' to SKILL.md: use at least one stated
number or limit to rule out a check, size or drop an option, or change
the order, and keep each move doable with the user's stated resources
(name any extra need in the Risk cell). Targets the Fit gap seen in
every eval round.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X9xscdGXL5xQmmxnwEf9T7
Fit stayed at 4.09; overall gap +0.8 vs +1.6 in round 4, within judge
noise. One run turned the rule into a wrong sample-size calculation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X9xscdGXL5xQmmxnwEf9T7
@davehallmon
davehallmon force-pushed the skill/fit-use-details branch from 5f968fc to 49463cf Compare October 4, 2026 18:30
@davehallmon
davehallmon merged commit caf874d into main Oct 4, 2026
1 check passed
@davehallmon
davehallmon deleted the skill/fit-use-details branch October 4, 2026 18:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant