Skip to content

Skill+evals: number guard and round 7 - #10

Merged
davehallmon merged 3 commits into
mainfrom
skill/number-guard
Oct 4, 2026
Merged

davehallmon merged 3 commits into
mainfrom
skill/number-guard

Conversation

@davehallmon

Copy link
Copy Markdown
Owner

What this changes

  • SKILL.md: adds a guard to "Use the user's details": derive a number only from facts the user gave, or name the assumption next to the figure. Never present a derived figure as known.
  • Round 7 logged in evals/runs/round-7_draft/, including calc-audit/, a blind audit of every derived number in rounds 6 and 7.
  • evals/README.md and CHANGELOG.md updated.

Round 7 results (same 11 problems, same baselines)

  • Calculation audit: unsupported derived numbers fell from 9 to 3. No arithmetic errors in either round. Two of the three left are effort estimates like "it costs an hour."
  • Honesty: 4.73, the highest of any round.
  • Overall: 3–1–7, gap 0.0. The Skill's mean (29.5) is unchanged from round 6; the baseline files scored higher this time. Judge noise this round was 1.64 points per pair, the highest yet. Five of the seven losses were by one point.
  • Fit: 4.18 (round 6: 4.09). Plain Claude: 4.73.
  • Rule checks: 10 of 11 pass. The receptionist Summary ran to four sentences.

Checklist

  • I read CONTRIBUTING.md.
  • I used active voice and short words.
  • I kept SKILL.md under 500 lines (127).
  • I updated README.md if needed (left for the 1.2.0 release PR).
  • I updated CHANGELOG.md.
  • I tested the Skill on real problems (eval round 7: 11 problems).
  • I ran python3 evals/check_rules.py problem-lens/examples/*.md (3 of 3 pass).

🤖 Generated with Claude Code

https://claude.ai/code/session_01X9xscdGXL5xQmmxnwEf9T7


Generated by Claude Code

davehallmon and others added 3 commits October 4, 2026 13:44
Adds a guard to 'Use the user's details': a derived figure that depends
on an unstated fact must name that assumption next to it, or not be
calculated. Responds to the round 6 onboarding run's unsupported A/B
test estimate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X9xscdGXL5xQmmxnwEf9T7
Unsupported derived numbers fell from 9 (round 6) to 3; no arithmetic
errors either round. Overall level with plain Claude (gap 0.0), within
judge noise. Updates evals/README and CHANGELOG.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X9xscdGXL5xQmmxnwEf9T7
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X9xscdGXL5xQmmxnwEf9T7
@davehallmon davehallmon mentioned this pull request Oct 4, 2026
@davehallmon
davehallmon added this pull request to stack #12 October 4, 2026 19:56
@davehallmon
davehallmon merged commit bfe31cd into main Oct 4, 2026
1 check passed
@davehallmon
davehallmon deleted the skill/number-guard branch October 4, 2026 19:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant