Skip to content

feat(skills): add adversarial-review and gate issue-ship on it - #248

Merged
paveg merged 3 commits into
mainfrom
feat/adversarial-review-skill
Sep 17, 2026
Merged

paveg merged 3 commits into
mainfrom
feat/adversarial-review-skill

Conversation

@paveg

@paveg paveg commented Sep 17, 2026 •

Copy link
Copy Markdown
Owner

What

A skill that tries to refute what a change claims about itself, and an issue-ship gate that refuses to open a PR until it passes.

File Change
skills/adversarial-review/SKILL.md new — extract claims from the diff / PR body / commit messages / subagent report into | claim | source | kind |, derive the implicit claims an attacker would test (Step 1b), route by kind, adjudicate, emit ADVERSARIAL: PASS or FAIL (n refuted, m unverifiable)
skills/issue-ship/SKILL.md Step 4 runs it after the implementer returns; FAIL sends refuted claims and evidence to a fresh implementer dispatch; Step 5 never starts on FAIL

Routing by claim kind — existing pieces do the checking, the skill only orchestrates:

kind check
behavior one skeptic agent per claim, parallel, cap 6 by blast radius
doc verify-doc-claims on the touched md files, table taken as-is
test break one condition in a throwaway worktree, suite must go red
count enumerate before/after, account for every item by name

Subagent verdicts stay hypotheses: a REFUTED is accepted only after the main session opens the cited evidence.

Step 1b (second commit; the third only unwraps the description onto one line) adds the attacker stance: stated claims are chosen by the author, so the author's blind spots never reach the table. Five angles — trust boundary, error paths, ordering, touched callers, removal — each become an implicit claim with source: implicit and ride the same routing under the same skeptic cap. A hypothetical that never ran is a row without evidence and does not count against the change. Security attack surface stays with pentest-parallel-prs.

Design choices

  • Mutation for test claims runs in the main session in a git worktree add throwaway, because skeptic is read-only. Alternative was marking such claims UNVERIFIABLE.
  • The cap of 6 skeptics is derived from workflow.md's ~10K-token spawn cost, not measured.
  • Description trimmed 490 → 298 chars (context-diet/scripts/measure.py) before commit; routing detail lives in the body.

Not in this PR

Behavioral verification of the skill (a run through empirical-prompt-tuning, or an end-to-end issue-ship pass). just test covers format and lint only.

🤖 Generated with Claude Code

adversarial-review extracts the claims a diff, PR body, or subagent
report makes and routes each to its decisive check: skeptic agents in
parallel for behavior, verify-doc-claims for docs, break-one-condition
in a throwaway worktree for tests, before/after enumeration for
counts. It returns a verdict table ending in ADVERSARIAL: PASS|FAIL.
issue-ship runs it after implementation and never opens the PR on
FAIL.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@changeset-bot

changeset-bot Bot commented Sep 17, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 101b12d

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@paveg

paveg commented Sep 17, 2026 •

Copy link
Copy Markdown
Owner Author

Test results

  • Target commit: 101b12deccd5a9c3079562d1934a276d9f0dba02
Check Command Result
Lint + render + format + hook/skill-script tests just test ✓ All checks passed!
Always-on cost of the new description context-diet/scripts/measure.py 332 chars (trimmed from 490 before the first commit; +34 for the attacker-stance wording)
Commit signature git log -1 --show-signature Good "git" signature (ED25519)

Not covered

  • The skill has not been executed: no empirical-prompt-tuning pass, no end-to-end issue-ship run through the new gate.
  • The throwaway-worktree mutation step for test claims is specified, not exercised.

paveg and others added 2 commits September 17, 2026 17:22
Stated claims are chosen by the author, so the author's blind spots
never reach the table. Step 1b switches stance to "how do I break
this", derives implicit claims from five angles (trust boundary, error
paths, ordering, touched callers, removal), and routes them through the
same checks under the same skeptic cap. Hypotheticals that never ran
stay rows without evidence and do not count against the change.
Security attack surface is handed to pentest-parallel-prs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@paveg
paveg enabled auto-merge (squash) September 17, 2026 08:24
@paveg
paveg merged commit e089f04 into main Sep 17, 2026
12 checks passed
@paveg
paveg deleted the feat/adversarial-review-skill branch September 17, 2026 08:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant