scaffold asks what the code guarantees, which produces tautological claims that canonise bugs

[raccioly]

Summary

scaffold asks the author to describe what the code does. That produces tautological claims that freeze current behaviour — bugs included — and quietly inverts what TestGuard proves.

The current prompt

Running scaffold on a middleware file emits draft claims shaped like this:

{
  "id": "GROUPAUTH-REQUIREGROUPACCESS",
  "severity": "medium",
  "statement": "TODO: state what `requireGroupAccess` in <file> guarantees (6 proposed faults; keep or drop each)",
  "faults": [
    { "id": "S1", "detail": "[line 34] Guard never triggers: `if (!req.user)` becomes `if (false)`." },
    { "id": "S2", "detail": "[line 39] Guard never triggers: `if (!req.resolvedGroupId)` becomes `if (false)`." }
  ]
}

and the CLI closes with:

Next: replace each TODO statement, drop proposals that are not claims, then move the claims into testguard.claims.json.

The status table says the same thing: "replace every TODO: statement with what the code guarantees".

The problem

Deriving faults from code is correct — a fault is just "what can I break here", and the code is the only place that answer lives.

Deriving the statement from code is not. The statement is the only part of a claim that carries intent, and the current prompt ("what this guarantees", present tense, pointing at a function the author is currently reading) reliably produces a paraphrase of the implementation:

"requireGroupAccess returns 403 when req.user is missing."

That statement is unfalsifiable by construction. It is true of the code because it was read off the code. Three consequences:

1. Bugs get canonised. If the function's group check is wrong today, a claim derived from it asserts the wrong behaviour, and a test written to kill the faults locks the bug in. TestGuard then reports killed — successfully defending a defect.
2. The proof inverts. TestGuard's value proposition is that tests prove the code meets its spec. A code-derived statement means the spec is back-derived from the code, and the apparatus degrades into an expensive change-detector.
3. Claims come out function-shaped. Real invariants cross files (see the separate issue on file-shaped claim IDs). A per-function statement can't express "through any route".

A good claim reads like a promise to a user or an auditor:

"A user without conversations:view_all can never read a conversation outside their assigned groups, through any route."

That spans several files — which is precisely the tell that it came from intent rather than from a function body.

Proposal

1. Change the prompt's tense and direction. Everywhere the TODO text appears (scaffold output, status table, skill file):

TODO: state what this is SUPPOSED to guarantee — from a spec, doc, ticket or
past bug, not from reading the branches below. If the only source you have is
this function's code, that is a sign this is not yet a claim.

2. Make provenance a required field at birth. scaffold should emit

"source": { "kind": "TODO: spec | doc | bug | incident | comment", "ref": "TODO: file#anchor, ticket ID, or commit SHA" }

and refuse (or loudly warn) on a claim moved into testguard.claims.json with kind: comment and no ref. The data model already tracks provenance — testguard claims prints a spec / comment column today — so this is surfacing an existing field at the moment it is cheapest to fill.

3. Offer intent-first entry points, not just file-first. scaffold <file> is inherently code-first. Two complements would change adoption shape:

• testguard scaffold --from-bug <commit|ticket> — a fixed bug is a proven missing claim: the bug shipped precisely because nothing asserted the invariant. The fix commit's diff gives you the target modules; the bug report gives you the statement in intent language for free.
• testguard scaffold --from-doc <path> — pull candidate statements from a spec/ADR and let the author bind them to modules.

4. Document the distinction prominently. One paragraph in the skill file: faults come from the code, claims come from intent. This is the single most important idea for a new user and it is currently implicit.

Acceptance

• A user following the prompts verbatim is steered away from writing a paraphrase of the function under their cursor.
• Claims lacking real provenance are visible as such before they are probed, not after.