Repository navigation
Rename rule checks: verify → validate, test → verify (0.13) #489
Description
Today verify and test ask two different kinds of question under names that don't say which is which. Found while working #485 (PR #488).
Current behavior
verifydecides from the rule files alone: zod over the vendored ast-grep schema, plus the house constraints inrules/constraints.ts(language:spelling, regex-needs-kind, TypeScript/Tsx globs, id matches directory). It never runs the engine on the rule.testrunsverify, then executes the rule against its fixtures.
That split leaves real gaps, all measured on main with ast-grep 0.45.3:
verifypasses rules ast-grep refuses.kind: readonly(an anonymous keyword token) gets{"ok":true}fromverify. ast-grep rejects it with exit 8,Kind `readonly` is invalid. The same holds for every keyword and for Markdown inline kinds such aslink. Zod can't catch this, since valid kinds depend on the grammar and the binary can't list them. Only an engine parse can.- One broken rule fails a different rule's tests, under the wrong name. With
kw/kw.ymlholdingkind: readonly,taskless test .taskless/rules/sg/inlinefailed withCannot parse rule .taskless/rules/sg/kw/kw.yml, reported underinline.checkis taken down by the same rule. - A top-level
strictness: astpassesverifyand is silently ignored by ast-grep. The pattern stays narrowed. (rule.strictnessis already caught by zod.)
Proposal
Name each command for the question it answers:
| new name | question | formerly | who runs it |
|---|---|---|---|
validate |
Is the rule well-formed? Decided from the files alone. | verify |
anything, any time; self-certifying |
verify |
Does the rule check what you currently want? | test |
an agent, with a human in the loop |
The names add real clarity. "Verify" and "test" both read as "run the rule", so an agent can't tell from the name which one gives the quick, deterministic answer. validate says "syntactically correct" and verify says "behaves as intended", which is how they are actually used: validate belongs in a generator's inner loop and in CI, and verify is the step where someone confirms the fixtures say what they meant.
validate becomes the home for every self-certifying check:
- zod and the existing structural constraints
- one ast-grep parse of just this rule, which catches invalid kinds and any other config error, reported under that rule (fixes gaps 1 and 2 for config errors)
- top-level
strictness:as an error (gap 3) - non-blocking notices for the heuristic quirks in
create-sg-rule's Quirks section:$$$next to a comma withoutstrictness: ast,follows/precedeswithoutstopBy: end, a calleeregexunder TypeScript/Tsx
Internal ast-grep rules over our own YAML are possible, but zod plus the one engine parse should cover nearly all of it.
verify runs validate first, then the fixtures, as test does today. Both already answer in the same JSON shape.
Compatibility
- Both commands shipped in
v0.12.0, so this is a 0.13 change consumers must react to. - Reusing the name
verifymeans an old caller gets the new meaning. Since the newverifyrunsvalidatefirst and keeps the same output shape, an old caller gets a stricter result, not a different kind of output. - Installed skills pin their CLI version (
install.cliVersion), so a project keeps 0.12 until it upgrades, and the upgrade rewrites the skill along with it. - The Cloud generator team, which adopted
taskless test, moves over by upgrading through the documented process. @taskless/cli/reference.jsontells evaluators to "Runtaskless verify" and labels constraintsenforcedBy: "verify" | "test". Those values change meaning, so the corpus's ownversiongoes 2 → 3.- Touches:
commands/verify.tsandcommands/names.ts, theverify-ruleagent topic and every recipe that names either command (create-*-rule,improve-rule,check,ci),rules/constraints.ts(enforcedBy),rules/reference.ts, and the installed skill.
Delivery
An OpenSpec change, stacked and merging forward, since each unit is safe on its own:
- The rename itself, plus recipe, skill and
reference.jsonv3 updates. - The new
validatechecks.
Refs #485
Activity
refactor(cli)!: rename `verify` to `validate`
refactor(cli)!: rename `test` to `verify`
feat(cli): validate an sg rule with ast-grep, and verify it alone
feat(cli): validate an sg rule with ast-grep, and verify it alone
refactor(cli)!: rename `verify` to `validate`
refactor(cli)!: rename `test` to `verify`
feat(cli): validate an sg rule with ast-grep, and verify it alone