Skip to content

Rename rule checks: verify → validate, test → verify (0.13) #489

Closed

Description

@theCodeDrift

Today verify and test ask two different kinds of question under names that don't say which is which. Found while working #485 (PR #488).

Current behavior

  • verify decides from the rule files alone: zod over the vendored ast-grep schema, plus the house constraints in rules/constraints.ts (language: spelling, regex-needs-kind, TypeScript/Tsx globs, id matches directory). It never runs the engine on the rule.
  • test runs verify, then executes the rule against its fixtures.

That split leaves real gaps, all measured on main with ast-grep 0.45.3:

  1. verify passes rules ast-grep refuses. kind: readonly (an anonymous keyword token) gets {"ok":true} from verify. ast-grep rejects it with exit 8, Kind `readonly` is invalid. The same holds for every keyword and for Markdown inline kinds such as link. Zod can't catch this, since valid kinds depend on the grammar and the binary can't list them. Only an engine parse can.
  2. One broken rule fails a different rule's tests, under the wrong name. With kw/kw.yml holding kind: readonly, taskless test .taskless/rules/sg/inline failed with Cannot parse rule .taskless/rules/sg/kw/kw.yml, reported under inline. check is taken down by the same rule.
  3. A top-level strictness: ast passes verify and is silently ignored by ast-grep. The pattern stays narrowed. (rule.strictness is already caught by zod.)

Proposal

Name each command for the question it answers:

new name question formerly who runs it
validate Is the rule well-formed? Decided from the files alone. verify anything, any time; self-certifying
verify Does the rule check what you currently want? test an agent, with a human in the loop

The names add real clarity. "Verify" and "test" both read as "run the rule", so an agent can't tell from the name which one gives the quick, deterministic answer. validate says "syntactically correct" and verify says "behaves as intended", which is how they are actually used: validate belongs in a generator's inner loop and in CI, and verify is the step where someone confirms the fixtures say what they meant.

validate becomes the home for every self-certifying check:

  • zod and the existing structural constraints
  • one ast-grep parse of just this rule, which catches invalid kinds and any other config error, reported under that rule (fixes gaps 1 and 2 for config errors)
  • top-level strictness: as an error (gap 3)
  • non-blocking notices for the heuristic quirks in create-sg-rule's Quirks section: $$$ next to a comma without strictness: ast, follows/precedes without stopBy: end, a callee regex under TypeScript/Tsx

Internal ast-grep rules over our own YAML are possible, but zod plus the one engine parse should cover nearly all of it.

verify runs validate first, then the fixtures, as test does today. Both already answer in the same JSON shape.

Compatibility

  • Both commands shipped in v0.12.0, so this is a 0.13 change consumers must react to.
  • Reusing the name verify means an old caller gets the new meaning. Since the new verify runs validate first and keeps the same output shape, an old caller gets a stricter result, not a different kind of output.
  • Installed skills pin their CLI version (install.cliVersion), so a project keeps 0.12 until it upgrades, and the upgrade rewrites the skill along with it.
  • The Cloud generator team, which adopted taskless test, moves over by upgrading through the documented process.
  • @taskless/cli/reference.json tells evaluators to "Run taskless verify" and labels constraints enforcedBy: "verify" | "test". Those values change meaning, so the corpus's own version goes 2 → 3.
  • Touches: commands/verify.ts and commands/names.ts, the verify-rule agent topic and every recipe that names either command (create-*-rule, improve-rule, check, ci), rules/constraints.ts (enforcedBy), rules/reference.ts, and the installed skill.

Delivery

An OpenSpec change, stacked and merging forward, since each unit is safe on its own:

  1. The rename itself, plus recipe, skill and reference.json v3 updates.
  2. The new validate checks.

Refs #485

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions