diff --git a/.changeset/runtime-fixture-shape.md b/.changeset/runtime-fixture-shape.md new file mode 100644 index 00000000..9dc53de7 --- /dev/null +++ b/.changeset/runtime-fixture-shape.md @@ -0,0 +1,17 @@ +--- +"@taskless/cli": patch +--- + +Document where a runtime rule's test fixtures go. + +`create-runtime-rule` described a rule directory as `captures/*.yml` plus +`check.ts` and said nothing about `.tests/`, so an agent authoring a runtime +rule had no statement of the fixture layout at all. `verify-rule` reported +runtime tests as "not run" without saying what was not being run. + +Both now state it: cases live in `.tests/pass/` and `.tests/fail/`, and a +case is a **directory** whose path is the `root` the harness hands the +check, so it may hold as many files as the case needs. That last part is +load-bearing rather than incidental — a runtime rule exists because its +evidence spans more than one file, so a layout allowing one file per case +could not express the rules the tier is for. diff --git a/packages/cli/src/agent/create-runtime-rule.txt b/packages/cli/src/agent/create-runtime-rule.txt index 821a87ed..c0b66ccd 100644 --- a/packages/cli/src/agent/create-runtime-rule.txt +++ b/packages/cli/src/agent/create-runtime-rule.txt @@ -1,4 +1,4 @@ -# Topic: create-runtime-rule (CLI v%(CLI_VERSION)s / topic v1) +# Topic: create-runtime-rule (CLI v%(CLI_VERSION)s / topic v2) ## You are here This is `create-runtime-rule`. It helps you write a runtime rule: a @@ -31,6 +31,22 @@ kinds of file: |------------------|---------------------------------------------------------------------------| | `captures/*.yml` | ast-grep capture rules that narrow which files the check looks at | | `check.ts` | a module whose default export receives those matches and returns findings | +| `.tests/` | fixtures, in `pass/` and `fail/` buckets | + +A fixture case is a DIRECTORY, not a file: `.tests/pass/case-1/`, and its +path is the `root` the harness hands the check, with every `match.file` +relative to it. So a case may hold as many files as the case needs. + +That is not a detail. A runtime rule exists because its evidence spans +more than one file, so a layout allowing one file per case could not +express the rules the tier is for. `pass/case-1/src/a.ts` alongside +`pass/case-1/src/b.ts` is the ordinary shape, not the exotic one. + +`pass`/`fail` rather than `valid`/`invalid`, matching Vale's buckets. +`valid`/`invalid` are key names inside an ast-grep test YAML and are +right for `sg` for one reason only: that file is handed to `sg test` +verbatim. Nothing hands a runtime fixture to ast-grep, so borrowing the +word here would make it mean two things. `verify` fails a runtime rule with no capture rule in `captures/`, because `check.ts` would then never be invoked. diff --git a/packages/cli/src/agent/verify-rule.txt b/packages/cli/src/agent/verify-rule.txt index 4cc03b4b..b9d511ce 100644 --- a/packages/cli/src/agent/verify-rule.txt +++ b/packages/cli/src/agent/verify-rule.txt @@ -35,7 +35,7 @@ What each engine is checked for: |-----------|--------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------| | `sg` | ast-grep schema, plus `id`/`language`/`severity`/`message`/`rule`, and `regex` accompanied by `kind` | the `valid`/`invalid` cases in `.tests/` | | `vale` | style parses, `extends` and `message` present, `level` in vocabulary, and the rule's `.vale.ini` enables `.` under a matcher | the `.tests/pass/` and `.tests/fail/` buckets | -| `runtime` | `check.ts` present, at least one capture rule in `captures/` | reported as not run: runtime tests need the server harness | +| `runtime` | `check.ts` present, at least one capture rule in `captures/` | reported as not run: the `.tests/pass/` and `.tests/fail/` cases need the server harness | ## What a path means