From 4c8045969aa521653737587841c82f885e80229f Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Mon, 31 Aug 2026 13:02:50 -0700 Subject: [PATCH 1/2] docs(agent): state where a runtime rule's fixtures go MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `create-runtime-rule` described a rule directory as `captures/*.yml` plus `check.ts` and stopped. An agent authoring a runtime rule was told nothing about `.tests/` at all, and `verify-rule` reported runtime tests as "not run" without naming what it was not running. The shape existed only inside a generator. Now stated in both, with the part that is load-bearing rather than cosmetic: a case is a DIRECTORY whose path is the `root` the harness hands the check, with every `match.file` relative to it, so a case may hold as many files as it needs. A runtime rule exists because its evidence spans more than one file — a layout allowing one file per case could not express the rules the tier is for. That distinction was not academic. The generator's payload builder capped fixture path depth at 3, which admits `pass/case-1/input.ts` and refuses `pass/case-1/src/a.ts`, so the first cross-file fixture would have been rejected as a path-safety error: a security refusal for a legitimate test. The cap was written as a runaway guard and had silently become a shape constraint. They raised it to 8 and recorded which of the two it is. Buckets are `pass`/`fail`, matching Vale's. `valid`/`invalid` are key names inside an ast-grep test YAML and are right for `sg` for exactly one reason — that file is handed to `sg test` verbatim. Nothing hands a runtime fixture to ast-grep, so reusing the word there would make it mean two things, the same collision `captures/` avoids by not being called `matchers/`. Agreed with the cloud team as F1 and F2; their generator writes this shape and the restore endpoint shares the derivation. --- .changeset/runtime-fixture-shape.md | 17 +++++++++++++++++ packages/cli/src/agent/create-runtime-rule.txt | 16 ++++++++++++++++ packages/cli/src/agent/verify-rule.txt | 2 +- 3 files changed, 34 insertions(+), 1 deletion(-) create mode 100644 .changeset/runtime-fixture-shape.md diff --git a/.changeset/runtime-fixture-shape.md b/.changeset/runtime-fixture-shape.md new file mode 100644 index 00000000..9dc53de7 --- /dev/null +++ b/.changeset/runtime-fixture-shape.md @@ -0,0 +1,17 @@ +--- +"@taskless/cli": patch +--- + +Document where a runtime rule's test fixtures go. + +`create-runtime-rule` described a rule directory as `captures/*.yml` plus +`check.ts` and said nothing about `.tests/`, so an agent authoring a runtime +rule had no statement of the fixture layout at all. `verify-rule` reported +runtime tests as "not run" without saying what was not being run. + +Both now state it: cases live in `.tests/pass/` and `.tests/fail/`, and a +case is a **directory** whose path is the `root` the harness hands the +check, so it may hold as many files as the case needs. That last part is +load-bearing rather than incidental — a runtime rule exists because its +evidence spans more than one file, so a layout allowing one file per case +could not express the rules the tier is for. diff --git a/packages/cli/src/agent/create-runtime-rule.txt b/packages/cli/src/agent/create-runtime-rule.txt index 821a87ed..c0a3ce13 100644 --- a/packages/cli/src/agent/create-runtime-rule.txt +++ b/packages/cli/src/agent/create-runtime-rule.txt @@ -31,6 +31,22 @@ kinds of file: |------------------|---------------------------------------------------------------------------| | `captures/*.yml` | ast-grep capture rules that narrow which files the check looks at | | `check.ts` | a module whose default export receives those matches and returns findings | +| `.tests/` | fixtures, in `pass/` and `fail/` buckets | + +A fixture case is a DIRECTORY, not a file: `.tests/pass/case-1/`, and its +path is the `root` the harness hands the check, with every `match.file` +relative to it. So a case may hold as many files as the case needs. + +That is not a detail. A runtime rule exists because its evidence spans +more than one file, so a layout allowing one file per case could not +express the rules the tier is for. `pass/case-1/src/a.ts` alongside +`pass/case-1/src/b.ts` is the ordinary shape, not the exotic one. + +`pass`/`fail` rather than `valid`/`invalid`, matching Vale's buckets. +`valid`/`invalid` are key names inside an ast-grep test YAML and are +right for `sg` for one reason only: that file is handed to `sg test` +verbatim. Nothing hands a runtime fixture to ast-grep, so borrowing the +word here would make it mean two things. `verify` fails a runtime rule with no capture rule in `captures/`, because `check.ts` would then never be invoked. diff --git a/packages/cli/src/agent/verify-rule.txt b/packages/cli/src/agent/verify-rule.txt index 4cc03b4b..b9d511ce 100644 --- a/packages/cli/src/agent/verify-rule.txt +++ b/packages/cli/src/agent/verify-rule.txt @@ -35,7 +35,7 @@ What each engine is checked for: |-----------|--------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------| | `sg` | ast-grep schema, plus `id`/`language`/`severity`/`message`/`rule`, and `regex` accompanied by `kind` | the `valid`/`invalid` cases in `.tests/` | | `vale` | style parses, `extends` and `message` present, `level` in vocabulary, and the rule's `.vale.ini` enables `.` under a matcher | the `.tests/pass/` and `.tests/fail/` buckets | -| `runtime` | `check.ts` present, at least one capture rule in `captures/` | reported as not run: runtime tests need the server harness | +| `runtime` | `check.ts` present, at least one capture rule in `captures/` | reported as not run: the `.tests/pass/` and `.tests/fail/` cases need the server harness | ## What a path means From 8024f3e8d811efcd03900691aa22fca70c5fbf25 Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Mon, 31 Aug 2026 13:11:33 -0700 Subject: [PATCH 2/2] docs(agent): bump create-runtime-rule to topic v2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The recipe gained a whole section it never had — the `.tests/` layout, the directory-per-case shape, and the bucket naming — while its header still said v1. Every other recipe that changed materially bumped: create-sg-rule is v4, create-vale-rule v3, improve-rule v4. The number is maintained by hand, so nothing else would have caught it, and a stale `topic vN` is exactly the failure CLAUDE.md records: an agent followed a recipe from an old `dist/` and authored rules against outdated guidance, missing the revision that documented the field it then got wrong. Deliberately NOT bumping `verify-rule`, whose change in this branch names the fixture buckets inside an existing row. The test there is whether acting on the stale copy is wrong, not whether the text differs: an agent reading the old verify-rule still learns runtime tests are not run locally, which remains true. An agent reading the old create-runtime-rule learns nothing about fixtures at all and invents a layout. Found by the review on #225. --- packages/cli/src/agent/create-runtime-rule.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/cli/src/agent/create-runtime-rule.txt b/packages/cli/src/agent/create-runtime-rule.txt index c0a3ce13..c0b66ccd 100644 --- a/packages/cli/src/agent/create-runtime-rule.txt +++ b/packages/cli/src/agent/create-runtime-rule.txt @@ -1,4 +1,4 @@ -# Topic: create-runtime-rule (CLI v%(CLI_VERSION)s / topic v1) +# Topic: create-runtime-rule (CLI v%(CLI_VERSION)s / topic v2) ## You are here This is `create-runtime-rule`. It helps you write a runtime rule: a