feat(agent): fixtures come from real code and pass the repo's linters

.changeset/fixture-quality.md

@@ -0,0 +1,5 @@
+---
+"@taskless/cli": patch
+---
+
+The rule-authoring recipes now tell the agent where a rule's test cases should come from. `create-sg-rule`, `create-vale-rule`, `create-runtime-rule`, and `improve-rule --anonymous` have it copy cases from real code in the repository before inventing any, and name the cases it invented in its report. They also have it read the linters `detect --json` finds and write cases those linters would accept. Apart from the construct the rule checks, cases stay in the repository's idiom, with no placeholder types such as `Record<string, never>` and no nesting a reviewer would send back. `create-remote-rule` and `improve-rule` apply the same standard to the cases and references they send to the generator. `verify` is unchanged and does not judge fixture style.

.taskless/rules/sg/no-pii-in-telemetry/.tests/no-pii-in-telemetry-20261014-test.yml

@@ -0,0 +1,59 @@
+id: no-pii-in-telemetry
+valid:
+  - |
+    posthog.identify({
+      distinctId,
+      properties: {
+        cli: anonymousId,
+        cliVersion: CLI_VERSION,
+        scaffoldVersion,
+        ghOwner,
+        ...dimensions,
+      },
+    });
+  - |
+    telemetry.capture("cli_report_published", {
+      ...tierCounts(report),
+      withheldFixtures: report.redaction.withheldFixtures,
+    });
+  - |
+    posthog.groupIdentify({
+      groupType: "organization",
+      groupKey: String(orgSubject),
+    });
+  - |
+    auth = {
+      user: whoami.user,
+      email: whoami.email,
+      orgs: whoami.orgs.map((o) => o.name),
+    };
+invalid:
+  - |
+    posthog.identify({
+      distinctId,
+      properties: {
+        cli: anonymousId,
+        cliVersion: CLI_VERSION,
+        scaffoldVersion,
+        email,
+        ...dimensions,
+      },
+    });
+  - |
+    telemetry.capture("cli_authenticated", { email: whoami.email });
+  - |
+    ph.capture({
+      distinctId,
+      event,
+      properties: {
+        ...properties,
+        cli: anonymousId,
+        displayName: whoami.user,
+      },
+    });
+  - |
+    posthog.groupIdentify({
+      groupType: "organization",
+      groupKey: String(orgSubject),
+      properties: { email: whoami.email },
+    });

.taskless/rules/sg/no-pii-in-telemetry/no-pii-in-telemetry.yml

@@ -9,11 +9,16 @@ note: |
 
   See .conventions/posthog.md — Privacy.
 rule:
-  kind: pair
-  has:
-    field: key
-    kind: property_identifier
-    regex: '^(email|userEmail|displayName|fullName|firstName|lastName|username)$'
+  any:
+    - kind: pair
+      has:
+        field: key
+        kind: property_identifier
+        regex: '^(email|userEmail|displayName|fullName|firstName|lastName|username)$'
+    # Shorthand (`{ email }`) is its own node kind, not a `pair`. The telemetry
+    # call sites write most properties this way (`scaffoldVersion, ghOwner,`).
+    - kind: shorthand_property_identifier
+      regex: '^(email|userEmail|displayName|fullName|firstName|lastName|username)$'
   inside:
     stopBy: end
     any:

openspec/changes/archive/2026-10-14-fixture-quality/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-10-14

openspec/changes/archive/2026-10-14-fixture-quality/design.md

@@ -0,0 +1,70 @@
+## Context
+
+See proposal.md, under Why, for the motivation. This section covers only what shapes the approach.
+
+- The authoring recipes are markdown files under `packages/cli/src/agent/`. They are rendered by `src/prompts/recipes.ts` through sprintf variables and embedded into the bundle at build time. `buildVariables` provides every variable on every render, and `prompts.test.ts` pins the full list of variable names.
+- Fixture guidance today is per engine and lives in each recipe's own fixture step:
+  - `create-sg-rule`: step 6, "Write the tests". Step 2 already says "search the codebase for real instances".
+  - `create-vale-rule`: step 5, "Write the fixtures", which has detailed near-miss guidance for `pass/`.
+  - `create-runtime-rule`: the "Testing a rule you wrote yourself" section, with directory-shaped cases.
+  - `improve-rule.anonymous`: step 5, "Update test cases".
+  - `create-remote-rule` (step 4, "Enrich it") and `improve-rule` (step 5, "Collect supporting references") write no fixtures. They submit cases to the generator. Step 4 already says "Real matches beat invented ones" and reads linter configs, but only to find exclusion patterns.
+- `detect --json` already returns `linters[]`, each with a `name` and `evidence` (the config paths). `route` runs detection as its step 1 through `%(DETECT_EVIDENCE)s`. That block has a mechanics-off form for callers that supply the evidence themselves.
+
+## Goals / Non-Goals
+
+**Goals:**
+
+- Write the fixture-quality guidance once and render it into every recipe that writes fixtures, so the recipes cannot drift apart.
+- Keep each recipe's engine-specific fixture guidance where it is (Vale near-misses, ast-grep arity and awaited-call cases, runtime case directories). The shared passage adds to that guidance and replaces none of it.
+
+**Non-Goals:**
+
+- Any change to `verify`, `validate`, or the fixture file formats.
+- The service-side generator's own fixture prompts. Whether it copies submitted `successCases` into fixtures word for word is the generator team's question.
+- New `detect` output. `linters[]` already carries the evidence that is needed.
+
+## Decisions
+
+### One shared variable, `%(FIXTURE_SOURCES)s`, rendered as a whole block
+
+The passage is a new `buildVariables` entry, rendered with `indentBlock` like `NAME_DESTINATION`, and dropped into each fixture step as a sub-paragraph.
+
+- Alternative: copy the paragraph into four recipes. Rejected. Four copies pass the three-copy threshold for factoring out, and the copies would drift the first time one recipe's fixture step is revised.
+- Alternative: a separate `agent fixtures` topic that each recipe tells the agent to fetch. Rejected. It costs an extra fetch at exactly the step where an agent is most likely to skip it, and the guidance is short enough to carry inline.
+
+The block holds only what is engine-agnostic: the source order, the linter read, and the rule about varying only on the rule's axis. Each recipe keeps the word for its buckets (`valid`/`invalid` or `pass`/`fail`). For that reason the passage says "every case" or "the failing bucket", never a key name.
+
+### Detection reuses `DETECT_EVIDENCE`'s two modes
+
+With mechanics on, the passage tells the agent to reuse `detect --json` output from routing if it has it, and otherwise to run it. With mechanics off, it uses the supplied-evidence wording, the same split `DETECT_EVIDENCE` makes, so a caller without the binary is never told to run it. Reusing earlier output matters because the common path is `route` → `create-*-rule` in one task, and running detection twice adds nothing.
+
+### Linters are read, never run
+
+The passage asks the agent to read the configuration of each detected linter that covers the rule's language, and to write passing fixtures that linter would accept. It tells the agent not to run a linter or formatter on a fixture.
+
+- Alternative: run the linter on every snippet. Rejected. A fixture is a fragment, and linting a fragment produces unused-variable and unresolved-import errors that have nothing to do with style. An agent fixing those pads the fixture with scaffolding that hides the case, which is the readability problem this change exists to fix.
+- Alternative: run linters only on fixtures that are complete files. This was in the first draft and is rejected after the 4.3 trial. A tool's output becomes something to satisfy, and satisfying it is an open-ended loop with no rule-level payoff. The trial also showed the run is unreliable in practice: this repository's ESLint config ignores `.taskless/` entirely, so a run returns an "ignored file" warning that reads like a pass.
+
+### Failing cases come from the nearest real code
+
+A rule usually exists because the repository should not contain what it flags, so the failing bucket often has no real instance to copy. The passage directs the agent to take the nearest real code and change only the construct the rule flags, and counts only a case built from nothing as invented. Without this, "copy before you invent" marks every failing case as invented, and the disclosure stops carrying information. The 4.3 trial showed the payoff: a copied call site, written in the repository's shorthand style (`{ email }`), exposed a form the rule did not match.
+
+### The awaited-generic case is conditional on the repository
+
+`create-sg-rule` step 6 asked for an awaited generic call among the `invalid:` cases whenever a rule keys on a callee. That conflicts with "every case is code the repository would accept" for a callee the repository never awaits. The step now asks for that form only for a callee the repository does await with a type argument.
+
+### Submitting recipes get a short pointer, not the block
+
+`create-remote-rule` and `improve-rule` get one added bullet in their enrich and reference steps: submitted cases follow the same sourcing and idiom bar, and passing cases should be ones the detected linters would accept. They do not get the full block, whose step about writing fixture files does not apply to them. One short bullet in each of two places stays under the three-copy threshold.
+
+## Risks / Trade-offs
+
+- [The block's indentation differs between recipes] → `indentBlock` takes one indent. All four fixture steps sit inside a numbered list at three spaces, except the runtime section. The implementation checks each placement in the rendered output and gives the runtime recipe its own placement if it needs one.
+- [The recipe parity tests compare rendered output against source] → Changing recipe text needs no test updates beyond the variable list, but running prettier over `src/` rewrites recipe markdown. Format only the files changed, by path.
+- [An agent ignores the guidance anyway] → This is a recipe change with no gate behind it, by decision. Before archiving, check an authored rule by hand against a repository with ESLint, ideally the user's metric-counter rule.
+- [The Vale near-miss guidance and "idiomatic" seem to pull in different directions] → They don't. A near-miss is idiomatic prose that varies on the rule's axis. The passage says so explicitly, so an agent does not read "idiomatic" as "correct prose, which proves nothing".
+
+## Open Questions
+
+- Could the user's actual rule and fixture file serve as a before/after example in the change's verification? This doesn't affect the approach.

openspec/changes/archive/2026-10-14-fixture-quality/proposal.md

@@ -0,0 +1,33 @@
+## Why
+
+A user reviewed a rule an agent authored with Taskless. The rule was correct, but its fixtures were not code their team would accept. A `valid:` case used `Record<string, never>` and deeply nested call chains, and broke their own ESLint config. In their words, "the safer code fails while the bad code passes". People read fixtures as examples of the codebase's style, so fixtures like these lose the reader's trust before anyone looks at what the rule checks.
+
+The recipes allow this. `create-sg-rule` asks for fixtures "drawn from real code where you can" and says nothing else about what a fixture should look like. None of the authoring recipes runs `detect` before writing fixtures, so none of them knows which linters the repository enforces.
+
+## What Changes
+
+- Every recipe in which the agent writes fixtures gets one shared fixture-quality passage. Those recipes are `create-sg-rule`, `create-vale-rule`, `create-runtime-rule`, and `improve-rule.anonymous`, which adds cases to an existing test file. The passage covers three things:
+  - **Real code first.** Fixtures are copied from real code in the repository and trimmed down to the case. When the repository has no instance, which is usual for the failing cases of a rule that forbids something, the agent takes the nearest real code and changes only what the rule flags. Invented snippets are the last resort, and the report says which cases were invented.
+  - **Respect the repository's linters.** The agent reads `linters[]` from `taskless detect --json`, reuses that output if routing already produced it, and reads each detected linter's config. It writes cases in both buckets that those linters would accept as source. It reads the linters and never runs a linter or formatter on a fixture: a fixture is a fragment, and fixing what a tool reports about a fragment is an endless loop.
+  - **A fixture differs only where the rule looks.** Apart from the thing the rule checks, both buckets are plain code in the repository's idiom. No placeholder types such as `Record<string, never>` or `any`, and no nesting a reviewer would send back. Near-misses stay near-misses along the rule's own axis.
+- `create-remote-rule` and `improve-rule` (logged in) apply the same quality bar to the `successCases`/`failureCases` and references they send to the generator, because the generator works from those inputs.
+- **Out of scope:** `verify` does not lint fixtures and does not fail a rule over fixture style. Changes to the service-side generator are also out of scope.
+
+**Delivery shape: single PR.** This changes recipe prose, adds one shared block to `src/prompts/recipes.ts`, and adds the tests for that block. The whole change fits in one reviewable diff, and none of it is unsafe on its own.
+
+## Capabilities
+
+### New Capabilities
+
+None.
+
+### Modified Capabilities
+
+- `cli-agent-authoring`: adds a requirement covering where fixtures come from, how the repository's linters constrain them, and the scope rule for what a fixture may vary. This is an ADDED requirement. No existing requirement is rewritten.
+
+## Impact
+
+- `packages/cli/src/agent/create-sg-rule.md`, `create-vale-rule.md`, `create-runtime-rule.md`, `improve-rule.anonymous.md`: the fixture steps pull in the shared passage.
+- `packages/cli/src/agent/create-remote-rule.md`, `improve-rule.md`: the enrich and reference steps apply the same bar to what is submitted.
+- `packages/cli/src/prompts/recipes.ts`: a new shared variable for the passage. With mechanics off, it falls back to the existing supplied-evidence wording.
+- Recipe tests (`prompts.test.ts`, `recipe-cross-references.test.ts`) and a changeset. Recipes ship inside the CLI bundle, so this is a `patch` release note.

openspec/changes/archive/2026-10-14-fixture-quality/specs/cli-agent-authoring/spec.md

@@ -0,0 +1,67 @@
+## ADDED Requirements
+
+### Requirement: Fixtures are real, repo-idiomatic code that varies only where the rule looks
+
+Every authoring recipe in which the agent writes or extends fixtures SHALL instruct the agent to:
+
+1. Take fixtures from real code in the repository, trimmed down to the case, before inventing any. Where the repository has no instance of a case, take the nearest real code and change only the construct the rule flags, borrowing names and values from elsewhere in the repository where needed. A fixture invented outright is allowed only when nothing nearby exists, and the agent's report SHALL say which fixtures were invented.
+2. Read the repository's detected linters before writing fixtures. The agent reuses `detect --json` output already in hand, or runs it otherwise. It writes fixtures in both buckets that those linters, as configured, would accept as source, including where their configuration ignores the rules directory. The agent reads the linters' configuration and SHALL NOT be directed to run a linter or formatter on a fixture.
+3. Keep every fixture, in both buckets, in the repository's idiom everywhere except the construct the rule checks. A fixture SHALL NOT use placeholder types, catch-all types, or nesting that the repository's own conventions would reject.
+
+A recipe through which the agent submits cases to the generator, instead of writing fixtures itself, SHALL apply the same three instructions to the cases and references it submits.
+
+Fixtures are read as examples of the repository's style. A `valid:` case that the repository's own linter rejects reads as Taskless approving bad code, and a reader who sees that stops trusting the rule before checking what it actually matches.
+
+Enforcing this is the recipe's job. `verify` SHALL NOT lint fixtures and SHALL NOT fail a rule over fixture style.
+
+#### Scenario: Real instances are used before invented ones
+
+- **WHEN** an agent follows an authoring recipe and the repository contains code exhibiting the case
+- **THEN** the recipe SHALL direct the agent to take that fixture from the repository rather than inventing one
+
+#### Scenario: A case with no real instance is adapted from nearby code
+
+- **WHEN** the repository contains no instance of a failing case, as is usual for a rule that forbids something
+- **THEN** the recipe SHALL direct the agent to take the nearest real code and change only the construct the rule flags
+
+#### Scenario: Invented fixtures are disclosed
+
+- **WHEN** the repository has no instance of a case and the agent invents a fixture for it
+- **THEN** the recipe SHALL direct the agent to name the invented fixtures in its report
+
+#### Scenario: Detected linters shape every case
+
+- **WHEN** `detect --json` reports one or more linters
+- **THEN** the recipe SHALL direct the agent to read each one's configuration before writing fixtures
+- **AND** to write fixtures in both buckets that those linters would accept
+
+#### Scenario: Linters are read, not run, against fixtures
+
+- **WHEN** the recipe directs the agent to respect the repository's linters
+- **THEN** it SHALL direct the agent to read their configuration
+- **AND** SHALL direct the agent not to run a linter or formatter on a fixture
+
+#### Scenario: Detection is not repeated when routing already ran it
+
+- **WHEN** the agent already has `detect --json` output from routing in the same task
+- **THEN** the recipe SHALL let the agent reuse it rather than run detection again
+
+#### Scenario: A fixture varies only on the rule's axis
+
+- **WHEN** the agent writes a near-miss `pass` fixture next to a `fail` fixture
+- **THEN** the recipe SHALL direct that the two differ only in the construct the rule checks
+
+#### Scenario: Placeholder types are not used to make a fixture compile
+
+- **WHEN** a fixture needs a type, value, or call that is not what the rule checks
+- **THEN** the recipe SHALL direct the agent to use one the repository itself would use, not a placeholder or catch-all type
+
+#### Scenario: Submitted cases meet the same bar
+
+- **WHEN** an agent builds a generation or improvement request for the service
+- **THEN** the recipe SHALL apply the same sourcing, linter, and idiom instructions to the submitted cases and references
+
+#### Scenario: verify does not judge fixture style
+
+- **WHEN** `verify` runs a rule whose `pass` fixture would fail the repository's linter
+- **THEN** `verify` SHALL report on the rule's behavior alone, and SHALL NOT fail the rule over that fixture's style