feat(agent): fixtures come from real code and pass the repo's linters

[theCodeDrift]

A user reviewed a rule an agent wrote with Taskless. The rule was correct, but its valid: fixtures used Record<string, never> and deeply nested call chains, and broke their own ESLint config. In their words, "the safer code fails while the bad code passes." People read fixtures as examples of the codebase's style, so a technically correct fixture the team would reject in review costs the rule their trust. The recipes allowed it: they asked for fixtures "drawn from real code where you can" and said nothing else.

What changes

• A shared passage, %(FIXTURE_SOURCES)s (src/prompts/recipes.ts), rendered into every recipe where the agent writes fixtures: create-sg-rule, create-vale-rule, create-runtime-rule, and improve-rule --anonymous. It tells the agent to:

  • copy before inventing. Fixtures come from real code. A rule usually forbids something the repository does not contain, so a failing case takes the nearest real code and changes only what the rule flags, borrowing names from elsewhere if needed. Only a case built from nothing counts as invented, and the report names those.
  • write every case to the repository's linters. The agent reads the configs of the linters detect --json reports that cover the rule's language, and writes cases those linters would accept as source. It never runs a linter or formatter on a fixture: fixing what a tool says about a fixture is an endless loop that improves nothing about the rule.
  • vary only where the rule looks. No placeholder or catch-all types, and no nesting a reviewer would send back.

  With mechanics: false it names no command.

• create-sg-rule step 6 asks for an await f<T>(...) failing case only for a callee the repository actually awaits with a type argument, instead of for every rule that keys on a callee.

• create-remote-rule and improve-rule hold the cases and references they send to the generator to the same standard.

• verify is unchanged. Fixture style is the recipe's job, not a gate.

• no-pii-in-telemetry now catches { email }. See the trial below.

• Topic versions are bumped on all six recipes. The change includes a patch changeset. The OpenSpec change fixture-quality adds one requirement to cli-agent-authoring and is archived in this PR. All 19 existing scenarios survive the archive, which adds 10 new ones.

Trial (task 4.3)

I asked agents to re-author no-pii-in-telemetry from scratch by following the built recipe. It is the closest analog in this repository to the user's metric rule: a rule about which keys may appear inside a telemetry call.

• First run: it worked, and its critique drove the second commit. Every failing case counted as "invented", running linters on fixtures went nowhere (eslint ignores .taskless/), and the awaited-generic instruction conflicted with writing idiomatic code.
• Second run, on the revised recipe, with a fresh agent that was not told about the first run's finding:
  • All four passing cases are copied verbatim from telemetry.ts, report.ts and info.ts.
  • All four failing cases are real calls with only the key changed. None was invented.
  • It read the eslint and prettier configs and ran neither. It skipped Vale, which does not cover TypeScript.
• Both runs independently found a real gap in the rule. This repository writes shorthand properties, and { email } is a shorthand_property_identifier, not a pair, so the rule never matched it. The old fixtures were invented in the posthog.capture({...}) shape from the docs and could not show this.
  • Measured: the rule on main fails the new shorthand case, the new rule passes all 8 cases, and check finds no hits in the repository.

Delivery shape

Single PR. The change is recipe prose, one shared variable and its tests, and one rule fix. None of it is unsafe to ship on its own.

Verification

• pnpm typecheck, pnpm lint, and pnpm --filter @taskless/cli test (126 files, 2296 tests) all pass.
• New tests in prompts.test.ts cover the shared passage in every fixture-writing recipe, the detect command when a CLI is available, and no command with mechanics off.