Skip to content

chore(prompts): formalize cross-prompt contract registry for develop + research machines #322

Description

@canesin

Summary

The develop and research workflow machines chain markdown-producing prompts: each machine's prompt tells its agent to write artifacts with specific section headings, and the next machine's prompt tells its agent to read/resolve those same headings. Today the section lists are hand-edited string literals in separate files. Nothing enforces that the producer's list and the consumer's list stay in sync, and two heading-based parsers silently fall back to UNKNOWN when a prompt renames a heading.

This issue proposes a single-source prompt contract registry that prompt builders import and interpolate, plus a small enforcement test that runs the hard parsers against synthetic artifacts built from the registry. Drift becomes impossible by construction.

Problem

Concrete drift today

src/machines/develop/plan-review.machine.js:91-101 is the producer:

Required sections (in order):
1. Critical Issues (Must Fix)
2. Over-Engineering Concerns — ...
3. Data Structure Review — ...
4. Concerns (Should Address)
5. Questions (Need Clarification)
6. Verdict (REJECT | REVISE | PROCEED WITH CAUTION | APPROVED)

src/machines/develop/implementation.machine.js:132-147 is the consumer:

## Step 1: Address Critique
Update ${paths.plan} to resolve every finding in the critique —
Critical Issues, Over-Engineering Concerns, Data Structure Review,
Concerns, and Questions.

These are two independent string literals. PR #321 caught (over five review rounds) that Data Structure Review was added as a required section in plan-review without updating Step 1 of implementation, so the agent skipped resolving that section entirely. Nothing in the test suite noticed.

Hard parser contracts

Two parsers key off heading text:

  • parsePlanVerdict (src/machines/develop/plan-review.machine.js:115) matches /^#{1,6}\s+(?:\d+\.\s+)?Verdict\b[^\n]*/gim against PLANREVIEW.md. Renaming Verdict in the prompt returns UNKNOWN, which downgrades to a fallback inline-verdict regex and then to failing the plan-review machine.
  • parseReviewVerdict (src/machines/develop/quality-review.machine.js:35) matches /^##\s+VERDICT:\s+(APPROVED|REVISE)\s*$/gm against REVIEW_FINDINGS.md. Renaming VERDICT breaks the workflow silently — parseReviewVerdict returns { verdict: null, findings: content }.

Neither drift would be caught by the current test suite.

Research workflow has the same pattern

  • issue-synthesis.machine.js produces a JSON backlog that issue-publish.machine.js consumes.
  • spec-architect.machine.js produces domains/decisions/phases consumed by spec-render.machine.js.
  • Each contract is maintained by independent prompt builders with no shared source of truth.

Proposal

1. src/machines/prompt-contracts.js — single source of truth

New module that exports one entry per artifact, with section names, descriptions, producer path, consumer paths, and parser references:

// src/machines/prompt-contracts.js
export const CONTRACTS = {
  "PLANREVIEW.md": {
    producedBy: "src/machines/develop/plan-review.machine.js",
    requiredSections: [
      { heading: "Critical Issues", suffix: "(Must Fix)",
        description: "..." },
      { heading: "Over-Engineering Concerns",
        description: "flag speculative optimizations..." },
      { heading: "Data Structure Review",
        description: "is the plan carrying the right data shapes?..." },
      { heading: "Concerns", suffix: "(Should Address)",
        description: "..." },
      { heading: "Questions", suffix: "(Need Clarification)",
        description: "..." },
      { heading: "Verdict",
        description: "REJECT | REVISE | PROCEED WITH CAUTION | APPROVED" },
    ],
    consumers: [
      {
        file: "src/machines/develop/implementation.machine.js",
        mustResolve: ["Critical Issues", "Over-Engineering Concerns",
                      "Data Structure Review", "Concerns", "Questions"],
      },
    ],
    parsedBy: [
      { file: "src/machines/develop/plan-review.machine.js",
        function: "parsePlanVerdict",
        headingMatches: "Verdict" },
    ],
  },
  "PLAN.md":           { /* ... */ },
  "REVIEW_FINDINGS.md":{ /* ... */ },
  "ISSUE.md":          { /* ... */ },
  // research:
  "research/issue-backlog.json":   { /* ... */ },
  "research/spec-architect.json":  { /* ... */ },
};

// Render helpers used by prompt builders:
export function renderRequiredSections(contract) {
  return contract.requiredSections
    .map((s, i) => {
      const head = s.suffix ? `${s.heading} ${s.suffix}` : s.heading;
      return `${i + 1}. **${head}** — ${s.description}`;
    })
    .join("\n");
}

export function renderCritiqueSectionList(contract) {
  // "Critical Issues, Over-Engineering Concerns, ..., and Questions"
  const names = contract.requiredSections
    .filter((s) => s.heading !== "Verdict")
    .map((s) => s.heading);
  const last = names.pop();
  return names.length ? `${names.join(", ")}, and ${last}` : last;
}

The registry is the only place the section names appear. Everything else derives from it.

2. Refactor prompt builders to import and interpolate

Touch every prompt builder that today has a hand-written required-sections list, and replace the literal with a call to the registry.

Producers (planning.machine.js, plan-review.machine.js, quality-review.machine.js, issue-draft.machine.js, issue-synthesis.machine.js, spec-architect.machine.js, issue-publish.machine.js):

import { CONTRACTS, renderRequiredSections } from "../prompt-contracts.js";

const prompt = `...
Required sections (in order):
${renderRequiredSections(CONTRACTS["PLANREVIEW.md"])}
...`;

Consumers (implementation.machine.js, quality-review.machine.js where it reads ISSUE.md/PLAN.md, and any research machines that read upstream artifacts):

import { CONTRACTS, renderCritiqueSectionList } from "../prompt-contracts.js";

const prompt = `...
## Step 1: Address Critique
Update ${paths.plan} to resolve every finding in the critique —
${renderCritiqueSectionList(CONTRACTS["PLANREVIEW.md"])}.
...`;

For conditional fragments (the difficulty ≥ 3 Red/Green TDD subsection in ISSUE.md, the planExhausted branch in quality-review, active-branches in planning), keep the conditionals inline in the template literal. Only the static section list comes from the registry.

3. test/prompt-contracts.test.js — enforcement

The test is deliberately small because import-interpolation does most of the work:

  • Registry shape test. Every entry has producedBy, requiredSections, and either consumers or parsedBy (or both). Every producedBy and consumers[i].file resolves to an existing file on disk.
  • Parser roundtrip test. For each entry with parsedBy, build a synthetic markdown artifact containing the headingMatches heading, call the parser, assert it returns a non-UNKNOWN/non-null value. This catches parser drift against the registry.
  • Rendered-output sanity. Call renderRequiredSections(entry) and assert the output contains every heading from the registry (a smoke test that the render helpers handle edge cases like empty descriptions or missing suffixes).

No grep-the-file-string check is needed — because the prompt builders import from the registry, the same string literally cannot diverge.

4. docs/prompt-contracts.md — human-readable

Short doc (< 100 lines) explaining:

  • Why the registry exists (the PR feat(prompts): weave Pike's rules into develop + research prompts #321 drift story in one paragraph)
  • How to add a new section to an existing artifact: update the registry entry → run npm testrenderRequiredSections regenerates the producer prompt, renderCritiqueSectionList regenerates the consumer prompt, the parser roundtrip test catches heading-name changes
  • How to add a new artifact: register it → wire producer and consumer to import the registry → add parser entry if applicable
  • A one-line link from DESIGN.md pointing here

Files

New:

  • src/machines/prompt-contracts.js — the registry + render helpers
  • test/prompt-contracts.test.js — the enforcement test
  • docs/prompt-contracts.md — the human-readable guide

Modified (prompt builders):

  • src/machines/develop/issue-draft.machine.js
  • src/machines/develop/planning.machine.js
  • src/machines/develop/plan-review.machine.js (both buildCritiqueRetryPrompt and basePromptBody)
  • src/machines/develop/implementation.machine.js
  • src/machines/develop/quality-review.machine.js (producer of REVIEW_FINDINGS.md; consumer of ISSUE.md/PLAN.md)
  • src/machines/research/issue-synthesis.machine.js
  • src/machines/research/spec-architect.machine.js
  • src/machines/research/issue-publish.machine.js (consumer of synthesis output)

Modified (docs):

  • DESIGN.md — add one-line link to docs/prompt-contracts.md

Scope

In scope:

  • src/machines/prompt-contracts.js with entries for develop artifacts (ISSUE.md, PLAN.md, PLANREVIEW.md, REVIEW_FINDINGS.md) and research artifacts (issue-backlog JSON, spec-architect JSON, issue-synthesis draft schema)
  • Render helpers (renderRequiredSections, renderCritiqueSectionList, and any others the prompt builders need) in the same module
  • Refactor of every prompt builder listed above to import from the registry
  • test/prompt-contracts.test.js with registry-shape, parser-roundtrip, and render-sanity assertions
  • docs/prompt-contracts.md
  • One-line reference from DESIGN.md

Out of scope:

  • Moving the PLAN.md Testing Strategy or ISSUE.md Requirements prose into the registry. Only static section names + short descriptions move — multi-paragraph prose stays in the template.
  • Fancy schema validation (zod / JSON-schema) for the registry itself — plain objects are fine, the test enforces shape.
  • Refactoring parsePlanVerdict / parseReviewVerdict to read the registry at runtime (they still use hardcoded regexes; the registry documents what they parse, the test verifies the match still works).
  • Adding the registry to the MCP tool surface (no external caller needs it).

Requirements (EARS)

  • The src/machines/prompt-contracts.js module shall export CONTRACTS with one entry per cross-prompt artifact in the develop and research workflows.
  • Every prompt builder that today hand-writes a required-sections list shall import section names from the registry via a render helper, so the section list appears exactly once in the codebase.
  • test/prompt-contracts.test.js shall fail when a registry entry names a non-existent producer/consumer file.
  • test/prompt-contracts.test.js shall fail when parsePlanVerdict or parseReviewVerdict cannot extract a verdict from a synthetic artifact built from the registry headings.
  • WHEN a contributor adds a new section to CONTRACTS, the producer and consumer prompts shall automatically reflect it on the next build (no further edits required).
  • docs/prompt-contracts.md shall describe the update workflow for adding sections and artifacts.
  • DESIGN.md shall link to docs/prompt-contracts.md.

Verification

  • npm test passes (new tests added, existing 806 still green)
  • npm run lint passes with no new warnings
  • Manual drift check: temporarily rename Data Structure ReviewData Shapes Review in the registry, run npm test, observe the rendered prompt changes everywhere consistently, and the parser roundtrip still passes (it only tests the Verdict heading); then revert.
  • Every .machine.js file under src/machines/develop/ and src/machines/research/ that produced or consumed a required-sections list now imports from prompt-contracts.js.
  • No grep of the codebase outside prompt-contracts.js finds the string "Critical Issues", "Over-Engineering Concerns", "Data Structure Review", etc.

Out of Scope (follow-up candidates)

  • Migrating hard parsers (parsePlanVerdict, parseReviewVerdict) to derive their regex from the registry at runtime. The current issue keeps them hardcoded but test-verified.
  • Generating docs/prompt-contracts.md from the registry at build time (keep it hand-written for now).

Reference

The drift pattern was surfaced across 5 rounds of codex_loop_review on PR #321 (feat/pike-rules-in-prompts). Commits 25f0d86..913e883 show each round and the specific contract that drifted.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions