Skip to content

Migrate eligible model stages to structured output #252

Description

@eshulman2

Problem

Several Forge stages request a specific machine-readable structure in their prompts and then parse free-form model text. This is fragile: formatting deviations cause avoidable parsing failures, retries, and unclear errors.

Forge currently does not use provider-enforced structured output. The model-policy capabilities field therefore correctly supports only tools; structured_output must not be introduced until Forge actively uses and validates it.

Goal

Use schema-enforced structured output for stages whose results are genuinely data-shaped, while retaining Markdown/text output for narrative artifacts.

Candidate stages

Evaluate at least:

  • Bug, automated-review, proposal-review, and task-takeover triage
  • Epic decomposition
  • Task generation
  • CI analysis decisions
  • Review-thread classification and other bounded decision outputs

Keep narrative stages such as PRDs, specifications, implementation plans, PR descriptions, and qualitative reviews as text/Markdown unless a clear schema boundary is identified.

Requirements

  • Define typed Pydantic response schemas for each migrated stage.
  • Apply structured output to the final response without disrupting the Deep Agent tool loop.
  • Verify behavior across every supported backend: Vertex AI Gemini, Vertex AI Anthropic, Google GenAI, and direct Anthropic.
  • Preserve useful retries and a clearly documented fallback for provider/schema failures.
  • Replace ad hoc text parsing only after equivalent behavior and error handling are covered by tests.
  • Produce actionable validation errors rather than silently accepting malformed output.
  • Keep Langfuse model and stage attribution consistent.
  • Introduce a structured_output model-connection capability only when runtime enforcement exists and a stage concretely requires it.
  • Document backend limitations and configuration requirements.

Acceptance criteria

  • Eligible stages return and consume validated typed objects.
  • Tool-enabled stages can still complete their tool loop before producing the final structured response.
  • Tests cover valid output, malformed output, retry/fallback behavior, and all supported provider adapters.
  • Existing workflow behavior remains backward compatible.
  • Remaining prompt-parsed stages are documented with the reason they were not migrated.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions