Skip to content

[Docs] Visualize supported scenarios and their verification evidence #62

Description

@szmyty

Outcome

Make flow's supported and tested scenarios easy to understand and showcase in the documentation. A reader should be able to browse a visual catalog, see what each scenario demonstrates, and follow its verification evidence without reading the test implementation.

Suite roadmap: #11
Builds on: #30 / PR #61 and the scenario contract in #28
Coordinates with: #13 and #34 (scenario coverage and reporting)
Scheduling: later documentation work; does not block #49, #31, or the existing implementation sequence.

Starting point

PR #61 introduces 81 acceptance scenarios covering resolution, contracts, artifacts, provider failures, and privacy. Use its merged versions of:

  • tests/fixtures/acceptance-scenarios.v1.json — stable scenario IDs, expected outcomes, budgets, and known gaps.
  • tools/run_acceptance_scenarios.py and its versioned normalized report — actual verification evidence and tested source identity.
  • docs/integrations/acceptance-scenarios.md — interpretation and claim limits.

Re-query the catalog and merged state when implementation begins; 81 is the initial snapshot, not a permanent hard-coded count.

Proposed scope

  • Add an overview matrix or gallery grouped by scenario family, with plain-language titles and explanations of why each case matters.
  • Give scenarios stable links and show the setup or injected fault, expected flow response, acceptance/refusal outcome, and relevant contract or evidence.
  • Include a few representative diagrams showing provider execution, validation decisions, and resulting outcomes. Choose the simplest presentation supported by the existing documentation stack; searchable/filterable views are optional.
  • Distinguish a test passing from a provider result being accepted: a correctly rejected invalid result is a passing test. Preserve complete, observed-empty, unavailable, incomplete, unsupported, invalid, and failed evidence states.
  • Clearly identify synthetic contract coverage, verified real-provider coverage, and planned or unproven capabilities. Resolution or transcript validation alone must not be presented as accepted execution.
  • Generate or validate IDs, counts, expected outcomes, and coverage from canonical data. Explanatory copy should be keyed by scenario ID rather than becoming a second hand-maintained coverage inventory.
  • Link observed verification to its tested revision and report identity. Missing, expired, stale, or mismatched evidence must not appear as currently verified. Use only portable normalized evidence.
  • Link the showcase from the README/integration documentation and leave room for later [FLO-13.4] Prove interruption, retry, resume, and authority state transitions #31–[FLO-13.7] Add clean-room compatibility, CI tiers, and regression promotion #34 scenario families without claiming their support early.

Acceptance criteria

  • Readers can browse all current catalog scenarios and understand representative success and refusal cases without opening Rust source.
  • Every displayed scenario maps to a stable canonical ID; IDs, counts, and expected outcomes stay synchronized through generation or a drift check.
  • The presentation distinguishes expected behavior, observed verification, and known coverage gaps.
  • Evidence links identify what revision was tested; unavailable or stale evidence is clearly labeled.
  • Diagrams have readable text equivalents; status does not rely on color alone.
  • The showcase builds locally using the existing docs tooling, with documented refresh instructions and no new paid CI dependency.

Boundaries

This issue owns documentation presentation. #34 continues to own coverage-reporting and release-gate semantics. It does not add runtime capabilities, a hosted dashboard, or new provider support.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions