Skip to content

Extract implementation-pipeline scenario and tests from the E2E harness #5334

Open

Description

@Trecek

Problem

The implementation-pipeline scenario is embedded in the general E2E harness, and its flow tests and GitHub fakes are embedded in the general harness test suite. This makes scenario changes harder to locate and couples pipeline-specific behavior to generic execution infrastructure.

This follows up the two deferred modularity findings from PR #5255: d36c3244883b (production cohesion) and 54e735ca6562 (test organization). They describe one coordinated refactor. The review fixes are already committed and pushed at f2b3d54fac5168652bc8991b97e39cd9f896d941; the configured test gate and task test-check both passed at that revision.

Current boundaries

  • scripts/e2e/e2e_harness.py implements seed validation, owned-PR discovery and validation, sandbox CI polling, ownership evidence, capacity checks, and branch/claim cleanup alongside generic scenario dispatch, execution, artifact collection, and redaction.
  • tests/infra/test_e2e_harness.py contains TestPipelineRecipeFlow, pipeline-specific response routing, and measurement stubs alongside generic harness tests.
  • Pipeline helpers depend on existing command/evidence helpers, Runner, run_fleet, and the sandbox path. Measurement stubs currently patch the session collector through the harness module namespace.
  • Cleanup is guaranteed by launch_recipe calling finish_pipeline from finally. That guarantee must survive extraction, including unexpected-exception paths.

Evidence at the reviewed revision:

Intended outcome

Give the implementation-pipeline scenario and its flow tests explicit ownership apart from generic harness behavior. A dedicated stdlib-only scenario module and a matching pipeline test module are the proposed shape; choose the smallest dependency boundary that makes the extraction coherent.

Keep generic dispatch, artifact handling, and redaction with the harness. Keep isolated lifecycle/parsing/measurement tests with test_e2e_sessions.py; pipeline integration tests should continue exercising those components together. Keep pipeline-only fakes local to the pipeline test module, and extract shared test helpers only where existing consumers actually need them.

Acceptance criteria

  • Scenario execution and flow tests are separated from generic harness responsibilities, with clear entry points and dependency ownership.
  • Existing command ordering, sandbox selection, result/envelope shapes, evidence artifacts, ownership checks, and cleanup-on-failure behavior are preserved.
  • Shared command/fleet execution dependencies are handled without circular imports, duplicated utilities, compatibility re-exports, or a generic callback framework.
  • Existing coverage is preserved, and patch targets still affect the production objects being exercised; test modules do not import helpers from other test modules.
  • Standalone script loading and stdlib-only operation remain supported; documentation and test-filter mappings are updated where needed.
  • Pre-commit and the configured test gate pass in the implementation worktree.

Review and size guidance

After implementing the refactor, launch two independent adversarial reviewers concurrently: one should challenge behavior preservation, failure handling, cleanup, and evidence contracts; the other should challenge module boundaries, dependency direction, test isolation, and whether the extraction actually improves cohesion. Wait for both reviewers before continuing; do not perform independent implementation, testing, or other work while either reviewer is running. Independently validate their findings and address accepted findings before completion.

The user asked whether this could fit below 200 changed lines. Moving the existing scenario and test suite will exceed that as a Git diff; keep new coordination logic small and report moved lines separately from substantive edits. Do not distort the design to meet an artificial total-diff limit.

Scope

This issue is the coordinated production/test extraction only. It does not reopen earlier PR review findings or add pipeline features, alter acceptance policy, or repair unrelated authentication/tooling issues encountered during review resolution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions