Skip to content

Flaky test architectural immunity: centralized timeouts, configurable preamble, AST guards #1032

Closed

Description

@Trecek

Summary

Implement structural defenses against timing-sensitive async subprocess test flakiness under pytest-xdist. This addresses a recurring class of pipeline aborts (3 documented occurrences: 2026-04-13, 2026-04-16, 2026-04-18) caused by tests with insufficient timeout headroom for xdist -n 4 CPU contention.

Related: #1031 (resolve-failures verdict classification — the pipeline-level fix for the same root cause)

Background

The affected tests share these properties:

  • Async subprocess orchestration via sys.executable with JSONL output monitoring
  • A mandatory 1.0s anyio.move_on_after preamble in _process_race.py:250 (stdout_session_id_ready) that always expires in tests because test scripts don't emit type=system records
  • Under xdist -n 4, asyncio.sleep(1.0) can overrun to 5-10s due to event loop scheduling jitter
  • Tests with timeout=60 (15x headroom) never flake; tests with timeout=15 (~1.5x headroom) flake under load

The timing budget for a Channel B test: 1s preamble + _phase1_timeout (up to 30s default) + drain + jitter headroom. Tests in test_process_channel_b.py with timeout=60 document this math in inline comments. The flaky test at line 531 uses timeout=15 without this analysis.

Requirements

REQ-FLAKY-001: Fix the immediate flaky test

Change timeout=15 to timeout=60 in test_channel_b_ignores_sub_skill_marker (tests/execution/test_process_channel_b.py:531), matching all sibling tests in TestChannelBSubSkillCollision that go through the same Channel B path.

Add the standard xdist load comment explaining the budget math, consistent with the comments at lines 155-158, 291-296, and 416-428 in the same file.

Also fix 4 latent under-budgeted Channel B tests that use timeout=30 when the budget math requires >= 31.5s:

  • test_channel_b_wins_then_channel_a_confirms_within_drain (line 132)
  • test_data_confirmed_false_set_on_drain_timeout (line 218)
  • test_channel_b_then_a_empty_result_data_confirmed_is_false (line 266)
  • test_channel_b_drain_timeout_produces_success_skill_result (line 351)

These are latent flake risks per the budget math at lines 155-158 (minimum = 31.5s) and should be migrated to timeout=60.

REQ-FLAKY-002: Centralized timeout tier system

Add a timeout tier system to tests/conftest.py that encodes the budget math for different test classes:

Tier Value Use Case
UNIT 10s Pure logic tests, no I/O
INTEGRATION 30s Tests with filesystem or subprocess, no Channel B
CHANNEL_B 60s Tests going through full session_log_dir + Channel B path

The tier values encode the documented timing budgets from the existing inline comments. The system should be a simple class or module-level constants — not an over-engineered abstraction. Tests should reference the tier constant rather than hardcoding timeout values for Channel B tests where the budget math is non-obvious.

Implementation notes:

  • Place in tests/conftest.py alongside existing xdist-aware fixtures
  • Do NOT retroactively convert all existing timeout values — only require tier constants for the Channel B tier where the budget math is non-obvious
  • UNIT and INTEGRATION tiers exist for documentation; hardcoded values in those ranges are acceptable
  • The previously proposed RECORDING tier (300s) is removed — those values in test_recording.py are arguments forwarded to a mock runner, not test wall-clock budgets
  • Tests that deliberately use short timeouts (0.5-3s) to provoke TIMED_OUT behavior are not covered by tiers; their values are intentional and correct

REQ-FLAKY-003: Make stdout_session_id_ready preamble timeout configurable

The _process_race.py:250 hardcoded anyio.move_on_after(1.0) is the primary jitter source in tests. Make this timeout configurable via a parameter to run_managed_async:

  • Add _session_id_timeout: float = 1.0 parameter (underscore prefix, consistent with existing _phase1_timeout, _phase1_poll, _phase2_poll parameters)
  • Tests can pass _session_id_timeout=0.01 to skip the wait cheaply when the test script doesn't emit type=system records
  • Production default remains 1.0 — no behavioral change for non-test callers

This reduces the timing budget for Channel B tests from 1s + phase1 + drain to 0.01s + phase1 + drain, giving tests more headroom within the same outer timeout.

Critical implementation constraints:

  • _session_id_timeout MUST be keyword-only with a default value. test_process_monitor.py:1177 calls _watch_session_log directly with 13 positional args — adding a new positional arg would break that call.
  • The underscore prefix is load-bearing: test_req_api_001_public_params_present in tests/contracts/test_protocol_satisfaction.py:321-347 filters out k.startswith("_") when checking the public API surface. A non-underscore name would require updating that contract test.
  • run_managed_sync does NOT need this parameter — it uses bare subprocess.Popen with no Channel B infrastructure.
  • Inline comments at lines 156, 295, 418 of test_process_channel_b.py cite the 1.0s preamble as part of their budget calculation. These should be updated to reference the default value.

REQ-FLAKY-004: AST guard for Channel B timeout minimums

Add a structural test that scans tests/execution/test_process_channel_b.py (and any future Channel B test files) for run_managed_async( calls and asserts that the associated timeout= value meets the Channel B minimum (60s).

Implementation approach:

  • Parse the test file's AST to find run_managed_async call sites
  • Extract the timeout keyword argument value (timeout is always keyword-only due to * in run_managed_async signature, and always a numeric literal in the codebase)
  • Use a compound heuristic to identify Channel B tests: require session_log_dir + completion_marker + completion_drain_timeout >= 0.3 to all be present
  • Assert timeout >= TimeoutTier.CHANNEL_B for matching calls
  • Include an explicit exemption set for known-intentional low-timeout Channel B tests (e.g., test_drain_window_times_out_when_no_session_jsonl at timeout=10 with completion_drain_timeout=0.2)
  • Assert timeout is an ast.Constant (literal) — if it becomes a variable, emit a separate violation
  • Place in tests/arch/ alongside existing AST-based guards (test_anyio_migration.py, test_subpackage_isolation.py)

This prevents future occurrences of the timeout=15 class of bug.

REQ-FLAKY-005: Make test scripts emit synthetic type=system records

Update the test subprocess script literals in tests/execution/test_process_channel_b.py to emit a synthetic type=system JSONL record to stdout at startup. This causes stdout_session_id_ready to fire immediately instead of timing out, eliminating the 1.0s preamble cost entirely.

Example addition to test scripts:

import json, sys
sys.stdout.write(json.dumps({"type": "system", "session_id": "session"}) + "\n")
sys.stdout.flush()

Critical: The session_id value MUST be "session" (matching the session.jsonl filename stem used by all Channel B test scripts). Using any other value (e.g., "test-session") causes Phase 1 identity-based selection in _session_log_monitor (_process_monitor.py:200-213) to fail (f.stem == expected_session_id → False), falling back to recency selection with spurious session_id_match_not_found warnings.

Scripts that MUST NOT receive the synthetic record:

  • CHANNEL_B_NO_STDOUT_SCRIPT — test_channel_b_wins_drain_timeout_still_kills asserts not result.stdout.strip(). Any stdout output breaks this assertion directly.
  • WRITE_RESULT_THEN_HANG_SCRIPT (conftest) — these tests pass no session_log_dir, so _extract_stdout_session_id never runs. Adding a record is pointless.
  • Inline script in test_drain_window_times_out_when_no_session_jsonl — this test deliberately has no session JSONL; adding a session ID would change Phase 1 behavior.

Scripts that CAN receive the synthetic record:

  • CHANNEL_B_THEN_A_CONFIRM_SCRIPT
  • CHANNEL_B_THEN_A_EMPTY_RESULT_SCRIPT
  • PROCESS_EXIT_THEN_CHANNEL_B_FIRES_SCRIPT
  • CHANNEL_B_SUB_SKILL_COLLISION_SCRIPT

REQ-FLAKY-006: Document --dist mode discrepancy

tests/CLAUDE.md documents --dist worksteal but the Taskfile only passes -n 4 (no --dist flag). The actual default is --dist load (round-robin) — xdist's plugin.py:319-320 auto-promotes dist="no" to dist="load" when -n N > 0. Research found that worksteal has open bugs:

Update tests/CLAUDE.md to document the actual --dist mode in use (load, the xdist default when -n 4 is given without --dist). Also update audit-tests/SKILL.md line 179 which repeats the same --dist worksteal claim. If worksteal was intended, add a note about the known issues and a rationale for the choice.

Files to Modify

  • tests/execution/test_process_channel_b.py — REQ-FLAKY-001 (timeout fix for 5 tests), REQ-FLAKY-005 (synthetic system records for 4 scripts)
  • tests/conftest.py — REQ-FLAKY-002 (timeout tier system)
  • src/autoskillit/execution/_process_race.py — REQ-FLAKY-003 (configurable _session_id_timeout, keyword-only)
  • src/autoskillit/execution/process.py — REQ-FLAKY-003 (thread parameter through run_managed_async only, NOT run_managed_sync)
  • tests/arch/test_channel_b_timeout_guard.py (new) — REQ-FLAKY-004 (AST guard with compound heuristic + exemption set)
  • tests/CLAUDE.md — REQ-FLAKY-006 (dist mode documentation)
  • .claude/skills/audit-tests/SKILL.md — REQ-FLAKY-006 (dist mode reference)

Out of Scope

  • Retry plugins (pytest-rerunfailures, pytest-retry) — masks the problem without fixing it. The codebase intentionally has no retry infrastructure.
  • Fake clock libraries (looptime) — only affects event loop time, not real subprocess time.sleep() calls. Not applicable to this test class.
  • Quarantine markers (pytest.mark.xfail) — acceptable only as a time-bounded stopgap, not a structural solution.
  • Retroactively converting all timeout values to tier constants — only Channel B tier needs the constant; UNIT/INTEGRATION hardcoded values are fine.
  • Fixing test_recording_sigterm.py structural race — separate issue; the readiness token in _lifespan.py:63 is emitted before the cancellation scope is wired. Different root cause from _process_race.py timing, no overlap with the proposed changes.
  • run_managed_sync changes — it uses bare subprocess.Popen with no Channel B infrastructure. Not affected.

Activity

  1. added this to the Test Path Filtering milestone on Apr 18, 2026
  2. added a commit that references this issue on Apr 18, 2026
    e4ac45d
  3. added
    stagedImplementation staged and waiting for promotion to main
    and removed
    in-progressIssue is being actively worked on
    on Apr 18, 2026
  4. added a commit that references this issue on May 8, 2026
    1079807
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    recipe:implementationRoute through implementation recipestagedImplementation staged and waiting for promotion to main

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions