Repository navigation
Description
Summary
Implement structural defenses against timing-sensitive async subprocess test flakiness under pytest-xdist. This addresses a recurring class of pipeline aborts (3 documented occurrences: 2026-04-13, 2026-04-16, 2026-04-18) caused by tests with insufficient timeout headroom for xdist -n 4 CPU contention.
Related: #1031 (resolve-failures verdict classification — the pipeline-level fix for the same root cause)
Background
The affected tests share these properties:
- Async subprocess orchestration via
sys.executablewith JSONL output monitoring - A mandatory 1.0s
anyio.move_on_afterpreamble in_process_race.py:250(stdout_session_id_ready) that always expires in tests because test scripts don't emittype=systemrecords - Under xdist
-n 4,asyncio.sleep(1.0)can overrun to 5-10s due to event loop scheduling jitter - Tests with
timeout=60(15x headroom) never flake; tests withtimeout=15(~1.5x headroom) flake under load
The timing budget for a Channel B test: 1s preamble + _phase1_timeout (up to 30s default) + drain + jitter headroom. Tests in test_process_channel_b.py with timeout=60 document this math in inline comments. The flaky test at line 531 uses timeout=15 without this analysis.
Requirements
REQ-FLAKY-001: Fix the immediate flaky test
Change timeout=15 to timeout=60 in test_channel_b_ignores_sub_skill_marker (tests/execution/test_process_channel_b.py:531), matching all sibling tests in TestChannelBSubSkillCollision that go through the same Channel B path.
Add the standard xdist load comment explaining the budget math, consistent with the comments at lines 155-158, 291-296, and 416-428 in the same file.
Also fix 4 latent under-budgeted Channel B tests that use timeout=30 when the budget math requires >= 31.5s:
test_channel_b_wins_then_channel_a_confirms_within_drain(line 132)test_data_confirmed_false_set_on_drain_timeout(line 218)test_channel_b_then_a_empty_result_data_confirmed_is_false(line 266)test_channel_b_drain_timeout_produces_success_skill_result(line 351)
These are latent flake risks per the budget math at lines 155-158 (minimum = 31.5s) and should be migrated to timeout=60.
REQ-FLAKY-002: Centralized timeout tier system
Add a timeout tier system to tests/conftest.py that encodes the budget math for different test classes:
| Tier | Value | Use Case |
|---|---|---|
| UNIT | 10s | Pure logic tests, no I/O |
| INTEGRATION | 30s | Tests with filesystem or subprocess, no Channel B |
| CHANNEL_B | 60s | Tests going through full session_log_dir + Channel B path |
The tier values encode the documented timing budgets from the existing inline comments. The system should be a simple class or module-level constants — not an over-engineered abstraction. Tests should reference the tier constant rather than hardcoding timeout values for Channel B tests where the budget math is non-obvious.
Implementation notes:
- Place in
tests/conftest.pyalongside existing xdist-aware fixtures - Do NOT retroactively convert all existing timeout values — only require tier constants for the Channel B tier where the budget math is non-obvious
- UNIT and INTEGRATION tiers exist for documentation; hardcoded values in those ranges are acceptable
- The previously proposed RECORDING tier (300s) is removed — those values in
test_recording.pyare arguments forwarded to a mock runner, not test wall-clock budgets - Tests that deliberately use short timeouts (0.5-3s) to provoke
TIMED_OUTbehavior are not covered by tiers; their values are intentional and correct
REQ-FLAKY-003: Make stdout_session_id_ready preamble timeout configurable
The _process_race.py:250 hardcoded anyio.move_on_after(1.0) is the primary jitter source in tests. Make this timeout configurable via a parameter to run_managed_async:
- Add
_session_id_timeout: float = 1.0parameter (underscore prefix, consistent with existing_phase1_timeout,_phase1_poll,_phase2_pollparameters) - Tests can pass
_session_id_timeout=0.01to skip the wait cheaply when the test script doesn't emittype=systemrecords - Production default remains
1.0— no behavioral change for non-test callers
This reduces the timing budget for Channel B tests from 1s + phase1 + drain to 0.01s + phase1 + drain, giving tests more headroom within the same outer timeout.
Critical implementation constraints:
_session_id_timeoutMUST be keyword-only with a default value.test_process_monitor.py:1177calls_watch_session_logdirectly with 13 positional args — adding a new positional arg would break that call.- The underscore prefix is load-bearing:
test_req_api_001_public_params_presentintests/contracts/test_protocol_satisfaction.py:321-347filters outk.startswith("_")when checking the public API surface. A non-underscore name would require updating that contract test. run_managed_syncdoes NOT need this parameter — it uses baresubprocess.Popenwith no Channel B infrastructure.- Inline comments at lines 156, 295, 418 of
test_process_channel_b.pycite the 1.0s preamble as part of their budget calculation. These should be updated to reference the default value.
REQ-FLAKY-004: AST guard for Channel B timeout minimums
Add a structural test that scans tests/execution/test_process_channel_b.py (and any future Channel B test files) for run_managed_async( calls and asserts that the associated timeout= value meets the Channel B minimum (60s).
Implementation approach:
- Parse the test file's AST to find
run_managed_asynccall sites - Extract the
timeoutkeyword argument value (timeoutis always keyword-only due to*inrun_managed_asyncsignature, and always a numeric literal in the codebase) - Use a compound heuristic to identify Channel B tests: require
session_log_dir+completion_marker+completion_drain_timeout >= 0.3to all be present - Assert
timeout >= TimeoutTier.CHANNEL_Bfor matching calls - Include an explicit exemption set for known-intentional low-timeout Channel B tests (e.g.,
test_drain_window_times_out_when_no_session_jsonlattimeout=10withcompletion_drain_timeout=0.2) - Assert
timeoutis anast.Constant(literal) — if it becomes a variable, emit a separate violation - Place in
tests/arch/alongside existing AST-based guards (test_anyio_migration.py,test_subpackage_isolation.py)
This prevents future occurrences of the timeout=15 class of bug.
REQ-FLAKY-005: Make test scripts emit synthetic type=system records
Update the test subprocess script literals in tests/execution/test_process_channel_b.py to emit a synthetic type=system JSONL record to stdout at startup. This causes stdout_session_id_ready to fire immediately instead of timing out, eliminating the 1.0s preamble cost entirely.
Example addition to test scripts:
import json, sys
sys.stdout.write(json.dumps({"type": "system", "session_id": "session"}) + "\n")
sys.stdout.flush()Critical: The session_id value MUST be "session" (matching the session.jsonl filename stem used by all Channel B test scripts). Using any other value (e.g., "test-session") causes Phase 1 identity-based selection in _session_log_monitor (_process_monitor.py:200-213) to fail (f.stem == expected_session_id → False), falling back to recency selection with spurious session_id_match_not_found warnings.
Scripts that MUST NOT receive the synthetic record:
CHANNEL_B_NO_STDOUT_SCRIPT—test_channel_b_wins_drain_timeout_still_killsassertsnot result.stdout.strip(). Any stdout output breaks this assertion directly.WRITE_RESULT_THEN_HANG_SCRIPT(conftest) — these tests pass nosession_log_dir, so_extract_stdout_session_idnever runs. Adding a record is pointless.- Inline script in
test_drain_window_times_out_when_no_session_jsonl— this test deliberately has no session JSONL; adding a session ID would change Phase 1 behavior.
Scripts that CAN receive the synthetic record:
CHANNEL_B_THEN_A_CONFIRM_SCRIPTCHANNEL_B_THEN_A_EMPTY_RESULT_SCRIPTPROCESS_EXIT_THEN_CHANNEL_B_FIRES_SCRIPTCHANNEL_B_SUB_SKILL_COLLISION_SCRIPT
REQ-FLAKY-006: Document --dist mode discrepancy
tests/CLAUDE.md documents --dist worksteal but the Taskfile only passes -n 4 (no --dist flag). The actual default is --dist load (round-robin) — xdist's plugin.py:319-320 auto-promotes dist="no" to dist="load" when -n N > 0. Research found that worksteal has open bugs:
- Hang at high core counts (xdist issue franchise: doctor checks (9 total — ambient env detection + infrastructure health) #884, open)
xdist_groupmarks not supported (issue franchise: release hygiene (version bump + hardcoded test strings + sync-plugin-version) #890)- Test ordering non-determinism (issue Promote integration to main (198 PRs, 182 issues, 102 fixes, 107 features, 1 infra, 7 tests, 4 docs) #925)
Update tests/CLAUDE.md to document the actual --dist mode in use (load, the xdist default when -n 4 is given without --dist). Also update audit-tests/SKILL.md line 179 which repeats the same --dist worksteal claim. If worksteal was intended, add a note about the known issues and a rationale for the choice.
Files to Modify
tests/execution/test_process_channel_b.py— REQ-FLAKY-001 (timeout fix for 5 tests), REQ-FLAKY-005 (synthetic system records for 4 scripts)tests/conftest.py— REQ-FLAKY-002 (timeout tier system)src/autoskillit/execution/_process_race.py— REQ-FLAKY-003 (configurable_session_id_timeout, keyword-only)src/autoskillit/execution/process.py— REQ-FLAKY-003 (thread parameter throughrun_managed_asynconly, NOTrun_managed_sync)tests/arch/test_channel_b_timeout_guard.py(new) — REQ-FLAKY-004 (AST guard with compound heuristic + exemption set)tests/CLAUDE.md— REQ-FLAKY-006 (dist mode documentation).claude/skills/audit-tests/SKILL.md— REQ-FLAKY-006 (dist mode reference)
Out of Scope
- Retry plugins (
pytest-rerunfailures,pytest-retry) — masks the problem without fixing it. The codebase intentionally has no retry infrastructure. - Fake clock libraries (
looptime) — only affects event loop time, not real subprocesstime.sleep()calls. Not applicable to this test class. - Quarantine markers (
pytest.mark.xfail) — acceptable only as a time-bounded stopgap, not a structural solution. - Retroactively converting all timeout values to tier constants — only Channel B tier needs the constant; UNIT/INTEGRATION hardcoded values are fine.
- Fixing
test_recording_sigterm.pystructural race — separate issue; the readiness token in_lifespan.py:63is emitted before the cancellation scope is wired. Different root cause from_process_race.pytiming, no overlap with the proposed changes. run_managed_syncchanges — it uses baresubprocess.Popenwith no Channel B infrastructure. Not affected.
Activity
Implementation Plan: Flaky Test Architectural Immunity — Centralized …
Implementation Plan: Flaky Test Architectural Immunity — Centralized …