tests(session-control): test_the_inter_stage_append_persists_before_returning_success flakes on Windows (executor hop outruns 20 zero-sleeps)

Summary

test/test_session_control.py::test_the_inter_stage_append_persists_before_returning_success failed on the Windows backend shard of an unrelated PR head with:

AssertionError: the immediate queue write must have been started for the slot whose queue now holds the acknowledged prompt: []

The test starts an executor-backed queue write via send_to_target and then polls asyncio.sleep(0) twenty times for flush_slot_now to have been called. On a loaded Windows runner the executor hop does not land within twenty loop iterations, so the (correct) write is reported as never started. Passes deterministically on Linux locally.

Evidence

Head: fix: inherit per-chat trust to subagents at every spawn depth #13109 at d80cadd -- Backend Tests (Windows) (2); the PR touches nothing under session_control.
The module's other tests in the same shard passed.

Suggested fix

Await the flush with a bounded real-time wait (e.g. poll with asyncio.sleep(0.01) up to ~1 s) instead of a fixed count of zero-sleeps, so the assertion is about behaviour rather than executor latency.

Automated triage summary — generated from the issue text alone, before any maintainer has read it. Treat every line as a starting point, not a verdict.

A session-control test that polls twenty asyncio.sleep(0) iterations for an executor-backed queue write to start fails intermittently on the Windows backend test shard, asserting the write was never started even though it is in flight. Seen on a loaded runner during an unrelated PR head; passes on Linux locally.

Worth adding

how often this has recurred (a link to more than one failing Windows run, if you have it)
the full pytest output or log excerpt around the failure, beyond the assertion line

Possibly already reported

test_diag_procs: the too-soon test leaves a 0.2s margin against a 1.0s floor, so a slow Windows runner fails it deterministically #14183 (test_diag_procs: the too-soon test leaves a 0.2s margin against a 1.0s floor, so a slow Windows runner fails it determin)
tests(windows): test_a_read_refused_as_too_soon_does_not_restart_the_window reads the real clock and flakes on slow runners #13886 (tests(windows): test_a_read_refused_as_too_soon_does_not_restart_the_window reads the real clock and flakes on slow runn)
tests(windows): test_diag_procs refused-read test lets its second scan read the real clock, red on two main pushes #13997 (tests(windows): test_diag_procs refused-read test lets its second scan read the real clock, red on two main pushes)

If one of these is the same problem, say so and we will close this as a duplicate.

_{Posted by issue-summary.yml. The model had no access to the repository — it read only this issue and the titles of recent ones.}

🤖 Kiro Crew Auto-Pipeline [operator: NicholasRBowers#a942f9ca]: Verified this against main at f6d8b7f and the diagnosis holds exactly. test/test_session_control.py:5650-5651 polls for _ in range(20): await asyncio.sleep(0), and the test's own comment immediately above the loop says "Started, not awaited, and it runs in an executor" — twenty zero-sleeps yield to the event loop but advance no real time, so a thread hop on a loaded runner can outlast the poll and a correct write is reported as never started. The assertion text and the empty [] in the traceback match that line byte-for-byte.

Scope: this is the only fixed-count zero-sleep poll in the module. The other asyncio.sleep(0) calls in test_session_control.py are bare single or double yields rather than waits on an executor, so the fix is one site.

Two notes for whoever picks this up, because the suggested fix already exists in-tree rather than needing to be invented:

test/test_dashboard_cron_run_guard.py:156 defines _wait_until(predicate, *, timeout=5.0) as a time.monotonic() deadline around await asyncio.sleep(0.01), raising AssertionError on expiry. That is the suggested shape exactly, and test/test_app_dev_mode.py:451 carries a second copy of the same idiom.
test_session_control.py does not currently import time, so a deadline-based wait needs that import added.

On the cross-reference: the PR #13109 link on this issue is the CI head whose Windows shard produced the failure, not a fix for it. It declares no issues and none of its 32 files are under session_control, which matches the report's own note that the PR touches nothing there.

Leaving this in needs-triage deliberately rather than relabelling it: this instance's dispatchers are paused, and moving it to auto-fixable would drop it out of other operators' candidate queries while it sits here unfixed.

🤖 Kiro Crew Auto-Pipeline [operator: CrysisDeu#0c98c3a2]: Confirmed on main. start_queue_persist (src/kiro_crew/dashboard/chat_delivery.py:898) hands flush_slot_now to loop.run_in_executor, so the fake flush the test installs is called from a worker thread after a real scheduling hop. The test waits for it with twenty asyncio.sleep(0) iterations, which is a fixed number of loop turns, not a bounded amount of time; on a loaded Windows runner the thread has not run when the loop has spun twenty times, and the correct write is reported as never started. Test-only fix: replace the count loop with the bounded real-time wait the same file already uses (deadline = time.monotonic() + 5, poll with await asyncio.sleep(0.01) until flushed is non-empty), so the assertion is about the write starting rather than executor latency. Starting an automated fix — this issue is now claimed by CrysisDeu#0c98c3a2's instance.

Second instance of this mechanism, different test and module

test/test_spawn_approval_channel_delivery_2381.py::TestASilentSendFailureFallsThroughAtOnce::test_a_real_message_id_still_arms_the_wait_and_honors_the_press failed on Backend Tests (Windows) (6) of an unrelated PR head with:

>       assert asyncio.run(_go()) is True
E       assert False is True

Recording it here rather than as a new issue because the mechanism is the one this issue names — a test that bounds an asynchronous hand-off by a fixed budget, failing on a loaded Windows runner while the code under test is correct. The budget style differs: this issue's test polls a fixed count of asyncio.sleep(0), mine polls 50 × asyncio.sleep(0.01), so 500 ms of real time rather than a count of loop iterations. Same shape, larger budget, still not enough.

Whoever picks this up decides whether to widen this issue to the class or split mine out; the two tests share no module.

Why it is a budget shortfall and not a logic break

The test starts deliver_spawn_approval, polls up to 500 ms for cli.sent, then presses. When the send has not landed inside that budget the press arrives before the waiter future is armed, the press is lost, and the gate returns something other than True — the window #13578 describes, reached here by runner latency rather than by a missing lifecycle.

Evidence:

The file hashes to shard 6 under scripts/ci_file_shards.py (sha256(path)[:8] % 8 + 1), so it ran on the failing shard.
The same shard concluded success on the immediately preceding head of the same PR, and neither the test nor src/kiro_crew/messaging/spawn_approval_delivery.py changed on main between the two heads. Same code, opposite result.
Backend Tests (3.12, *) was 8/8 on the failing head, so Linux did not reproduce it.
Backend Tests (Windows) (6) is success right now on three other open PRs.
The shard reported 1 failed, 16348 passed, so the session completed and no other test in that class failed.

Not the rename-contention family

Checked explicitly, because the prior Windows flake on this PR was that family: the failing shard's log contains zero occurrences of atomic_write, WinError or Access is denied, against a control term from the same log that returned 3. So this is not #14107 / #14212.

Not #14060 either

#14060 reports a sibling test in the same class (test_it_registers_no_wait_so_the_window_is_never_spent) failing on pristine main with assert ['wait'] == [] — a seam-versus-expectation disagreement, deterministic and on both platforms. That is a different failure, and main has since landed #14056 over that file. Mine is Windows-only, non-deterministic, and asserts the opposite side of the branch.

Suggested fix

The same remedy this issue proposes, applied to the press path: wait for cli.sent on a budget sized for a loaded runner, or arm the waiter before the prompt is dispatched so the press cannot precede it. The second removes the timing dependence rather than enlarging it.

Thanks for the report. This issue has been triaged by Crew Captain and will be assigned to a maintainer, who will follow up here.

Seen again on Windows, on PR #23939 (head 4e2b10e). That PR changes only one docstring, in src/kiro_crew/apps/builtins/issue_radar/backend/http_routes/pr_actions.py.

Run: https://github.com/kirodotdev/KiroCrew/actions/runs/40181401789/job/121135467970
OS: Windows (Backend Tests (Windows) (7))
Shard result: 1 failed, 22462 passed

FAILED test/test_session_control_steer_delivery.py::test_the_inter_stage_append_persists_before_returning_success - AssertionError: the immediate queue write must have been started for the slot whose queue now holds the acknowledged prompt: [<kiro_crew.dashboard.state_parts.chat_slot._ChatSlot objec

The failed job gets one rerun.

🤖 AI-assisted comment 🤖

Another hit, this time on Linux. PR #23961 (head 3c1e063, which touches only apps/registry_pipeline/manifests.py). Run: https://github.com/kirodotdev/KiroCrew/actions/runs/40181405024/job/121136036687. OS: ubuntu-latest, Python 3.12, Backend Tests (3.12, 7). Failing line: test/test_session_control_steer_delivery.py:495, AssertionError: the immediate queue write must have been started for the slot whose queue now holds the acknowledged prompt.

Seen again on an unrelated PR (#23845), whose diff does not touch session_control / steer / queue code. test/test_session_control_steer_delivery.py::test_the_inter_stage_append_persists_before_returning_success (line 533) failed on Backend Tests (Windows, 7) and failed again on the rerun, with:

AssertionError: the immediate queue write must have been started for the slot whose queue now holds the acknowledged prompt: []

Seen again on PR #23976 (test-only change, does not touch this file): https://github.com/kirodotdev/KiroCrew/actions/runs/40181410460/job/121135766938 at 9434ca0e95. Rerunning the failed job.

Seen again on Windows: https://github.com/kirodotdev/KiroCrew/actions/runs/40183859530/job/121142783126 (PR #23979, head 6c5dabd, Backend Tests (Windows) (7)). Failing line: AssertionError: the immediate queue write must have been started for the slot whose queue now holds the acknowledged prompt. PR #23979 touches only pi fixtures and the pi pin gate, nothing in session control.

Seen again on Linux, not only Windows: job 121153737811, Backend Tests (3.12, 7) on PR #17624 at 0557fde, 2026-10-25. It failed with AssertionError: the immediate queue write must have been started for the slot whose queue now holds the acknowledgement. 1 failed, 24,295 passed in that shard. The PR does not touch session control, and the file passes 13 of 13 locally on the same commit.

[operator: bolichen97] Seen again on Windows: https://github.com/kirodotdev/KiroCrew/actions/runs/40186898636/job/121152148029 (Backend Tests (Windows) shard 7/8, PR #23998, head ed6eb95, a frontend-only diff).

Failing line: AssertionError: the immediate queue write must have been started for the slot whose queue now holds the acknowledged prompt: [] in test/test_session_control_steer_delivery.py::test_the_inter_stage_append_persists_before_returning_success.

Another Windows report of this flake.

Run: https://github.com/kirodotdev/KiroCrew/actions/runs/40192216031/job/121168222233
OS: Windows (Backend Tests (Windows) shard 7/8)
PR: test(ui): accept either path separator in the auto-out refusal test #24004, head 42c38f5. This PR changes only one frontend test regex (website/src/uiLocations/uiIndex.test.ts), with no backend code.
Failing line: FAILED test/test_session_control_steer_delivery.py::test_the_inter_stage_append_persists_before_returning_success - AssertionError: the immediate queue write must have been started for the slot whose queue now holds the acknowledged prompt: []
Shard result: 1 failed, 22462 passed.

I will rerun the failed job once.

🤖 AI-assisted comment 🤖

Seen on Linux too, fork PR #9749 (40dd276862), Backend Tests (3.12, 7) (https://github.com/kirodotdev/KiroCrew/actions/runs/40189994603/job/121166002351): test_the_inter_stage_append_persists_before_returning_success - AssertionError: the immediate queue write must have been started ...: []. The PR touches no session-control code.

Flake ledger: seen again on Windows.

Run: https://github.com/kirodotdev/KiroCrew/actions/runs/40203548597/job/121202162854 (Backend Tests (Windows) (7), PR fix(chat): let Escape cancel an in-flight prompt optimize #24035, head fd667a63298a)
OS: Windows (CodeBuild runner, Python 3.12.10, xdist worker gw0)
Failing line: test\test_session_control_steer_delivery.py:533: AssertionError: the immediate queue write must have been started for the slot whose queue now holds the acknowledged prompt: []

The PR changes only frontend files (website/src/...), so it does not touch this path. The 20 asyncio.sleep(0) yields ran out before the executor write started, which is the shape this issue describes. Rerunning the failed job once.

🤖 AI-assisted comment 🤖

Navigation Menu

Repository navigation

tests(session-control): test_the_inter_stage_append_persists_before_returning_success flakes on Windows (executor hop outruns 20 zero-sleeps) #14218

Summary

Evidence

Suggested fix

Second instance of this mechanism, different test and module

Why it is a budget shortfall and not a logic break

Not the rename-contention family

Not #14060 either

Suggested fix

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

tests(session-control): test_the_inter_stage_append_persists_before_returning_success flakes on Windows (executor hop outruns 20 zero-sleeps) #14218

Description

Summary

Evidence

Suggested fix

Activity

github-actions commented on Sep 26, 2026

NicholasRBowers commented on Sep 26, 2026

CrysisDeu commented on Sep 26, 2026

chenmingwei23 commented on Sep 26, 2026

Second instance of this mechanism, different test and module

Why it is a budget shortfall and not a logic break

Not the rename-contention family

Not #14060 either

Suggested fix

KiroCrewCaptain commented on Oct 8, 2026

iamwhatever commented on Oct 25, 2026

IngridMorstrad commented on Oct 25, 2026

chenmingwei23 commented on Oct 25, 2026

iamwhatever commented on Oct 25, 2026

iamwhatever commented on Oct 25, 2026

Pearcekieser commented on Oct 25, 2026

bolichen97 commented on Oct 25, 2026

iamwhatever commented on Oct 25, 2026

amadsalmon commented on Oct 25, 2026

iamwhatever commented on Oct 25, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions