Flaky: chat.spec.ts › displays streaming response passes only on a retry in E2E (stub ACP backend, offline)

Flake ledger entry.

Test: test/test_playwright_e2e.py::test_dashboard_playwright_suite — Playwright spec website/playwright/chat.spec.ts:62 › Chat Page E2E Tests › displays streaming response (@needs-agent)
OS: ubuntu-latest (ubuntu-24.04 runner image 20261004.327), job E2E (stub ACP backend, offline)
Run: https://github.com/kirodotdev/KiroCrew/actions/runs/37669843004/job/112967280075 (on PR feat: add a stacked spend-over-time chart to the usage tab #17159)
Failing line: chat.spec.ts:69 — expect(page.locator('.msg-content').first()).toBeVisible({ timeout: 5000 }) timed out on the first attempt and passed on the retry, so the MAX_FLAKY_SPECS=0 ceiling failed the job: 1 spec(s) passed only on a retry, ceiling is 0: chat.spec.ts > Chat Page E2E Tests > displays streaming response (322 passed, 1 flaky, 15.7m)

PR #17159 adds a usage chart to the Settings › Usage tab and touches no chat, message or streaming code. The other completed runs of this job today passed (the one other red today was an apt mirror timeout installing lsof, not this spec). Shape: the first streamed message's .msg-content is not visible within 5 s under runner load, then renders fine on the retry.

Seen again on PR #17847, run 37706201811, job 113081986965 (ubuntu, E2E stub ACP backend, offline): AssertionError: 1 spec(s) passed only on a retry, ceiling is 0: chat.spec.ts > Chat Page E2E Tests > displays streaming response at test/test_playwright_e2e.py:133. That PR changes only a backend config warning.

Seen again on #17922 (head 351721e, a change with no website or chat diff): run 37709918003, job 113094206364.
OS: ubuntu (E2E stub ACP backend, offline). Failing line: expect(locator).toBeVisible() failed in chat.spec.ts > Chat Page E2E Tests > displays streaming response, passed on retry; the 0-flaky ceiling failed the job.

Seen again on #17930 (head 6a4152e): run 37711645441, job 113099766980 (ubuntu, E2E stub ACP backend, offline): chat.spec.ts:69 .msg-content stayed hidden on the first attempt and passed on the retry, tripping the zero-flaky ceiling. The same spec passed on this PR's previous head.

Another occurrence, same shape.

Run: https://github.com/kirodotdev/KiroCrew/actions/runs/37713444396/job/113107421314 (on PR fix(stt): end dictation when Transcribe drops the stream #17925)
OS: ubuntu-latest, job E2E (stub ACP backend, offline)
Failing line: chat.spec.ts:69 — expect(locator).toBeVisible() timed out on the first attempt and passed on the retry; the MAX_FLAKY_SPECS=0 ceiling failed the job (322 passed, 1 flaky, 10.2m)

PR #17925 changes only the Transcribe branch of dashboard/stt_stream.py and its tests; it touches no chat, message or streaming-render code.

Seen again on #17828 (head 6d98123, a backend-test-only change): run 37714471308, job 113108622440. Same shape: chat.spec.ts:69 .msg-content not visible within 5000 ms on the first attempt, passed on the retry, so the zero-flaky ceiling failed the job. A fix for the spec itself is being prepared as a dedicated PR.

Root cause and a candidate fix for the spec PR, in case it helps.

Verified on main at caf10f9. page.locator('.msg-content').first() matches the oldest bubble in the transcript, which is the user prompt this test just sent, not the reply. When the new turn scrolls that prompt above the fold, ChatPage.tsx makes it the pinned-prompt stand-in and hides the real row with visibility: hidden by design. Playwright then reports Received: hidden while the reply is already rendered. The assertion passes or fails on scroll timing and never looks at the response.

What removed the flake locally was waiting for this attempt's own reply:

a prompt unique to each attempt, built from testInfo, so a reply left in the shared slot by an earlier spec, an earlier --repeat-each copy or a failed first attempt cannot satisfy the wait;
a wait for a [data-role="assistant"] row after that prompt's [data-display-index] row, containing the stub's reply text;
a named ceiling for a lost turn in place of the bare 5 s.

const STUB_REPLY = 'pong from the fake ACP backend' // REPLY_TEXT in fake_acp_backend.py
const STUB_TURN_CEILING_MS = 20000 // cold first turn measured ~8.2 s, warm ~0.7 s

test('displays streaming response', async ({ page }, testInfo) => {
  const prompt = `Hello ${testInfo.testId}-${testInfo.repeatEachIndex}-${testInfo.retry}`
  const messageInput = page.getByPlaceholder(/message/i)
  await messageInput.fill(prompt)
  await page.keyboard.press('Enter')

  const reply = page
    .locator(`[data-display-index]:has-text(${JSON.stringify(prompt)}) ~ [data-display-index] [data-role="assistant"]`)
    .filter({ hasText: STUB_REPLY })
  await expect(reply.last()).toBeVisible({ timeout: STUB_TURN_CEILING_MS })
})

Local results against the same stub ACP gateway the E2E job uses, --repeat-each=30:

current spec: failed 1 of 30 at 1 worker, 1 of 30 at 1 worker pinned to 1 CPU, and 2 of 30 at 2 workers pinned to 1 CPU. Every failure was Received: hidden on the user bubble.
spec above: 30 of 30 at 1 worker, at 4 workers, and at 1 worker pinned to 1 CPU. The full chat.spec.ts file also passed 12 of 12.
not yet run on the final version: 2 workers pinned to 1 CPU. An earlier draft hit the 30 s test timeout once in that combination, so that one is still open.

Two more sightings on PRs with no chat or website diff, both Received: hidden on the first attempt and passing on the retry:

Happy to share the full patch if it is useful for your PR.

Seen again on #17930 (head ca95faf): run 37717678564, job 113118857264, ubuntu, E2E stub ACP backend, offline: chat.spec.ts:69 passed only on the retry. Second time on this PR, on two different heads.

Seen again on #17973 (head 486f3de, a docs-only one-docstring change in shell_audit_log.py — cannot affect a frontend chat spec).

Run: https://github.com/kirodotdev/KiroCrew/actions/runs/37722132542/job/113132896921
OS: ubuntu (CodeBuild us-east-1), lane E2E (stub ACP backend, offline)
Same shape: chat.spec.ts › Chat Page E2E Tests › displays streaming response passed only on a retry; flaky_count=1 trips the MAX_FLAKY_SPECS ceiling (0) in test_playwright_e2e.py:133. Underlying: expect(page.locator('.msg-content').first()).toBeVisible({ timeout: 5000 }) timed out at chat.spec.ts:69.

Rerunning the failed job once per the flake protocol.

Seen again on PR #18016 (diff touches only i18n labels and the title control's spinner text, not chat streaming): https://github.com/kirodotdev/KiroCrew/actions/runs/37741961046/job/113195711667 — ubuntu, E2E (stub ACP backend, offline). Failing line: playwright/chat.spec.ts:69 expect(page.locator('.msg-content').first()).toBeVisible({ timeout: 5000 }), passed on retry; the flaky ceiling (0) failed the job.

Thanks for the report. This issue has been triaged by Crew Captain and will be assigned to a maintainer, who will follow up here.

test(e2e): give chat.spec's streaming check its own slot

test(e2e): give chat.spec's streaming check its own slot (#18038)

Navigation Menu

Repository navigation

Flaky: chat.spec.ts › displays streaming response passes only on a retry in E2E (stub ACP backend, offline) #17857

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Flaky: chat.spec.ts › displays streaming response passes only on a retry in E2E (stub ACP backend, offline) #17857

Description

Activity

bolichen97 commented on Oct 8, 2026

bolichen97 commented on Oct 8, 2026

bolichen97 commented on Oct 8, 2026

amadsalmon commented on Oct 8, 2026

bolichen97 commented on Oct 8, 2026

jeeshofone commented on Oct 8, 2026

bolichen97 commented on Oct 8, 2026

cruisercohen commented on Oct 8, 2026

bolichen97 commented on Oct 8, 2026

KiroCrewCaptain commented on Oct 8, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions