Playwright instrument: 2 repos write zero traces (retries:0 + on-first-retry), and no repo in the fleet sets globalTimeout

[rubenvdlinde]

Summary

Two Playwright instrument defects, swept across all 20 repos in the fleet that ship a playwright.config.ts. Neither changes a verdict — both change whether you can see why a verdict happened.

1. Two repos write ZERO traces, ever

trace: 'on-first-retry' only produces a trace when a retry happens. A repo with retries: 0 never retries, so it never writes a trace — the config reads as if tracing is configured while producing nothing.

Verified against origin/development, not a local checkout:

repo	retries	trace	traces written
nldesign	0 (explicit, line 18)	on-first-retry	none, ever
scholiq	key absent → Playwright default 0	on-first-retry	none, ever

scholiq is the worse of the two: there is no retries key at all, so nothing in the file hints that tracing is disabled.

This exact pair was found in openregister earlier, where a second defect compounded it (traces written to a path the upload step did not glob), with the result that every trace on disk was of a green run.

For the record, the shared workflow's globs are not a defect — ConductionNL/.github/.github/workflows/quality.yml@main lines 2682-2689 upload both server/apps/<app>/test-results/ and server/apps/<app>/tests/e2e/test-results/, and the report step covers both playwright-report/ locations. Every repo's outputDir is matched. The only defect is the retries/trace pair.

Fleet state of the trace setting:

• retain-on-failure (correct): app-versions, procest
• on-first-retry with retries >= 1 (works, but only traces the retried attempt): decidesk, docudesk, doriath, hermiq, hrmq, larpingapp, nextcloud-app-template, openbuild, opencatalogi, openconnector, openregister, petstore, pipelinq, portaliq, shillinq, softwarecatalog
• on-first-retry with retries == 0 (dead): nldesign, scholiq

Suggested fix: retain-on-failure everywhere. It is strictly more informative than on-first-retry and does not depend on the retry count.

2. No repo in the fleet sets globalTimeout

globalTimeout is absent from all 20 configs.

The shared workflow's playwright job is timeout-minutes: 45 (quality.yml line 2002). When that fires the job is cancelled, which means:

• no verdict — a cancelled job is not a failure and not a pass, and it surfaces as "fail" in gh pr checks while carrying no information;
• no artifacts — the trace upload is if: failure() and the report upload is if: always(), and neither runs on a cancelled job. So the one run you most need to read is the one that leaves nothing behind.

A globalTimeout set inside the config (roughly 38-40 minutes, i.e. comfortably under 45) makes Playwright stop itself and exit with a tally, so the run produces both a count and its artifacts. Runs cancelled at ~45m16s have already been observed in this fleet.

Why this matters more than it looks

Both defects are invisible in exactly the situation they are meant to cover. A green run needs no trace and finishes well inside 45 minutes, so nothing draws attention to either setting until a run goes red or long — at which point the diagnostic you reach for is the one that was never produced.

Filed while burning down E2E failures across pipelinq / decidesk / openbuild / nldesign. The per-repo fixes are going in with those PRs; this issue covers the repos outside that scope, principally scholiq.