E2E is not a required check anywhere: 0 of 20 fleet repos gate on Playwright, and 17 have no branch protection at all

[rubenvdlinde]

Summary

The stated goal is "everything tested through Playwright E2E". The wiring is done — 20 fleet repos ship a playwright.config.ts and run E2E Tests (Playwright) in the shared Code Quality workflow. Not one of them requires that job to pass before a merge.

A pull request whose E2E job is red merges into development today in every repo in the fleet, and into main and beta in all but three.

This is deliberate-looking enough that it needs a decision rather than a silent fix, so I am filing it instead of changing protection.

Measured

GET /repos/ConductionNL/<repo>/branches/development/protection, all 20 repos that ship a Playwright config:

repo	required checks on development
nldesign	PHP Quality, CSS Quality, Branch Policy Check
pipelinq	PHP Quality, Frontend Quality, Branch Policy Check
procest	PHP Quality, Frontend Quality, Branch Policy Check
app-versions, decidesk, docudesk, doriath, hermiq, hrmq, larpingapp, nextcloud-app-template, openbuild, opencatalogi, openconnector, openregister, petstore, portaliq, scholiq, shillinq, softwarecatalog	branch not protected — no required checks at all

• 0 of 20 require E2E Tests (Playwright).
• 17 of 20 have no branch protection on development whatsoever.

main and beta are the same picture. Spot-checked across the four repos currently being burned down:

repo	main	beta
pipelinq	PHP Quality, Frontend Quality, Branch Policy Check	same
nldesign	PHP Quality, CSS Quality, Branch Policy Check	same
decidesk	not protected	not protected
openbuild	not protected	not protected

Where protection does exist it requires the static-analysis jobs and the branch-policy check, and never E2E.

Why this matters right now

Four repos are actively red on E2E as of 2026-08-06 development:

repo	passed	failed	skipped	did not run
decidesk	56	53	62	0
pipelinq	140	31	40	1
openbuild	168	5	76	0
nldesign	97	2	1	10

Those failures reached development because nothing stopped them. The suites are doing their job — they are finding real defects — but the finding has no consequence, so the red state persists and accumulates. A test suite that cannot block a merge is a report, not a gate.

The decision, stated honestly

Turning E2E into a required check today would block every merge in four repos, because those suites are genuinely red. That is not a reason to skip the decision, but it does mean sequencing matters:

1. Burn the suites to green (in progress).
2. Then make E2E Tests (Playwright) a required status check on development, per repo, as each one reaches green — so no repo is blocked on another repo's debt.
3. Promote to main/beta once development has held green for a while.

Open questions that need a human:

• Is the absence of protection on 17 repos intentional? The fleet runs agent-driven merges, and --admin bypasses required checks anyway, so protection may have been dropped on purpose to keep automation unblocked. If so, the honest fix is different: keep protection off, but make a red E2E block the automation rather than the branch.
• Should the skipped counts gate too? decidesk is skipping 62 of 171 and openbuild 76 of 249. A suite can go "green" by skipping everything, so a required check on failures alone does not deliver the stated goal. A skip ratio ceiling may be the more meaningful gate.
• did not run is a third state. nldesign left 10 tests with no verdict. Whatever gate is chosen must treat "not measured" as not-passing.

Related

• Playwright instrument: 2 repos write zero traces (retries:0 + on-first-retry), and no repo in the fleet sets globalTimeout #188 — Playwright trace and globalTimeout instrument defects (2 repos write zero traces; no repo bounds the run under the 45-minute job timeout, so a cancelled run leaves no artifacts and no tally).
• playwright-coverage-threshold has never gated: warning-only, not line coverage, and zero scenarios scores 100% #189 — playwright-coverage-threshold has never gated: warning-only, not line coverage, and zero scenarios scores 100%.

Together those three are the same shape: the E2E stack is fully built and produces no enforceable consequence at any layer — not the coverage threshold, not the trace artifacts, not the merge.