fix(e2e): traces that are never written, a cascade that eats 10 verdicts, and two specs measuring the wrong thing

[rubenvdlinde]

Burndown against the measured baseline of run 31086399980 (job 92566906706, development): 97 passed / 2 failed / 1 skipped / 10 did not run out of 110.

The two reported failures were both real, and neither was the failure it named. The ten no-verdicts were the larger problem.

1. Zero traces have ever been written in this repo

playwright.config.ts paired retries: 0 with trace: 'on-first-retry'. There is no first retry when there are no retries, so the trace file is never produced — the shared workflow's if: failure() upload has been faithfully uploading a screenshot and an error-context.md and no trace at all. Every failure in this repo's history was debugged blind.

Fixed to retain-on-failure, which is the setting that matches retries: 0. (Raising retries would also have produced traces, at the price of letting a flake pass on the second attempt — the opposite of what this suite is for.)

2. A cancelled job is not a verdict

The shared Playwright job caps at timeout-minutes: 45. A job killed by that cap prints no tally and uploads nothing. globalTimeout: 38m inside the config makes Playwright exit on its own clock first, so the tally and the artifacts always exist. The full suite runs in 5.9m, so this is roughly 6x headroom and cannot mask a regression — it can only turn a silent cancellation into a reported timeout.

3. The cascade: one failure, ten missing verdicts

lasuite-parity declared test.describe.configure({ mode: 'serial' }), with the stated rationale that "a flake isolates to one element, not the whole suite". Serial mode does the opposite of that: when one test in a serial group fails, Playwright abandons every later test in the group. On the baseline run the lasuite: header app name row timed out and took ten further tests down with it — three remaining lasuite rows, all six cunningham rows, and the unified-search modal check. They reported did not run: no pass, no fail, no verdict, and nothing in the summary distinguishing them from tests nobody has written yet.

The isolation the comment wanted comes from the one-test-per-element split, which is orthogonal and is kept. Nothing in the block depends on a previous test — every test navigates, reads its own CSRF token, sets its own token set and reloads — and workers: 1 / fullyParallel: false mean removing the mode introduces no concurrency whatsoever. The tests still run one at a time in the same order. The only change is that a failure stops being contagious.

4. Both failing specs were measuring something other than what they named

lasuite: header app name — written against NC34, executed against stable31

The row waited 15s for #header .app-menu__current-app-name, on the grounds that "Nextcloud 34 renders the current app's name" that way. True of NC34. This job installs nextcloud-test-refs[0], which defaults to stable31 — and app-menu__current-app-name appears in no stable31 or stable32 source file. The current-app button is a 34-era addition. The run's own page snapshot confirms it: stable31's header is an icon-only app list with no app-name text anywhere.

So the row could only ever time out, and it reported that timeout under the name "header app name matches the Cunningham reference" — which reads like a parity regression. This is the third selector in this row's history to blow the same 15s timeout for the same reason, and the first two were diagnosed as wrong selectors rather than as a version mismatch.

The declaration block under test (element-overrides.css:151-153) covers three selectors, and one of them is live on stable31: #header .header-start .header-appname, in core/templates/layout.public.php. Same block, same declared values. The row now measures there, on a public link share the test creates itself — public is one of CssInjectionService's five valid render contexts, so the theme is injected. The share is created inside the test rather than in beforeAll on purpose: a throwing beforeAll fails every test in the describe, so a transient sharing hiccup would have reddened all thirteen rows and told you nothing about twelve of them.

selector-liveness — nobody was activating the theme it measures

The file's docblock says "Requires the lasuite set to be active". Nothing ever activated it. The CI seed sets rijkshuisstijl; the only spec that switches to lasuite is lasuite-parity, which runs earlier alphabetically and dutifully restores the baseline in its afterAll.

Its two tests failed differently on that, and the difference is the lesson:

• The selector sweep is fail-closed, so it skipped, printing lasuite element-overrides.css was not served on any surface. That is one line in a 110-test summary and renders as 1 skipped. This repo's single most valuable guard — the one written because five consecutive defects were all dead selectors — had never once executed.
• The geometry lock has no such guard, so it measured stock Nextcloud chrome and asserted La Suite's full-bleed shell against it. It reported x = 8: Nextcloud's own 8px inset, correctly present on a page the theme was not applied to. The failure named a theming regression that did not exist. The failure screenshot from the baseline run is stock Nextcloud blue, with the blue frame plainly visible down the left edge.

A guard that skips on a fixture nobody built and a guard that fails on one are the same bug wearing two faces. Both are fixed by building the fixture where the requirement is stated: snapshot the active set, activate lasuite, restore afterwards — the same pattern lasuite-parity already uses. The fail-closed skip is deliberately kept; if it fires again it means this hook broke, and that must stay visible rather than silently becoming a green sweep over zero selectors.

Expect this to surface new failures. With the sweep actually running, #header .app-menu__current-app-name and #header .app-menu__current-app .button-vue__text are dead on stable31 and are not in ALLOWED. That is the guard doing its job for the first time, and each entry it reports needs a stated reason rather than a silent pass.

5. tsc could not typecheck a browser test suite

tsconfig.json declared lib: ["ES2020"] with no DOM, so every document, window, getComputedStyle and CSSStyleRule in tests/e2e/ was an error — 27 of them, all the missing lib. With DOM added the entire suite typechecks clean, after typing three implicitly-any page parameters in app-theming.spec.ts. That makes npx tsc --noEmit a usable positive control on spec code, which it was not before.

Not included

No .skip, no test.fixme, no continue-on-error, no relaxed assertion, no deleted spec, and no raised timeout. The only timeout added is globalTimeout, which is a ceiling rather than a budget.

---

Added after the first CI cycle

6. A dev-only CVE had switched the entire test tier off

The first run of this branch (31096112500) reported:

quality / Security (composer)      failure
quality / E2E Tests (Playwright)   SKIPPED
quality / PHPUnit                  SKIPPED
quality / Integration (Newman)     SKIPPED

CVE-2026-67434 / GHSA-hmqg-cxww-wqhq in squizlabs/php_codesniffer was added to roave/security-advisories dev-latest this morning, between the baseline run (31086399980, 08:49Z, security green, 110 tests executed) and that one (11:18Z). The shared workflow gates the test tier on needs.security.result != 'failure', so a CVE in a dev-only code formatter — one that never runs in a browser, never ships, and cannot touch a rendered page — took E2E, PHPUnit and Newman down with it.

A skipped job renders in the checks list as a grey tick, not a red X, and the Quality Report aggregates it as "not failed". So the repo's whole test tier went dark and nothing in the run summary said so. That is the permanently-pending dead-gate shape: present on every run, executing on none.

Upstream published the patch the same morning (3.13.6, 01:52Z) and the constraint here is already ^3.9, so only the lock moved. composer audit after the bump: No security vulnerability advisories found.

Reported rather than worked around: the bump is the real fix, but the coupling deserves its own look. "A CVE exists in a dev tool" and "the tests cannot be trusted to run" are not the same statement.

7. The unified-search modal check has never once asserted anything

#header .unified-search__button matches nothing on any supported Nextcloud. From core/src/views/UnifiedSearch.vue at each ref:

ref	markup
stable31	<div class="header-menu unified-search-menu"> then <NcButton class="header-menu__trigger">
stable32	<div class="unified-search-menu"> then <NcHeaderButton id="unified-search">
NC34	identical to stable32

So the count was always 0 and the test skipped with a message that reads like a benign version quirk — "unified-search trigger not present in header on this NC version" — while the truth is a locator that is wrong on every release. The modal border-radius assertion it exists to make has never run. The locator now covers all three spellings, scoped to the search menu rather than a page-wide role query, and the fail-closed skip is kept for a Nextcloud that genuinely ships no trigger.

The dead spelling came from this app's own CSS. #header .unified-search__button is a live rule in element-overrides.css that styles nothing, and selector-liveness's ALLOWED list currently excuses it as a "pre-Vue header class retained as fallback for older Nextcloud releases" — which is backwards. It is not an old class; it is not any class. Left in place deliberately so the sweep, now that it actually runs, reports it on its own evidence rather than on mine.

The systemic finding underneath all of this

css/systems/lasuite/element-overrides.css was authored against Nextcloud 34, and this repo's e2e installs stable31. The header block alone contains app-menu__current-app-name, app-menu__current-app .button-vue__text, unified-search__button, unified-search-input, unified-search-input__button, unified-search-input__label and unified-search__input — every one of them 34-era or invented, none present on the version under test. The La Suite header treatment is largely inert on the Nextcloud this suite actually exercises.

That is a product decision, not a test fix, and it is left honestly visible: reconciling the theme against stable31 needs a design round with live measurement on both versions, not a selector search-and-replace. The selector-liveness sweep — running for the first time in this PR — is the instrument that will enumerate it precisely.

Mutation proof

Each repaired assertion was proven able to fail, on a throwaway branch carrying three deliberate defects in element-overrides.css:

mutation	expected failure	proves
header app name font-weight 700 to 400	lasuite: header app name — font-weight: expected 700, got 400	the row reads the shipped rule on the public-share surface
content shell inset 0 to 17px	selector-liveness geometry — x = 17	the geometry lock measures the activated theme (17 is deliberately not 8, the stock inset the unfixed spec reported)
a selector matching nothing anywhere	the sweep reports it under unexplainedDead	the sweep runs — it had never executed once in this repo's history

[github-actions]

Quality Report — ConductionNL/nldesign @ e3c1fc3

Check	PHP	Vue	Security	License	Tests
lint	✅
phpcs	✅
phpmd	✅
psalm	✅
phpstan	✅
phpmetrics	✅
eslint		✅
stylelint		✅
build		✅
check-manifest		✅
test-l10n		✅
composer			❌	✅ 100/100
npm			✅	✅ 2/2
PHPUnit					⏭️
Newman					⏭️
Playwright					⏭️
Hydra gates					❌

---

Quality workflow — 2026-08-06 11:26 UTC

Download the full PDF report from the workflow artifacts.

[github-actions]

Quality Report — ConductionNL/nldesign @ bebffa8

Check	PHP	Vue	Security	License	Tests
lint	✅
phpcs	✅
phpmd	✅
psalm	✅
phpstan	✅
phpmetrics	✅
eslint		✅
stylelint		✅
build		✅
check-manifest		✅
test-l10n		✅
composer			✅	✅ 100/100
npm			✅	✅ 2/2
PHPUnit					✅
Newman					❌
Playwright					❌
Hydra gates					❌

---

Quality workflow — 2026-08-06 11:43 UTC

Download the full PDF report from the workflow artifacts.