ci(e2e): run the 433-test suite and make it return a verdict by rubenvdlinde · Pull Request #280 · ConductionNL/scholiq

rubenvdlinde · 2026-08-05T17:37:35Z

Supersedes and absorbs #254 (closed; its whole tree is merged here, its measurements are recorded in its closing comment).

The E2E Tests (Playwright) job had never executed on this repo. enable-playwright was not set to false with a reason — it was absent, so the job reported skipped on every run this repo has ever produced while tests/e2e/ grew to 33 spec files and 433 tests. The Quality Report renders skipped exactly like a pass.

The verdict — run 31041359993


collected	433 in 33 files
reached	433 (404 + 2 + 27 — nothing was left unrun)
passed	404
failed	2
skipped	27
duration	32.0 min (job 35m28s of a 45-min cap)
workers	4
conclusion	`failure` — a real verdict

The previous attempt on this branch (31030901245) was cancelled at 45m28s. That is not a red — a killed job prints no tally, writes no HTML report, and skips both artifact uploads (if-no-files-found: ignore). This run uploaded a 7.75 MB report.

How the budget was fixed — every number measured, all four in the strict direction

	before	now	evidence
`globalTimeout`	none	38 min	below the 45-min job cap, so Playwright stops itself and prints a tally instead of being killed
`retries`	1	0	a retry only ever turns red into green; run 30889902343 spent a minute replaying one failure against a budget it was already close to
`timeout`	60 s	40 s	slowest passing test measured at 4 workers = 22.4 s; 40 s is 1.79× that
`workers`	6	4	see below

run 30835724202  workers 4 -> 24.2 min wall | 94.0 min test-time | mean 13.8 s | slowest PASS 22.4 s
run 30889902343  workers 6 -> 29.2 min wall | 169.7 min test-time | mean 25.0 s | slowest PASS 37.4 s

Going 4 → 6 workers made the wall clock 21 % worse and inflated every individual test by 81 %. The runner has 4 cores and already hosts php -S with PHP_CLI_SERVER_WORKERS=8; the 5th and 6th Chromium do not get a core, they take one from the server answering their own requests.

Note what the contention did to the headline number: the "slowest passing test" reads 37.4 s at 6 workers but 22.4 s at 4. Sizing the timeout under the wrong worker count would have justified keeping the loose 60 s.

No timeout was raised. No assertion was relaxed. No .skip, test.fixme, continue-on-error or if: false was added, and no spec was deleted.

Negative controls — proof the suite can actually fail

1. Bundle truncation. Branch feature/e2e-negative-control-bundle (run 31041400714, since deleted, never merged, no PR) appended && : > js/scholiq-main.js to the build. Truncated, not deleted — global-setup's checks use fs.existsSync(), and a deleted file is also the case the existing content-type check already catches.

Result: the bundle served 200 application/javascript 0 and the job went red at the seed step, 5m56s in, before Playwright started:

[ci-seed] warm bundle /apps/scholiq/js/scholiq-main.js?v=04c8bc4f-0 -> 200 application/javascript 0
::error::The Scholiq frontend bundle served as JavaScript but is only 0 bytes (floor 1048576).

Note the content type: application/javascript. The type check could not have seen this — which is why this PR adds the size floor. The pre-existing half of the gate covers the other case: a missing Nextcloud asset does not 404, it is served through index.php as the NC HTML error page with HTTP 200 and Content-Type: text/html, so every status-code check in the pipeline reads a completely absent frontend as success.

Because the gate fails the job before Playwright runs, no tests flipped — the suite never started. That is the correct behaviour for a broken frontend (433 selector timeouts blaming the specs is strictly worse evidence), and it is why the gate lives in the seed step.

2. Mutation of the code under test. The one failure this PR fixes was re-run against the real dialog and passes; the two failures below are the suite discriminating on a dependency change nothing else in the pipeline could see (next section).

The 2 remaining failures are a DEPENDENCY REGRESSION, not spec debt

Both passed in both of #254's measured runs and fail here. The cause is not this PR's config:

#254's tree      @conduction/nextcloud-vue 3.0.0-vue3.4   -> 404-405 passing
this PR's tree   @conduction/nextcloud-vue 2.2.0-vue3.3   -> those two fail

#278 repinned off 3.0.0-vue3.6 because it was unpublished. The whole 3.0.0 line is now gone from the registry (npm view lists only 2.2.0-vue3.1 … 2.2.0-vue3.4), so #254's green runs were installing a version that no longer exists — served from the Actions npm cache. The repin was necessary. This job is the first thing in the pipeline capable of seeing what it cost, which is the whole argument for the gate.

1. accessibility-axe-scan.spec.ts:66 — Dashboard, scrollable-region-focusable (serious, WCAG 2.1 SC 2.1.1 / 2.1.3)

html:   <div class="cn-data-table__scroll">
target: div[aria-label="manage-courses"] > .grid-stack-item-content > .cn-widget-wrapper
        > .cn-widget-wrapper__content[role="region"] > .cn-table-container > .cn-data-table__scroll

cn-data-table__scroll is @conduction/nextcloud-vue's own DOM, not scholiq markup. CnDataTable's scroll container carries no tabindex, so a keyboard-only user cannot scroll the table. This is a real a11y defect in the shared library affecting every app that puts a CnDataTable in a widget, and it cannot be fixed from this repo. 2.2.0-vue3.4 is published and is the current vue3 dist-tag (one patch ahead of the pin) — whether it fixes this is a dependency decision, deliberately not taken here.

2. credential-verify.spec.ts:21 — 2 console errors on a deliberately unknown id

Error fetching scholiq-Credential/test-id: Proxy(Object)
Error fetching Credential/test-id: Proxy(Object)

The test navigates to test-id on purpose ("shows valid/invalid status for unknown credential") and also asserts no fatal JS errors. The OpenRegister object store console.errors on the resulting 404, once per schema-slug resolution attempt. A healthy, specified path is therefore indistinguishable from an outage in the console. Same defect class as #254's MyLearningRecordView fix ("a user with no learner profile is an empty state, not an error"), but the console.error is in OpenRegister's store, not in scholiq. Left red deliberately — the honest fixes are upstream or a fixture-based rewrite, and neither is worth weakening the assertion for.

Hydra Gates: RED, but strictly BETTER than the base

Hydra Gates fails — and it fails on development too. Compared properly:

	base `development` @39ad3ea (run 31035634115)	this PR
gates failed	18	11
gate-58 networkidle waits	67	43

The PR's 11 are a strict subset of the base's 18 — comm -13 returns empty. This PR introduces zero new gate failures. The base run is genuinely red (56 s, named findings), not vacuous — no SCOPE WAS EMPTY, no unresolved diff base — so "pre-existing" is supported by evidence rather than assumed.

Six of the eleven (12, 32, 38, 40, 43, 44) are app-wide accessibility debt that predates this branch. They are not the cause of axe failure #1, which is in the vendored library's DOM.

Coverage

The first run also tripped the coverage guard: Tests: 898, Assertions: 4126, Skipped: 5 all passed, then 71.16% → 71.15%, FAIL: Coverage dropped by 0.01%. That drop was this PR's — porting #254 brought an untested catch into LearningRecordShareVerifyController::fetchObject(). Fixed by covering it, not by moving the baseline. The five existing tests could not have covered it: the shared mock returns null for an unknown id, so not_found came from the $obj === null branch and the catch was never entered — the two paths give the identical 404 JSON, which is exactly how an untested fail-closed branch hides.

What else this fixes

src/manifest.json: openconnector was a HARD dependency. It was declared as a bare string, which CnAppRoot treats as required — if the app is not installed and enabled, the shell switches to the blocking dependency-missing phase and nothing renders. Scholiq calls openconnector from exactly two places, LtiToolPlacementController (OIDC launch forward) and PaymentTransactionController (payment initiation), both optional integrations. A school running Scholiq without LTI or online payments got a completely blank app. Now { "id": "openconnector", "required": false }. A product defect, independent of CI, and the most user-visible thing in this PR.

The suite would otherwise have run LOGGED OUT. global-setup.ts wrote its auth storageState to a CWD-relative path. Playwright resolves testDir against the config directory but storageState/globalSetup against the CWD; a mismatch does not error, it silently runs all 433 tests unauthenticated and the assertions then pass or fail for the wrong reason. Anchored to the app root.

ci-seed.sh is now the playwright-seed-command. occ app:enable cannot be trusted to provision the register: the repair step runs with no user session (OR RBAC can deny it), swallows its exception as "Non-fatal", occ exits 0, and the non-forced import advances the recorded version without applying the register. Measured on this run:

[ci-seed] seed status: full
[ci-seed] schemas with seeded rows: 32 (floor 15)
[ci-seed] bundle verified as JavaScript, 11726186 bytes (floor 1048576).

Green-but-dead shapes, counted

294 assertions vanish silently if the seed does not complete. index-pages.spec.ts gates three expect.softs per page (no-JS-error, nav-present, ≥1 row) behind SEEDED, across 98 index pages. The step now runs under bash -e and ci-seed.sh exits non-zero on a bad provision, so in CI this branch can no longer be taken silently.
82 of 98 row-count assertions are vacuous even on a fully successful seed. The seeder creates rows for 32 schemas; only 16 of the 98 index pages have one of those as config.schema. Worth a follow-up — the seed's coverage, not its success, is the limit.
27 skipped = 26 data-conditional test.skip(!fixture, …) + 2 test.describe.skip blocks ("live run deferred": peer-and-self-assessment, personal-timetable). All pre-existing; none added here.
The one silent-skip guard in a test this PR touches (accessibility-conformance's outer if (isVisible), which made the whole body optional) is now a hard assertion.

The failure that was fixed, with its mechanism

accessibility-conformance.spec.ts — "a user fills and submits the AccessibilityFeedback create form" — failed in both of #254's runs and now passes.

It reported Expected: enabled / Received: undefined, which reads like the Create button does not exist. It does — the ARIA snapshot has button "Create" [disabled]. Received: undefined is what an assertion reports when the test was already out of time. The real failure was 60 seconds earlier, swallowed by .catch(() => {}):

waiting for getByRole('option', { name: /serious/i }).first()
  - locator resolved to <li role="option" … id="vs-nc-vue-123__option-1">
  - element is visible, enabled and stable
  - <input id="nc-vue-124" placeholder="Tenant UUID (multi-tenant isolation)."/>
    from <div role="dialog" aria-modal="true" …> subtree intercepts pointer events
  - retrying click action  (× 56, for the full test timeout)

The NcSelect dropdown is rendered inline in the form rather than teleported, and the field immediately after it paints on top of it. Only the first option is reachable with a mouse; serious is option-1. Another real defect in the create dialog, affecting every required enum on every CnDetailPage create form.

Severity is now chosen from the keyboard — immune to the overlap, and the appropriate interaction for an accessibility-conformance spec. The swallow is replaced by two assertions: the option must be visible after typing, then hidden after Enter (the listbox closing is the proof the Enter was taken as a selection). The second is only meaningful because of the first — toBeHidden on an element that never rendered passes for free.

For a human to decide

@conduction/nextcloud-vue pin. 2.2.0-vue3.4 is the current vue3 dist-tag, one patch ahead of the pinned 2.2.0-vue3.3. Not touched here. Does it fix CnDataTable's unfocusable scroll region?
cn-data-table__scroll has no keyboard access (WCAG 2.1 SC 2.1.1, serious). Fleet-wide, fix belongs in nc-vue.
NcSelect dropdowns are painted over by the next field in CnDetailPage create dialogs — mouse users can only pick the first option. Fleet-wide, fix belongs in nc-vue.
credential-verify: should the OR object store console.error on an expected 404, or should the view own the not-found path?
Seed coverage: 32 of 118 schemas get rows, so 82 of 98 index pages assert nothing about their data.
18 pre-existing Hydra gate failures on development, six of them accessibility.

The `E2E Tests (Playwright)` job had never run on this repo — `enable-playwright` was absent from the caller entirely, so 34 spec files and ~260 tests sat in the tree with no gate behind them. What this adds: * `tests/e2e/playwright.config.ts` — a CI-only config. The shared workflow resolves `${playwright-test-path}/playwright.config.ts` first and passes no `--project`, so every project in the chosen config runs; this one declares only the regression project, and writes its HTML report and test-results to the APP ROOT, which is where the workflow's upload step looks. The root config writes them one level deeper — with `if-no-files-found: ignore` that mismatch uploads an EMPTY artifact, i.e. a red run with no report to read. The root config is untouched and stays the local entry point. * `tests/e2e/ci-seed.sh` — explicit force-import of the scholiq register over the admin HTTP API, the example-data seed, a hard verification that the register and six core schemas exist, a warm-up, and a bundle gate. The `occ app:enable` repair path cannot be trusted on its own: an IRepairStep runs with no user session (OR RBAC can deny it), it swallows its exception as "Non-fatal", `occ` still exits 0, and the non-forced import advances the recorded version WITHOUT applying the register. The bundle gate exists because a missing bundle returns HTTP 200 text/html, never 404 — it reads the real src out of the rendered app page (assets are at /apps/ on CI but /custom_apps/ in docker dev images) and asserts the content type is javascript. * `tests/e2e/base-url.ts` — now accepts NEXTCLOUD_URL and NC_BASE_URL as well. The shared workflow exports BASE_URL / NEXTCLOUD_URL / NC_BASE_URL and does NOT export PLAYWRIGHT_BASE_URL. Still no localhost:8080 fallback. * `tests/e2e/global-setup.ts` — the auth storageState path is anchored to the repo root instead of the process CWD (a CWD-relative path does not fail, it runs the whole suite logged OUT), and the seed is skipped when ci-seed.sh already ran it. * `.github/workflows/code-quality.yml` — `pull_request: types: [opened, reopened, synchronize]` (without `synchronize` every commit after the first merged unchecked behind a stale green tick), a top-level permissions block (a caller cannot grant a reusable workflow more than it holds, and capping at read makes the call fail to START), a concurrency group, and `github.event.created != true` so a branch-creation push does not produce a run that looks green because it ran nothing. Real defect fixed on the way: `src/manifest.json` declared `openconnector` as a bare string, which CnAppRoot treats as a HARD dependency — absence switches the entire shell to the blocking `dependency-missing` phase. Scholiq calls openconnector from exactly two places (LtiToolPlacementController's OIDC launch forward and PaymentTransactionController's payment initiation), both optional integrations. A school running Scholiq without LTI or online payments got a completely unusable app. It is now `{ id, required: false }` — a soft dependency whose absence surfaces a dismissible in-shell notice and degrades only those two features. Rationale recorded in src/App.vue. Refs: opencatalogi#787, opencatalogi#790

First CI run of the job (30796644591) failed in the seed step, exactly where it was designed to. Everything below is a defect it surfaced. 1. settings/load returns HTTP 200 with success:false. The endpoint answered 200 and {"success":false,"message":"SchemaMapper::loadSchema(): Argument #1 ($identifier) must be of type string|int, array given"} — importing nothing. A status-code-only gate would have declared provisioning green and handed the suite an empty register. ci-seed.sh now parses the body and annotates. 2. OpenRegister 500s on any schema with a top-level allOf. Measured: the three schemas that carry one (lesson, grade-entry, portfolio-entry) all failed with 500; all 115 without one succeeded. 3 of 3 vs 0 of 115. The seeder now retries once with the top-level allOf stripped and says so loudly — that drops only conditional if/then validation, keeps every property and the base required list, and is the difference between an index page that exists and one that 404s for a reason unrelated to the code under test. The underlying defect is OpenRegister's, and cannot be fixed from this repo. 3. index-pages.spec.ts asserted rows for schemas that had none. It carried a hand-written SEEDED_SCHEMAS set of 32 names; the seeder actually created rows for 17. The other 15 creates were failing OpenRegister's required- property validation, being logged as a warning, and swallowed. Both halves are fixed: the fixture bodies now satisfy the register (Assignment.dueAt is date-time not date; GradeScale.kind; GradeEntry/FinalGrade curriculumPlanId+gradeScaleId; LearningPlan.coordinatorId; LearningPlanEvaluation.evaluatedAt+evaluatedBy; Signature.subjectKind; AttendanceRecord.sessionId; Regulation.audienceScope; Credential issuerDid+signature+openbadges3Payload and a UUID learnerId; XapiStatement.stored; DataMappingProfile.direction+sourceSchema; DataExchangeJob.requestedAt; plus Portfolio/PortfolioEntry), and the assertion set is now DERIVED from what the seeder really created (.e2e-state/seeded-schemas.json) so the two can never disagree again. ci-seed.sh gates the floor so a silent collapse to zero still fails. 4. Pretty URLs 404 under the CI php -S. The shared workflow runs a bare `php -S` with no router, so pretty URLs are not rewritten: the built-in server only falls back to index.php for paths that do NOT exist, and server/apps/<app>/ exists without an index.php. Five spec files used the short /apps/... form (29 of 34 already used /index.php/apps/...); they are now consistent. Tracked upstream as ConductionNL/.github#125. 5. /Settings is not a route. Two specs navigated to /apps/scholiq/Settings while the manifest declares /settings — vue-router paths are case-sensitive, so the heading those tests assert on could never appear. 6. Marker files under test-results/ are deleted before any spec can read them. Playwright's createRemoveOutputDirsTask removes every project outputDir at the start of the run — after the workflow's seed step, before globalSetup. The seed state now lives in .e2e-state/. Also: TOTAL_SCHEMAS in the seeder said 35; the register declares 118, so every ratio it logged was wrong. It is now derived from the register.

Run 30798535945: 389 passed, 17 failed, 27 skipped (1.0h). Every failure is fixed at the cause; no assertion was weakened, skipped or given a longer timeout. 1. `#/…` navigates to a route that does not exist (11 failures) src/main.js builds the router with `createWebHistory(generateUrl('/apps/scholiq'))` — HISTORY mode. vue-router strips the base from location.pathname and then appends the untouched hash, so `/index.php/apps/scholiq/#/admissions/review-board` resolves to the location `/#/admissions/review-board`, which matches no declared route. <router-view> renders nothing. The run proves it: every `#/` spec failed with Received string: "Keyboard navigation help / Skip to app navigation / …" — the Nextcloud chrome with an empty app body — while index-pages.spec.ts and detail-pages.spec.ts, which use the plain path form, passed 206/206 on the same instance. Fixed in the 7 failing specs plus credential-verify, whose docstring described this as a 'hash-route component mount timing' gap that 'may' appear 'in some test runs'. It was neither intermittent nor a timing gap. 2. shell.spec.ts asserted a role-gated menu item is always present (1) The manifest gates Compliance on visibleIf: { "user.primaryRole": { in: ["compliance-officer", "hr"] } } and the CI session resolves primaryRole to the default `learner`. Hiding it is correct behaviour. Inverted into the opposite assertion rather than dropped: for a non-gated role the entry MUST be absent — which proves visibleIf is enforced, something the old assertion could not have detected being broken. 3. dashboard.spec.ts measured Nextcloud's own chrome and its own BEM children (2) `[class*="dashboard-page"]` also matches the 25 BEM sub-elements @conduction/nextcloud-vue ships for CnDashboardPage (__header, __content, __title, … — verified in the published dist CSS), so ONE dashboard counted as 5 and was reported as dashboard-in-dashboard nesting. Now counts the `.cn-dashboard-page` host token exactly. `nav a, .app-navigation a, [role="navigation"] a` selects NC's global header too. The captured DOM shows the 3 matches were: NC's logo link 'Go to Dashboard' -> /index.php, NC's Dashboard app link -> /index.php/apps/ dashboard/, and the one real Scholiq entry. It would have counted 2 with the Scholiq nav entirely absent. Now scoped to #app-navigation-vue. 4. nextcloud-app.spec.ts pointed at the wrong surface entirely (4) Every assertion there ('Scholiq Settings', the OpenRegister section, the register combobox, 'Credential Signing') targets AdminRoot.vue, which src/settings.js mounts on the NEXTCLOUD ADMIN page. It navigated to /apps/scholiq/Settings, which is neither: /Settings is not a declared route, and the in-app /settings page is a different surface — it rendered a generic 'Settings' heading and a disabled Save button, with no 'Scholiq Settings' anywhere. Now /index.php/settings/admin/scholiq (sectionId defaults to the app id in OpenRegister's AppHost Bootstrap; Application.php passes no override). Still outstanding and NOT fixed here: 18 further spec files navigate with the same dead `#/` form. They pass today only because their assertions are permissive or they skip on absent data, so they are green-but-dead. Converting them needs its own verification pass and is filed separately.

⚠️ This merge is not housekeeping — it un-blocks CI entirely. development moved to cb101f1 (#257, workflow hardening) and edited the same `permissions:` block this branch edits. GitHub could no longer compute the PR's merge ref, so PR #254 went CONFLICTING and **every `pull_request` workflow stopped firing**: the push of 991e132 produced no Code Quality run at all, only CodeQL. The PR page shows no red — it shows nothing, which is the failure mode where a missing gate is indistinguishable from a passing one. Conflict resolved as a union, because both sides are load-bearing: * issues/pull-requests: write — development's least-privilege pass; without them the Quality Report job cannot post its sticky PR comment. * contents/actions: write, packages: read — required by jobs the CALLED workflow declares (coverage-baseline update, journeydoc capture, SBOM). A caller cannot grant a reusable workflow more than it holds, so capping those at read makes the call fail to START. development at cb101f1 shows exactly that: two startup_failure runs back to back.

…ed-e2e

The previous measurement (run 30810841150: 326 passed / 80 failed) was taken INSIDE a broken window. ConductionNL/.github#125 merged at 08:49:30Z and its router exempted other PHP entry points with `is_file(__DIR__ . $path)`, which matches /status.php (the one case it verified) but never matches /ocs/v2.php/cloud/user, because that carries path info — so the entire OCS surface fell through to index.php and 404'd. That is exactly the two signatures I measured: dashboard-main.js fetchApiWidgets hits /ocs/v2.php/apps/dashboard/..., and 'core: Failed to load user status' hits /ocs/v2.php/apps/user_status/... — which is why 10 spec files I never touched flipped with byte-identical URLs. ConductionNL/.github#127 merged at 12:11:39Z and fixed it (exempt by prefix, plus a gate asserting /ocs/v2.php/cloud/user is not 404). A plain re-run replays the ORIGINAL workflow resolution and would not pick that up, so this is a fresh commit. Baseline for comparison is the 08:44:01Z pre-router run 30798535945: 389 passed / 17 failed / 27 skipped — compared by failing test NAME, not count.

Run 30817505312 came back `cancelled`, not failed. The E2E job ran 13:21:24 -> 14:06:42 and the 'Run Playwright tests' step was killed at exactly 45 minutes — the `timeout-minutes: 45` the shared workflow gained in ConductionNL/.github#261. That cap was sized from suites of '4-10 min' (planix 0.8, doriath 4.2, opencatalogi 10.0). Scholiq declares 276 manifest pages; the suite is 433 tests and took 1.0h then 1.4h single-worker. It was never going to fit, and a cancelled job is not a result — it is the failure mode where a gate produces no verdict at all. So the suite now runs 4 workers with fullyParallel. This is safe by construction, not by hope: every spec is read-only against a dataset ci-seed.sh provisions BEFORE the run, so no spec creates, mutates or deletes objects and workers cannot race. The single shared mutable artefact is the auth storageState, which globalSetup writes once before any worker starts and workers only read. Server-side capacity is already there — the shared workflow sets PHP_CLI_SERVER_WORKERS=8, so `php -S` is no longer the single-request bottleneck that motivated `workers: 1`. 4 rather than 8 because each Playwright worker drives a full Chromium plus the SPA's boot fan-out; matching the PHP worker count 1:1 would just re-queue the requests.

Three distinct root causes, none of them the test suite. 1. PascalCase schema segments in OpenRegister object-API URLs (10 of 12). SchemaMapper::findBySlugInIds() matches on LOWER(slug), so a SINGLE-word title accidentally resolves ('Course' -> 'course') while a MULTI-word one cannot ('AccessibilityStatement' -> 'accessibilitystatement', which is not the declared slug 'accessibility-statement'). ObjectService::setSchema() then rethrows DoesNotExistException and the request 404s. That is why some calls in the same file worked and others did not. Fixed every multi-word segment across 20 views to the slug declared in lib/Settings/scholiq_register.json — all 20 target slugs verified present before use. Single-word segments are left alone: they resolve correctly. This follows the precedent set by e5b78e9 ("resolve register/schema ids to slugs so 9 dead listeners can run") and matches the declarative manifest pages and SubjectChoicePicker.vue, which were already correct. 2. LearningRecordShareVerify returned HTML where JSON was promised. fetchObject() was the one method on the controller with no try/catch around ObjectService::find(). On a #[PublicPage] an escaping exception makes Nextcloud render printExceptionErrorPage(), so the caller got an HTML error page — and the view called resp.json() with no resp.ok guard, producing "SyntaxError: Unexpected token '<'". Both halves fixed: the controller fails closed to the specified denied response, the view renders the denied state instead of logging a client crash. 3. networkidle never settles on Nextcloud (ADR-074 rule 4 / hydra gate 58). The notification poll keeps a request in flight for the whole session, so waitForLoadState('networkidle') silently burns its full 30s out of a 60s test budget; the customary .catch(() => {}) hides the throw but not the cost. The symptom is a bare "Test timeout exceeded" attributed to whatever call was in flight, indistinguishable from an app outage. Replaced with an explicit readiness assertion on the Vue root in the affected tests only. No assertion was weakened, skipped or given a longer timeout. Two stale specs rewritten, with evidence: - nextcloud-app: asserted <th>Feature</th>, but Scholiq's own AI-feature table was removed and delegated to Hermiq (ADR-005). No <th> exists in any settings view, so the assertion could only ever fail. Now asserts the delegation branch the app actually renders. - dashboard: counted /Dashboard/i nav entries expecting <=1, but the manifest legitimately declares "Risk dashboard", "Skills gap dashboard" and "BSA risk dashboard" beside the single "Dashboards" landing page, and hasText matches descendants. Now matches the accessible name exactly and asserts the real invariant — no per-role dashboard entries (ADR-009 section 6).

…ed-e2e # Conflicts: # .github/workflows/code-quality.yml

…comments The previous commit's comments said OpenRegister "matches on LOWER(slug), so a single-word title resolves and a multi-word one cannot". That explanation is wrong and dangerous to generalise. SchemaMapper::findBySlugInIds() lowercases BOTH sides (openregister lib/Db/SchemaMapper.php:586), so the actual invariant is: strtolower(<url segment>) === strtolower(<declared slug>) Casing never breaks resolution. STRUCTURE does. 'AccessibilityStatement' lowercases to 'accessibilitystatement', which is not the declared slug 'accessibility-statement' — the hyphen is the difference. Prefixes and underscores break it identically. This matters because "kebab-case the schema name" is NOT a valid rule and applying it elsewhere would create the exact bug being fixed here: scholiq is the fleet outlier in declaring hyphenated slugs for most of its schemas, and even inside scholiq AiFeature declares its slug as literally 'AiFeature'. Other apps declare camelCase or PascalCase slugs. No code changes — every call site was already derived by looking the slug up in lib/Settings/scholiq_register.json, and all 54 segments now in src/ were re-verified to resolve to the correct schema under the invariant above.

…n error MyLearningRecordView threw on any non-2xx from /api/learning-records/me, including 404. A 404 there means the signed-in Nextcloud user has no bound LearnerProfile — the normal case for an admin or staff account — so the component logged '[MyLearningRecordView] loadRecord error' on a perfectly healthy session. In the console that is indistinguishable from a real outage, and the e2e suite rightly treats it as fatal. Render the component's already-declared loadError branch with an accurate message instead of throwing. Real failures (5xx, network) still throw and still log.

…ed-e2e

…f-contradictory test Both live in accessibility-conformance.spec.ts; neither was an app bug. 1. `getByLabel(/^Description$/i)` matched nothing and burned the full 60 s test timeout. The rendered accessible name of a REQUIRED field carries the required marker — the control is `textbox "Description *"` (measured, run 30835724202 error-context.md). The unanchored `/Affected Surface/i` one line above matched fine, which is exactly why only this one field looked "missing". Anchor now tolerates the marker. The same test then never actually submitted: `tenant_id` is in AccessibilityFeedback's `required` list, was never filled, and CnDetailPage keeps Create DISABLED until it is — so `.click().catch(() => {})` was a silent no-op and the test asserted on a submit that had not happened. It now fills the tenant and ASSERTS the Create button becomes enabled, which is the direct proof that every required field is satisfied. 2. "limitation detail route resolves ... not a blank/404 shell" navigated to `00000000-0000-0000-0000-000000000000` while also asserting no console errors. Those cannot both hold: the object store `console.error`s `Error fetching <type>/<id>` on the resulting 404, once per schema-slug resolution attempt. The page itself was fine — the run's error-context shows the registered component mounted with heading "Accessibility limitation", the Data widget, and all 11 fields as `—`. It now creates an AccessibilityStatement + AccessibilityLimitation fixture and asserts the page renders that record's OWN field value, which is a strictly stronger bar than "the component mounted". Also: workers 4 -> 6. The suite took 24.2 min against a 20-minute budget. The config's justification for parallelism ("no spec creates, mutates or deletes objects") was false — nextcloud-app.spec.ts POSTs settings and PUTs notification preferences, and this spec creates rows. Replaced with the reason parallelism is ACTUALLY safe (no spec asserts an exact global collection count) plus a re-audit trigger.

`enable-playwright` was absent from the caller's `with:` block entirely, and the shared workflow defaults it to false, so "E2E Tests (Playwright)" has reported `skipped` on every run this repo has produced. tests/e2e/ meanwhile grew to 34 spec files / 433 tests. A skipped job renders in the Quality Report the same way a passing one does, so nothing ever said those tests were not running. Adds `enable-playwright: true` + `playwright-test-path: tests/e2e`, and ships tests/e2e/playwright.config.ts so the workflow's first config lookup hits it. The run step passes no `--project`, so the chosen config decides what runs; the root config additionally writes its HTML report to `test-results/playwright-report`, which matches neither path the upload step collects — with `if-no-files-found: ignore` that uploads an empty artifact exactly when a red run needs a report. The new config writes report and output to the app root and declares only the chromium project. The root playwright.config.ts is untouched and stays the local-dev entry point. No seed command is wired: global-setup.ts already runs seed-example-data.mjs, which imports the scholiq register over the admin API.

github-actions · 2026-08-05T18:38:48Z

Quality Report — ConductionNL/scholiq @ `907556e`

Check	PHP	Vue	Security	License	Tests
lint	✅
phpcs	✅
phpmd	✅
psalm	✅
phpstan	✅
phpmetrics	✅
eslint		✅
stylelint		✅
build		✅
check-specs		✅
composer			✅	✅ 100/100
npm			✅	✅ 750/750
PHPUnit					✅
Newman					⏭️
Playwright					❌
Hydra gates					❌

Quality workflow — 2026-08-05 18:38 UTC

Download the full PDF report from the workflow artifacts.

Deduplicates the two competing E2E-enablement PRs. #280 (this branch) is the current one — rebased on today's development — but it is a bare config: its last run was CANCELLED at the shared workflow's 45-minute cap (45m28s), which is not a verdict, and the kill also skips the artifact upload, so it measured nothing at all. #254 is stale (last touched Aug 4) and CONFLICTING, but it is the branch that actually produced verdicts. Its content is merged here wholesale rather than cherry-picked: the seed script, the base-url env names, the storageState anchoring, the fixture repairs, the ~20 src/views slug fixes and the src/manifest.json openconnector fix are all load-bearing for a run that means anything. Conflicts, both resolved as a union of the two intents: * .github/workflows/code-quality.yml — kept this branch's newer inputs (frontend-checks, enable-coverage-guard, enable-hydra-gates) and took #254's `playwright-seed-command: bash apps/scholiq/tests/e2e/ci-seed.sh`. This branch's comment claimed a seed command would run the import twice; that is no longer true — ci-seed.sh writes a marker and global-setup.ts skips its own seed when it sees it. The comment now records why occ's repair step cannot be trusted instead (no user session, swallowed exception, exit 0, non-forced import advances the version WITHOUT applying the register) and why the bundle gate is not redundant with an HTTP status check. * tests/e2e/playwright.config.ts — rewritten, see the next commit.

…wright-e2e

…found Two things, both sized on measurements from the runs this branch inherits. ── 1. THE JOB MUST RETURN A TALLY, NOT `cancelled` ────────────────────────── #280's last run (31030901245) was killed at the shared workflow's 45-minute cap after 45m28s. A killed job prints no tally, writes no HTML report, and skips both artifact uploads (`if-no-files-found: ignore`), so it reported `cancelled` — which is neither a pass nor a fail. Four changes, every one in the strict direction, none of them a relaxed assertion: globalTimeout: 38 min (NEW). Playwright now stops the run itself, below the job cap, prints `N passed / N failed / N did not run`, and flushes the report. A red run WITH evidence instead of a silent kill. ~7 min is left for the report writer and the two uploads (measured ~90 s for 7.0 + 6.1 MB). retries: 1 -> 0. A retry can only turn red into green, never green into red, so it carries no information about correctness — it hides instability and doubles the price of every failure. Run 30889902343 spent a minute replaying one failing test against a budget it was already close to. `trace: 'on-first-retry'` -> `'retain-on-failure'` so the FIRST failure still ships a trace. timeout: 60_000 -> 40_000. Sized on the slowest PASSING test measured at the worker count this config uses: 22.4 s (run 30835724202, detail-pages.spec.ts AccessibilityFeedback). 40 s is 1.79x that. No passing test was ever near 60 s — the per-test cap is a tax paid almost entirely by failures, which sit on it. workers: 6 -> 4. This one reverses a change made on the other branch, and the two runs disagree with the reasoning that motivated it: run 30835724202 workers 4 -> 24.2 min wall | 94.0 min test-time | 408 tests, mean 13.8 s, slowest PASS 22.4 s run 30889902343 workers 6 -> 29.2 min wall | 169.7 min test-time | 407 tests, mean 25.0 s, slowest PASS 37.4 s Going 4 -> 6 made the WALL CLOCK 21% WORSE and inflated every individual test by 81%. The runner has 4 cores and already hosts `php -S` with PHP_CLI_SERVER_WORKERS=8; the 5th and 6th Chromium do not get a core, they take one from the server answering their own requests. 4 is the measured optimum, not a compromise. Note what the contention was doing to the headline number: the "slowest passing test" looked like 37.4 s at 6 workers and 22.4 s at 4 — measuring the timeout budget under the wrong worker count would have justified keeping 60 s. ── 2. THE ONE FAILURE: AN OPTION THAT CANNOT BE CLICKED ───────────────────── `accessibility-conformance.spec.ts` — "a user fills and submits the AccessibilityFeedback create form" — failed in BOTH measured runs. It was reported as Error: the Create button must become enabled once every required field is filled … Expected: enabled Received: undefined which reads like the Create button does not exist. It does. The ARIA snapshot from the failing run has `button "Create" [disabled]`, and three required fields empty. `Received: undefined` is what an assertion reports when the test was ALREADY out of time — the real failure happened 60 seconds earlier and was swallowed by a `.catch(() => {})`. From the trace: the `serious` option resolves, Playwright calls it "visible, enabled and stable", and then refuses to click it for the entire test timeout: <input id="nc-vue-124" placeholder="Tenant UUID (multi-tenant isolation)."> from <div role="dialog" aria-modal="true" …> subtree intercepts pointer events The NcSelect dropdown (`vs__dropdown-menu`, `vs-nc-vue-123`) is rendered inline in the form rather than teleported, and the field immediately after it — Tenant ID, `nc-vue-124` — paints on top of it. Only the FIRST option is reachable with a mouse; `serious` is option-1. That is a real defect in the create dialog, it is not specific to this schema or this app (every required enum on every CnDetailPage create form is affected), and it is reported separately — fixing it is not in this repo. Severity is now chosen with the KEYBOARD, which is immune to the overlap and is in any case the interaction an accessibility-conformance spec should be exercising: click to open, type to filter, Enter to take the highlighted option. `keyboard.type` rather than `fill` because vue-select marks its search input readonly when the select is not searchable and `fill` throws on that. No assertion was weakened. Two were ADDED, and the swallow was deleted: * the `serious` option must be VISIBLE after typing; * it must then be HIDDEN after Enter — the listbox closing is the proof the Enter was taken as a selection rather than ignored. That check is only meaningful because the option was asserted visible first: `toBeHidden` on an element that never rendered passes for free. ── 3. A GREEN-BUT-DEAD GUARD IN THE SAME TEST ─────────────────────────────── The whole body of that test sat inside `if (await affectedSurfaceField.isVisible().catch(() => false)) { … }`. If the create form failed to mount, the test asserted NOTHING and reported green. That was defensible while the register import was best-effort. It is not any more: ci-seed.sh is now the workflow's `playwright-seed-command`, it verifies the register, the six core schemas and the seeded-row floor, and the step runs under `bash -e`, so a bad provision fails the job outright. An unmounted form can now only mean the form is broken — which is what the test is for. The guard is an assertion.

The gate already handled the hard case — a MISSING Nextcloud asset does not 404, it is served through index.php as the NC error page with HTTP 200 and `Content-Type: text/html`, so every status-code check in the pipeline reads a completely absent frontend as success. That is why the gate asserts the content type. It only covered half of it. A TRUNCATED bundle — webpack exiting 0 after writing an empty chunk, an artifact uploaded before the write completed — is still served with HTTP 200 AND `Content-Type: application/javascript`, at zero bytes. Status: fine. Type: JavaScript. SPA: mounts nothing. Every UI spec then fails on a selector timeout pointing at the specs instead of at the build. `%{size_download}` was already being fetched and printed; nothing looked at it. The gate now parses the three fields separately and enforces a 1 MB floor. The real bundle measured 11,688,751 bytes on run 30889902343, so the floor is two orders of magnitude below the true size and far above anything a truncation leaves behind. Verified both directions before pushing: "200 application/javascript 11688751" passes, "200 application/javascript 0" trips the floor.

github-actions · 2026-08-05T19:57:11Z

Quality Report — ConductionNL/scholiq @ `a4f3958`

Check	PHP	Vue	Security	License	Tests
lint	✅
phpcs	✅
phpmd	✅
psalm	⏭️
phpstan	⏭️
phpmetrics	✅
eslint		✅
stylelint		✅
build		❌
check-specs		✅
composer			✅	⏭️
npm			⏭️	✅ 750/750
PHPUnit					❌
Newman					❌
Playwright					❌
Hydra gates					❌

Quality workflow — 2026-08-05 19:57 UTC

Download the full PDF report from the workflow artifacts.

github-actions · 2026-08-05T20:44:25Z

Quality Report — ConductionNL/scholiq @ `1996c6e`

Check	PHP	Vue	Security	License	Tests
lint	✅
phpcs	✅
phpmd	✅
psalm	✅
phpstan	✅
phpmetrics	✅
eslint		✅
stylelint		✅
build		✅
check-specs		✅
composer			✅	✅ 100/100
npm			✅	✅ 750/750
PHPUnit					❌
Newman					⏭️
Playwright					❌
Hydra gates					❌

Quality workflow — 2026-08-05 20:44 UTC

Download the full PDF report from the workflow artifacts.

…endpoint The E2E run went red on the coverage guard, not on a test: `Tests: 898, Assertions: 4126, Skipped: 5` all passed, then Coverage baseline: 71.16% Coverage current: 71.15% FAIL: Coverage dropped by 0.01% That drop is this PR's. Porting #254 brought a try/catch into `LearningRecordShareVerifyController::fetchObject()` — `find()` THROWS for an id that does not resolve and, via `ObjectService::setSchema()`, for a schema the register never imported; on a `#[PublicPage]` an escaping exception makes Nextcloud render `printExceptionErrorPage()`, so the caller gets an HTML error page where it asked for JSON and the verify view dies on `SyntaxError: Unexpected token '<'`. The fix is right; it arrived with no test, so the `catch` was dead weight in the coverage denominator. Note WHY the five existing tests could not have covered it. The shared ObjectService mock in setUp() RETURNS null for an unknown id, so `testUnknownShareIsDenied` produces `not_found` through the `$obj === null` branch and never enters the `catch` at all. The two paths are indistinguishable from the response alone — same JSON, same 404 — which is exactly how an untested fail-closed branch hides. This test installs a mock that THROWS instead, which is the only way the behaviour is really exercised. No baseline was moved and no threshold was relaxed.

github-actions · 2026-08-05T21:36:18Z

Quality Report — ConductionNL/scholiq @ `b0f9dd0`

Check	PHP	Vue	Security	License	Tests
lint	✅
phpcs	✅
phpmd	✅
psalm	✅
phpstan	✅
phpmetrics	✅
eslint		✅
stylelint		✅
build		✅
check-specs		✅
composer			✅	✅ 100/100
npm			✅	✅ 750/750
PHPUnit					✅
Newman					⏭️
Playwright					❌
Hydra gates					❌

Quality workflow — 2026-08-05 21:36 UTC

Download the full PDF report from the workflow artifacts.

rubenvdlinde and others added 14 commits August 3, 2026 10:13

Merge remote-tracking branch 'origin/development' into ci/enable-shar…

1f0955b

…ed-e2e

Merge remote-tracking branch 'origin/development' into ci/enable-shar…

1352ddb

…ed-e2e # Conflicts: # .github/workflows/code-quality.yml

Merge remote-tracking branch 'origin/development' into ci/enable-shar…

1126846

…ed-e2e

rubenvdlinde added 4 commits August 5, 2026 21:42

Merge remote-tracking branch 'origin/development' into ci/enable-play…

0dfd597

…wright-e2e

rubenvdlinde mentioned this pull request Aug 5, 2026

ci(e2e): enable the shared E2E Tests (Playwright) job #254

Closed

2 tasks

rubenvdlinde changed the title ~~ci: run the e2e suite — 34 spec files that never executed~~ ci(e2e): run the 433-test suite and make it return a verdict Aug 5, 2026

Navigation Menu

Uh oh!

ci(e2e): run the 433-test suite and make it return a verdict - #280

ci(e2e): run the 433-test suite and make it return a verdict#280
rubenvdlinde wants to merge 19 commits into
developmentfrom
ci/enable-playwright-e2e

rubenvdlinde commented Aug 5, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 5, 2026

Uh oh!

github-actions Bot commented Aug 5, 2026

Uh oh!

github-actions Bot commented Aug 5, 2026

Uh oh!

github-actions Bot commented Aug 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

rubenvdlinde commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The verdict — run 31041359993

How the budget was fixed — every number measured, all four in the strict direction

Negative controls — proof the suite can actually fail

The 2 remaining failures are a DEPENDENCY REGRESSION, not spec debt

Hydra Gates: RED, but strictly BETTER than the base

Coverage

What else this fixes

Green-but-dead shapes, counted

The failure that was fixed, with its mechanism

For a human to decide

Uh oh!

github-actions Bot commented Aug 5, 2026

Quality Report — ConductionNL/scholiq @ 907556e

Uh oh!

github-actions Bot commented Aug 5, 2026

Quality Report — ConductionNL/scholiq @ a4f3958

Uh oh!

github-actions Bot commented Aug 5, 2026

Quality Report — ConductionNL/scholiq @ 1996c6e

Uh oh!

github-actions Bot commented Aug 5, 2026

Quality Report — ConductionNL/scholiq @ b0f9dd0

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

rubenvdlinde commented Aug 5, 2026 •

edited

Loading

Quality Report — ConductionNL/scholiq @ `907556e`

Quality Report — ConductionNL/scholiq @ `a4f3958`

Quality Report — ConductionNL/scholiq @ `1996c6e`

Quality Report — ConductionNL/scholiq @ `b0f9dd0`