Skip to content

06-testing.md

Latest commit

 

History

History
397 lines (329 loc) · 21.7 KB

06-testing.md

File metadata and controls

397 lines (329 loc) · 21.7 KB
Raw

06 — Testing Standards

Status: Active Derives from: ADR 0003 — Tenant Isolation Defense in Depth, ADR 0010 — Cross-Module Communication. Public-read additions derive from ADR-0052. Public-renderer controls derive from ADR-0053 and ADR-0054. The seeded browser/audit proof derives from ADR-0056. Native admission, stock-browser fallback and HMR proof controls derive from ADR-0055. UI localization controls derive from ADR-0027 and the Accepted P02d-6 package.

Test pyramid, conventions, and what every change must cover.

Test Pyramid

---
title: LearnStack Test Pyramid
---
flowchart TB
  e2e[End-to-end / Playwright<br/>handful of golden flows]
  contract[Contract & API tests<br/>OpenAPI + provider fakes]
  integration[Integration tests<br/>Testcontainers Postgres]
  arch[Architecture tests<br/>module boundaries + tenant invariants]
  unit[Unit tests<br/>domain + application + UI logic]

  unit --> arch --> integration --> contract --> e2e
Loading

Text fallback (for renderers without Mermaid support — pyramid base → top):

  • Unit tests (base layer, widest) — domain + application + UI logic.
  • Architecture tests — module boundaries + tenant invariants.
  • Integration tests — Testcontainers Postgres. Valkey, SeaweedFS and the rest arrive with the phase that ships something calling them (ADR-0035).
  • Contract & API tests — OpenAPI + provider fakes.
  • End-to-end / Playwright (top, narrowest) — handful of golden flows.

We invest most at unit + integration. Architecture tests are zero-flake. E2E covers only what cannot be proven below.

Backend Test Types

Type Project Tool
Unit LearnStack.Tests.Unit xUnit, FluentAssertions
Integration LearnStack.Tests.Integration xUnit + WebApplicationFactory (Docker-free host tests) and, from Packet 6, Testcontainers marked [Trait("Requires","Docker")] — CI runs the two halves in separate jobs by that trait
Architecture LearnStack.Tests.Architecture xUnit with NetArchTest (Mono.Cecil), reflection and source scans
API contract LearnStack.Tests.Contract The published OpenAPI document; the snapshot-diff gate activates in Phase 02d, per § API Contract Tests
End-to-end Shared demo lifecycle (P02d-7) Playwright library + Lighthouse; Phase 06 owns the broader golden-flow suite, per § End-to-End Tests

Unit Tests

  • Cover domain aggregates, use cases (with mocked repos), value objects, pure helpers.
  • No I/O. No HttpClient. No database.
  • One scenario per test. Arrange / Act / Assert.
  • Naming: MethodName_Scenario_ExpectedOutcome.

Integration Tests

LearnStack.Tests.Integration holds three populations, and which one a test belongs to is decided by what it needs, not by what it is about:

  • Host tests — a real HTTP pipeline through WebApplicationFactory, no Docker. Everything that is a property of the API surface lives here: routing, the error shape, idempotency, limits, the tenancy edge. These run in the required backend CI job alongside the unit suite.
  • Data tests — real Postgres via Testcontainers, connected as learnstack_app, one database per test collection — cases that write either roll back or clean up after themselves, because the fixture's seeded row counts are what the isolation assertions compare against. Not Valkey and not SeaweedFS: nothing the backend runs calls either, and both sit behind the gated compose profile (ADR-0035). Everything that is a property of the schema lives here, and every schema-level tenant-isolation invariant does. Packet 6 shipped the fixtures, the four-role provisioning suite, both migration chains, the policies, a two-tenant seed and the schema-level isolation suite; Packet 7 re-runs those cases one layer up — the third population below. All of them connect as learnstack_app, because a test run as the owner or as a BYPASSRLS role passes against inert policies. They run in the separate backend-integration job, split from the Docker-free tests by [Trait("Requires","Docker")].
  • Request-level isolation tests — a WebApplicationFactory<Program> pointed at a Testcontainers PostgreSQL. Packet 7 introduces the population, because it is the only shape in which the real middleware chain, the real EF query filters and the real RLS policies run against one another in one request, connected as learnstack_app. Packet 7 originally drives test-only controllers because it ships no production endpoint. P02d-4 adds actual public Education GET/HEAD cases in PublicEducationHttpTests, including mapped-host scope and uniform hidden resources. Test-only probes remain separate from the production endpoint set. These live under Database/ with the rest of the Docker-bound suite and carry the same trait.

Where a Docker-bound test lives, and what it takes, is what routes it. Every_Database_Test_Carries_The_Docker_Trait scans the whole of LearnStack.Tests.Integration and requires the trait on every test class under Database/ and on every one elsewhere that takes a container fixture — SchemaFixture, PostgresFixture or the SharedSchema collection, as ApiHandlerCompositionTests does. It scanned Database/ alone until Packet 9's fifth review, which left a class beside the HTTP suites unchecked. A missing trait does not fail the class, it runs it in the backend job: both CI runners are ubuntu-latest and both carry a Docker socket, so the container starts and the test passes — nothing goes red, and the Docker suite stops being where the Docker tests live. Its companion, The_Docker_Trait_Sweep_Can_Actually_Fail, feeds the predicate the shapes it must and must not flag. And neither backend job can pass having run nothing: each ends with scripts/assert-tests-ran.py, because a filter that matches no test exits 0.

All three: real module configuration, no mocked repositories, and coverage of the happy path and the edges.

An isolation test connects as learnstack_app. One that runs as learnstack_migration, learnstack_platform or learnstack_outbox_admin passes even when every policy is inert, and therefore proves nothing.

Architecture Tests

Non-skippable: no architecture test carries Skip, and the suite runs in the required backend job on every PR. The rules — canonical names, what each asserts, and whether it runs yet — are the architecture-test catalogue. This section keeps no second list: the table that stood here named checks, such as a Hangfire payload scan and a provider-SDK reflection rule, that are registered for later phases and were read as running.

API Contract Tests

  • Validate every endpoint against the published OpenAPI.
  • Breaking changes require a version bump.
  • P02d-4 Step 4 delivered: the contract suite compares the production composition's served document with backend/openapi/v1.json, preserving array order. Eight GET/HEAD operations, exact parameters/statuses and recursive DTO allowlists have clean and planted offending controls. The required frontend job regenerates the locked SDK and refuses drift; fake transports exercise all four generated GET wrappers. Pinned CLI fixtures cover ADR-0024 compatibility, with explicit companions for the tool's open-enum gaps. Snapshot admission controls fail closed on missing/deleted snapshots and failed base verification. The packet record records executed counts, reviews and the live required-check rollout.

End-to-End Tests

P02d-7 Step 3 implements the narrow seeded catalog → course → lesson browser proof over the running production API/native renderer. Its delivery record records the passing local nine-page walkthrough and 27 Lighthouse audits; both independent Step 3 reviews are complete. The packet's Step 4 record owns real CI receipts and required-check activation; local proof alone does not establish activation. The proof uses Playwright's library and managed Chromium, without a separate Playwright Test project. It supplements G33's human walkthrough and preserves P6's recorded manual smoke; it does not replace either with automated accessibility scores.

Phase 06 owns the broader MVP vertical slice (tenant → page → course → enrollment → live session), approximately ten high-signal golden flows and the full Playwright/axe harness. P02d-5/6's real-API HTML/RSC integration cases remain separate from browser evidence. P02d-7's proof adds actual linked navigation, computed themes, ordered fields, TLS refusal controls and bounded audits over unchanged seeded product routes.

Run the shared proof from a fresh checkout without root .env, web .env.local or artifacts/public-demo, with Docker and the documented toolchain available:

(cd frontend && pnpm install --frozen-lockfile)
(cd frontend && pnpm exec playwright install --with-deps chromium)
make demo DEMO_MODE=ci

The lifecycle provisions a unique tmpfs-backed Compose project, generates private disposable TLS/configuration, seeds through normal commands and stops its Keycloak before browser navigation. Fixed application/Compose ports must be free. Review the sanitized artifacts/public-demo/receipt.json, all 27 JSON/HTML report pairs and browser screenshots; proof and owned cleanup must both pass. Existing artifacts must be moved aside before a repeat run. Private environments, certificates, keys, browser profiles and raw process logs are excluded from artifacts. Human foreground make demo retains manual hosts/CA/leaf setup and persistent local data. The local test workflow owns the complete setup and artifact instructions.

The frontend workspace pins Playwright and Lighthouse as development tools. Lighthouse's transitive development axe-core@4.14.0 does not supply a standalone application axe dependency or the full Phase 06 axe suite. There is no pnpm test:e2e, pnpm test:a11y or @playwright/test project.

Public renderer fixture ownership

P02d-5 remediation Step 3 implements test-owned process/tree cleanup, positive child environments, explicit TLS verification and sticky bounded private-output scanning. Setup and termination share one owner; SIGINT/SIGTERM, explicit control closure and EPIPE must stop owned children before removing private files. Fixed port collisions fail; a healthy response alone cannot establish owned readiness. Cleanup joins final child output before its containment verdict. An unresolved process/listener cleanup retains the private tree and fails; it cannot claim confirmed teardown. Envelope boundaries at chunk ends require a delimiter or EOF.

Controls must fail for omitted cleanup, early split leaks followed by large output, disabled trust verification, stale listeners and unclosed upgrades. Healthy builds receive nonempty private configuration; a real public-only Client Component and a successfully built leaking mutant make emitted-asset containment falsifiable. The remediation record owns execution/review evidence. These are socket/build controls, not P7 browser or Lighthouse delivery.

Frontend Test Types

Type Tool
Unit (logic) Vitest
Component Testing Library + Vitest
Server actions Vitest + Next.js test helpers
E2E P02d-7: Playwright library; Phase 06: broader Playwright Test harness
Visual regression Chromatic / Playwright snapshot
Accessibility P02d-7: Lighthouse automated score; Phase 06: full axe-core via Playwright suite

Rules:

  • Components with non-trivial state have unit tests.
  • Snapshot tests only for stable visual primitives.
  • Playwright covers the same golden flows the backend E2E covers, from the user's perspective.
  • Visual regression covers the public renderer and page-builder block output.

P02d-6 G38(b,c)/G43 — Steps 1–4 implemented, 2026-10-10. Vitest covers synchronous views and pure mappings; all 31 active recommended JSX accessibility rules have blocking actual-config dirty/clean controls. Async pages have production HTML/RSC proofs against unchanged product routes in the existing required real API/Next Docker fixture; keep P5's synthetic transport probe in a separate mode so its catch-all cannot shadow product routes. Preserve learnstack_app, RLS, read-only transactions, native HTTPS, closed hop headers and fixture containment. Step 4 adds deterministic overlapping host/locale lanes, authored RTL content, whole-theme fallback, next-document freshness and one-shot content-failure modes. The delivery record owns execution, review evidence and the completed manual accessibility smoke.

Assert visible HTML/DOM separately from Flight: status, lang/dir, headings, theme, attribution, both hosts/locales, cross-tenant absence, overlapping concurrent host/locale requests, freshness in one process, controlled failures, both opaque pagination surfaces, exact schema swaps and inert authored values. Test-owned inventory exceeds the page bound; test-owned enabled RTL content proves direction and labelled UI fallback without rewriting historical seed inventory. Verify the approved local 307→404 chain, changed status URL, no direct internal bypass and HTTP 200 noindex known content-failure states. Under ADR-0055, completed product documents cost two API calls, fixed status/scaffold documents one and followed missing documents three. Metadata/layout/page/UI reuse adds no bootstrap; the next document re-reads. HEAD/RSC/prefetch and Flight fallback need separately proven counts. The Step 3 record records passing production replacement proofs; both independent Step 3 review rounds are complete. These HTTP/HTML/RSC proofs are not browser E2E evidence.

ADR-0055's separate test-only stock-Next navigation probe observes a failed Flight followed by a fresh document request and counts that chain separately. It requires an installed stock Chromium browser, uses an isolated disposable profile with certificate-specific public-key trust, and fails if the browser is absent. The required CI integration job checks /usr/bin/google-chrome and its version. Development HMR controls separately check recompilation and fresh admission in one native process. The Step 3 record owns their actual execution and review status; P7's browser demo and Lighthouse, and Phase 06's full Playwright/axe, remain separate.

Applicable jsx-a11y rules run at error severity with actual-config planted controls. Catalogue checks cover nonempty/equal key sets, valid ICU and argument agreement, checked callsite coverage and bounded formatter failure, with planted offenders. Canonical names/statuses live in the architecture-test catalogue; acceptance does not mark them Implemented. Manual keyboard, focus, 320 CSS px reflow/zoom, long strings, contrast and real screen-reader smoke evidence must name commit, environment, both hosts/locales and assistive technology. If pending, P6 completion is not claimed. Full Playwright Test/axe remains Phase 06; the narrow seeded demo/browser/Lighthouse proof is delivered by P02d-7. G38(a)'s transport proofs and G38(d)'s guarded runner are already delivered.

Tenant Isolation Tests

Mandatory for every PR touching tenant-owned data:

Use the shipped examples:

  • PublicEducationHttpTests resolves actual mapped hosts and asserts exact-locale public catalog/detail behavior, foreign/draft/restricted hiding and sibling-organization refusal. Course_draft_branch_foreign_and_unknown_are_uniformly_hidden and Organization_host_never_serves_a_sibling_organization exercise the production public GET/HEAD contract.
  • EducationIsolationTests independently tests raw app-role RLS and EF-filter behavior on all four Education tables, including unresolved scope, insert scope, reporting reads versus writes, immutable organizations and parent constraints.

Fixtures apply every chain through MigrationChains, which owns current chain/history discovery. The current schema has five chains, 21 data tables and five history tables; tests must not copy a fixed partial chain list. Follow add-integration-test for fixture selection and transaction-scoped context.

Rules:

  • Every module owning tenant-scoped data has at least one such test.
  • Failures block PRs.
  • Regression tests are kept indefinitely.

Test Data

  • Builders / factories for entity construction.
  • No global mutable fixtures.
  • Seed only what a test needs.
  • Factories live beside the suite that uses them — SchemaFixture and the seeder's composition for the integration suite. There is no shared test-kit project; one earns its place when two suites need the same builder.

Coverage Targets

  • Domain code: ≥ 90% line, ≥ 80% branch.
  • Application code: ≥ 80% line.
  • Infrastructure adapters: ≥ 70% line.
  • UI components: behavior coverage, not lines.

No CI job collects coverage today, so these numbers are a local target and gate nothing. The architecture, isolation and contract suites are the hard gates. Collection lands with the release pipeline in Phase 11; until it does, a statement that coverage "blocks" anything is false.

Test Speed

  • Unit suite < 30 s locally.
  • Single-module integration suite < 2 min locally.
  • Full CI < 15 min.

If a test gets slow, fix the test before the suite.

Flaky Tests

  • A flaky test is a bug. Triage immediately.
  • Quarantined tests re-enabled within one sprint or deleted.
  • A Skip fails CI: scripts/assert-tests-ran.py refuses any backend run whose results report a case that did not run (No_Architecture_Test_Is_Skippable). P02d-5 G38(d) extends the runner to frontend workspaces: pnpm test from frontend/ discovers test packages and checks actual nonempty Vitest JSON outcomes, rejecting failures, skipped/todo cases, missing scripts/reports and omitted discovered files. Symlinked workspace packages and test sources fail explicitly rather than disappearing from discovery. Real clean/planted runner controls prove refusal.

Live Classroom Testing

  • Token issuance — correct identity, role, TTL, grants.
  • Authorization — only authorized users can join.
  • Provider abstraction contract — fake provider validates the interface.
  • Webhook handling — signature verification, idempotency, tenant scope.
  • Attendance computation — synthetic event streams.
  • Recording policy — consent flow asserted; recording does not start without consent.

Real LiveKit is not required in CI; a fake provider satisfies the contract.

Background Jobs

  • Hangfire jobs are unit-tested by invoking the job class directly.
  • Tenant context propagation asserted in integration test for every job.
  • Idempotency: same job invoked twice produces the same effect.

CI Pipeline

  1. Restore + build, warnings as errors.
  2. Static analysis — the Roslyn analyzers run in the build; ESLint and TypeScript in the frontend job.
  3. Unit, architecture and contract tests, and the Docker-free integration tests — the backend job.
  4. The Docker-bound integration tests — the backend-integration job.
  5. Frontend unit and component tests — the frontend job.
  6. Narrow seeded browser/Lighthouse proof — the public demo + lighthouse job invokes the shared make demo DEMO_MODE=ci lifecycle, per § End-to-End Tests. It retains real admission budgets and never retries a 429 or audits a refusal page. The packet's Step 4 record owns the real PR receipt and seventh required check activation. Phase 06 owns the broader Playwright Test/axe golden flows; any infrastructure-flake retry there requires logged justification.
  7. Coverage report — none yet, per § Coverage Targets.

A failure in any step that runs fails the pull request. Which checks block a merge is a repository setting, and CONTRIBUTING § Branch protection lists them.

Forbidden

  • Tests depending on real external services (Stripe, LiveKit Cloud, Postmark) without sandbox + recorded interactions.
  • Tests sharing mutable state.
  • Tests depending on execution order.
  • Tests sleeping > 1 s without a polling assertion.
  • Tests asserting on log output strings.
  • Tests catching and ignoring exceptions.

Test Names Read Like Specs

Good:

  • PublishCourse_WhenDraftHasNoLessons_ReturnsValidationError
  • JoinSession_WhenLearnerNotEnrolled_Returns403

Bad:

  • Test1, ShouldWork, CourseTests_Publish