Skip to content

testing.md

Latest commit

 

History

History
190 lines (163 loc) · 13.4 KB

testing.md

File metadata and controls

190 lines (163 loc) · 13.4 KB
Raw

Testing

This is the binding test discipline for Relavium. It follows from engine-first and from the build-in-house principle: because we own the core layers (@relavium/llm, packages/core), we own the tests that keep them honest. The tools are pinned in tech-stack.md (Vitest for unit, Playwright for e2e) and are not restated here.

Engine-first test discipline

The engine is the critical path, so it carries the test burden. packages/shared, packages/llm, and packages/core are tested to a high bar before any surface consumes them — the same order as the build order in architectural-principles.md. A surface bug is a surface bug; an engine bug is every surface's bug, so the engine is where we spend test effort first.

The CLI (apps/cli) doubles as an integration harness: it exercises the engine under real conditions with no UI, so a regression in WorkflowEngine / AgentRunner surfaces in CLI tests before it ever reaches the desktop. The engine regression harness (2.K) is the workflow-level extension of the per-provider conformance discipline below: it runs committed example workflows through relavium run … --json and asserts on the NDJSON event stream + exit code, offline and deterministic on every PR. Its fixture + scenario format is the one canonical home — reference/cli/regression-harness.md — and the recorded-LLM agent fixtures it grows into follow the same regenerate-not-hand-edit rule as the conformance fixtures.

Unit tests — Vitest

  • packages/core — the workflow engine is unit-tested exhaustively: YAML parsing and validation, node execution, the run-event sequence, fallback-chain policy, checkpoint/resume, and cost accumulation. Run events are asserted against the canonical run-event schema by their colon-namespaced names (the RunEvent union — that schema is the one home for the names, not restated here) and by their sequenceNumber ordering — never the legacy dotted names.
  • packages/llm — the seam types, the ToolNormalizer, the CostTracker pricing table, the FallbackChain runner, and each adapter's normalization logic (system-prompt placement, tool-schema reshaping, stop-reason mapping, usage extraction, streaming-chunk folding) are unit-tested with no live network. Adapter unit tests run against recorded fixtures (see below).
  • packages/shared — Zod schemas are tested for both accept and reject cases; a schema that never rejects a bad input is untested.

Tests live beside the code (*.test.ts) and never reach across the LLM seam: a core test asserts on Relavium types only, never on a vendor SDK shape (see code-style-typescript.md). One sanctioned extension: *.test.tsx is the convention for ink-mounted component tests under apps/cli's renderer layer — a JSX file is required to render an ink component through ink-testing-library and assert on captured frames / driven stdin (ADR-0068). It is the same .test. prefix (not the rejected .spec. suffix), collected by the root Vitest config alongside *.test.ts.

Security-critical primitive tests (direct, negative-case)

Security primitives are tested directly and adversarially, never only transitively through a happy-path caller — the dangerous input is exactly the one the happy path never exercises, so coverage-% is not a substitute. Each carries its own unit suite with the malicious/edge inputs spelled out:

  • The shared SSRF range-primitive (the one parser behind provider baseURL, the http_request tool, MCP server URLs, and the multimodal media url carrier): asserts it blocks the cloud metadata IP 169.254.169.254, link-local (169.254/16), loopback (127/8, ::1), private ranges (10/8, 172.16/12, 192.168/16), CGNAT (100.64/10), an IPv4-mapped IPv6 form of any of the above, a credentials-in-URL, and a non-HTTPS scheme — and re-checks the post-DNS-resolution IP and per-hop redirect targets, not just the literal hostname. A runtime-derived base URL is re-checked the same way.
  • The path-resolve guard (the Rust CAS / file layer): realpath + commonpath fail-closed rejects a .. traversal, an absolute path outside the run/session dir, and a symlink that escapes it.
  • The keychain bridge: the raw key is never returned from an IPC command and never appears in a command result; only a key reference crosses to the WebView.
  • read_media / byte delivery: an oversize upload is rejected; a cross-session handle read is denied (scope-set authz). The model-facing tool now performs authorization and metadata lookup only — it takes a whole handle, returns a text descriptor, and delivers the media as a top-level part on a synthesized message (ADR-0089 §1, landed in W5/CR-50), so it has no Range to exercise and its tests assert instead that no range parameter is accepted or advertised.
  • Range validation (byte delivery): a negative, reversed (end < start), or out-of-bounds Range is rejected. Re-homed here from the clause above when CR-50 landed, per ADR-0089 §1: the caller is the desktop read_media(ref) display command (1.AH), the only one that still delivers bytes. The engine-pure primitive it must use (validateByteRange, fail-closed against the host-populated byteLength) ships and is unit-tested in @relavium/shared; this clause is an open acceptance criterion for 1.AH, not a satisfied one — stated so the requirement is visibly waiting for its caller rather than quietly gone with the code that used to meet it.
  • INLINE_MEDIA_CEILING + per-message caps: an over-ceiling base64 part, an over-count message, and an over-aggregate-bytes message are each rejected (the inputs the happy path never sends).

These are acceptance criteria for the workstreams that build them (the shared SSRF primitive at 1.AE; the MediaStore / read_media / Rust CAS at 1.AF/1.AH) — green direct tests, not coverage-only. A normalized LlmError.message/code being secret-free is likewise asserted by each adapter's secret-safety unit test (a planted secret → a secret-free surfaced LlmError) and by llm-error.test.ts (the makeLlmError→scrubSecrets choke-point backstop), not assumed (see security-review.md).

Per-provider conformance tests

Each @relavium/llm adapter must pass one shared conformance suite — a single spec run against every provider — proving the adapter honors the seam contract: streams text, calls a tool and returns a normalized tool_call, returns usage, maps stop reasons to the canonical enum, and surfaces errors as a classified LlmError (see error-handling.md). This suite is the single biggest leverage point for the in-house abstraction: it converts provider drift from a production incident into a red CI run.

It runs in two modes:

  • On every PR — recorded fixtures. The suite replays committed request/response fixtures (including streamed SSE transcripts) so PR CI is fast, deterministic, offline, and uses no API keys or quota. Fixtures are checked in and reviewed like code; a fixture must be regenerated, not hand-edited, when a provider's wire format changes.
  • Nightly — live APIs. The same suite runs against the real Anthropic, OpenAI, Google, and DeepSeek endpoints on a schedule, using keys from CI secrets (never committed, never logged). A nightly failure is the early-warning signal that a provider changed its contract under us. New model ids and pricing-table entries are validated here.

Implementation qualification — 2026-10-10: the ordinary shared response suites use hand-authored synthetic seeds; Gemini's older suite uses a transport double. W8 adds a separate 24-case canonical installed-SDK HTTP matrix for Anthropic, OpenAI, DeepSeek and Gemini: generate/stream structured output, named tools and two-turn continuations, including applicable signatures. Independently authored static request expectations verify method, endpoint, query, protocol headers and structural JSON. Its internal exchange harness owns projection, sequence, recording and credential safeguards. The ordinary LLM test task runs this matrix in required CI.

pnpm check:wire-mutations independently changes three actual production-lowering sites in isolated byte copies and requires wire mismatches, then exact restoration and a passing full matrix after each change. The runner guide owns its procedure and report format. This explicit offline author/reviewer command supplements the ordinary CI matrix; it does not authorize provider calls.

Live cases remain separate smoke/effort probes, not execution of the complete offline matrix or an automatic fixture recorder. Synthetic protocol seeds are reviewed as authored test code; the earlier regeneration rule applies to genuine captured evidence. The five genuine W7 overflow artifacts retain distinct immutable provenance. Neither these tests nor their synthetic fixtures claim complete live-provider coverage or unknown-usage accounting acceptance. The W8 register records staged review and acceptance rather than inferring closure from test count.

End-to-end tests — Playwright

The desktop app (apps/desktop, Tauri v2) has Playwright e2e coverage for the critical user journeys: create/import a workflow, configure an agent and provider key, run a workflow and watch the live run-event stream render, hit and resolve a human gate, and read run history and cost. E2e tests drive the real app, not mocked stores. They cover journeys, not exhaustive logic — exhaustive logic belongs in engine unit tests.

Coverage expectations

  • packages/core, packages/llm, and packages/mcp: high line and branch coverage, floor ≥ 90%, because branch coverage is what catches the error/fallback/edge paths that matter here — and packages/mcp fences a security-critical seam (the SDK + node:child_process). The JSON-Schema→Zod compiler it used to own now lives in @relavium/shared and is tested there. Coverage is a floor and a signal, not the goal — an uncovered branch is a question to answer, not a number to game.
    • Where the floor blocks a merge: packages/llm and packages/mcp only. packages/core is measured and reported by the same run but does not fail CI — its branch margin is +0.83 (90.83, measured 2026-07-29) and Phase 2.5.5's Waves 1–3 edit core heavily, so gating merges on a sub-1-point margin costs more than it catches. It is promoted once Wave 3's test-coverage items land. A local pnpm coverage enforces all three; pnpm coverage:enforced is the CI subset. Stating this here rather than only in ci.yml is the point: the standard previously claimed an enforcement the pipeline did not perform.
  • Every bug fix lands with a regression test that fails before the fix.
  • Surfaces (apps/*, packages/ui): smoke + critical-journey coverage; deep logic is pushed down into the engine and tested there.

CI gate

PRs must pass: typecheck, lint (code-style-typescript.md), all unit tests, and the fixture-mode conformance suite. The live conformance suite and the desktop e2e suite run nightly and on release branches. A red required check blocks merge.

pnpm ci:workspaces is the common local/required-CI workspace gate. It completes lint and typecheck before running all workspace test tasks with one Turbo task at a time. Each package retains its Vitest pool and its tests' native child-process races; no test is filtered and no readiness or case deadline is extended. This prevents independent package pools from multiplying the same CPU budget while linters and typecheckers also run. Required CI retains synthetic native-worker timings separately from cached task results; a local or advisory pass does not clear a failed required run. The full pnpm run ci adds the remaining build, formatting, isolation, smoke and causal checks.

Test isolation is part of that gate (pnpm lint:test-isolation, tools/test-isolation/check.mjs). A root run must collect this repo's tests and nothing else. A second checkout of the repo inside the working tree — an agent worktree, a scratch clone — is otherwise collected in full: measured once at 234 foreign test files out of 472, with its sources in the coverage denominator. The guard fails on any collected file whose ancestor carries its own pnpm-workspace.yaml, so it stays true even when the exclusion list in vitest.config.ts is the thing that is wrong. It also asserts every workspace still yields tests, because the cheapest way to pass an isolation check is an over-broad exclude that goes green by not running.