testing.md

Testing Caffold

Run the smallest test that exercises the changed owner first. Before handoff, expand to every boundary affected by the change. A mocked browser test, protocol/schema check, live agent run, and installed macOS application check provide different evidence and must be reported separately.

Setup

The npm package lives in frontend/. Run every JavaScript and browser command from that directory; Rust and release commands stay at the repository root.

Use Node.js 26 LTS and the committed npm lockfile:

cd frontend
npm ci
npx playwright install chromium

npm ci also installs the Lit release the server embeds and the TypeScript compiler, so it comes before the Rust checks as well as the browser suites. Cargo builds run tsc on Node.js to compile the frontend's TypeScript modules and stop on a type error, and npm run test:unit runs the unit tests from tsc's output.

Rust checks use the 1.96 toolchain pinned by rust-toolchain.toml and bounded by Cargo.toml's minimum supported version.

Rust

cargo test
cargo fmt --check
cargo clippy --all-targets -- -D warnings

Prefer an owner-focused cargo test <name> while iterating. Run the full suite when the change crosses storage, process, HTTP, Git, or agent integration boundaries.

Coverage is supporting evidence, not a replacement for boundary tests. When cargo-llvm-cov is installed, measure the affected production path and inspect its missing lines:

cargo llvm-cov --summary-only
cargo llvm-cov report --text --show-missing-lines

Caffold does not currently enforce a repository-wide coverage percentage. Explain the production paths measured and any integration boundary the coverage run did not exercise.

The opt-in macOS capture check blocks the sole Tokio worker and log writer and verifies a real stack report. It needs /usr/bin/sample permission and must run with --exact because it owns the process's global log subscriber:

cargo test --locked -p caffold --lib app::stall::tests::a_blocked_runtime_automatically_saves_real_macos_thread_stacks -- --ignored --exact --nocapture

JavaScript and browser tests

Commands are owned by the thing they verify rather than by one package, so each one records where it runs from and what it needs:

Command	Run from	Requires	Boundary
npm run typecheck	frontend/	Node	TypeScript types of the frontend's .ts modules
npm run lint	frontend/	Node	Biome formatting and lint rules for .ts and .css files, plus TypeScript import order
npm run test:unit	frontend/	Node	all focused frontend Node unit tests
npm run test:contract	frontend/	Node	frontend policy and browser-infrastructure contracts, plus the repository, release, and protocol contracts that have not yet moved to an owner
npm run test:e2e	frontend/	Node, Chromium, a built server	deterministic fixture-backed Playwright coverage
cargo test --test codex_protocol -- --ignored	repository root	installed Codex CLI	Codex CLI schema compatibility without authentication or model usage
cargo test -p caffold --lib codex -- --ignored --list	repository root	Rust test toolchain	discover Codex backend live checks; execution requirements depend on the selected test
npm run test:codex-live	frontend/	authenticated Codex CLI	authenticated Codex browser coverage with model usage
cargo test -p caffold-runner --test live -- --ignored	repository root	authenticated Claude CLI	that Claude still returns an unanswered permission request to a client that reattaches, with model usage
cargo test -p caffold --test claude_live -- --ignored --test-threads=1	repository root	authenticated Claude CLI	what a person sees when the backend is replaced or the runner is killed under a working Claude Task, that each permission decision does what it says, that the agent reaches the tool Caffold serves it, and that the installation reports its status, with model usage
cargo test -p caffold --lib -- agent::grok	repository root	Rust test toolchain	the Grok driver against a scripted leader that answers as grok 1.0.30 did: creation, turns, approvals, history, worktree moves, reconnection, and the Settings report, without an installed Grok
cargo test -p caffold --test antigravity_live -- --ignored --test-threads=1	repository root	authenticated macOS arm64 ACP installation	official package, cached Google sign-in, native history, permissions and backend replacement; the prompt scenario consumes model usage
cargo test -p caffold --lib voice -- --ignored --list	repository root	Rust test toolchain	discover the voice live checks; a transcription check needs a Whisper model file or a provider API key and a speech recording, and the Grok key check needs only internet access
node --test docs/tests/*.test.mjs	repository root	Node	documentation index, links, entrypoints, and this command index
node --test scripts/tests/*.test.mjs	repository root	Node	release version tooling
desktop/macos/test-contracts	repository root	Node	macOS packaging, release, and installer contracts, from desktop/macos/tests/
desktop/macos/test-runtime	repository root	macOS, Xcode tools	Swift wrapper lifecycle and external stack capture
desktop/macos/test-system-status	repository root	macOS, Xcode tools	Swift system-status behavior
desktop/macos/test-updater	repository root	macOS, Xcode tools	Swift updater behavior

The macOS contracts verify shell scripts, workflow definitions, and documentation, so they run on any platform and belong in the ordinary pull request checks. Only the Swift programs and the one packaging-metadata check need a macOS host; that check skips itself elsewhere.

Every test belongs to the thing it verifies. Owners outside the frontend package keep their Node contracts in a tests/ directory of their own and are invoked by path: docs/tests/ verifies the documentation, scripts/tests/ verifies the repository tooling, and desktop/macos/tests/ verifies the macOS application. The frontend package keeps its own under frontend/tests/contracts/.

The workspace has two Rust members, and each keeps its tests with it. The server is caffold/, and caffold/tests/ holds its Cargo integration tests and the fixtures the backend shares: native-agent HTTP/process live suites, the Codex protocol contract, and the stub Codex CLI that readiness tests and the browser server both use. The shared process runner is runners/caffold/, and runners/caffold/tests/ holds its own: a suite driven by a stand-in agent, which is deterministic and spends no model usage, and the opt-in live check above. cargo test from the repository root covers every member, so neither needs its own job.

A test written in Node does not make it frontend material, and a contract on the backend's own boundary belongs to the backend even when a different harness would be easier.

Continuous integration follows the same ownership. Pull-request checks run one job per owner, and each job name says both whose it is and what kind of verification it performs:

Job	Reusable workflow in .github/workflows/	Verifies
Frontend Tests	frontend-tests.yml	TypeScript types, formatting and lints of .ts and .css files, colocated units, and frontend contracts
Documentation Contracts	documentation-contracts.yml	the documentation index, links, and command index, and the user manual's strict build
Repository Tooling Tests	repository-tooling-tests.yml	the release version tooling, by calling it
macOS Packaging Contracts	macos-packaging-contracts.yml	packaging, release, and installer definitions
Browser Tests / project	browser-tests.yml	browser behavior, one job per Playwright project
Rust Checks	rust-checks.yml	formatting, lints, and the Rust suites

A failing check therefore names its owner without being opened. The contract jobs read shell scripts, workflow definitions, and documentation rather than running a macOS application, so macOS Packaging Contracts needs no macOS host and runs on the ordinary Ubuntu runner.

Each owner keeps its ubuntu-26.04 check in its own reusable workflow. CI uses Node.js 26 LTS, matching the repository .nvmrc. checks.yml and release.yml call each one directly, so the commands, project matrix, toolchains, and failure artifacts have one definition per owner. The entrypoint workflows choose which checks to run and enforce their dependencies. Every shared check tests the caller's commit directly, without release-specific inputs or candidate restoration.

A pull request or main push that changes only files under website/ skips checks.yml: the user manual's pages, images, and build settings reach no other owner's check, so website-checks.yml runs Documentation Contracts alone. A change that also touches anything else runs both workflows, and a release always runs every check.

After all shared checks pass, .github/workflows/macos-release.yml prepares the release in one macOS job. For a new release it bumps the version and commits the candidate locally, then runs Swift application tests, Rust tests and lints, and the arm64 package build and verification. It uploads the package before pushing the verified candidate. Browser and other Ubuntu checks therefore run before the version bump; macOS checks and packaging verify the final version.

The macOS workflow can also be dispatched manually from main to verify and package that commit. Standalone runs, release dry-run, and resume do not bump or push the version. Only the Release entrypoint publishes GitHub and Homebrew releases after the macOS job succeeds. A contract follows reusable workflow calls to ensure every suite still runs where it can run.

Focused Node unit tests live beside their owning frontend module as name.test.js. They may import that module directly, but must not require test-only exports from a public entry point. Production ownership scans, the static import graph, the Rust asset table, and the service-worker shell inventory exclude colocated *.test.js files.

Contracts that compare multiple production owners belong to the owner they verify. Frontend policy, asset, layout, and browser-infrastructure contracts live in frontend/tests/contracts/, beside the Playwright configuration, its Node support modules, and the browser suites under frontend/tests/e2e/ and frontend/tests/live/.

Run an individual contract directly with node --test while iterating. A cross-cutting frontend or release change should run every affected boundary rather than relying on test:e2e alone.

Browser tests

npm run test:e2e

Each regular Playwright invocation selects an available loopback port, starts its own Caffold server with reuseExistingServer: false, and uses isolated fixture data. It runs the desktop, foldable, and phone viewport projects and the server-changes and server-steady projects. Its Codex responses are deterministic fixtures, so it does not prove compatibility with the installed Codex app-server. Set CAFFOLD_E2E_PORT to an available port only for targeted diagnostics; an occupied override fails instead of attaching to that server.

Every browser test declares its viewport coverage with a Playwright tag; the frontend review policy's Browser Test Ownership section decides which. @server-changes and @server-steady tests run once, in the desktop browser. When touch-only behavior needs one touch viewport, use foldable unless the phone's single-pane contract is relevant. Tags are Playwright test details, not title suffixes, and a runtime project-name skip is not a coverage declaration.

Pull-request, main, and release checks run the five projects in the same independent Ubuntu matrix jobs. Each job starts its own server, selects only its coverage tags, and uses one worker against its own fixture workspace. All five jobs must pass. Tests have no retries, and each project uploads its failure artifacts. Tracing keeps the existing on-first-retry setting, so ordinary runs with zero retries do not record traces. For a targeted diagnostic run, enable tracing explicitly with --trace on. The ordinary local command still runs the complete tagged suite in one Playwright invocation.

Test-server ports belong to individual Playwright runs, including runs in other worktrees. Do not stop a process merely because it owns a port used by an older test command; let Playwright shut down its own server, or retry a failed run.

For layout changes, inspect the generated screenshots under test-results and exercise the relevant desktop, foldable, and phone projects. For fixture or shared-state changes, compare normal parallel execution with --workers=1.

Foreground recovery changes require the adjacent unit tests, the owning Playwright lifecycle spec, and, when platform signals are affected, the installed-Android checks in mobile-pwa-testing.md. These are separate unit, browser-integration, and platform evidence.

Application update changes require the update module's Rust tests and desktop/macos/test-updater. A change to how caffold update calls Homebrew or replaces, starts, or restores the app also requires the Homebrew checks in app-update-testing.md before it merges.

PWA build-handoff changes require the adjacent unit tests, frontend/tests/contracts/service-worker.test.mjs, and frontend/tests/e2e/app-shell-update.spec.js. The loopback lifecycle server provides real Chromium service-worker replacement coverage. Its core recovery case keeps an old document alive after the target controller takes over, drops the first navigation, verifies that page resume does not repeat it automatically, and requires a second explicit Reload action to replace the document. No handoff target is persisted across documents; a fresh page reconstructs update availability from browser and server state. The loopback test arms its exact waiting worker through a fixture-only MessageChannel and waits for that worker to acknowledge the production activation request. It then uses ServiceWorker.stopAllWorkers inside the isolated browser context to clear the active worker's internal pending-event boundary without waiting for CDP version events. It rearms the page's original waiting worker, verifies its build ID again, and invokes skipWaiting() in that build's Playwright service-worker runtime without another extendable message event. CDP does not select or activate the waiting worker. Activation completion is the page's original worker object's statechange. The gate records every boundary so the waiting-to-active edge does not infer progress from scheduling latency or registration-slot polling. Production activation behavior remains separate contract and mocked-browser evidence. Report unit, complete-shell inventory, deterministic browser, and real-browser evidence separately.

User manual scenarios

frontend/tests/e2e/manual/ owns the screenshots in the user manual and the README. Each test renders one state the manual shows from frontend/tests/e2e/support/manual-fixture.js, a sample workspace with a fixed clock, time zone, theme, and typefaces, and checks what the manual says about that state before it captures the image. A renamed label or a removed control therefore fails its scenario; a purely visual change does not. Run them with:

npm run test:e2e -- tests/e2e/manual

Each scenario writes <name>-<project>.png under test-results, which every run empties. These are candidate images, not committed visual baselines. Open each one before using it, because a passing scenario can still capture an error, an empty state, or a clipped control. Then copy the images of that passing run, unchanged and under the same names, to website/docs/assets/screenshots/, where the manual and the README reference them. Documentation Review defines when an image has to be replaced.

Name a scenario and its image after the surface they show, so that a search for the surface finds both. Keep the sample content short and representative; edge cases and layout stress data belong in the owning behavioral fixtures.

Codex compatibility and live tests

To verify status against an already running daemon without starting, restarting, or updating it, run:

cargo test -p caffold --lib agent::codex::tests::live_status_reads_the_current_daemon_versions -- --ignored --exact

This opens a temporary proxy connection and reads status and version facts. It creates no thread or turn and consumes no model usage.

The installed CLI compatibility check does not authenticate or start a Codex session and does not consume model usage:

cargo test --test codex_protocol -- --ignored

It runs codex app-server generate-ts --experimental and verifies the schema contract required by Caffold. The command requires a supported Codex CLI on the local executable search path.

The browser task loop uses a real authenticated Codex installation and consumes model usage:

npm run test:codex-live

The browser run prints a usage summary and writes test-results/codex-live-usage.json. Per-test and per-model token breakdowns come from the final cumulative thread/tokenUsage/updated notification for each thread. The report also snapshots account lifetime tokens and both the overall and named model rate-limit windows before and after the suite.

Model-backed Codex live checks use explicit models with low reasoning effort, except the browser Task-naming scenario, which uses high. The browser Task, Task-naming, Fast-mode, and multimodal scenarios use gpt-6-luna. Rust reconnect and MCP replacement/restart checks also use gpt-6-luna. The browser current-plan scenario and Rust clarification and MCP approval checks use gpt-6.1-sol.

Rate-limit usedPercent values have integer resolution, so a non-zero live run can legitimately report a 0pp change. Account snapshots can also include other Codex activity on the same subscription during the run; use the thread token totals when the live suite itself must be isolated precisely.

By default each invocation selects an available loopback port and starts an isolated server with runtime files below target/. It creates real Codex threads and archives the threads it records during teardown. It is serialized and separate from the deterministic browser suite. CAFFOLD_LIVE_PORT pins an available local port for diagnostics. CAFFOLD_LIVE_URL and CAFFOLD_LIVE_CWD are advanced overrides; do not point them at the installed application's data directory.

Use node scripts/dev/probe-codex-app-server.mjs THREAD_ID for an explicit maintainer probe of resume/read/page latency and payload size. The probe does not send a prompt, but it does resume the supplied thread through a temporary app-server connection.

Codex backend live checks are ignored by the ordinary Rust run and live beside the modules they verify. List them without starting a provider or using a model:

cargo test -p caffold --lib codex -- --ignored --list

Replace TEST_NAME with a full name from that list to run one check:

cargo test -p caffold --lib TEST_NAME -- --ignored --exact

The selected test's #[ignore] reason and setup define its installation, authentication, configuration, and process requirements. Checks that submit model turns consume usage; protocol and empty-thread checks that submit no turns do not. Scenario details and assertions stay in the owning test code. Backend checks provide evidence for their integration boundary; the browser suite verifies the user-facing Task loop.

Claude compatibility and live tests

The runner's deterministic suite uses a stand-in process and spends no model usage. Its ignored live check verifies the one boundary a stand-in cannot: whether the installed Claude CLI redelivers an unanswered permission request after the client reattaches.

cargo test -p caffold-runner --test live -- --ignored

The backend live suite drives the shipped Caffold server and runner against an authenticated Claude CLI:

cargo test -p caffold --test claude_live -- --ignored --test-threads=1

It covers Task execution and approvals, transcript recovery, backend replacement during a turn, stale-child cleanup after a runner is killed, Caffold-served tools, managed-worktree movement, and installation status. It spends model usage and is serialized because the scenarios share the installed account and exercise process-start ordering. Re-run it when changing the minimum Claude version, protocol reader, runner lifecycle, approval mapping, or served tools.

Grok checks

The Grok driver keeps its deterministic suite inline, beside the code it verifies, and plays the leader from answers recorded from grok 1.0.30; the recordings live under caffold/src/agent/grok/fixtures/. The browser suite serves /api/grok/status from a fixture. There is no ignored live Grok suite: checking the installed CLI is a manual run of caffold serve against a throwaway directory, where Settings → Grok and a Grok Task show what the installation does. Such a run starts Caffold's Grok leader on ~/.grok/leader-caffold.sock and leaves it running.

Antigravity compatibility and live tests

The deterministic suite covers the pinned native record reader, ACP input and approval state, runner replay, and remote callback validation. Browser cases are settings-antigravity.spec.js and tasks/antigravity-input.spec.js.

On an authenticated macOS arm64 host, build both shipped binaries before the opt-in public HTTP checks:

cargo build -p caffold -p caffold-runner
cargo test -p caffold --test antigravity_live official_install_auth_catalog_and_empty_task_lifecycle -- --ignored --nocapture
cargo test -p caffold --test antigravity_live native_prompt_permission_survives_display_reconstruction -- --ignored --nocapture

The first downloads and verifies the official pinned ACP release, uses cached personal Google authentication, reads model choices, and creates/archives/ restores/deletes an empty Task without model prompts. It also replaces the AGY runner over HTTP, checks process isolation and preserved authentication, and reopens the same native conversation. The second spends one short prompt and checks native permissions, backend process replacement and exact-ID history, plus context retention across active and idle backend replacement. Neither signs out or copies credentials. Complete Google sign-in in Settings first; a missing cached sign-in cancels the test's job. Each test reports its temporary evidence directory and retains bindings on failure so cleanup can use exact native IDs. A passing deterministic suite does not establish these official package, account or native runtime boundaries.

Voice provider checks

The voice suite keeps its deterministic tests inline and answers OpenAI, Gemini, and Grok requests from local stand-in servers, so it needs no model file, API key, or internet access. Its ignored live checks verify what a stand-in cannot. The transcription checks use a 16 kHz mono 16-bit PCM WAV recording that contains speech:

CAFFOLD_WHISPER_MODEL=/path/to/ggml-large-v3-turbo.bin CAFFOLD_WHISPER_WAV=/path/to/speech.wav \
  cargo test -p caffold --lib live_pinned_model_transcribes_a_real_wav -- --ignored
CAFFOLD_OPENAI_API_KEY=... CAFFOLD_VOICE_WAV=/path/to/speech.wav \
  cargo test -p caffold --lib live_openai_transcribes_a_real_wav -- --ignored
CAFFOLD_GEMINI_API_KEY=... CAFFOLD_VOICE_WAV=/path/to/speech.wav \
  cargo test -p caffold --lib live_gemini_transcribes_a_real_wav -- --ignored
CAFFOLD_GROK_API_KEY=... CAFFOLD_VOICE_WAV=/path/to/speech.wav \
  cargo test -p caffold --lib live_grok_transcribes_a_real_wav -- --ignored

The OpenAI, Gemini, and Grok checks send the recording to that provider and spend the key's account usage.

One more check needs only internet access. It sends xAI a key that does not exist and confirms that Caffold reads the answer as a rejected key:

cargo test -p caffold --lib live_grok_rejects_an_incorrect_key -- --ignored

macOS application tests

desktop/macos/ owns its own tests. Their sources live in desktop/macos/tests/, beside the production Swift they compile against.

The Swift programs require macOS and the Xcode command-line tools:

desktop/macos/test-runtime
desktop/macos/test-system-status
desktop/macos/test-updater

Each compiles the production wrapper source together with its test program. The runtime test verifies owned-child shutdown and real stack capture during HTTP failure. It requires local sockets and permission for /usr/bin/sample to inspect disposable children.

The packaging, release, and local-install contracts need only Node:

desktop/macos/test-contracts

The local-install contract uses controlled fake system tools to verify that an orphaned bundled server blocks replacement and that rollback stops a failed runtime before restoring the backup. The one check that reads real packaging metadata skips itself off macOS arm64.

When changing the application wrapper, process lifecycle, packaging, updater, or installer, also run:

desktop/macos/package-app build

An actual replacement of /Applications/Caffold Server.app is direct runtime validation, not an automated test. Perform it only when the change needs user review in the installed application, then verify the expected build ID and port owner as described in the macOS development guide.