diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 9551fac..3ab5c3d 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -140,6 +140,30 @@ jobs: path: dist merge-multiple: true + - uses: actions/setup-node@v6 + with: + node-version: 20 + + # Static contract on the build artifacts, before anything is attested + # or published: exact 5-artifact matrix, per-format magic bytes, and + # every checksum matches its file. Catches a corrupted/truncated/ + # wrong-architecture artifact — fail_on_unmatched_files below only + # catches a totally missing one. See tests/releasecontract/README.md. + - name: Verify the release artifact contract + run: | + node tests/releasecontract/verify-release.mjs \ + --directory dist \ + --version "${GITHUB_REF_NAME}" \ + --report artifacts/desktop-release-contract/report.json + + - name: Upload the release contract report + if: always() + uses: actions/upload-artifact@v7 + with: + name: release-contract-report + path: artifacts/desktop-release-contract/ + if-no-files-found: warn + - name: Attest release artifacts uses: actions/attest-build-provenance@v3 with: diff --git a/README.md b/README.md index 014b846..e5bf9c0 100644 --- a/README.md +++ b/README.md @@ -70,6 +70,7 @@ The dashboard opens immediately. If you don't have a ready Podman runtime yet, a ## Documentation - [`DEVELOPMENT.md`](./DEVELOPMENT.md) — full dev environment setup, testing (including previewing onboarding screens without touching real Podman state), and the command reference +- [`TESTING.md`](./TESTING.md) — the full test-layer breakdown (source, release contract, native smoke, manual) and what's deliberately not built yet - [`RELEASING.md`](./RELEASING.md) — tagging, the CI release pipeline, and the review-gate policy - [`AGENT.md`](./AGENT.md) — architecture, non-negotiable rules, and current build status (the primary reference for contributing code) - [`DESIGN.md`](./DESIGN.md) — screen-by-screen UI/UX reference diff --git a/RELEASING.md b/RELEASING.md index f9aac50..a571f4d 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -75,15 +75,11 @@ to inspect or change the required reviewers. developer" warning; Windows shows a SmartScreen "unknown publisher" prompt. Fine for early alpha/internal use; worth fixing (via GitHub Environment secrets + the relevant Tauri signing config) before pointing this at the - general public. -- **No release-artifact contract check.** Nothing yet asserts the expected - installer files actually exist with the expected names/checksums before - publishing — `release.yml`'s `fail_on_unmatched_files: true` catches a - totally missing platform, but not a subtly wrong one. See - `reference/desktop-hardening-migration-PLAN.md`'s Phase 7 for the sibling - repo's `tests/releasecontract` pattern if this becomes worth porting. -- **No `TESTING.md`.** This file assumes `ci.yml`'s green checkmark is - sufficient evidence; there's no separate document defining test layers, - promotion gates, or required manual sign-off yet. Fine at this repo's - current stage (single maintainer, no external users depending on - releases); revisit if that changes. + general public. `tests/manual/published-artifact.md` covers verifying + this is the *only* warning shown. + +See [`TESTING.md`](TESTING.md) for the full test-layer breakdown and its +own "Deliberately not done yet" list (self-hosted hardware runners, +`run.ps1`, destructive host-reset tooling, post-publication +re-verification) — a release-artifact contract check and this doc both +used to be listed here as gaps; both are done now. diff --git a/TESTING.md b/TESTING.md new file mode 100644 index 0000000..fc21a72 --- /dev/null +++ b/TESTING.md @@ -0,0 +1,89 @@ +# Testing + +The authoritative source for what test layers exist, what each one proves, +and what's deliberately not built yet. Modeled on the sibling repo +(`omnideck/desktop`)'s `TESTING.md`, scaled down to this repo's actual +scope and maturity — see "Deliberately not done yet" below for what was +cut and why. + +A passing build proves only its stated layer. Compilation is not +installation, a cross-build is not native execution, and an unexecuted +manual procedure is `blocked` coverage, not a pass. + +## Test layers + +| Layer | Implementation | Where it runs | Proves | +|---|---|---|---| +| Source | `test.yml` (reusable): sidecar fetch/verify, policy tests, typecheck, `cargo fmt`/`test`/`clippy` | Every PR, push to `main`, and tag push | Code compiles, security-posture invariants hold, sidecar checksums match the pin | +| Release contract | `tests/releasecontract/verify-release.mjs` | `release.yml`'s `publish` job, before attestation/publish | The 5-artifact matrix is exactly right: correct filenames, per-format magic bytes, checksums match | +| Native packaged smoke | `tests/hardware/run.sh` + `validate-proof.mjs` | Manual, on a real machine with a display session (see `tests/hardware/README.md`) | A real installed/launched build actually reaches and correctly parses real CLI output on that OS — not a fixture | +| Manual journey | `tests/manual/*.md` | Manual, on a disposable VM or dedicated machine | First-run setup, recovery/interruption handling, published-artifact trust experience | + +Hosted CI (source + release contract) runs on every relevant push/PR/tag. +Native packaged smoke and manual journeys are run by hand before a real +release goes out to actual users — see `RELEASING.md`. + +## Automated security boundary + +Source tests (`tests/policy.test.mjs`) keep the following invariants +release-blocking: + +- the dashboard window's capability is an enumerated allowlist + (`dashboard-bridge`), never `core:default`; +- the onboarding window's capability (`onboarding-bridge`) is scoped to + `"windows": ["onboarding"]` only — never `"main"`, which is the dashboard + here (this exact mistake was caught once already; see the policy test's + own comment); +- every onboarding command re-checks `window.label() == "onboarding"` + server-side, not just the capability grant; +- `bootstrap()` only reveals the onboarding window when setup is actually + needed — never unconditionally (this was a real, caught-and-fixed + regression); +- the CLI sidecar is pinned by checksum for all 6 target triples, and the + runtime version check is a floor (`>= v0.10.0`), not an exact match; +- sidecar process output is bounded and every operation has a timeout. + +## Release gating + +`release.yml`'s `test` → `build` → `publish` chain (see `RELEASING.md`) is +the actual gate: a tag that fails any version-field check, any source +check, or the release-artifact contract never reaches `publish`. Stable +tags additionally require review in the `release` GitHub Environment; +alpha/beta/rc tags publish unattended via `release-preview`. + +There is no formal alpha/beta/RC/stable promotion ladder with named gates +per channel (contrast the sibling's, which has one) — this repo doesn't +have the release volume or team size to justify that process yet. The tag +naming convention (`-alpha.N`/`-beta.N`/`-rc.N`/bare) already exists and +GitHub correctly treats non-bare tags as prereleases; formalize gates per +channel if/when that distinction needs to mean something more than "not +GA." + +## Deliberately not done yet + +- **Self-hosted hardware runners.** The sibling's opt-in native-smoke CI + workflow runs on dedicated, always-on self-hosted machines. Standing + that up is a real infrastructure/ops commitment (cost, machine + maintenance, security surface of a self-hosted runner), not something to + build speculatively — `tests/hardware/run.sh` exists and is verified + working; running it is manual for now. +- **`run.ps1`** (Windows equivalent of `tests/hardware/run.sh`). Cheap to + write, but not written — and critically, not *tested*, since there's no + Windows machine available while writing this. `run.sh` needed two real + fixes (a broken `pgrep -x` length limit, an AppImage process-reparenting + cleanup bug) despite being ported from already-working code; don't trust + a straight port of `run.ps1` without running it for real first. +- **`visual-platform.md`** (theme/DPI/multi-monitor/accessibility/platform + fit). Worth adding once this app has real usage on more than one + platform — right now only Linux has been built, run, and verified. +- **Destructive host-reset tooling** (the sibling's + `scripts/release-test/reset-host.*`, for resetting a test machine's + WSL/Podman state between manual runs). Real, useful, and genuinely + dangerous if done wrong (the sibling's version has real safety rails — + typed confirmation, dry-run, administrator checks) — not built + speculatively alongside everything else here. +- **Post-publication re-verification workflow** (downloading the *public* + release assets fresh and re-running the release contract + attestation + check, rather than trusting the build-time artifacts). `tests/manual/ + published-artifact.md` covers this by hand; automating it is a + reasonable, cheap follow-up once it's worth not doing by hand every time. diff --git a/package.json b/package.json index 139f33b..afe54ce 100644 --- a/package.json +++ b/package.json @@ -14,7 +14,7 @@ "fetch:sidecars": "node scripts/fetch-sidecars.mjs", "verify:sidecars": "node scripts/verify-sidecars.mjs", "checksums": "node scripts/checksums.mjs", - "test:policy": "node --test tests/*.test.mjs", + "test:policy": "node --test tests/*.test.mjs tests/releasecontract/*.test.mjs tests/hardware/*.test.mjs", "typecheck": "tsc --noEmit", "test:rust": "cargo test --manifest-path src-tauri/Cargo.toml --locked", "lint:rust": "cargo clippy --manifest-path src-tauri/Cargo.toml --locked --all-targets -- -D warnings", diff --git a/tests/hardware/README.md b/tests/hardware/README.md index 0d39da3..2f0b6cd 100644 --- a/tests/hardware/README.md +++ b/tests/hardware/README.md @@ -7,29 +7,50 @@ Launches a real packaged build (not `tauri dev`) with `run_packaged_smoke`/`record_packaged_smoke` in `src-tauri/src/lib.rs`. `validate-proof.mjs` checks the pinned CLI version/commit against `vendor-manifest.json`, runtime schema 4, the exact read-only operations, -and the application hash. +and the application hash. `validate-proof.test.mjs` unit-tests the +validator itself against synthetic fixtures. -Ported from the sibling repo's `tests/hardware`. Not yet ported: its -`run.sh`/`run.ps1`/`reset-host` harness scripts and the opt-in self-hosted -CI workflow — those assume real per-platform installers and a release -pipeline this repo doesn't have yet (Phase 6/7 of -`reference/desktop-hardening-migration-PLAN.md`). For now, run this by hand -against a real build: +Ported from the sibling repo's `tests/hardware`, now including `run.sh` +(build a real installer, run it with the smoke env var, wait for the proof, +validate, clean up): ```sh -npm run build:appimage -export OMNIDECK_DESKTOP_SMOKE_FILE=/tmp/omnideck-smoke-proof.json -./src-tauri/target/release/bundle/appimage/Omnideck_*.AppImage --appimage-extract-and-run & -sleep 5 -kill %1 -node tests/hardware/validate-proof.mjs \ - --proof /tmp/omnideck-smoke-proof.json \ - --application ./src-tauri/target/release/bundle/appimage/Omnideck_*.AppImage \ - --report /tmp/omnideck-smoke-report.json \ +npm run build:linux # or build:windows / build:macos +bash tests/hardware/run.sh \ + --application src-tauri/target/x86_64-unknown-linux-gnu/release/bundle/appimage/Omnideck_*.AppImage \ --require-ready # only if the test machine's Podman runtime is already ready ``` Requires an actual display session (X11 or Wayland) — this launches the -real GUI process, not a headless check. A missing target or unexecuted step -is blocked coverage, never a pass. Generated evidence belongs under -`artifacts/desktop-hardware/` (not committed). +real GUI process, not a headless check. Verified directly against a real +build (not just written and assumed correct) — two real bugs were caught +and fixed doing that: + +- `pgrep -x omnideck-desktop`'s "refuse to run if already running" check + silently never matched anything at all — the kernel truncates `comm` to + 15 bytes, and `omnideck-desktop` is 16. Switched to `pgrep -f` matching + the full command line instead, which isn't length-limited. +- An AppImage launched via `--appimage-extract-and-run` reparents away from + the PID this script tracks during extraction, so cleanup only killing + that one PID left a fully running app + WebKit process tree behind. + `pkill -f` as a cleanup fallback catches it regardless of the actual + process-tree relationship. + +Both fixes use the `[o]mnideck-desktop` bracket form, not +`omnideck-desktop` — the standard guard against `pgrep -f`/`pkill -f` +matching their own invocation (their own argv literally contains the +search string). + +Not yet ported: `run.ps1` (Windows), `reset-host.sh`/`reset-host.ps1`, and +the opt-in self-hosted-runner CI workflow that drives all of this +automatically. `run.ps1` is a reasonable, cheap follow-up (same shape as +`run.sh`, just PowerShell) but hasn't been written *or tested* — don't +trust a straight port of it without running it on real Windows first, the +same way `run.sh` needed two real fixes despite being ported from +already-working code. The self-hosted-runner workflow is a real +infrastructure/ops decision (dedicated always-on test hardware), not +something to stand up speculatively — see `TESTING.md`. + +A missing target or unexecuted step is blocked coverage, never a pass. +Generated evidence belongs under `artifacts/desktop-hardware/` (not +committed). diff --git a/tests/hardware/run.sh b/tests/hardware/run.sh new file mode 100755 index 0000000..fca9d04 --- /dev/null +++ b/tests/hardware/run.sh @@ -0,0 +1,126 @@ +#!/usr/bin/env bash +# Launches a packaged desktop executable with OMNIDECK_DESKTOP_SMOKE_FILE +# set, waits for the read-only smoke proof lib.rs's run_packaged_smoke() +# writes, validates it, and terminates the host. Ported from the sibling +# repo's tests/hardware/run.sh, adapted to this repo's binary name +# (omnideck-desktop, not omnideck) and without OMNIDECK_DESKTOP_USER_DATA +# (this app has no isolated-user-data-directory feature — it uses the +# CLI's/Podman's normal shared state, per the multi-instance model). +set -euo pipefail + +script_dir="$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)" +desktop_root="$(CDPATH= cd -- "$script_dir/../.." && pwd)" +application="" +output_directory="${OMNIDECK_DESKTOP_SMOKE_OUTPUT_DIR:-}" +timeout_seconds=45 +require_ready=false + +usage() { + echo "Usage: $0 --application PATH [--output DIR] [--timeout SECONDS] [--require-ready]" +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --application) application="${2:?Missing application path}"; shift 2 ;; + --output) output_directory="${2:?Missing output directory}"; shift 2 ;; + --timeout) timeout_seconds="${2:?Missing timeout}"; shift 2 ;; + --require-ready) require_ready=true; shift ;; + --help|-h) usage; exit 0 ;; + *) echo "Unknown option: $1" >&2; usage >&2; exit 1 ;; + esac +done + +[[ -n "$application" ]] || { usage >&2; exit 1; } +[[ -x "$application" ]] || { echo "Application is not executable: $application" >&2; exit 1; } +[[ "$timeout_seconds" =~ ^[0-9]+$ ]] || { echo "Timeout must be an integer." >&2; exit 1; } +(( timeout_seconds >= 5 && timeout_seconds <= 300 )) || { echo "Timeout must be between 5 and 300 seconds." >&2; exit 1; } +command -v node >/dev/null 2>&1 || { echo "Node.js is required to validate the packaged smoke proof." >&2; exit 1; } + +# -f, not -x: the kernel truncates comm to 15 bytes, and "omnideck-desktop" +# is 16 — pgrep -x can never match it (confirmed directly: it prints "pattern +# ... longer than 15 characters will result in zero matches" and always +# exits 1, silently disabling this safety check entirely). -f matches +# against the full command line instead, which isn't length-limited. +# "[o]mnideck-desktop" (not "omnideck-desktop") is the standard pgrep +# self-match guard: pgrep's own argv literally contains the search string, +# so an unguarded pattern matches the pgrep/pkill invocation itself — +# confirmed directly, this pattern always "found" a match with none +# actually running. The bracket makes the *string* "[o]mnideck-desktop" +# (this command's own argv) not satisfy the *regex* `[o]mnideck-desktop`, +# while still matching a real process's plain "omnideck-desktop". +if pgrep -f '[o]mnideck-desktop' >/dev/null 2>&1; then + echo "Close every existing Omnideck process before running packaged smoke." >&2 + exit 1 +fi +if [[ "$(uname -s)" == "Linux" && -z "${DISPLAY:-}${WAYLAND_DISPLAY:-}" ]]; then + echo "A real X11 or Wayland session is required for native desktop smoke." >&2 + exit 1 +fi + +run_id="${GITHUB_RUN_ID:-local-$$}" +if [[ -z "$output_directory" ]]; then + output_directory="$desktop_root/../artifacts/desktop-hardware/$(uname -s | tr '[:upper:]' '[:lower:]')-$run_id" +fi +mkdir -p "$output_directory" +proof_path="$output_directory/smoke-proof.json" +report_path="$output_directory/report.json" +stdout_path="$output_directory/host.stdout.log" +stderr_path="$output_directory/host.stderr.log" +rm -f -- "$proof_path" + +application="$(CDPATH= cd -- "$(dirname -- "$application")" && pwd)/$(basename -- "$application")" + +# AppImage-specific: the plain FUSE-mount launch has been observed to exit +# silently within a few seconds on at least one real environment (see +# AGENT.md's AppImage runtime notes and scripts/run-appimage.sh, which +# always passes this too) — --appimage-extract-and-run sidesteps FUSE +# entirely. Harmless to always pass for an AppImage; irrelevant for the +# other platforms' installers, which don't take this flag at all. +application_args=() +if [[ "$application" == *.AppImage ]]; then + application_args+=(--appimage-extract-and-run) +fi + +OMNIDECK_DESKTOP_SMOKE_FILE="$proof_path" \ + "$application" "${application_args[@]}" >"$stdout_path" 2>"$stderr_path" & +application_pid=$! + +# Confirmed directly: an AppImage launched via --appimage-extract-and-run +# reparents away from $application_pid during extraction (its real PPID ends +# up some unrelated adopting process, not this script) — killing only the +# tracked PID here left a fully running, un-cleaned-up omnideck-desktop + +# WebKit process tree behind in testing. pkill -f as a fallback catches it +# regardless of the process-tree relationship. +cleanup() { + if kill -0 "$application_pid" >/dev/null 2>&1; then + kill "$application_pid" >/dev/null 2>&1 || true + wait "$application_pid" >/dev/null 2>&1 || true + fi + pkill -f '[o]mnideck-desktop' >/dev/null 2>&1 || true +} +trap cleanup EXIT + +deadline=$((SECONDS + timeout_seconds)) +while [[ ! -f "$proof_path" ]]; do + if ! kill -0 "$application_pid" >/dev/null 2>&1; then + echo "The desktop host exited before writing a packaged smoke proof." >&2 + exit 1 + fi + if (( SECONDS >= deadline )); then + echo "The desktop host did not write a packaged smoke proof within $timeout_seconds seconds." >&2 + exit 1 + fi + sleep 0.25 +done + +validation=( + node "$script_dir/validate-proof.mjs" + --proof "$proof_path" + --application "$application" + --report "$report_path" +) +if [[ "$require_ready" == "true" ]]; then + validation+=(--require-ready) +fi +"${validation[@]}" +echo "Evidence: $report_path" diff --git a/tests/hardware/validate-proof.test.mjs b/tests/hardware/validate-proof.test.mjs new file mode 100644 index 0000000..4ce57c8 --- /dev/null +++ b/tests/hardware/validate-proof.test.mjs @@ -0,0 +1,88 @@ +import assert from "node:assert/strict"; +import { spawnSync } from "node:child_process"; +import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import test from "node:test"; +import { fileURLToPath } from "node:url"; + +const testRoot = path.dirname(fileURLToPath(import.meta.url)); +const validator = path.join(testRoot, "validate-proof.mjs"); +const vendor = JSON.parse( + await readFile(path.join(testRoot, "../../src-tauri/binaries/vendor-manifest.json"), "utf8"), +); + +async function fixture(overrides = {}) { + const root = await mkdtemp(path.join(os.tmpdir(), "omnideck-hardware-proof-")); + const application = path.join(root, "omnideck-fixture"); + const proof = path.join(root, "proof.json"); + const report = path.join(root, "report.json"); + await writeFile(application, Buffer.alloc(1024, 1)); + await writeFile( + proof, + JSON.stringify({ + cliVersion: vendor.version, + cliCommit: vendor.commit, + schemaVersion: 4, + ready: true, + operations: ["--version", "--json runtime status"], + mutation: false, + ...overrides, + }), + ); + return { root, application, proof, report }; +} + +function run(paths, extraArgs = []) { + return spawnSync( + process.execPath, + [validator, "--proof", paths.proof, "--application", paths.application, "--report", paths.report, ...extraArgs], + { encoding: "utf8" }, + ); +} + +test("accepts the exact read-only packaged smoke proof", async (t) => { + const paths = await fixture(); + t.after(() => rm(paths.root, { recursive: true, force: true })); + const result = run(paths, ["--require-ready"]); + assert.equal(result.status, 0, result.stderr); + const report = JSON.parse(await readFile(paths.report, "utf8")); + assert.equal(report.result, "pass"); + assert.equal(report.proof.mutation, false); + assert.equal(report.application.size, 1024); +}); + +test("rejects a smoke proof that reports a mutation", async (t) => { + const paths = await fixture({ mutation: true }); + t.after(() => rm(paths.root, { recursive: true, force: true })); + const result = run(paths); + assert.notEqual(result.status, 0); + assert.match(result.stderr, /packaged smoke must remain read-only/); +}); + +test("rejects a CLI version that doesn't match the vendor manifest", async (t) => { + const paths = await fixture({ cliVersion: "v0.9.0" }); + t.after(() => rm(paths.root, { recursive: true, force: true })); + const result = run(paths); + assert.notEqual(result.status, 0); + assert.match(result.stderr, /does not match the vendor manifest/); +}); + +test("rejects the wrong runtime status schema version", async (t) => { + const paths = await fixture({ schemaVersion: 3 }); + t.after(() => rm(paths.root, { recursive: true, force: true })); + const result = run(paths); + assert.notEqual(result.status, 0); + assert.match(result.stderr, /schema version 4/); +}); + +test("--require-ready rejects a not-ready runtime; without it, not-ready still passes", async (t) => { + const paths = await fixture({ ready: false }); + t.after(() => rm(paths.root, { recursive: true, force: true })); + const strict = run(paths, ["--require-ready"]); + assert.notEqual(strict.status, 0); + assert.match(strict.stderr, /runtime was not ready/); + + const lenient = run(paths); + assert.equal(lenient.status, 0, lenient.stderr); +}); diff --git a/tests/manual/README.md b/tests/manual/README.md index 5806c30..f2462a6 100644 --- a/tests/manual/README.md +++ b/tests/manual/README.md @@ -16,16 +16,16 @@ Required procedures: - [Clean-machine first run](clean-first-run.md) - [Recovery lifecycle](recovery-lifecycle.md) - -Not yet ported from the sibling: `published-artifact.md` (needs a real -release/CI pipeline — Phase 6/7 of -`reference/desktop-hardening-migration-PLAN.md`, not done yet), -`hosted-app-behavior.md` (specific to the sibling's single hosted-instance -webview, which this repo's dashboard model doesn't have — the closest -analog here is the per-Deck instance webview, DESIGN.md #7, already covered -by this repo's existing manual verification practice, not a new document), -and `visual-platform.md` (worth adding once this app ships on more than one -platform — right now only Linux has been built/run at all). +- [Published artifact and trust experience](published-artifact.md) + +Not yet ported from the sibling: `hosted-app-behavior.md` (specific to the +sibling's single hosted-instance webview, which this repo's dashboard model +doesn't have — the closest analog here is the per-Deck instance webview, +DESIGN.md #7, already covered by this repo's existing manual verification +practice, not a new document), and `visual-platform.md` (worth adding once +this app has real usage on more than one platform — right now only Linux +has been built/run and verified at all, including the one real published +release so far). Every execution must record: diff --git a/tests/manual/published-artifact.md b/tests/manual/published-artifact.md new file mode 100644 index 0000000..e4580ad --- /dev/null +++ b/tests/manual/published-artifact.md @@ -0,0 +1,48 @@ +# Published artifact and trust experience + +Ported from the sibling repo's `published-artifact.md`. Verify the package +users actually download, including browser reputation and the operating +system's unsigned-package warning — this repo's installers aren't +code-signed yet (no Apple notarization or Windows Authenticode cert; see +`RELEASING.md`'s "Current known gaps"). + +## Purpose + +Confirm a real published release is trustworthy and installable through +the normal user path, not just that CI produced files. + +## Procedure + +1. Record the target release tag, source commit, OS, architecture, and + package format. +2. Download the package and its `.sha256` through a normal browser from the + GitHub release page (`https://github.com/omnideck-dev/desktop/releases`). + Do not substitute a local build or Actions artifact. +3. Verify the SHA-256 independently: + ```sh + sha256sum -c Omnideck__.sha256 + ``` + and verify build provenance: + ```sh + gh attestation verify Omnideck__ -R omnideck-dev/desktop + ``` + Record both results. (Confirmed working against a real published + release while writing this doc — `gh attestation verify` exits `0` with + no output in a non-interactive shell; run it in a real terminal for the + human-readable verification table.) +4. Record any browser download warnings. Confirm the filename, version, + format, and architecture are correct. +5. Open the package through the normal user path and record SmartScreen + (Windows), Gatekeeper (macOS), or Linux desktop/package-manager + warnings. +6. Confirm any warning is attributable to the documented unsigned-build + status, not corruption, a wrong architecture, or malformed packaging. +7. Complete installation, or for the AppImage, set only its executable bit + and launch it directly (no installation step). + +## Pass criteria + +Checksum and provenance both pass, the OS recognizes the intended package, +the documented unsigned-build warning is the only warning shown, and no +unexpected publisher, architecture, corruption, or duplicate-launch warning +appears. diff --git a/tests/releasecontract/README.md b/tests/releasecontract/README.md new file mode 100644 index 0000000..3156c2e --- /dev/null +++ b/tests/releasecontract/README.md @@ -0,0 +1,34 @@ +# Desktop release contract + +Treats the published desktop packages as artifacts, not source-build output. +Verifies the exact 5-package matrix (`.github/workflows/release.yml`'s +3-target build), one matching sha256sum-compatible checksum per package, +nonempty package files, container format signatures (PE for NSIS, UDIF for +DMG, ELF for AppImage, ar for deb, RPM magic bytes), and AppImage executable +architecture. + +Run it against a directory containing the release assets: + +```sh +node tests/releasecontract/verify-release.mjs \ + --directory dist \ + --version v0.5.0-alpha.2 \ + --report artifacts/desktop-release-contract/report.json +``` + +This is a static, non-installing contract. It does not prove that an +installer can actually be installed, that a GUI can reach a display server, +or that the bundled sidecar can execute. Those requirements belong to +[`../hardware`](../hardware/README.md) and [`../manual`](../manual/README.md). + +`release.yml`'s `publish` job runs this contract against the downloaded +build artifacts before attesting/publishing anything — a totally missing +platform was already caught by `fail_on_unmatched_files: true` on the +publish step, but this catches a subtly wrong one (corrupted, truncated, or +wrong-architecture artifact; a checksum that doesn't match its file). + +Not yet ported from the sibling repo: a post-publication re-verification +against the *public* release assets (downloading them fresh and checking +GitHub attestation, not just the build-time artifacts). Worth adding as a +manually-dispatched workflow once this becomes a real concern — see +`TESTING.md`. diff --git a/tests/releasecontract/verify-release.mjs b/tests/releasecontract/verify-release.mjs new file mode 100644 index 0000000..28460a6 --- /dev/null +++ b/tests/releasecontract/verify-release.mjs @@ -0,0 +1,157 @@ +// Static release-artifact contract: asserts a release directory contains +// exactly the expected installer set for this repo's 3-target build matrix +// (see .github/workflows/release.yml), with correct per-format magic bytes +// and matching checksums — not just "some files exist" +// (`fail_on_unmatched_files: true` on the publish step already catches a +// totally missing platform; this catches a subtly wrong one, e.g. a +// corrupted or empty artifact, or a checksum that doesn't match its file). +// +// Ported from the sibling repo's tests/releasecontract/verify-release.mjs, +// scaled from its 10-artifact/6-target matrix down to this repo's 5-artifact/ +// 3-target one (see reference/desktop-hardening-migration-PLAN.md's Phase 6 +// — deliberately scoped down from the sibling's linux/windows/macos × +// x64/arm64 matrix to one target per OS). +import assert from "node:assert/strict"; +import { createHash } from "node:crypto"; +import { mkdir, readFile, readdir, writeFile } from "node:fs/promises"; +import path from "node:path"; +import { pathToFileURL } from "node:url"; + +const VERSION_PATTERN = /^v(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(?:-[0-9A-Za-z]+(?:[.-][0-9A-Za-z]+)*)?$/; +const PACKAGE_PATTERN = /\.(?:AppImage|deb|dmg|exe|rpm)$/i; + +function expectedArtifacts(version) { + const bareVersion = version.slice(1); + return [ + { name: `Omnideck_${bareVersion}_x64-setup.exe`, platform: "windows", architecture: "x64", format: "nsis" }, + { name: `Omnideck_${bareVersion}_aarch64.dmg`, platform: "macos", architecture: "arm64", format: "dmg" }, + { name: `Omnideck_${bareVersion}_amd64.AppImage`, platform: "linux", architecture: "x64", format: "appimage" }, + { name: `Omnideck_${bareVersion}_amd64.deb`, platform: "linux", architecture: "x64", format: "deb" }, + { name: `Omnideck-${bareVersion}-1.x86_64.rpm`, platform: "linux", architecture: "x64", format: "rpm" }, + ]; +} + +async function filesBelow(root) { + const files = []; + async function visit(directory) { + for (const entry of await readdir(directory, { withFileTypes: true })) { + const candidate = path.join(directory, entry.name); + if (entry.isDirectory()) await visit(candidate); + else files.push(candidate); + } + } + await visit(root); + return files; +} + +function sha256(contents) { + return createHash("sha256").update(contents).digest("hex"); +} + +function verifyMagic(descriptor, contents) { + assert(contents.length >= 1024, `${descriptor.name} is unexpectedly small`); + if (descriptor.format === "nsis") { + assert.equal(contents.subarray(0, 2).toString("ascii"), "MZ", `${descriptor.name} is not a PE executable`); + return; + } + if (descriptor.format === "dmg") { + assert.equal(contents.subarray(-512, -508).toString("ascii"), "koly", `${descriptor.name} has no UDIF trailer`); + return; + } + if (descriptor.format === "appimage") { + assert.deepEqual([...contents.subarray(0, 4)], [0x7f, 0x45, 0x4c, 0x46], `${descriptor.name} is not ELF`); + assert.equal(contents[5], 1, `${descriptor.name} is not a little-endian ELF image`); + const expectedMachine = descriptor.architecture === "x64" ? 62 : 183; + assert.equal(contents.readUInt16LE(18), expectedMachine, `${descriptor.name} contains the wrong ELF architecture`); + return; + } + if (descriptor.format === "deb") { + assert.equal(contents.subarray(0, 8).toString("ascii"), "!\n", `${descriptor.name} is not a Debian archive`); + return; + } + assert.deepEqual([...contents.subarray(0, 4)], [0xed, 0xab, 0xee, 0xdb], `${descriptor.name} is not an RPM`); +} + +export async function verifyReleaseDirectory({ directory, version }) { + assert.match(version, VERSION_PATTERN, "version must be a SemVer tag such as v0.5.0-alpha.2"); + const root = path.resolve(directory); + const files = await filesBelow(root); + const byName = new Map(); + for (const file of files) { + const name = path.basename(file); + assert(!byName.has(name), `duplicate release filename: ${name}`); + byName.set(name, file); + } + + const expected = expectedArtifacts(version); + const expectedNames = new Set(expected.map(({ name }) => name)); + const publishedPackages = [...byName.keys()].filter((name) => PACKAGE_PATTERN.test(name)); + assert.deepEqual( + publishedPackages.sort(), + [...expectedNames].sort(), + "published desktop package matrix does not match the supported five artifacts", + ); + + const artifacts = []; + for (const descriptor of expected) { + const artifactPath = byName.get(descriptor.name); + const checksumName = `${descriptor.name}.sha256`; + const checksumPath = byName.get(checksumName); + assert(artifactPath, `missing release artifact: ${descriptor.name}`); + assert(checksumPath, `missing checksum: ${checksumName}`); + + const contents = await readFile(artifactPath); + verifyMagic(descriptor, contents); + const digest = sha256(contents); + const checksum = (await readFile(checksumPath, "utf8")).trim(); + const match = checksum.match(/^([a-fA-F0-9]{64})\s{2}([^\r\n]+)$/); + assert(match, `${checksumName} must contain one sha256sum-compatible line`); + assert.equal(match[2], descriptor.name, `${checksumName} names the wrong artifact`); + assert.equal(match[1].toLowerCase(), digest, `${descriptor.name} checksum mismatch`); + + artifacts.push({ + ...descriptor, + sha256: digest, + size: contents.length, + }); + } + + return { + schemaVersion: 1, + release: version, + result: "pass", + artifactCount: artifacts.length, + artifacts, + }; +} + +function parseArguments(arguments_) { + const result = {}; + for (let index = 0; index < arguments_.length; index += 1) { + const argument = arguments_[index]; + if (!["--directory", "--version", "--report"].includes(argument)) { + throw new Error(`unknown argument: ${argument}`); + } + result[argument.slice(2)] = arguments_[index + 1]; + index += 1; + } + assert(result.directory, "--directory is required"); + assert(result.version, "--version is required"); + return result; +} + +export async function main(arguments_ = process.argv.slice(2)) { + const options = parseArguments(arguments_); + const report = await verifyReleaseDirectory(options); + if (options.report) { + const reportPath = path.resolve(options.report); + await mkdir(path.dirname(reportPath), { recursive: true }); + await writeFile(reportPath, `${JSON.stringify(report, null, 2)}\n`, "utf8"); + } + process.stdout.write(`Verified ${report.artifactCount} desktop packages for ${report.release}.\n`); + return report; +} + +if (process.argv[1] && import.meta.url === pathToFileURL(path.resolve(process.argv[1])).href) { + await main(); +} diff --git a/tests/releasecontract/verify-release.test.mjs b/tests/releasecontract/verify-release.test.mjs new file mode 100644 index 0000000..a06ca17 --- /dev/null +++ b/tests/releasecontract/verify-release.test.mjs @@ -0,0 +1,92 @@ +import assert from "node:assert/strict"; +import { createHash } from "node:crypto"; +import { mkdtemp, rm, writeFile } from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import test from "node:test"; +import { verifyReleaseDirectory } from "./verify-release.mjs"; + +const version = "v1.2.3-alpha.4"; +const bareVersion = version.slice(1); +const artifacts = [ + ["nsis", `Omnideck_${bareVersion}_x64-setup.exe`, "x64"], + ["dmg", `Omnideck_${bareVersion}_aarch64.dmg`, "arm64"], + ["appimage", `Omnideck_${bareVersion}_amd64.AppImage`, "x64"], + ["deb", `Omnideck_${bareVersion}_amd64.deb`, "x64"], + ["rpm", `Omnideck-${bareVersion}-1.x86_64.rpm`, "x64"], +]; + +function fixture(format, architecture) { + const contents = Buffer.alloc(2048); + if (format === "nsis") contents.write("MZ", 0, "ascii"); + if (format === "dmg") contents.write("koly", contents.length - 512, "ascii"); + if (format === "appimage") { + contents.set([0x7f, 0x45, 0x4c, 0x46], 0); + contents[5] = 1; + contents.writeUInt16LE(architecture === "x64" ? 62 : 183, 18); + } + if (format === "deb") contents.write("!\n", 0, "ascii"); + if (format === "rpm") contents.set([0xed, 0xab, 0xee, 0xdb], 0); + return contents; +} + +async function makeRelease() { + const root = await mkdtemp(path.join(os.tmpdir(), "omnideck-release-contract-")); + for (const [format, name, architecture] of artifacts) { + const contents = fixture(format, architecture); + const digest = createHash("sha256").update(contents).digest("hex"); + await writeFile(path.join(root, name), contents); + await writeFile(path.join(root, `${name}.sha256`), `${digest} ${name}\n`); + } + return root; +} + +test("accepts the complete desktop package matrix with matching checksums", async (t) => { + const root = await makeRelease(); + t.after(() => rm(root, { recursive: true, force: true })); + const report = await verifyReleaseDirectory({ directory: root, version }); + assert.equal(report.result, "pass"); + assert.equal(report.artifactCount, 5); + assert.deepEqual( + new Set(report.artifacts.map(({ platform }) => platform)), + new Set(["windows", "macos", "linux"]), + ); +}); + +test("rejects a package whose checksum does not match", async (t) => { + const root = await makeRelease(); + t.after(() => rm(root, { recursive: true, force: true })); + const name = `Omnideck_${bareVersion}_x64-setup.exe`; + await writeFile(path.join(root, `${name}.sha256`), `${"0".repeat(64)} ${name}\n`); + await assert.rejects(verifyReleaseDirectory({ directory: root, version }), /checksum mismatch/); +}); + +test("rejects a missing artifact", async (t) => { + const root = await makeRelease(); + t.after(() => rm(root, { recursive: true, force: true })); + await rm(path.join(root, `Omnideck-${bareVersion}-1.x86_64.rpm`)); + await rm(path.join(root, `Omnideck-${bareVersion}-1.x86_64.rpm.sha256`)); + await assert.rejects(verifyReleaseDirectory({ directory: root, version }), /does not match/); +}); + +test("rejects an AppImage with the wrong executable architecture", async (t) => { + const root = await makeRelease(); + t.after(() => rm(root, { recursive: true, force: true })); + const name = `Omnideck_${bareVersion}_amd64.AppImage`; + const contents = fixture("appimage", "arm64"); + const digest = createHash("sha256").update(contents).digest("hex"); + await writeFile(path.join(root, name), contents); + await writeFile(path.join(root, `${name}.sha256`), `${digest} ${name}\n`); + await assert.rejects(verifyReleaseDirectory({ directory: root, version }), /wrong ELF architecture/); +}); + +test("rejects a corrupted/truncated artifact", async (t) => { + const root = await makeRelease(); + t.after(() => rm(root, { recursive: true, force: true })); + const name = `Omnideck_${bareVersion}_amd64.deb`; + const contents = Buffer.alloc(10); // too small, not even a real ar archive + const digest = createHash("sha256").update(contents).digest("hex"); + await writeFile(path.join(root, name), contents); + await writeFile(path.join(root, `${name}.sha256`), `${digest} ${name}\n`); + await assert.rejects(verifyReleaseDirectory({ directory: root, version }), /unexpectedly small/); +});