feat(evals): score the iOS app's design from real screenshots with a model council

evals/README.md

@@ -9,6 +9,7 @@ Write or run an eval when a change affects what a model sees or does: agent inst
 | [mcp-tool-use](mcp-tool-use/)   | Whether agents use T3's MCP tools correctly, including device routing and recovery, in fake threads. |
 | [thread-titles](thread-titles/) | Whether generated thread titles name the right subject.                                              |
 | [test-signal](test-signal/)     | Whether a cheap model running the test-signal check flags low-signal tests.                          |
+| [mobile-design](mobile-design/) | How a model council scores the iOS app's design.                                                     |
 
 ## Run an eval
 

evals/mobile-design/README.md

@@ -0,0 +1,53 @@
+# mobile-design
+
+Scores the design of the T3 Code iOS app on iPhone, iPhone Duo and iPad. Code cannot judge design, so a council of models scores screenshots. This eval makes those screenshots the same way every round, and writes the brief that every round asks.
+
+A run captures:
+
+- **Scenes**: every screen in `SCENES` (`cases.ts`), on each pose in `POSES`. The poses are iPhone 17 Pro dark and light, the largest accessibility text size and landscape; iPhone Duo closed and open; and iPad Pro 13 in landscape and portrait. Every pose captures every scene, except the large-text and landscape passes, which capture a few. A scene's `source` names the complaint it guards, when there is one.
+- **Desktop pairs**: the same data in the desktop web app at 1440×900, next to the iPad and iPhone captures.
+- **Flows**: common tasks walked step by step on the dark iPhone, with the taps and distinct screens each took.
+
+The data comes from the showcase fixture (`scripts/mobile-showcase-environment.ts`): three environments, seeded threads, and live threads run by a scripted Codex provider (`scripts/mobile-showcase-provider.ts`). That provider is how a capture shows a stream in progress, tool calls, an approval, a question and background work without a real model.
+
+## Run it
+
+Mobile capture needs macOS with Xcode, and a simulator runtime with the iPhone Duo device type (Xcode 27.1 or later). Desktop capture needs the `shot` skill. Run both on the same Mac: on Linux, `/etc/machine-info` names the machine in the desktop captures.
+
+1. From the repo root, run `vp i`.
+2. From `evals/`, capture and build the pack:
+
+   ```bash
+   pnpm mobile-design --out runs/<name> --capture mobile --capture desktop
+   ```
+
+   The first run builds the showcase app, which takes about 10 minutes. Add `--skip-build` to reuse it. The capture creates its own simulators, named `T3 Design …`, and deletes them when it finishes.
+
+3. Run the council. The run writes `runs/<name>/council.json` with the brief and the panel: Claude Opus 5.5 and Codex GPT-6.1 Sol, both at `xhigh`. Give each member the brief at `runs/<name>/pack/brief.md`, read-only, with one `delegate_task` each. Add `--panel opus,sol,muse` to add Muse Wonder Full for rounds about copy.
+4. Save each reply as `runs/<name>/scores/<member>.md`, then:
+
+   ```bash
+   pnpm mobile-design --out runs/<name> --summarize --baseline runs/<earlier>
+   ```
+
+   `summary.md` has the median per axis and device class, each reviewer's scores, the change against the baseline, and each reviewer's top five fixes.
+
+To split the work across Macs, for example when only one has the Duo runtime, run each with its own `--pose` list, then copy the `captures/` folders into one run, join the `shots` lists in their `captures.json`, and build the pack with `--pack`.
+
+Other flags: `--pose <id>` and `--case <scene id>` capture a subset, `--no-flows` skips the flows, `--pack` rebuilds the pack from earlier captures. `captures/captures.json` lists any scene that did not reach its ready state; look at those screenshots before you send the pack.
+
+## Blind review
+
+The pack holds only screenshots, the brief and the flow table. The commit under review is in `run.json`, outside the pack. Do not tell reviewers which build is new.
+
+## Add a scene or flow
+
+Add it to `SCENES` or `FLOWS` in `cases.ts`, with its `source`. Steps open deep links (`t3code://…`) or tap by accessibility label. When a scene needs a new state, add it to the fixture: a seeded thread in `SHOWCASE_THREADS`, or a live thread in `SHOWCASE_LIVE_THREADS` with the steps the scripted provider plays.
+
+## Reference screenshots
+
+`references/README.md` lists the Codex and Claude iOS screenshots the comparison axis needs. A maintainer captures them on a device; every image in that folder goes into the pack.
+
+## Time and cost
+
+A full capture on an idle M-series Mac takes about 90 minutes: about 30 minutes per device class, plus 5 for desktop pairs. On a busy host it can take twice that. A council round is two `xhigh` reviews of about 115 images.

evals/mobile-design/brief.md

@@ -0,0 +1,51 @@
+# Design review: the T3 Code mobile app
+
+You are a senior product designer. Review the T3 Code mobile app from the screenshots in this pack. This is a read-only review. Do not edit files, start servers, or open browsers or simulators.
+
+T3 Code is a client for coding agents. People use the phone and tablet apps to watch agents work on their own machines, answer the agent's questions, approve commands, review diffs and start new work. The desktop app is the reference for the product's look. The mobile app should feel like the same product, adapted to touch and to each screen size.
+
+## What is in the pack
+
+Image paths are relative to the folder that holds this brief. Read each image before you score.
+
+- `screens/<shape>/<scene>.png`: one scene on one device shape. Each tile is labelled with its device, appearance and text size.
+- `desktop/<scene>.png`: the same data in the desktop web app at 1440×900, next to the iPad and iPhone captures, for the "conformance with desktop" axis. The desktop app is connected to one machine, so its lists are shorter than the phone's.
+- `flows/<task>.png`: a common task, one frame per step. The label above each frame names the action that led to it.
+
+The data is a fixture. Thread titles, projects and machine names are made up, and project paths point at temporary folders. Do not score the fixture's wording or paths. Do score how the app lays them out, for example how it truncates a long path.
+
+Screens:
+
+{{SCREENS}}
+
+Tasks, with the taps and distinct screens each one took:
+
+{{FLOWS}}
+
+References for the Codex and Claude iOS apps:
+
+{{REFERENCES}}
+
+## How to score
+
+Score each axis from 1 to 10 for each device class. 10 is the best mobile app you know for this kind of work. 7 is solid but clearly behind the best.
+
+Axes, by id:
+
+{{AXES}}
+
+Device classes, by id:
+
+{{DEVICES}}
+
+Then say what feels worse, blander or less clear than it should, and what works well and must stay as it is. Name the screen and device for each.
+
+Then list the five fixes that would raise the scores most, most important first. Each fix names the screen and device, what is wrong, and the change you would make. Be specific: "the question panel leaves a 120 pt scroll area on the closed Duo; let it take the space above the composer" is useful, "improve spacing" is not.
+
+## Reply format
+
+Write your review in plain prose and short lists. End the reply with exactly one fenced `json` block in this shape, with your numbers and your five fixes:
+
+```json
+{{JSON}}
+```

evals/mobile-design/capture-desktop.ts

@@ -0,0 +1,130 @@
+// @effect-diagnostics nodeBuiltinImport:off - Host-side capture uses Node subprocess and filesystem APIs directly.
+
+import type * as NodeChildProcess from "node:child_process";
+import * as NodeFSP from "node:fs/promises";
+import * as NodeOS from "node:os";
+import * as NodePath from "node:path";
+
+import {
+  issuePairingCredential,
+  startShowcaseEnvironments,
+  stopProcess,
+} from "../../scripts/mobile-showcase.ts";
+import { SHOWCASE_ENVIRONMENTS } from "../../scripts/mobile-showcase-environment.ts";
+import { SCENES, type SceneId } from "./cases.ts";
+import { expandPath, linkTargets, run } from "./ios.ts";
+
+const REPO_ROOT = NodePath.resolve(import.meta.dirname, "../..");
+const WEB_ROOT = NodePath.join(REPO_ROOT, "apps/web");
+const VIEWPORT = "1440x900";
+// Dark mode is the primary look; the web app reads it from these keys before it renders.
+const DARK_MODE = [
+  ["t3code:theme", "dark"],
+  ["t3code:theme-appearance-mode", "dark"],
+  ["t3code:theme-follow-system", "false"],
+]
+  .map(([key, value]) => `localStorage.setItem(${JSON.stringify(key)}, ${JSON.stringify(value)})`)
+  .join("; ");
+
+export interface DesktopShot {
+  readonly scene: SceneId;
+  readonly file: string;
+  readonly problem?: string;
+}
+
+/** The shot skill's CLI; set SHOT or pass --shot when it lives elsewhere. */
+export async function findShot(explicit: string | undefined): Promise<string | null> {
+  const home = NodeOS.homedir();
+  const candidates = [
+    explicit,
+    process.env.SHOT,
+    NodePath.join(home, ".claude/skills/shot/scripts/shot"),
+    NodePath.join(home, ".codex/skills/shot/scripts/shot"),
+  ].filter((value): value is string => Boolean(value));
+  for (const candidate of candidates) {
+    if (
+      await NodeFSP.access(candidate).then(
+        () => true,
+        () => false,
+      )
+    )
+      return candidate;
+  }
+  return null;
+}
+
+/**
+ * Captures the desktop half of each "conforms to desktop" pair: the built web client, served by
+ * the same seeded environments the phone uses, at a desktop window size.
+ */
+export async function captureDesktop(input: {
+  readonly outDirectory: string;
+  readonly shot: string;
+  readonly scenes: ReadonlySet<SceneId> | null;
+  readonly skipBuild: boolean;
+  readonly log: (message: string) => void;
+}): Promise<ReadonlyArray<DesktopShot>> {
+  const built = await NodeFSP.access(NodePath.join(WEB_ROOT, "dist/index.html")).then(
+    () => true,
+    () => false,
+  );
+  if (!input.skipBuild || !built) {
+    input.log("building the web client");
+    await run("vp", ["build"], { cwd: WEB_ROOT, timeoutMs: 900_000 });
+  }
+  const rootDir = await NodeFSP.mkdtemp(NodePath.join(NodeOS.tmpdir(), "t3-design-web-"));
+  const servers: NodeChildProcess.ChildProcess[] = [];
+  const shots: DesktopShot[] = [];
+  try {
+    input.log("starting the fixture environments");
+    const environments = await startShowcaseEnvironments(
+      rootDir,
+      (server) => servers.push(server),
+      SHOWCASE_ENVIRONMENTS,
+      { liveThreads: true },
+    );
+    const targets = linkTargets(environments);
+    for (const scene of SCENES) {
+      if (!("desktop" in scene) || (input.scenes && !input.scenes.has(scene.id))) continue;
+      // A thread opens on the environment that owns it; other pages use the first one.
+      const threadId = /\{thread:([^}]+)\}/.exec(scene.desktop.path)?.[1];
+      const owner = threadId ? targets.get(threadId)?.environmentId : undefined;
+      const environment =
+        environments.find((candidate) => candidate.environmentId === owner) ?? environments[0]!;
+      const path = `/${expandPath(scene.desktop.path.slice(1), targets)}`;
+      const credential = await issuePairingCredential(environment.baseDir);
+      const file = NodePath.join("captures", "desktop", `${scene.id}.png`);
+      await NodeFSP.mkdir(NodePath.join(input.outDirectory, "captures", "desktop"), {
+        recursive: true,
+      });
+      const result = await run(
+        input.shot,
+        [
+          `http://127.0.0.1:${environment.port}/pair#token=${credential}`,
+          "--size",
+          VIEWPORT,
+          "--sleep",
+          "2500",
+          "--eval",
+          `${DARK_MODE}; location.assign(${JSON.stringify(path)})`,
+          "--sleep",
+          "4000",
+          ...("click" in scene.desktop ? ["--click", scene.desktop.click, "--sleep", "1500"] : []),
+          "--out",
+          NodePath.join(input.outDirectory, file),
+        ],
+        { allowFailure: true, timeoutMs: 120_000 },
+      );
+      const problem =
+        result.code === 0
+          ? undefined
+          : result.stderr.trim().split("\n")[0] || `shot exited with code ${result.code}`;
+      if (problem) input.log(`desktop/${scene.id}: ${problem}`);
+      shots.push({ scene: scene.id, file, ...(problem ? { problem } : {}) });
+    }
+  } finally {
+    await Promise.all(servers.map(stopProcess));
+    await NodeFSP.rm(rootDir, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });
+  }
+  return shots;
+}