Explorations: one place chooses the capture strategy, and ARCHITECTURE describes strategies

ARCHITECTURE.md

@@ -29,10 +29,11 @@ This file describes the source layout and the rules that keep it in order.
 | `review` | Compares base and head evidence, and prepares the local report assets. `ci` calls `evidence-assessment.ts` and `capture-review.ts`. | `evidence-assessment.ts`, `capture-review.ts`, `visual-evidence.ts`, `runtime-evidence.ts`, `evidence-comparator.ts`, `reviewer-changes.ts` |
 | `report` | The report artifact, the pull-request comment, and the Viewer links. | `artifact.ts`, `pr-comment*.ts`, `viewer-links.ts` |
 | `ci` | The pipeline behind `source:verify`, `capture:review` and `capture:baseline`. | `pipeline.ts`, `verified-plan.ts`, `artifact-cache.ts` |
+| `binary` | Explores an app with no source (`explore:binary`, `explore:session`): the archive, the device session, the explorer, and the strategies that drive it. | `run.ts` (entry), `explorer.ts`, `strategy.ts`, `replay.ts`, `flows.ts`, `archive.ts`, `report.ts` |
 | `cli` | Argument parsing, help, and the command handlers. | `args.ts`, `help.ts`, `*-commands.ts` |
 | `map` | The local map store, build catalogs and comparisons, and the server for `expo-explore view`. | `store.ts`, `build-workspace.ts`, `compare.ts`, `viewer-server.ts` |
 | `contracts` | The modules that `viewer/src` imports from `src/`, by value or by type: data shapes, validators and projections that the CLI and the Viewer share. `capture.ts` also defines the review types that the Viewer reads, such as `PublicReview`, and `review-validation.ts` validates them. | `capture.ts`, `review.ts`, `review-validation.ts`, `source-validation.ts` |
-| `shared` | Small Node utilities that several folders use, and the pinned `agent-device` version that capture and review both check. | `ci-errors.ts`, `canonical-json.ts`, `bounded-bytes.ts`, `png-evidence.ts`, `agent-device-version.ts`, `timings.ts` (a capture job's timings and its summary, CI-FLOW §4.0) |
+| `shared` | Small Node utilities that several folders use, and the pinned `agent-device` version that capture and review both check. | `ci-errors.ts`, `canonical-json.ts`, `bounded-bytes.ts`, `png-evidence.ts`, `archive-entries.ts` (tar.gz and zip readers), `agent-device-version.ts`, `timings.ts` (a capture job's timings and its summary, CI-FLOW §4.0) |
 
 ## Dependency rule
 
@@ -44,14 +45,24 @@ Each layer imports only from the layers below it.
 3. `source` imports `source/route-providers`.
 4. `capture` imports `capture/agent`, `config` and `map`. `capture/agent` imports `capture` and `config`.
 5. `review` imports `capture` and `map`.
-6. `ci` imports `config`, `source`, `capture`, `capture/agent`, `review` and `report`.
-7. `cli` imports `ci`, `map`, `config` and `report`. `src/cli.ts` imports `cli`, `map` and `shared`.
+6. `ci` imports `config`, `source`, `capture`, `capture/agent`, `review` and `report`. `binary` imports `config`, `capture`, `capture/agent`, `map` and `report`.
+7. `cli` imports `ci`, `binary`, `map`, `config` and `report`. `src/cli.ts` imports `cli`, `map` and `shared`.
 
 Every folder may import `contracts/`, `shared/` and `src/types.ts`.
 The exception is `contracts/`: its value imports stay inside `contracts/` or go to `src/types.ts`.
 The only cycle is `capture` and `capture/agent`, which are one feature.
 Do not add a new import that points up a layer.
 
+## Capture strategies
+
+A capture decides three things: what to look at, how to get there, and what to record.
+
+- **What to look at.** For a review, `source` plans the routes from the app's source (`capture-plan.ts`). An exploration has no source; its strategy decides as it goes.
+- **How to get there.** In a review, `capture/route-capture.ts` opens each planned route by deep link, or, in agent mode, a model navigates to it (`capture/agent/runtime.ts`). In an exploration, a strategy drives the explorer's tools: a model exploring on its own, a replay plan (`binary/replay.ts`), or named flows (`binary/flows.ts`).
+- **What to record.** The host records the evidence, never the strategy: `capture/route-capture.ts` in a review, `binary/explorer.ts` in an exploration. The host also owns the device, the budgets, refusing destructive and app-leaving actions, and redacting secrets. Both write an `ExploreMap` (`src/types.ts`, `schema/screen-map.v1.schema.json`). A review adds the capture bundle (`contracts/capture.ts`); an exploration adds the report in `contracts/standalone-report.ts`.
+
+To add an exploration strategy, add a case to `StrategyChoice` in `binary/strategy.ts`: what it checks before any device starts, and the session it runs. The session is given the explorer's tools and calls them, as the model sessions, `createReplaySession` and `createFlowSession` do. Reviews have no such seam yet: the deep-link loop and agent mode are two branches of `capturePlannedRoutes`.
+
 ## The contracts boundary
 
 `viewer/src` imports `src/` only through `src/contracts/` and `src/types.ts`.

src/binary/app-url.ts

@@ -0,0 +1,17 @@
+import { CiError } from "../shared/ci-errors.js";
+
+/** Schemes the OS routes to a browser, the store, Settings, or another app: opening one hands the session away from the app under test. */
+const SYSTEM_URL_SCHEMES = /^(https?|ftp|file|javascript|data|blob|mailto|tel|telprompt|facetime(-audio)?|sms|mms|imessage|itms(-apps|-services|-beta)?|macappstore|app-settings|prefs|shortcuts|maps|music|videos|photos-redirect|x-apple-[a-z-]+|market|intent|content|android-app|geo|package):$/i;
+
+const urlError = (code: string, message: string, nextAction: string) => new CiError({ code, stage: "session", message, nextAction });
+
+/** Only the app's own URLs may be opened; an https page or a store link would hand the session to another app. */
+export function assertAppUrl(url: string, appSchemes: readonly string[] = []): void {
+  let parsed: URL;
+  try { parsed = new URL(url); } catch { throw urlError("AGENT_URL_INVALID", `Not a URL: ${url}`, "Pass a custom-scheme app URL."); }
+  if (SYSTEM_URL_SCHEMES.test(parsed.protocol)) throw urlError("AGENT_URL_NOT_ALLOWED", `${parsed.protocol} URLs leave the app; only the app's own URL schemes are allowed.`, "Use the app's own URL scheme.");
+  const scheme = parsed.protocol.slice(0, -1).toLowerCase();
+  if (appSchemes.length && !appSchemes.includes(scheme)) {
+    throw urlError("AGENT_URL_NOT_ALLOWED", `${parsed.protocol} is not a URL scheme this app registers (${appSchemes.join(", ")}).`, "Use one of the app's own URL schemes.");
+  }
+}

src/binary/archive.ts

@@ -2,9 +2,7 @@ import { createHash } from "node:crypto";
 import { createReadStream } from "node:fs";
 import { lstat, mkdir, open, readdir, readFile, realpath, writeFile, chmod } from "node:fs/promises";
 import { basename, join, relative, resolve, sep } from "node:path";
-import { createGunzip } from "node:zlib";
-import { Reader, Uint8ArrayWriter, ZipReader, configure } from "@zip.js/zip.js";
-import { CiError } from "../shared/ci-errors.js";
+import { MAX_ARCHIVE_ENTRIES, MAX_METADATA_BYTES, archiveError, readZipEntry, safeRelativePath, walkTar, zipEntries } from "../shared/archive-entries.js";
 import { APP_ICON_SOURCE_LIMIT, isWebp, shrinkAppIcon, webpSize } from "../capture/app-icon.js";
 import { resolveResourceStrings } from "./android-resources.js";
 import type { Platform } from "../types.js";
@@ -38,14 +36,8 @@ export type InspectedArchive = {
 };
 
 export const DEFAULT_MAX_ARCHIVE_BYTES = 1024 * 1024 * 1024;
-const MAX_METADATA_BYTES = 4 * 1024 * 1024;
-const MAX_ARCHIVE_ENTRIES = 200_000;
 const MAX_RESOURCE_TABLE_BYTES = 32 * 1024 * 1024;
 
-export function archiveError(code: string, message: string, nextAction: string, details?: Record<string, unknown>): CiError {
-  return new CiError({ code, stage: "archive", message, nextAction, ...(details ? { details } : {}) });
-}
-
 /** The archive kind from its name; the content check follows and may still reject it. */
 export function archiveKind(path: string): ArchiveKind {
   const name = basename(path).toLowerCase();
@@ -231,63 +223,6 @@ async function hashDirectory(root: string, maxBytes: number): Promise<{ sha256:
   return { sha256: hash.digest("hex"), bytes };
 }
 
-// ---- tar.gz -------------------------------------------------------------------------------------
-
-type TarVisitor = (entry: { name: string; type: string; size: number; mode: number }, contents: () => Promise<Buffer>, skip: () => Promise<void>) => Promise<boolean | void>;
-
-/** Walk a gzip tarball once; `visit` returns true to stop. Long GNU names are honoured; nothing is written. */
-export async function walkTar(path: string, visit: TarVisitor): Promise<void> {
-  const stream = createReadStream(path).pipe(createGunzip());
-  const iterator = stream[Symbol.asyncIterator]();
-  let buffer = Buffer.alloc(0);
-  let done = false;
-  const fill = async (needed: number) => {
-    while (buffer.length < needed && !done) {
-      const next = await iterator.next();
-      if (next.done) { done = true; break; }
-      buffer = Buffer.concat([buffer, next.value as Buffer]);
-    }
-  };
-  const take = async (count: number): Promise<Buffer> => {
-    await fill(count);
-    if (buffer.length < count) throw archiveError("ARCHIVE_INVALID", "The tarball ended unexpectedly.", "Pass a complete .tar.gz archive.");
-    const out = buffer.subarray(0, count); buffer = buffer.subarray(count); return Buffer.from(out);
-  };
-  const drop = async (count: number) => {
-    let remaining = count;
-    while (remaining > 0) {
-      await fill(Math.min(remaining, 1 << 20));
-      const step = Math.min(remaining, buffer.length);
-      if (!step) throw archiveError("ARCHIVE_INVALID", "The tarball ended unexpectedly.", "Pass a complete .tar.gz archive.");
-      buffer = buffer.subarray(step); remaining -= step;
-    }
-  };
-  let longName: string | undefined;
-  let entries = 0;
-  try {
-    for (;;) {
-      await fill(512);
-      if (buffer.length < 512 || buffer.subarray(0, 512).every(byte => byte === 0)) return;
-      const header = await take(512);
-      const field = (offset: number, length: number) => header.subarray(offset, offset + length).toString("utf8").replace(/\0.*$/s, "");
-      const size = parseInt(field(124, 12).trim() || "0", 8);
-      const type = field(156, 1) || "0";
-      const prefix = field(345, 155);
-      const name = longName ?? (prefix ? `${prefix}/${field(0, 100)}` : field(0, 100));
-      longName = undefined;
-      const padded = Math.ceil(size / 512) * 512;
-      if (type === "L") { longName = (await take(padded)).subarray(0, size).toString("utf8").replace(/\0+$/, ""); continue; }
-      if (++entries > MAX_ARCHIVE_ENTRIES) throw archiveError("ARCHIVE_TOO_LARGE", "The tarball has too many entries.", "Pass a smaller archive.");
-      let consumed = false;
-      const stop = await visit({ name, type, size, mode: parseInt(field(100, 8).trim() || "0", 8) },
-        async () => { consumed = true; return (await take(padded)).subarray(0, size); },
-        async () => { consumed = true; await drop(padded); });
-      if (!consumed) await drop(padded);
-      if (stop) return;
-    }
-  } finally { await iterator.return?.(undefined); }
-}
-
 async function tarEntry(path: string, match: (name: string) => boolean): Promise<{ name: string; contents: Buffer } | undefined> {
   let found: { name: string; contents: Buffer } | undefined;
   await walkTar(path, async (entry, contents) => {
@@ -299,12 +234,6 @@ async function tarEntry(path: string, match: (name: string) => boolean): Promise
   return found;
 }
 
-export function safeRelativePath(name: string): string | undefined {
-  const parts = name.replace(/^\.\//, "").split("/").filter(part => part.length);
-  if (!parts.length || parts.some(part => part === "." || part === ".." || part.includes("\\") || part.includes("\0"))) return undefined;
-  return parts.join("/");
-}
-
 /**
  * Unpack the iOS bundle out of a .tar.gz or simulator .ipa into `destination` for installation.
  * Paths are confined to the destination; symbolic and hard links are rejected (an app bundle has none).
@@ -347,60 +276,6 @@ export async function extractIosBundle(archive: InspectedArchive, destination: s
   return appDirectory;
 }
 
-// ---- zip ----------------------------------------------------------------------------------------
-
-type ZipEntry = Awaited<ReturnType<ZipReader<unknown>["getEntries"]>>[number];
-
-/** zip.js reads the central directory and entries by range, so a 200 MB APK is never loaded whole. */
-class FileRangeReader extends Reader<unknown> {
-  constructor(private readonly path: string) { super(path); }
-  override async init(): Promise<void> {
-    this.size = (await lstat(this.path)).size;
-  }
-  /** Each range opens its own descriptor: nothing stays open after the entries are read, so no handle leaks when a caller stops early. */
-  override async readUint8Array(index: number, length: number): Promise<Uint8Array> {
-    const out = new Uint8Array(length);
-    const handle = await open(this.path, "r");
-    try {
-      let offset = 0;
-      while (offset < length) {
-        const { bytesRead } = await handle.read(out, offset, length - offset, index + offset);
-        if (bytesRead === 0) break;
-        offset += bytesRead;
-      }
-      return offset === length ? out : out.subarray(0, offset);
-    } finally { await handle.close(); }
-  }
-}
-
-const readers = new WeakMap<object, FileRangeReader>();
-
-/** The ABIs an APK ships native libraries for (`lib/<abi>/`), without reading its manifest. */
-export async function apkAbis(path: string): Promise<string[]> {
-  return [...new Set((await zipEntries(path)).map(entry => /^lib\/([^/]+)\//.exec(entry.filename)?.[1]).filter((abi): abi is string => !!abi))].sort();
-}
-
-async function zipEntries(path: string): Promise<ZipEntry[]> {
-  configure({ useWebWorkers: false });
-  const reader = new FileRangeReader(path);
-  const zip = new ZipReader(reader, { useWebWorkers: false });
-  try {
-    const entries = await zip.getEntries();
-    if (entries.length > MAX_ARCHIVE_ENTRIES) throw archiveError("ARCHIVE_TOO_LARGE", "The archive has too many entries.", "Pass a smaller archive.");
-    for (const entry of entries) readers.set(entry, reader);
-    return entries;
-  } catch (error) {
-    if (error instanceof CiError) throw error;
-    throw archiveError("ARCHIVE_INVALID", `${basename(path)} is not a readable ZIP archive: ${error instanceof Error ? error.message : String(error)}`, "Pass a valid .ipa or .apk.");
-  }
-}
-
-async function readZipEntry(entry: ZipEntry, maxBytes = MAX_METADATA_BYTES): Promise<Uint8Array> {
-  if (entry.directory || !entry.getData) throw archiveError("ARCHIVE_INVALID", `${entry.filename} is not a file entry.`, "Pass a valid archive.");
-  if (entry.uncompressedSize > maxBytes) throw archiveError("ARCHIVE_INVALID", `${entry.filename} is too large.`, "Pass a valid archive.");
-  return entry.getData(new Uint8ArrayWriter(), { checkSignature: true, useWebWorkers: false });
-}
-
 // ---- plist --------------------------------------------------------------------------------------
 
 /** Property list reader for Info.plist: XML (Xcode default) or binary (`bplist00`). */