Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,23 @@
# Changelog

## 2026-08-31

### Changes

- Prima: every command now prints `### Artifacts` naming the files it just wrote — the ARIA tree and
the html always, plus the screenshot and network log when they were captured. Until now those files
were written on every command but only named when a `check` came back with a CONTRADICTION, so the
envelope offered a hash and nothing to suggest there was a page on disk worth reading. The ARIA tree
carries values and checked states, so one `grep` over it often answers a question that would
otherwise cost another `verify` or `ask`.
- Prima: `do` now names the per-step captures by file rather than only their directory, so the
`diff.yaml` recording what each step changed is visible from the envelope.
- Prima: `prima status <hash>` no longer needs a browser. It only reads recorded files, but it used to
open a session first and failed with "No browser to drive" once the run it was meant to explain had
finished. It also looks the hash up across every recorded site instead of assuming the most recent
one, and lists everything else kept under the hash. A hash that really is missing now reports the
directory it looked in.

## 2026-08-30

### Changes
Expand Down
29 changes: 21 additions & 8 deletions boat/prima/src/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,9 @@ const checkHelp = dedent`
as false.
Outcomes are settled against a screenshot of the whole page: what a user can see is
the proof, and the run log only says what was done. CONTRADICTION means the two
disagree - reported with both sides rather than settled one way, and ### Artifacts
then names the html, aria and screenshot on disk so you can judge it yourself. Not
finding something in the picture is not enough on its own; that is "not verified".
disagree - reported with both sides rather than settled one way, so read the html,
aria and screenshot named under ### Artifacts and judge it yourself. Not finding
something in the picture is not enough on its own; that is "not verified".
ok: follows those outcomes - false when one FAILED or CONTRADICTED, or when the run
could not complete, which is reported as such rather than as an app failure.
Page problems seen on the way appear under ### Answer, not as step failures.
Expand All @@ -53,6 +53,13 @@ const verifyHelp = dedent`
could not be expressed, which is not the same as false.
`;

const statusHelp = dedent`
Reads the files a command recorded, so it needs no browser and outlives the session.
The hash is looked up across every recorded site. ### Artifacts names every file kept
under it: the aria tree, the html, the screenshot and network log when they were
captured, and the per-step captures of a do run.
`;

const reportHelp = dedent`
Commands are logged as they run, so the report needs no browser and outlives the session.
The most recent session is reported unless --pw-session names another.
Expand Down Expand Up @@ -122,7 +129,7 @@ function primaFor(options: any): Prima {
return new Prima(buildOptions(options));
}

async function runPrima(options: any, command: string, run: (prima: Prima) => Promise<EnvelopeData>, record = true): Promise<void> {
async function runPrima(options: any, command: string, run: (prima: Prima) => Promise<EnvelopeData>): Promise<void> {
setQuietMode(!isVerboseMode());
trackActivityLine();
const prima = primaFor(options);
Expand All @@ -136,7 +143,7 @@ async function runPrima(options: any, command: string, run: (prima: Prima) => Pr
envelope = await prima.toolFailureEnvelope(command, error);
}

if (record) prima.record(envelope, Date.now() - startedAt);
prima.record(envelope, Date.now() - startedAt);
clearActivityLine();
console.log(renderEnvelope(envelope));
await prima.stop().catch(() => {});
Expand Down Expand Up @@ -211,9 +218,15 @@ export function createPrimaCommands(name = 'prima'): Command {
process.exit(0);
});

addCommonOptions(cmd.command('status <hash>').description('Show the artifacts and page detail recorded for an earlier command')).action(async (hash, options) => {
await runPrima(options, `status ${hash}`, (prima) => prima.status(hash), false);
});
addCommonOptions(cmd.command('status <hash>').description('Show the artifacts and page detail recorded for an earlier command'))
.addHelpText('after', `\n${statusHelp}`)
.action(async (hash, options) => {
setQuietMode(!isVerboseMode());
const prima = primaFor(options);
const envelope = await prima.status(hash).catch((error: unknown) => prima.toolFailureEnvelope(`status ${hash}`, error));
console.log(renderEnvelope(envelope));
process.exit(envelope.ok ? 0 : 1);
});

addCommonOptions(cmd.command('report').description('Turn every command of a session into one html and markdown report'))
.addHelpText('after', `\n${reportHelp}`)
Expand Down
44 changes: 35 additions & 9 deletions boat/prima/src/envelope.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
import { mkdirSync, writeFileSync } from 'node:fs';
import { existsSync, mkdirSync, readdirSync, writeFileSync } from 'node:fs';
import path from 'node:path';

export const STATUS_FILE = 'status.json';

const ARTIFACT_FILES = { aria: 'aria.yml', html: 'page.html', screenshot: 'page.png', network: 'network.jsonl' };

const EXPECTATION_LABELS = {
passed: 'PASSED ',
failed: 'FAILED ',
Expand Down Expand Up @@ -33,35 +37,48 @@ export interface EnvelopeData {
failure?: { error: string; compactAria?: string };
instance: InstanceInfo;
status?: string;
artifacts?: { aria: string; html: string; screenshot?: string; network?: string };
artifacts?: ArtifactPaths;
}

export function renderEnvelope(data: EnvelopeData): string {
const sections = [renderResult(data), renderPage(data), renderValue(data), renderChanges(data), renderSteps(data), renderExpectations(data), renderWarning(data), renderOutcome(data), ...renderFailure(data), renderInstance(data), renderArtifacts(data)];
return sections.filter((section) => section).join('\n\n');
}

export function writeArtifacts(dir: string, snapshot: { aria: string | null; html: string | null; screenshot?: Buffer; requests: unknown[] }): { aria: string; html: string; screenshot?: string; network?: string } {
export function writeArtifacts(dir: string, snapshot: { aria: string | null; html: string | null; screenshot?: Buffer; requests: unknown[] }): ArtifactPaths {
mkdirSync(dir, { recursive: true });
const paths: { aria: string; html: string; screenshot?: string; network?: string } = {
aria: path.resolve(dir, 'aria.yml'),
html: path.resolve(dir, 'page.html'),
const paths: ArtifactPaths = {
aria: path.resolve(dir, ARTIFACT_FILES.aria),
html: path.resolve(dir, ARTIFACT_FILES.html),
};
writeFileSync(paths.aria, snapshot.aria ?? '', 'utf-8');
writeFileSync(paths.html, snapshot.html ?? '', 'utf-8');

if (snapshot.screenshot) {
paths.screenshot = path.resolve(dir, 'page.png');
paths.screenshot = path.resolve(dir, ARTIFACT_FILES.screenshot);
writeFileSync(paths.screenshot, snapshot.screenshot);
}

if (!snapshot.requests.length) return paths;

paths.network = path.resolve(dir, 'network.jsonl');
paths.network = path.resolve(dir, ARTIFACT_FILES.network);
writeFileSync(paths.network, snapshot.requests.map((request) => `${JSON.stringify(request)}\n`).join(''), 'utf-8');
return paths;
}

export function readArtifacts(dir: string): ArtifactPaths {
const paths: ArtifactPaths = { aria: path.resolve(dir, ARTIFACT_FILES.aria), html: path.resolve(dir, ARTIFACT_FILES.html) };
if (existsSync(path.resolve(dir, ARTIFACT_FILES.screenshot))) paths.screenshot = path.resolve(dir, ARTIFACT_FILES.screenshot);
if (existsSync(path.resolve(dir, ARTIFACT_FILES.network))) paths.network = path.resolve(dir, ARTIFACT_FILES.network);

const recorded = [...Object.values(ARTIFACT_FILES), STATUS_FILE];
const files = readdirSync(dir)
.filter((entry) => !recorded.includes(entry))
.sort();
if (files.length) paths.files = files;
return paths;
}

function renderResult(data: EnvelopeData): string {
const lines = [`ok: ${data.ok}`, `command: ${data.command}`];
if (data.used?.length) lines.push(`used: ${data.used.join('; ')}`);
Expand Down Expand Up @@ -100,7 +117,7 @@ function renderSteps(data: EnvelopeData): string | null {
lines.push(`${index + 1}. ${mark} ${step.label}`);
for (const line of (step.proof || '').split('\n').filter(Boolean)) lines.push(` ${line}`);
});
if (data.stepFiles) lines.push('', `page after each step: ${data.stepFiles}`);
if (data.stepFiles) lines.push('', `page after each step: ${data.stepFiles}/<n>-<step>.{aria.yaml,html,diff.yaml}`);
return section('Steps', lines.join('\n'));
}

Expand Down Expand Up @@ -178,6 +195,7 @@ export function renderArtifacts(data: EnvelopeData): string | null {
const lines = [`aria: ${data.artifacts.aria}`, `html: ${data.artifacts.html}`];
if (data.artifacts.screenshot) lines.push(`screenshot: ${data.artifacts.screenshot}`);
if (data.artifacts.network) lines.push(`network: ${data.artifacts.network}`);
if (data.artifacts.files?.length) lines.push(`also here: ${data.artifacts.files.join(', ')}`);
return section('Artifacts', lines.join('\n'));
}

Expand All @@ -189,3 +207,11 @@ function align(label: string, marker: string, width: number): string {
function section(title: string, body: string): string {
return `### ${title}\n${body}`;
}

export interface ArtifactPaths {
aria: string;
html: string;
screenshot?: string;
network?: string;
files?: string[];
}
33 changes: 23 additions & 10 deletions boat/prima/src/prima.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ import { browserErrorMessage } from '../../../src/utils/browser-errors.ts';
import { pluralize } from '../../../src/utils/logger.ts';
import { mdq } from '../../../src/utils/markdown-query.ts';
import { safeFilename } from '../../../src/utils/strings.ts';
import { type EnvelopeData, type InstanceInfo, writeArtifacts } from './envelope.ts';
import { type EnvelopeData, type InstanceInfo, STATUS_FILE, readArtifacts, writeArtifacts } from './envelope.ts';
import { isFunctionExpression, takePwValue, toCodeceptWrapper } from './pw-parser.ts';
import { type PwServerDescriptor, readDescriptors, selectDescriptor } from './pw-registry.ts';
import { type SessionRun, latestSessionFile, readSession, recordCommand, sessionFile, sessionsDir } from './session-log.ts';
Expand Down Expand Up @@ -376,7 +376,6 @@ export class Prima {

const problems = [...unreached.map((expectation) => `not reached: ${expectation.text}`), ...contradicted.map((expectation) => `the picture and the run disagree about: ${expectation.text}`)];
if (problems.length) envelope.failure = { error: problems.join('\n') };
if (contradicted.length) envelope.artifacts = this.artifacts;

if (!test.hasFinished || test.isSkipped) {
envelope.ok = false;
Expand Down Expand Up @@ -982,14 +981,17 @@ export class Prima {
}

private async successEnvelope(command: string, used: string[], result: ActionResult, previousState: WebPageState | null): Promise<EnvelopeData> {
const changes = await this.pageChanges(result, previousState, used[0]);
const status = await this.saveStatus(result);
return {
ok: true,
command,
used,
page: this.pageBlock(result, previousState),
changes: await this.pageChanges(result, previousState, used[0]),
changes,
instance: await this.instanceInfo(),
status: await this.saveStatus(result),
status,
artifacts: this.artifacts,
};
}

Expand All @@ -998,24 +1000,28 @@ export class Prima {
const failure: EnvelopeData['failure'] = { error: browserErrorMessage(error) };
if (result.ariaSnapshot) failure.compactAria = compactAriaSnapshot(result.ariaSnapshot, true);

const status = await this.saveStatus(result);
return {
ok: false,
command,
page: this.pageBlock(result, previousState),
failure,
instance: await this.instanceInfo(),
status: await this.saveStatus(result),
status,
artifacts: this.artifacts,
};
}

private async reportEnvelope(command: string, result: ActionResult, previousState: WebPageState | null, outcome: Partial<EnvelopeData>): Promise<EnvelopeData> {
const status = await this.saveStatus(result);
return {
ok: true,
command,
page: this.pageBlock(result, previousState),
...outcome,
instance: await this.instanceInfo(),
status: await this.saveStatus(result),
status,
artifacts: this.artifacts,
};
}

Expand Down Expand Up @@ -1060,24 +1066,31 @@ export class Prima {
}

async status(hash: string): Promise<EnvelopeData> {
if (!this.artifactsDir) {
const sites = listSites();
const site = sites.find((candidate) => existsSync(path.join(candidate.dir, 'output', 'prima', hash))) || sites[0];
if (site && !this.configBaseUrl()) this.sessionUrl = site.url;
await this.loadConfig();
}

const dir = this.statusDir(hash);
const statusFile = path.join(dir, 'status.json');
if (!existsSync(statusFile)) return this.toolFailureEnvelope(`status ${hash}`, `No command was recorded under ${hash}. Every envelope prints its own hash on the Instance line.`);
const statusFile = path.join(dir, STATUS_FILE);
if (!existsSync(statusFile)) return this.toolFailureEnvelope(`status ${hash}`, `No command was recorded under ${dir}. Every envelope prints its own hash on the Instance line.`);

const saved = JSON.parse(readFileSync(statusFile, 'utf-8'));
return {
ok: true,
command: `status ${hash}`,
page: saved.page,
instance: await this.instanceInfo(),
artifacts: { aria: path.join(dir, 'aria.yml'), html: path.join(dir, 'page.html') },
artifacts: readArtifacts(dir),
};
}

private async saveStatus(result: ActionResult): Promise<string> {
const hash = this.statusHash();
await this.writeSnapshot(result);
writeFileSync(path.join(this.statusDir(hash), 'status.json'), JSON.stringify({ page: this.pageBlock(result, null) }), 'utf-8');
writeFileSync(path.join(this.statusDir(hash), STATUS_FILE), JSON.stringify({ page: this.pageBlock(result, null) }), 'utf-8');
return hash;
}

Expand Down
41 changes: 39 additions & 2 deletions boat/prima/tests/envelope.test.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
import { describe, expect, test } from 'bun:test';
import { mkdtempSync, readFileSync, rmSync } from 'node:fs';
import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import path from 'node:path';
import { type EnvelopeData, renderEnvelope, writeArtifacts } from '../src/envelope.ts';
import { type EnvelopeData, readArtifacts, renderEnvelope, writeArtifacts } from '../src/envelope.ts';

const base: EnvelopeData = {
ok: true,
Expand Down Expand Up @@ -161,13 +161,50 @@ describe('renderEnvelope', () => {
expect(out).not.toContain('### Artifacts');
});

test('a successful action names the files it recorded, so they invite reading', () => {
const out = renderEnvelope(base);
expect(out).toContain('### Artifacts');
expect(out).toContain('aria: /tmp/x/aria.yml');
expect(out).toContain('html: /tmp/x/page.html');
expect(out).toContain('network: /tmp/x/network.jsonl');
expect(out).not.toContain('screenshot:');
});

test('files recorded beside the named ones are listed too', () => {
const out = renderEnvelope({ ...base, artifacts: { ...base.artifacts!, files: ['1-click-add.aria.yaml', '1-click-add.diff.yaml'] } });
expect(out).toContain('also here: 1-click-add.aria.yaml, 1-click-add.diff.yaml');
});

test('step captures name the files written for each step, not just their directory', () => {
const out = renderEnvelope({ ...base, steps: [{ label: "I.click('Add')", ok: true, proof: '' }], stepFiles: '/tmp/x' });
expect(out).toContain('page after each step: /tmp/x/<n>-<step>.{aria.yaml,html,diff.yaml}');
});

test('open tabs alone are evidence enough for a running browser', () => {
const out = renderEnvelope({ ...base, instance: { name: 'default', tabs: 2, others: [] } });
expect(out).toContain('| running');
expect(out).not.toContain('browser: not running');
});
});

describe('readArtifacts', () => {
test('reports the files a recorded command left behind, and only those', () => {
const dir = mkdtempSync(path.join(tmpdir(), 'prima-'));
writeArtifacts(dir, { aria: '- button "Login"', html: '<html></html>', requests: [] });
writeFileSync(path.join(dir, 'status.json'), '{}', 'utf-8');
writeFileSync(path.join(dir, '1-click-add.diff.yaml'), 'added: []', 'utf-8');

const found = readArtifacts(dir);

expect(found.aria).toBe(path.join(dir, 'aria.yml'));
expect(found.html).toBe(path.join(dir, 'page.html'));
expect(found.network).toBeUndefined();
expect(found.screenshot).toBeUndefined();
expect(found.files).toEqual(['1-click-add.diff.yaml']);
rmSync(dir, { recursive: true, force: true });
});
});

describe('writeArtifacts', () => {
test('writes aria, html and network files and returns absolute paths', () => {
const dir = mkdtempSync(path.join(tmpdir(), 'prima-'));
Expand Down
2 changes: 2 additions & 0 deletions boat/prima/tests/prima.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,8 @@ describe('Prima.pw', () => {
expect(existsSync(path.join(artifactsRoot, envelope.status!, 'aria.yml'))).toBe(true);
expect(existsSync(path.join(artifactsRoot, envelope.status!, 'page.html'))).toBe(true);
expect(existsSync(path.join(artifactsRoot, envelope.status!, 'network.jsonl'))).toBe(false);
expect(envelope.artifacts?.aria).toBe(path.join(artifactsRoot, envelope.status!, 'aria.yml'));
expect(envelope.artifacts?.network).toBeUndefined();
expect(envelope.instance.name).toBe('default');
});

Expand Down
4 changes: 3 additions & 1 deletion docs/reference/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -685,7 +685,9 @@ html: /home/you/.explorbot/sites/app.example.com/output/prima/2026-08-04T10-04-2

`used:` is code that already executed: for `go` the CodeceptJS step it ran, for `pw` the Playwright expression you passed, which a CodeceptJS test needs wrapped in `I.usePlaywrightTo(...)`. Log lines can precede the envelope, so start parsing at the first `###` line.

`### Changes` renders on every action envelope, saying `no change` when the tree is identical — so a successful command proves what it did instead of leaving you to check. `check` and `do` report per step rather than in aggregate: `### Steps` names each step with the code it ran and what proved it, and `page after each step:` points at the captures. `check` adds `### Expected outcomes`; `ask`, `research`, and `verify` add `### Answer`, `### Research`, or `### Assertions`; `pw` adds `### Value` when its expression returns one. `network:` appears under `### Artifacts` only when requests were captured, and everything else recorded for a command is behind `prima status <hash>`.
`### Changes` renders on every action envelope, saying `no change` when the tree is identical — so a successful command proves what it did instead of leaving you to check. `check` and `do` report per step rather than in aggregate: `### Steps` names each step with the code it ran and what proved it, and `page after each step:` names the per-step captures. `check` adds `### Expected outcomes`; `ask`, `research`, and `verify` add `### Answer`, `### Research`, or `### Assertions`; `pw` adds `### Value` when its expression returns one.

`### Artifacts` names the files every command leaves on disk: the aria tree and the html always, `screenshot:` and `network:` when a screenshot was taken or requests were captured. The aria tree carries values and checked states, so one `grep` over it settles a question the envelope did not answer, without asking the page again. `prima status <hash>` reprints those paths for an earlier command and lists whatever else was kept under the hash; it only reads recorded files, so it needs no browser and outlives the session.

**A failed action is a failure.** Nothing is retried along a different route and no other element is substituted, so `ok: true` means the action you asked for is the one that landed. A failure adds `### Failure` with the error and the compact ARIA of the page, so you can retarget from the envelope itself instead of opening the artifact files.

Expand Down
Loading
Loading