From edeb8631baf9e5aca10cb3a3fd7b1094f90cf51e Mon Sep 17 00:00:00 2001 From: DavertMik Date: Fri, 7 Aug 2026 12:56:12 +0300 Subject: [PATCH 01/39] Make prima an executor you can trust: no substitution, refs, proofs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A `pw` call on a selector that did not exist returned ok: true, healed: true, having clicked an unrelated control. Heal is removed outright — with it the --no-heal flag, the healed: line and the healing-attempts block — so no code path can reach an element other than the one asked for. `do` now drives by the refs Playwright puts in the page snapshot. New clickRef and hoverRef tools in boat/prima/src/tools.ts take a ref, resolve it through WebElement to an attribute-based clickXPath, and require exactly one match. One instruction becomes one command instead of a fallback ladder, and a stale ref is reported as stale rather than replaced by a guess. click and hover in src/ai/tools.ts are untouched. A tool's schema is shared with every caller, and Tester never receives ref-bearing snapshots, so it must not be shown a ref parameter it could only fill by inventing one. The ref tools live in the boat that uses them; core only lends its result-shaping helpers so they are not copied. A new context() tool returns the page with fresh refs when one dies mid-run, and drops to raw markup on a second call. ### Changes now renders on every action, including "no change" — it was structurally impossible alongside an answer, a research map or a verdict, so a successful click proved nothing. Refs never reach the diff or state hash: an identical page whose refs merely renumber produced six phantom entries. Also fixed, each found against a live app: - Attach discovery keyed on a workspaceDir field no release of @playwright/cli writes, so prima reported no browser while a session was open. Attached sessions now connect through the daemon's own Playwright build; connect succeeds across builds but ariaSnapshot does not. - ARIA compaction kept only the first bracket group, so every control carrying [pressed], [disabled] or [checked] lost its ref — exactly the controls worth acting on. - A redirect that only appends query parameters counted as failed navigation, costing three minutes of retries on a page that had loaded correctly. - verify reported claims it could not phrase as failing checks, and remembered them as such. It now separates the two, and can assert control state. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 49 +++ boat/prima/src/cli.ts | 17 +- boat/prima/src/envelope.ts | 61 ++- boat/prima/src/prima.ts | 215 ++++++---- boat/prima/src/pw-registry.ts | 9 +- boat/prima/src/tools.ts | 84 ++++ boat/prima/tests/envelope.test.ts | 36 +- boat/prima/tests/prima.test.ts | 128 +++--- boat/prima/tests/pw-registry.test.ts | 47 +-- boat/prima/tests/tools.test.ts | 118 ++++++ bun.lock | 6 +- .../2026-08-06-prima-vs-playwright-cli.md | 268 ++++++++++++ .../specs/2026-08-07-prima-fixes-design.md | 394 ++++++++++++++++++ package.json | 2 +- rules/navigator/verification-actions.md | 17 + src/ai/navigator.ts | 10 +- src/ai/tools.ts | 13 +- src/utils/aria.ts | 3 +- src/utils/url-matcher.ts | 3 + src/utils/web-element.ts | 9 + tests/integration/prima-heal.test.ts | 160 ------- tests/integration/prima-smoke.test.ts | 8 +- tests/unit/aria.test.ts | 21 + tests/unit/url-matcher.test.ts | 13 + 24 files changed, 1269 insertions(+), 422 deletions(-) create mode 100644 boat/prima/src/tools.ts create mode 100644 boat/prima/tests/tools.test.ts create mode 100644 docs/superpowers/reviews/2026-08-06-prima-vs-playwright-cli.md create mode 100644 docs/superpowers/specs/2026-08-07-prima-fixes-design.md delete mode 100644 tests/integration/prima-heal.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 73c66c11..6aeef7c2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,54 @@ # Changelog +## 2026-08-07 + +### Prima drives by element refs, and never substitutes your target + +A failed action now fails. Previously a `pw` call on a selector that did not exist could be +"healed" into clicking a different element and still report `ok: true` — so `ok: true` did not +mean your own action landed. Automatic retry along a different route is gone entirely, together +with the `--no-heal` flag that used to switch it off, and the `healed:` line and +`### Healing attempts` block in the envelope. + +`prima do` now works from the refs Playwright puts in the page snapshot. New `clickRef` and +`hoverRef` steps take the ref of the element they want instead of composing a locator and hoping, +so one instruction becomes one command rather than a list of fallbacks. A ref that no longer +resolves is reported as stale instead of being replaced by a guess. The existing `click` and +`hover` steps are unchanged and still available for pages without refs. + +### Configuration + +- **`ai.agents.prima.researchAfterVisits`** — how many visits to a page before `prima do` works + from that page's stored research map instead of its accessibility tree. Default: `3`. + +### Changes + +- [Prima] `### Changes` now appears on every action, showing what the accessibility tree gained, + lost or toggled — or saying `no change` outright. It previously went missing whenever the + command also produced an answer, a research map or a verdict, so a successful click proved + nothing. +- [Prima] Attaching to a running `playwright-cli` session works again. Prima was matching on a + field that no release of `playwright-cli` writes, so it reported no browser while one was open + and told you to open the session you already had. +- [Prima] Attached sessions are driven through the browser's own Playwright build, which is what + makes reading the page work across versions. +- [Prima] New `context()` step for `do`: when a ref goes stale mid-run it returns the page again + with fresh refs, and drops to raw markup if asked a second time on the same page. +- [Prima] `click` and `fill` report themselves in the envelope instead of appearing as `do`. +- [Prima] `network.jsonl` is listed only when requests were actually recorded, instead of always + pointing at an empty file. +- [Navigator] `verify` now separates "this claim is false" from "no assertion can express this + claim". A claim it cannot phrase is no longer reported as a failing check, and is no longer + remembered as one. +- [Navigator] Verification can assert whether a control is enabled, disabled, checked, selected or + expanded. Only presence and text could be asserted before, so state claims always failed. +- Navigation: a redirect that only adds query parameters — a session id, a workspace flag — counts + as arriving. Reaching such a page previously burned minutes of retries and could still end in a + failure while the page sat correctly loaded. +- Page snapshots keep every attribute on an element. A control marked `[pressed]` or `[disabled]` + used to lose its ref, which hid exactly the controls worth acting on. +- Playwright updated to 1.62. + ## 2026-08-06 ### Global Installation diff --git a/boat/prima/src/cli.ts b/boat/prima/src/cli.ts index 25612859..ee8247ae 100644 --- a/boat/prima/src/cli.ts +++ b/boat/prima/src/cli.ts @@ -30,21 +30,22 @@ const helpContract = dedent` Fall back to click/fill/do whenever research left you no locator to hold. ENVELOPE - ### Result ok, command, healed, used + ### Result ok, command, used ### Page url, title, state hash, visit count - ### Changes what the accessibility tree gained or lost + ### Changes what the accessibility tree gained or lost, or 'no change' + ### Steps per-instruction outcome of do, each with the change that proves it ### Answer | ### Research | ### Verdict output of ask, research, verify - ### Failure error, reasoning, healing attempts, compact ARIA of the page + ### Failure error, compact ARIA of the page ### Instance the browser you are on and the other instances running ### Artifacts paths to the full aria.yml, page.html and network.jsonl used: is code that already executed - CodeceptJS steps to copy as they are, except for pw, whose Playwright expression a test needs inside I.usePlaywrightTo(...). Log lines can precede the envelope; start parsing at the first ### line. - HEALING AND FAILURE - A failed action is retried by AI along a different route; healed: true means the - outcome was reached another way and used: holds the code that worked. - --no-heal skips that and fails fast. + FAILURE + A failed action fails. Nothing is retried along a different route and no other + element is ever substituted for the one you asked for, so ok: true always means + your own action landed. Failures print compact ARIA inline, so retarget from the envelope itself and open the artifact files only when the inline snapshot is not enough. @@ -78,7 +79,6 @@ function buildOptions(options: any): PrimaOptions { path: options.path, instance: options.instance, session: options.session, - heal: options.heal, ephemeral: options.ephemeral, framework: options.framework, noVision: options.vision === false, @@ -99,7 +99,6 @@ function addCommonOptions(cmd: Command): Command { .option('-p, --path ', 'Working directory path') .option('-i, --instance ', 'Browser instance to drive') .option('--session [file]', 'Persist cookies and storage to a session file') - .option('--no-heal', 'Fail immediately instead of letting AI retry a failed action') .option('--ephemeral', 'Keep no state between runs; applies to config-free runs, where output goes to a temp directory') .option('--framework ', 'Not active yet: framework the reported code targets, codeceptjs or playwright') .option('--url ', 'Page to open when the session has no page yet') diff --git a/boat/prima/src/envelope.ts b/boat/prima/src/envelope.ts index 46e68ffc..1368be0b 100644 --- a/boat/prima/src/envelope.ts +++ b/boat/prima/src/envelope.ts @@ -9,59 +9,48 @@ export interface InstanceInfo { others: Array<{ name: string; tabs: number }>; } -export interface HealAttempt { - code: string; - outcome: string; -} - export interface EnvelopeData { ok: boolean; command: string; - healed?: boolean; - healNote?: string; used?: string[]; page: { url: string; previousUrl?: string; title: string; state: string; visits: number }; changes?: string | null; + steps?: Array<{ instruction: string; proof: string | null }>; answer?: string; research?: string; verdict?: { passed: boolean; evidence: string; code: string }; - failure?: { error: string; attempts: HealAttempt[]; reasoning?: string; compactAria?: string }; + failure?: { error: string; compactAria?: string }; instance: InstanceInfo; - artifacts?: { aria: string; html: string; network: string }; + artifacts?: { aria: string; html: string; network?: string }; } export function renderEnvelope(data: EnvelopeData): string { - const sections = [renderResult(data), renderPage(data), renderOutcome(data), ...renderFailure(data), renderInstance(data.instance), renderArtifacts(data)]; + const sections = [renderResult(data), renderPage(data), renderChanges(data), renderSteps(data), renderOutcome(data), ...renderFailure(data), renderInstance(data.instance), renderArtifacts(data)]; return sections.filter((section) => section).join('\n\n'); } -export function writeArtifacts(dir: string, snapshot: { aria: string | null; html: string | null; requests: unknown[] }): { aria: string; html: string; network: string } { +export function writeArtifacts(dir: string, snapshot: { aria: string | null; html: string | null; requests: unknown[] }): { aria: string; html: string; network?: string } { mkdirSync(dir, { recursive: true }); - const paths = { + const paths: { aria: string; html: string; network?: string } = { aria: path.resolve(dir, 'aria.yml'), html: path.resolve(dir, 'page.html'), - network: path.resolve(dir, 'network.jsonl'), }; writeFileSync(paths.aria, snapshot.aria ?? '', 'utf-8'); writeFileSync(paths.html, snapshot.html ?? '', 'utf-8'); + + if (!snapshot.requests.length) return paths; + + paths.network = path.resolve(dir, 'network.jsonl'); writeFileSync(paths.network, snapshot.requests.map((request) => `${JSON.stringify(request)}\n`).join(''), 'utf-8'); return paths; } function renderResult(data: EnvelopeData): string { const lines = [`ok: ${data.ok}`, `command: ${data.command}`]; - const healed = renderHealed(data); - if (healed) lines.push(healed); if (data.used?.length) lines.push(`used: ${data.used.join('; ')}`); return section('Result', lines.join('\n')); } -function renderHealed(data: EnvelopeData): string | null { - if (data.healed === undefined) return null; - if (data.healNote) return `healed: ${data.healed} (${data.healNote})`; - return `healed: ${data.healed}`; -} - function renderPage(data: EnvelopeData): string { const { url, previousUrl, title, state, visits } = data.page; const urlLabel = `url: ${url}`; @@ -73,8 +62,21 @@ function renderPage(data: EnvelopeData): string { return section('Page', lines.join('\n')); } +function renderChanges(data: EnvelopeData): string | null { + if (data.changes === undefined || data.changes === null) return null; + return section('Changes', data.changes); +} + +function renderSteps(data: EnvelopeData): string | null { + if (!data.steps?.length) return null; + const lines = data.steps.map((step, index) => { + if (!step.proof) return `${index + 1}. ${step.instruction} — unproven`; + return `${index + 1}. ${step.instruction} — proven by ${step.proof}`; + }); + return section('Steps', lines.join('\n')); +} + function renderOutcome(data: EnvelopeData): string | null { - if (data.changes) return section('Changes', data.changes); if (data.answer) return section('Answer', data.answer); if (data.research) return section('Research', data.research); if (!data.verdict) return null; @@ -84,17 +86,7 @@ function renderOutcome(data: EnvelopeData): string | null { function renderFailure(data: EnvelopeData): Array { if (!data.failure) return []; - const lines = [`error: ${data.failure.error}`]; - if (data.failure.reasoning) lines.push(`reasoning: ${data.failure.reasoning}`); - return [section('Failure', lines.join('\n')), renderAttempts(data.failure.attempts), renderCompactAria(data.failure.compactAria)]; -} - -function renderAttempts(attempts: HealAttempt[]): string | null { - if (!attempts?.length) return null; - const labels = attempts.map((attempt, index) => `${index + 1}. ${attempt.code}`); - const width = Math.max(...labels.map((label) => label.length)) + 3; - const lines = labels.map((label, index) => align(label, `→ ${attempts[index].outcome}`, width)); - return section(`Healing attempts (${attempts.length})`, lines.join('\n')); + return [section('Failure', `error: ${data.failure.error}`), renderCompactAria(data.failure.compactAria)]; } function renderCompactAria(compactAria?: string): string | null { @@ -127,7 +119,8 @@ function tabsLabel(tabs: number): string { function renderArtifacts(data: EnvelopeData): string | null { if (!data.artifacts) return null; - const lines = [`aria: ${data.artifacts.aria}`, `html: ${data.artifacts.html}`, `network: ${data.artifacts.network}`]; + const lines = [`aria: ${data.artifacts.aria}`, `html: ${data.artifacts.html}`]; + if (data.artifacts.network) lines.push(`network: ${data.artifacts.network}`); return section('Artifacts', lines.join('\n')); } diff --git a/boat/prima/src/prima.ts b/boat/prima/src/prima.ts index 94338944..acab8ca7 100644 --- a/boat/prima/src/prima.ts +++ b/boat/prima/src/prima.ts @@ -1,10 +1,11 @@ import { createRequire } from 'node:module'; import path from 'node:path'; +import { tool } from 'ai'; import dedent from 'dedent'; +import { z } from 'zod'; import * as playwright from 'playwright'; import type { Browser } from 'playwright'; import { ActionResult } from '../../../src/action-result.ts'; -import type { Navigator } from '../../../src/ai/navigator.ts'; import { actionRule, locatorRule } from '../../../src/ai/rules.ts'; import { createCodeceptJSTools } from '../../../src/ai/tools.ts'; import { getAliveEndpoint, launchServer, listInstances, stopServer } from '../../../src/browser-server.ts'; @@ -12,19 +13,28 @@ import { ConfigMissingError, ConfigParser, type ExplorbotConfig, outputPath } fr import { ExplorBot } from '../../../src/explorbot.ts'; import type { WebPageState } from '../../../src/state-manager.ts'; import { Task } from '../../../src/test-plan.ts'; +import { getPreviousResearch } from '../../../src/ai/researcher/cache.ts'; import { compactAriaSnapshot } from '../../../src/utils/aria.ts'; import { browserErrorMessage } from '../../../src/utils/browser-errors.ts'; import { pluralize } from '../../../src/utils/logger.ts'; -import { type EnvelopeData, type HealAttempt, type InstanceInfo, writeArtifacts } from './envelope.ts'; +import { type EnvelopeData, type InstanceInfo, writeArtifacts } from './envelope.ts'; import { isFunctionExpression, toCodeceptWrapper } from './pw-parser.ts'; +import { createRefTools } from './tools.ts'; import { type PwServerDescriptor, readDescriptors, selectDescriptor } from './pw-registry.ts'; const MAX_INSTRUCTION_ITERATIONS = 6; +const DEFAULT_RESEARCH_AFTER_VISITS = 3; +const CONTEXT_HTML_CAP = 6000; const MAX_TOOL_ROUNDTRIPS = 5; const AI_AGENT_NAME = 'prima'; const CONNECT_TIMEOUT = 3000; const requireLib = createRequire(import.meta.url); +function cap(text: string, max: number): string { + if (text.length <= max) return text; + return `${text.slice(0, max)}\n[...truncated; ${text.length - max} chars omitted...]`; +} + export class Prima { private options: PrimaOptions; private bot: ExplorBot; @@ -65,7 +75,7 @@ export class Prima { const validation = isFunctionExpression(expression); if (!validation.valid) return this.toolFailureEnvelope(command, validation.error!); - const previousState = this.bot.stateManager().getCurrentState(); + const previousState = await this.baselineState(); let result: ActionResult | null = null; let executionError: unknown = null; @@ -76,23 +86,24 @@ export class Prima { executionError = error; } - if (executionError) return this.heal(command, expression, executionError, previousState); + if (executionError) return this.failureEnvelope(command, executionError, previousState); result ||= await this.capturedResult(previousState); return this.successEnvelope(command, [expression], result, previousState); } - async do(instructions: string[]): Promise { - const command = `do ${instructions.map((instruction) => `"${instruction}"`).join(' ')}`; + async do(instructions: string[], label?: string): Promise { + const command = label || `do ${instructions.map((instruction) => `"${instruction}"`).join(' ')}`; const guard = await this.aiGuard(command); if (guard) return guard; const provider = this.bot.getProvider(); - const previousState = this.bot.stateManager().getCurrentState(); + const previousState = await this.baselineState(); const conversation = provider.startConversation(this.instructionSystemPrompt(), AI_AGENT_NAME); const task = new Task(instructions.join('; '), previousState?.url || ''); - const tools = createCodeceptJSTools({ explorer: this.bot.getExplorer(), stateManager: this.bot.stateManager(), ai: provider }, task); - conversation.addUserText(this.instructionPrompt(instructions, await this.capturedResult(previousState))); + const deps = { explorer: this.bot.getExplorer(), stateManager: this.bot.stateManager(), ai: provider }; + const tools = { ...createCodeceptJSTools(deps, task), ...createRefTools(deps, task), context: this.contextTool() }; + conversation.addUserText(await this.instructionPrompt(instructions, await this.capturedResult(previousState))); const used: string[] = []; let failure: { code: string; message: string } | null = null; @@ -104,7 +115,7 @@ export class Prima { const state = this.bot.stateManager().getCurrentState(); if (iteration > 1 && state && state.hash !== contextHash) { contextHash = state.hash; - conversation.addUserText(this.pageContext(ActionResult.fromState(state))); + conversation.addUserText(await this.pageContext(ActionResult.fromState(state))); } const invoked = await provider.invokeConversation(conversation, tools, { maxToolRoundtrips: MAX_TOOL_ROUNDTRIPS, agentName: AI_AGENT_NAME }).catch((error: unknown) => { @@ -132,8 +143,8 @@ export class Prima { if (aiError) return this.failureEnvelope(command, aiError, previousState); if (failure) { - const envelope = await this.heal(command, failure.code || instructions.join('; '), failure.message, previousState); - envelope.used = [...used, ...(envelope.used || [])]; + const envelope = await this.failureEnvelope(command, failure.message, previousState); + envelope.used = used; return envelope; } @@ -147,11 +158,11 @@ export class Prima { } async click(target: string): Promise { - return this.do([`click ${target}`]); + return this.do([`click ${target}`], `click ${target}`); } async fill(field: string, value: string): Promise { - return this.do([`fill ${field} with value: ${value}`]); + return this.do([`fill ${field} with value: ${value}`], `fill ${field} ${value}`); } async ask(question: string): Promise { @@ -173,6 +184,10 @@ export class Prima { const result = await this.capturedResult(previousState); const verification = await this.bot.agentNavigator().verifyState(assertion, result); const codes = verification.successfulCodes || []; + if (verification.inexpressible) { + return this.reportEnvelope(command, result, previousState, { ok: false, verdict: { passed: false, evidence: 'not checked: no assertion can express this claim, so this is not a statement about the page', code: '' } }); + } + const verdict = { passed: verification.verified, evidence: this.verdictEvidence(verification.verified, codes), code: codes.join('\n') }; return this.reportEnvelope(command, result, previousState, { ok: verification.verified, verdict }); } @@ -208,7 +223,7 @@ export class Prima { navigationError = error; } - if (navigationError) return this.heal(command, code, navigationError, previousState); + if (navigationError) return this.failureEnvelope(command, navigationError, previousState); const used: string[] = []; if (isUrl) used.push(code); @@ -282,7 +297,7 @@ export class Prima { async toolFailureEnvelope(command: string, error: unknown): Promise { const state = this.bot.getCurrentState(); const instance = await this.instanceInfo().catch(() => ({ name: this.instanceName(), tabs: 0, others: [] })); - const failure: EnvelopeData['failure'] = { error: `tool: ${browserErrorMessage(error)}`, attempts: [] }; + const failure: EnvelopeData['failure'] = { error: `tool: ${browserErrorMessage(error)}` }; if (error instanceof ConfigMissingError) failure.error = browserErrorMessage(error); if (state?.ariaSnapshot) failure.compactAria = compactAriaSnapshot(state.ariaSnapshot, true); @@ -310,11 +325,12 @@ export class Prima { if (this.options.endpoint) { const endpoint = this.options.endpoint; const browserName = config.playwright.browser || 'chromium'; - if (await this.attachToEndpoint({ file: '', title: '', endpoint, workspaceDir: '', browserName, playwrightLib: '' })) return; + const known = readDescriptors().find((descriptor) => descriptor.endpoint === endpoint); + if (await this.attachToEndpoint({ file: '', title: '', endpoint, workspaceDir: '', browserName, playwrightLib: known?.playwrightLib || '' })) return; throw new Error(dedent` No browser answered at ${endpoint}. - Check the endpoint of the running session, or drop --endpoint to attach to the - playwright-cli browser of this workspace. + Check the endpoint of the running session, or drop --endpoint to let prima pick + the playwright-cli session itself. `); } @@ -324,7 +340,7 @@ export class Prima { if (!match && candidates.length) { const titles = candidates.map((candidate) => candidate.title).join(', '); throw new Error(dedent` - Several playwright-cli sessions are open for this workspace: ${titles} + Several playwright-cli sessions are open: ${titles} Pick one with --pw-session . `); } @@ -342,7 +358,7 @@ export class Prima { private async discover(descriptors = readDescriptors()): Promise<Discovery> { const title = this.options.pwSession ?? process.env.PLAYWRIGHT_CLI_SESSION; - const opts = { workspaceDir: this.workspaceDir(), title }; + const opts = { title }; const alive = new Map<PwServerDescriptor, Browser>(); for (const candidate of selectDescriptor(descriptors, opts).candidates) { @@ -372,9 +388,9 @@ export class Prima { } private async connectDescriptor(descriptor: PwServerDescriptor): Promise<Browser | null> { - const connected = await this.connectWith(playwright, descriptor); + const connected = await this.connectWith(this.descriptorLib(descriptor), descriptor); if (connected) return connected; - return this.connectWith(this.descriptorLib(descriptor), descriptor); + return this.connectWith(playwright, descriptor); } private async connectWith(lib: any, descriptor: PwServerDescriptor): Promise<Browser | null> { @@ -394,13 +410,10 @@ export class Prima { private attachmentLabel(descriptor: PwServerDescriptor): string { if (!descriptor.title) return `endpoint ${descriptor.endpoint}`; + if (!descriptor.workspaceDir) return `playwright-cli session "${descriptor.title}"`; return `playwright-cli session "${descriptor.title}", workspace ${descriptor.workspaceDir}`; } - private workspaceDir(): string { - return path.resolve(this.options.path || process.cwd()); - } - private async connectOwnInstance(): Promise<boolean> { return !!(await getAliveEndpoint(this.instanceName())); } @@ -425,49 +438,6 @@ export class Prima { return URL.canParse(value); } - private async heal(command: string, expression: string, error: unknown, previousState: WebPageState | null): Promise<EnvelopeData> { - if (this.options.heal === false) return this.failureEnvelope(command, error, previousState); - - const navigator = this.healNavigator(); - if (!navigator) { - const envelope = await this.failureEnvelope(command, error, previousState); - envelope.healed = false; - envelope.healNote = this.aiUnavailableNote(); - return envelope; - } - - const message = dedent` - I tried to run this command on the page: ${expression} - But it failed with: ${browserErrorMessage(error)} - Reach the same outcome on the current page in a different way. - `; - - const attempts: Array<{ code: string; error?: string }> = []; - const failedResult = await this.capturedResult(previousState); - const resolved = await navigator.resolveState(message, failedResult, { onAttempt: (attempt) => attempts.push(attempt) }).catch(() => false); - - if (!resolved) { - const healAttempts = attempts.map((attempt) => ({ code: attempt.code, outcome: attempt.error || 'ok' })); - return this.failureEnvelope(command, error, previousState, healAttempts); - } - - const used = attempts.filter((attempt) => !attempt.error).map((attempt) => attempt.code); - const result = await this.capturedResult(this.bot.stateManager().getCurrentState()); - const envelope = await this.successEnvelope(command, used, result, previousState); - envelope.healed = true; - envelope.healNote = `recovered after ${attempts.length} ${pluralize(attempts.length, 'attempt')}`; - return envelope; - } - - private healNavigator(): Navigator | null { - if (this.aiUnavailable()) return null; - try { - return this.bot.agentNavigator?.() ?? null; - } catch { - return null; - } - } - private aiUnavailable(): string | null { try { if (this.bot.getProvider?.()) return null; @@ -477,12 +447,6 @@ export class Prima { return this.bot.aiFailureReason?.() || 'no AI model is configured'; } - private aiUnavailableNote(): string { - const reason = this.aiUnavailable(); - if (!reason) return 'ai unavailable'; - return `ai unavailable: ${reason}`; - } - private async aiGuard(command: string): Promise<EnvelopeData | null> { const reason = this.aiUnavailable(); if (!reason) return null; @@ -504,40 +468,108 @@ export class Prima { 5. Stop calling tools when every instruction is done, or when an instruction cannot be performed on this page — say what is missing instead. </approach> + <scope> + Do only what the instructions ask. An action that looks helpful but was not asked for is out of scope — report it as something you noticed, never perform it. + Continuing past the last instruction is a failure, even when the next step seems obvious. + </scope> + + <proof> + An instruction is done only when a change on the page shows it. After each action read the reported change and decide which part of it proves the instruction. + When nothing observable followed, say the instruction is unproven and why. Do not restate the action as if it were the outcome. + </proof> + + <refs> + Page context lists refs for the elements you can act on. When your target has one, act on it with the ref tools rather than composing a locator — the ref names that exact element. + Fall back to the locator tools only for a target the context describes without a ref. + Refs belong to the context they came in. Once new context arrives the earlier refs are dead — use only the ones in the newest context you were given. + When a ref no longer resolves, or the element an instruction needs is missing from your newest context, call context() to look again and act on what it returns. + Never adapt a ref from an earlier context, invent one, or fall back to a locator you guessed. + </refs> + ${locatorRule} ${actionRule} `; } - private instructionPrompt(instructions: string[], result: ActionResult): string { + private async instructionPrompt(instructions: string[], result: ActionResult): Promise<string> { const list = instructions.map((instruction, index) => `${index + 1}. ${instruction}`).join('\n'); return dedent` <instructions> ${list} </instructions> - ${this.pageContext(result)} + ${await this.pageContext(result)} `; } - private pageContext(result: ActionResult): string { + private contextTool(): any { + let refreshed = false; + return tool({ + description: dedent` + Look at the page again when the refs you hold no longer resolve, or when the element an instruction needs is not in the context you were given. + The first call returns the page as it is now, with fresh refs that replace every ref you were holding. + A later call on the same page drops to the raw markup, for elements the accessibility tree does not describe. + Do not call it to confirm an action worked — the change is already reported back to you. + `, + inputSchema: z.object({ + reason: z.string().describe('Which element you cannot reach and what you already tried'), + }), + execute: async () => { + const result = await this.capturedResult(this.bot.stateManager().getCurrentState()); + if (!refreshed) { + refreshed = true; + return { success: true, context: await this.pageContext(result) }; + } + return { success: true, context: cap(await result.simplifiedHtml(), CONTEXT_HTML_CAP) }; + }, + }); + } + + private async pageContext(result: ActionResult): Promise<string> { const experience = this.bot.experienceTracker?.()?.renderExperienceTocFor?.(result) || ''; + const map = this.researchMap(result); + if (map) { + return dedent` + <page_ui_map url="${result.url}" title="${result.title}"> + ${map} + </page_ui_map> + + ${experience} + `; + } + return dedent` <page url="${result.url}" title="${result.title}"> - ${compactAriaSnapshot(result.ariaSnapshot, true)} + ${compactAriaSnapshot(await this.refAriaSnapshot(result), true)} </page> ${experience} `; } + private researchMap(result: ActionResult): string { + if (this.bot.stateManager().getVisitCount(result.url) < this.researchAfterVisits()) return ''; + return getPreviousResearch(result.getStateHash()); + } + + private researchAfterVisits(): number { + const configured = this.bot.getConfig?.()?.ai?.agents?.prima?.researchAfterVisits; + if (typeof configured === 'number') return configured; + return DEFAULT_RESEARCH_AFTER_VISITS; + } + + private async refAriaSnapshot(result: ActionResult): Promise<string | null> { + const snapshot = await Promise.resolve(this.bot.getExplorer()?.withPage?.((page: any) => page.locator('body').ariaSnapshot({ mode: 'ai' }))).catch(() => null); + return snapshot || result.ariaSnapshot; + } + private executedCodes(code: unknown): string[] { if (typeof code !== 'string') return []; return code .split('\n') .map((line) => line.trim()) - .filter((line) => line); + .filter((line) => line && !line.startsWith('//')); } private visionEnabled(): boolean { @@ -603,11 +635,10 @@ export class Prima { }; } - private async failureEnvelope(command: string, error: unknown, previousState: WebPageState | null, attempts: HealAttempt[] = []): Promise<EnvelopeData> { + private async failureEnvelope(command: string, error: unknown, previousState: WebPageState | null): Promise<EnvelopeData> { const result = await this.capturedResult(previousState); - const failure: EnvelopeData['failure'] = { error: browserErrorMessage(error), attempts }; + const failure: EnvelopeData['failure'] = { error: browserErrorMessage(error) }; if (result.ariaSnapshot) failure.compactAria = compactAriaSnapshot(result.ariaSnapshot, true); - if (attempts.length) failure.reasoning = [...new Set(attempts.map((attempt) => attempt.outcome))].join('; '); return { ok: false, @@ -650,10 +681,19 @@ export class Prima { }; } - private async pageChanges(result: ActionResult, previousState: WebPageState | null, code: string): Promise<string | null> { - if (!previousState) return null; + private async baselineState(): Promise<WebPageState | null> { + const existing = this.bot.stateManager?.()?.getCurrentState(); + if (existing) return existing; + + const result = await Promise.resolve(this.bot.getExplorer?.()?.capture?.()).catch(() => null); + if (!result) return null; + return this.bot.stateManager?.()?.updateState(result) ?? null; + } + + private async pageChanges(result: ActionResult, previousState: WebPageState | null, code: string): Promise<string> { + if (!previousState) return 'no snapshot was captured before this command, so nothing could be compared'; const toolResult = await result.toToolResult(ActionResult.fromState(previousState), code); - return toolResult.pageDiff?.ariaChanges ?? null; + return toolResult.pageDiff?.ariaChanges || 'no change'; } private async writeSnapshot(result: ActionResult): Promise<EnvelopeData['artifacts']> { @@ -692,7 +732,6 @@ export interface PrimaOptions { path?: string; instance?: string; session?: string | boolean; - heal?: boolean; ephemeral?: boolean; framework?: 'codeceptjs' | 'playwright'; noVision?: boolean; diff --git a/boat/prima/src/pw-registry.ts b/boat/prima/src/pw-registry.ts index 43168717..b689d796 100644 --- a/boat/prima/src/pw-registry.ts +++ b/boat/prima/src/pw-registry.ts @@ -20,9 +20,8 @@ export function readDescriptors(dir = registryDir()): PwServerDescriptor[] { return descriptors; } -export function selectDescriptor(descriptors: PwServerDescriptor[], opts: { workspaceDir: string; title?: string }): { match?: PwServerDescriptor; candidates: PwServerDescriptor[] } { - const workspaceDir = path.resolve(opts.workspaceDir); - const candidates = descriptors.filter((descriptor) => path.resolve(descriptor.workspaceDir) === workspaceDir); +export function selectDescriptor(descriptors: PwServerDescriptor[], opts: { title?: string } = {}): { match?: PwServerDescriptor; candidates: PwServerDescriptor[] } { + const candidates = descriptors; if (!candidates.length) return { candidates }; if (opts.title) { @@ -53,13 +52,13 @@ function parseDescriptor(file: string): PwServerDescriptor | null { return null; } - if (!data?.endpoint || !data?.title || !data?.workspaceDir) return null; + if (!data?.endpoint || !data?.title) return null; return { file, title: data.title, endpoint: data.endpoint, - workspaceDir: data.workspaceDir, + workspaceDir: data.workspaceDir || '', browserName: data.browser?.browserName || DEFAULT_BROWSER, playwrightLib: data.playwrightLib || '', }; diff --git a/boat/prima/src/tools.ts b/boat/prima/src/tools.ts new file mode 100644 index 00000000..86082227 --- /dev/null +++ b/boat/prima/src/tools.ts @@ -0,0 +1,84 @@ +import { tool } from 'ai'; +import dedent from 'dedent'; +import { z } from 'zod'; +import { ActionResult } from '../../../src/action-result.ts'; +import type { ToolDeps } from '../../../src/ai/agent.ts'; +import { commitNote, failedToolResult, successToolResult } from '../../../src/ai/tools.ts'; +import { type Task, TestResult } from '../../../src/test-plan.ts'; +import { WebElement } from '../../../src/utils/web-element.ts'; + +const REF_INPUT_DESCRIPTION = 'Ref of the target element, copied exactly as the current page snapshot writes it inside [ref=...]. Use it whenever the snapshot lists one for your target. Only refs from the latest snapshot resolve; never adapt or invent one.'; + +export async function refToXPath(explorer: any, ref: string): Promise<{ xpath?: string; error?: string }> { + if (!WebElement.isAriaRef(ref)) return { error: `"${ref}" is not a ref. Pass a ref exactly as the snapshot writes it inside [ref=...], or use commands instead.` }; + + const element = await explorer.withPage((page: any) => WebElement.fromAriaRef(page, ref)); + if (!element) return { error: `Ref ${ref} matches no element. It belongs to an older snapshot of this page.` }; + + const xpath = element.clickXPath; + if (!xpath) return { error: `Ref ${ref} resolved to an element with no distinctive attributes to target.` }; + + const matches = await explorer.withPage((page: any) => page.locator(`xpath=${xpath}`).count()); + if (matches !== 1) return { error: `Ref ${ref} resolved to ${xpath}, which matches ${matches} elements instead of exactly one.` }; + + return { xpath }; +} + +export function createRefTools({ explorer, stateManager }: ToolDeps, task: Task) { + const runRef = async (action: 'clickRef' | 'hoverRef', ref: string, explanation: string, toCommand: (xpath: string) => string) => { + const activeNote = task.startNote(explanation); + const resolved = await refToXPath(explorer, ref); + + if (resolved.error) { + activeNote.commit(TestResult.FAILED); + return failedToolResult(action, resolved.error, { + suggestion: 'Do not retry with a guessed locator. Take a fresh look at the page and use a ref it lists, or report that the element is not there.', + }); + } + + const previousState = ActionResult.fromState(stateManager.getCurrentState()!); + const runner = explorer.action(); + const command = toCommand(resolved.xpath!); + const success = await runner.attempt(command, explanation); + const toolResult = await ActionResult.fromState(stateManager.getCurrentState()!).toToolResult(previousState, command); + + if (!success) { + await commitNote(activeNote, TestResult.FAILED, toolResult, runner); + return failedToolResult(action, `${command} failed on the element behind ${ref}`, { ...toolResult, code: command }, runner.lastError); + } + + await commitNote(activeNote, TestResult.PASSED, toolResult, runner); + return successToolResult(action, { ...toolResult, code: command }, runner); + }; + + return { + clickRef: tool({ + description: dedent` + Click the element a page snapshot gave you a ref for. + + Use this whenever the snapshot lists a ref for your target — the ref names that exact + element, so no locator and no fallback list is needed. + Use click() instead when you are working from markup that carries no refs. + `, + inputSchema: z.object({ + ref: z.string().describe(REF_INPUT_DESCRIPTION), + explanation: z.string().describe('Why you are clicking this element'), + }), + execute: async ({ ref, explanation }) => runRef('clickRef', ref, explanation, (xpath) => `I.click(${JSON.stringify(xpath)})`), + }), + + hoverRef: tool({ + description: dedent` + Move the mouse to the element a page snapshot gave you a ref for, to reveal hover-only controls. + + Use this whenever the snapshot lists a ref for your target. It does not click. + Use hover() instead when you are working from markup that carries no refs. + `, + inputSchema: z.object({ + ref: z.string().describe(REF_INPUT_DESCRIPTION), + explanation: z.string().describe('Why you are hovering this element'), + }), + execute: async ({ ref, explanation }) => runRef('hoverRef', ref, explanation, (xpath) => `I.moveCursorTo(${JSON.stringify(xpath)})`), + }), + }; +} diff --git a/boat/prima/tests/envelope.test.ts b/boat/prima/tests/envelope.test.ts index 0071cef6..0c3dbad2 100644 --- a/boat/prima/tests/envelope.test.ts +++ b/boat/prima/tests/envelope.test.ts @@ -32,26 +32,21 @@ describe('renderEnvelope', () => { expect(out).not.toContain('(changed:'); }); - test('failure envelope renders attempts, reasoning and compact aria', () => { + test('failure envelope renders the error and compact aria, and nothing about retries', () => { const out = renderEnvelope({ ...base, ok: false, failure: { error: "locator 'text=Login' not found", - attempts: [ - { code: "I.click('Login')", outcome: 'not visible' }, - { code: 'scroll + retry', outcome: 'covered by cookie banner' }, - ], - reasoning: 'element hidden behind consent overlay', compactAria: '- button "Accept all"', }, }); expect(out).toContain('### Failure'); - expect(out).toContain('### Healing attempts (2)'); - expect(out).toContain("1. I.click('Login')"); - expect(out).toContain('→ not visible'); + expect(out).toContain("locator 'text=Login' not found"); expect(out).toContain('### Current page (compact ARIA)'); expect(out).toContain('- button "Accept all"'); + expect(out).not.toContain('Healing'); + expect(out).not.toContain('healed'); }); test('answer replaces changes for ask', () => { @@ -75,9 +70,26 @@ describe('renderEnvelope', () => { expect(out).toContain("I.see('Dashboard')"); }); - test('healed success carries note', () => { - const out = renderEnvelope({ ...base, healed: true, healNote: 'dismissed overlay first' }); - expect(out).toContain('healed: true (dismissed overlay first)'); + test('changes render on every action envelope, including when nothing moved', () => { + expect(renderEnvelope({ ...base, changes: 'no change' })).toContain('### Changes\nno change'); + }); + + test('changes render alongside a verdict rather than replacing it', () => { + const out = renderEnvelope({ ...base, changes: 'no change', verdict: { passed: true, evidence: 'I.seeElement()', code: 'I.seeElement()' } }); + expect(out).toContain('### Changes'); + expect(out).toContain('### Verdict'); + }); + + test('steps report per-instruction proof and name what stayed unproven', () => { + const out = renderEnvelope({ + ...base, + steps: [ + { instruction: 'open the account menu', proof: 'added menu "Account"' }, + { instruction: 'choose the settings entry', proof: null }, + ], + }); + expect(out).toContain('1. open the account menu — proven by added menu "Account"'); + expect(out).toContain('2. choose the settings entry — unproven'); }); test('attached instance renders attached browser line', () => { diff --git a/boat/prima/tests/prima.test.ts b/boat/prima/tests/prima.test.ts index 2552f96b..8a507a83 100644 --- a/boat/prima/tests/prima.test.ts +++ b/boat/prima/tests/prima.test.ts @@ -29,6 +29,7 @@ function fakeState(over: Record<string, unknown> = {}) { getStateHash: () => 'login_h1_login', ariaSnapshot: '- textbox "Email"\n- button "Sign in"', combinedHtml: () => '<form></form>', + simplifiedHtml: async () => '<form><button>Sign in</button></form>', toToolResult: async () => ({ pageDiff: { urlChanged: false, ariaChanges: null } }), ...over, }; @@ -77,12 +78,14 @@ function fakePrima(options: Record<string, unknown> = {}) { }, }), capture: async () => after, + withPage: async (fn: (page: any) => any) => fn({ locator: () => ({ ariaSnapshot: async () => `${(prima as any).bot.stateManager().getCurrentState().ariaSnapshot}\n- button "Refreshed" [ref=e7]` }) }), }), stateManager: () => ({ getCurrentState: () => fakeState(), getVisitCount: () => 1, }), getCurrentState: () => fakeState(), + getConfig: () => ({}), requestStore: () => ({ getRequests: () => [] }), getProvider: () => ({ chat: async () => '' }), }; @@ -121,7 +124,7 @@ describe('Prima.pw', () => { expect(envelope.changes).toContain('heading "Dashboard"'); expect(existsSync(envelope.artifacts!.aria)).toBe(true); expect(existsSync(envelope.artifacts!.html)).toBe(true); - expect(existsSync(envelope.artifacts!.network)).toBe(true); + expect(envelope.artifacts!.network).toBeUndefined(); expect(envelope.instance.name).toBe('default'); }); @@ -138,7 +141,6 @@ describe('Prima.pw', () => { const envelope = await prima.pw("({ page }) => page.click('text=Login')"); expect(envelope.ok).toBe(false); expect(envelope.failure?.error).toContain("locator 'text=Login' not found"); - expect(envelope.failure?.attempts).toEqual([]); expect(envelope.page.url).toBe('https://app.example.com/login'); expect(envelope.artifacts).toBeTruthy(); }); @@ -172,76 +174,34 @@ describe('Prima.pw', () => { }); }); -describe('Prima heal', () => { - test('failed pw heals via navigator and reports healed envelope', async () => { +describe('Prima failure never substitutes a target', () => { + test('a failed pw fails without consulting the navigator', async () => { const { prima } = fakePrima(); - (prima as any).bot.getExplorer = failingExplorer; - (prima as any).bot.agentNavigator = () => ({ - resolveState: async (_msg: string, _result: unknown, opts: any) => { - opts?.onAttempt?.({ code: "I.click('Login')", error: 'not visible' }); - opts?.onAttempt?.({ code: "I.click('#login-btn')" }); - return true; - }, - }); - const envelope = await prima.pw("({ page }) => page.click('text=Login')"); - expect(envelope.ok).toBe(true); - expect(envelope.healed).toBe(true); - expect(envelope.healNote).toBeTruthy(); - expect(envelope.used).toEqual(["I.click('#login-btn')"]); - }); - - test('exhausted heal returns failure envelope with attempts and compact aria', async () => { - const { prima } = fakePrima(); - (prima as any).bot.getExplorer = failingExplorer; - (prima as any).bot.agentNavigator = () => ({ - resolveState: async (_msg: string, _result: unknown, opts: any) => { - opts?.onAttempt?.({ code: "I.click('Login')", error: 'not visible' }); - return false; - }, - }); - const envelope = await prima.pw("({ page }) => page.click('text=Login')"); - expect(envelope.ok).toBe(false); - expect(envelope.failure?.error).toContain("locator 'text=Login' not found"); - expect(envelope.failure?.attempts).toEqual([{ code: "I.click('Login')", outcome: 'not visible' }]); - expect(envelope.failure?.reasoning).toBe('not visible'); - expect(envelope.failure?.compactAria).toContain('button'); - expect(envelope.artifacts).toBeTruthy(); - }); - - test('heal disabled skips navigator entirely', async () => { - const { prima } = fakePrima({ heal: false }); let called = false; (prima as any).bot.getExplorer = failingExplorer; (prima as any).bot.agentNavigator = () => { called = true; return { resolveState: async () => true }; }; + const envelope = await prima.pw("({ page }) => page.click('text=Login')"); + expect(called).toBe(false); expect(envelope.ok).toBe(false); - expect(envelope.healed).toBeUndefined(); - expect(envelope.failure?.attempts).toEqual([]); + expect(envelope.used).toBeUndefined(); + expect(envelope.failure?.error).toContain("locator 'text=Login' not found"); expect(envelope.failure?.compactAria).toContain('button'); + expect(envelope.artifacts).toBeTruthy(); }); - test('unusable ai provider skips healing and notes it in the envelope', async () => { + test('a failed pw carries no healed marker', async () => { const { prima } = fakePrima(); - let called = false; (prima as any).bot.getExplorer = failingExplorer; - (prima as any).bot.getProvider = () => { - throw new Error('AI provider is not configured'); - }; - (prima as any).bot.agentNavigator = () => { - called = true; - return { resolveState: async () => true }; - }; + const envelope = await prima.pw("({ page }) => page.click('text=Login')"); - expect(called).toBe(false); - expect(envelope.ok).toBe(false); - expect(envelope.healed).toBe(false); - expect(envelope.healNote).toContain('ai unavailable'); - expect(envelope.healNote).toContain('AI provider is not configured'); - expect(envelope.failure?.error).toContain("locator 'text=Login' not found"); + + expect(envelope).not.toHaveProperty('healed'); + expect(envelope).not.toHaveProperty('healNote'); }); }); @@ -491,20 +451,20 @@ describe('Prima.do', () => { expect(calls).toBe(6); }); - test('routes a failed tool execution through heal', async () => { + test('a failed tool execution fails the command instead of reaching for another element', async () => { const { prima } = fakePrima(); + let called = false; (prima as any).bot.getProvider = () => fakeProvider(async () => ({ toolExecutions: [toolExecution("I.click('Login')", false, 'element not found')] })); - (prima as any).bot.agentNavigator = () => ({ - resolveState: async (_msg: string, _result: unknown, opts: any) => { - opts?.onAttempt?.({ code: "I.click('#login-btn')" }); - return true; - }, - }); + (prima as any).bot.agentNavigator = () => { + called = true; + return { resolveState: async () => true }; + }; const envelope = await prima.do(['click the login link']); - expect(envelope.ok).toBe(true); - expect(envelope.healed).toBe(true); - expect(envelope.used).toEqual(["I.click('#login-btn')"]); + + expect(called).toBe(false); + expect(envelope.ok).toBe(false); + expect(envelope.failure?.error).toContain('element not found'); }); test('reports a failure when the model performs no action and keeps its explanation', async () => { @@ -554,6 +514,25 @@ describe('Prima.do', () => { expect(prompts.length).toBe(1); }); + test('context() hands back refs first and drops to markup only when asked again', async () => { + const { prima } = fakePrima(); + let captured: any; + (prima as any).bot.getProvider = () => + fakeProvider(async (_conversation: unknown, tools: any) => { + captured = tools.context; + return { toolExecutions: [] }; + }); + + await prima.do(['open the invoices page']); + + const first = await captured.execute({ reason: 'the ref no longer resolves' }); + expect(first.context).toContain('ref=e7'); + + const second = await captured.execute({ reason: 'still cannot reach it' }); + expect(second.context).not.toContain('ref=e7'); + expect(second.context).toContain('<form>'); + }); + test('click is a single-instruction alias over do', async () => { const { prima } = fakePrima(); const received: string[][] = []; @@ -795,27 +774,28 @@ describe('Prima.go', () => { const envelope = await prima.go('/billing'); expect(envelope.ok).toBe(false); - expect(envelope.healed).toBe(false); - expect(envelope.healNote).toContain('ai unavailable'); + expect(envelope).not.toHaveProperty('healed'); expect(envelope.failure?.error).toContain('AI-assisted recovery is unavailable'); }); - test('navigation error is routed through heal', async () => { + test('navigation error fails instead of reaching the target another way', async () => { const { prima } = fakePrima(); + let resolved = false; (prima as any).bot.agentNavigator = () => ({ visit: async () => { throw new Error('Navigation to /billing failed'); }, - resolveState: async (_message: string, _result: unknown, opts: any) => { - opts?.onAttempt?.({ code: "I.click('Billing')" }); + resolveState: async () => { + resolved = true; return true; }, }); const envelope = await prima.go('/billing'); - expect(envelope.ok).toBe(true); - expect(envelope.healed).toBe(true); - expect(envelope.used).toEqual(["I.click('Billing')"]); + + expect(resolved).toBe(false); + expect(envelope.ok).toBe(false); + expect(envelope.failure?.error).toContain('Navigation to /billing failed'); }); test('go cannot run before a session exists and never launches a browser', async () => { diff --git a/boat/prima/tests/pw-registry.test.ts b/boat/prima/tests/pw-registry.test.ts index 09ed9dce..a217e7ab 100644 --- a/boat/prima/tests/pw-registry.test.ts +++ b/boat/prima/tests/pw-registry.test.ts @@ -26,14 +26,21 @@ describe('readDescriptors', () => { expect(list[0].playwrightLib).toBe('/lib/pw'); }); - test('skips descriptors without an endpoint, a title or a workspace', () => { + test('skips descriptors without an endpoint or a title', () => { const dir = mkdtempSync(path.join(tmpdir(), 'pwb-')); - writeDescriptor(dir, 'a', { title: 'default' }); writeDescriptor(dir, 'b', { workspaceDir: '/work/app' }); writeDescriptor(dir, 'c', { title: 'default', workspaceDir: '/work/app', endpoint: undefined }); expect(readDescriptors(dir)).toEqual([]); }); + test('keeps descriptors that carry no workspaceDir, as playwright-cli writes them', () => { + const dir = mkdtempSync(path.join(tmpdir(), 'pwb-')); + writeDescriptor(dir, 'a', { title: 'default' }); + const list = readDescriptors(dir); + expect(list.length).toBe(1); + expect(list[0].workspaceDir).toBe(''); + }); + test('returns nothing when the registry directory is missing', () => { expect(readDescriptors(path.join(tmpdir(), 'pwb-missing-registry'))).toEqual([]); }); @@ -41,28 +48,25 @@ describe('readDescriptors', () => { describe('selectDescriptor', () => { const dir = mkdtempSync(path.join(tmpdir(), 'pwb-')); - writeDescriptor(dir, 'a', { title: 'default', workspaceDir: '/work/app' }); - writeDescriptor(dir, 'b', { title: 'auth', workspaceDir: '/work/app' }); - writeDescriptor(dir, 'c', { title: 'default', workspaceDir: '/work/other' }); + writeDescriptor(dir, 'a', { title: 'default' }); + writeDescriptor(dir, 'b', { title: 'auth' }); const all = readDescriptors(dir); - test('explicit title wins within workspace', () => { - const { match } = selectDescriptor(all, { workspaceDir: '/work/app', title: 'auth' }); - expect(match?.title).toBe('auth'); + test('explicit title wins', () => { + expect(selectDescriptor(all, { title: 'auth' }).match?.title).toBe('auth'); }); - test('default title picked for workspace when present', () => { - const { match } = selectDescriptor(all, { workspaceDir: '/work/app' }); - expect(match?.title).toBe('default'); + test('default title picked when no title is asked for', () => { + expect(selectDescriptor(all).match?.title).toBe('default'); }); - test('single survivor for workspace picked without title', () => { - const { match } = selectDescriptor(all, { workspaceDir: '/work/other' }); - expect(match?.workspaceDir).toBe('/work/other'); + test('the single live session is picked without a title', () => { + const single = all.filter((descriptor) => descriptor.title === 'auth'); + expect(selectDescriptor(single).match?.title).toBe('auth'); }); - test('no workspace match returns candidates empty', () => { - const { match, candidates } = selectDescriptor(all, { workspaceDir: '/elsewhere' }); + test('no descriptors returns no match and no candidates', () => { + const { match, candidates } = selectDescriptor([]); expect(match).toBeUndefined(); expect(candidates.length).toBe(0); }); @@ -70,19 +74,14 @@ describe('selectDescriptor', () => { test('ambiguity returns no match with candidates listed', () => { const noDefault = all.filter((d) => d.title !== 'default'); const extra = [...noDefault, { ...noDefault[0], title: 'second' }]; - const { match, candidates } = selectDescriptor(extra, { workspaceDir: '/work/app' }); + const { match, candidates } = selectDescriptor(extra); expect(match).toBeUndefined(); expect(candidates.length).toBe(2); }); - test('unknown title in a populated workspace returns the workspace candidates', () => { - const { match, candidates } = selectDescriptor(all, { workspaceDir: '/work/app', title: 'missing' }); + test('unknown title returns every candidate', () => { + const { match, candidates } = selectDescriptor(all, { title: 'missing' }); expect(match).toBeUndefined(); expect(candidates.map((candidate) => candidate.title).sort()).toEqual(['auth', 'default']); }); - - test('workspace paths are compared resolved', () => { - const { match } = selectDescriptor(all, { workspaceDir: '/work/app/../app' }); - expect(match?.title).toBe('default'); - }); }); diff --git a/boat/prima/tests/tools.test.ts b/boat/prima/tests/tools.test.ts new file mode 100644 index 00000000..4f17d289 --- /dev/null +++ b/boat/prima/tests/tools.test.ts @@ -0,0 +1,118 @@ +import { beforeEach, describe, expect, it } from 'bun:test'; +import { ActionResult } from '../../../src/action-result.ts'; +import { createCodeceptJSTools } from '../../../src/ai/tools.ts'; +import { createRefTools } from '../src/tools.ts'; +import { ConfigParser } from '../../../src/config.ts'; +import { Task } from '../../../src/test-plan.ts'; + +beforeEach(() => { + ConfigParser.resetForTesting(); + ConfigParser.setupTestConfig(); +}); + +const state = new ActionResult({ url: '/page', title: 'Page', html: '<html></html>', ariaSnapshot: '' }); + +function buildTools(page: any, attempted: string[]) { + const action = { + lastError: null as Error | null, + attempt: async (command: string) => { + attempted.push(command); + return true; + }, + saveScreenshot: async () => null, + }; + const explorer = { + action: () => action, + withPage: (fn: (page: any) => any) => fn(page), + } as any; + const stateManager = { getCurrentState: () => state } as any; + return createRefTools({ explorer, stateManager, ai: {} as any }, new Task('t', '/page')); +} + +function pageWith(element: any, matches: number) { + return { + locator: (selector: string) => { + if (selector.startsWith('xpath=')) return { count: async () => matches }; + return { + count: async () => (element ? 1 : 0), + first: () => ({ evaluate: async () => element }), + }; + }, + }; +} + +const saveButton = { tag: 'button', allAttrs: { id: 'save-button' }, text: 'Save', outerHTML: '<button id="save-button">Save</button>', x: 1, y: 1 }; + +describe('clickRef', () => { + it('resolves a ref to an attribute-based xpath and clicks it', async () => { + const attempted: string[] = []; + const tools = buildTools(pageWith(saveButton, 1), attempted); + + const result = await tools.clickRef.execute({ ref: 'f1e13', explanation: 'click save' }); + + expect(result.success).toBe(true); + expect(attempted).toEqual(['I.click("//*[@id=\\"save-button\\"]")']); + expect(result.code).toContain('save-button'); + }); + + it('fails without guessing when the ref matches nothing', async () => { + const attempted: string[] = []; + const tools = buildTools(pageWith(null, 0), attempted); + + const result = await tools.clickRef.execute({ ref: 'f1e99', explanation: 'click save' }); + + expect(result.success).toBe(false); + expect(result.message).toContain('matches no element'); + expect(attempted).toEqual([]); + }); + + it('fails when the resolved xpath is ambiguous', async () => { + const attempted: string[] = []; + const tools = buildTools(pageWith(saveButton, 3), attempted); + + const result = await tools.clickRef.execute({ ref: 'f1e13', explanation: 'click save' }); + + expect(result.success).toBe(false); + expect(result.message).toContain('matches 3 elements'); + expect(attempted).toEqual([]); + }); + + it.each(['e13', 'f1e13', 'f12e345'])('accepts the ref shape %p that playwright emits', async (ref) => { + const attempted: string[] = []; + const tools = buildTools(pageWith(saveButton, 1), attempted); + + const result = await tools.clickRef.execute({ ref, explanation: 'click save' }); + + expect(result.success).toBe(true); + }); + + it('rejects a ref that is not snapshot-shaped', async () => { + const attempted: string[] = []; + const tools = buildTools(pageWith(saveButton, 1), attempted); + + const result = await tools.clickRef.execute({ ref: 'button.save', explanation: 'click save' }); + + expect(result.success).toBe(false); + expect(result.message).toContain('is not a ref'); + expect(attempted).toEqual([]); + }); + + it('leaves the locator tools alone — they take no ref', async () => { + const ladder = createCodeceptJSTools({ explorer: {} as any, stateManager: {} as any, ai: {} as any }, new Task('t', '/page')); + + expect(Object.keys(ladder.click.inputSchema.shape)).toEqual(['commands', 'explanation']); + expect(Object.keys(ladder.hover.inputSchema.shape)).toEqual(['commands', 'explanation']); + }); +}); + +describe('hoverRef', () => { + it('resolves a ref to a moveCursorTo command', async () => { + const attempted: string[] = []; + const tools = buildTools(pageWith(saveButton, 1), attempted); + + const result = await tools.hoverRef.execute({ ref: 'f1e13', explanation: 'reveal row actions' }); + + expect(result.success).toBe(true); + expect(attempted).toEqual(['I.moveCursorTo("//*[@id=\\"save-button\\"]")']); + }); +}); diff --git a/bun.lock b/bun.lock index 2a159374..7bb52d82 100644 --- a/bun.lock +++ b/bun.lock @@ -52,7 +52,7 @@ "ora-classic": "^5.4.2", "parse5": "^8.0.0", "pixelmatch": "^7.2.0", - "playwright": "^1.60", + "playwright": "^1.62", "pngjs": "^7.0.0", "react": "^19.1.1", "sambanova-ai-provider": "^1.2.2", @@ -2153,7 +2153,7 @@ "pkce-challenge": ["pkce-challenge@5.0.1", "", {}, "sha512-wQ0b/W4Fr01qtpHlqSqspcj3EhBvimsdh0KlHhH8HRZnMsEa0ea2fTULOXOS9ccQr3om+GcGRk4e+isrZWV8qQ=="], - "playwright": ["playwright@1.60.0", "", { "dependencies": { "playwright-core": "1.60.0" }, "optionalDependencies": { "fsevents": "2.3.2" }, "bin": { "playwright": "cli.js" } }, "sha512-hheHdokM8cdqCb0lcE3s+zT4t4W+vvjpGxsZlDnikarzx8tSzMebh3UiFtgqwFwnTnjYQcsyMF8ei2mCO/tpeA=="], + "playwright": ["playwright@1.62.1", "", { "dependencies": { "playwright-core": "1.62.1" }, "optionalDependencies": { "fsevents": "2.3.2" }, "bin": { "playwright": "cli.js" } }, "sha512-0M+L3LAD8/nm554LOla9Ayx0j0tmFZ0FBcoQ7F1VuVHpM/XpiC8RcDzBQB8W5+hA8L22THxELzeF+2WcUzvcLg=="], "playwright-core": ["playwright-core@1.54.1", "", { "bin": { "playwright-core": "cli.js" } }, "sha512-Nbjs2zjj0htNhzgiy5wu+3w09YetDx5pkrpI/kZotDlDUaYk0HVA5xrBVPdow4SAUIlhgKcJeJg4GRKW6xHusA=="], @@ -3209,7 +3209,7 @@ "path-scurry/lru-cache": ["lru-cache@10.4.3", "", {}, "sha512-JNAzZcXrCt42VGLuYz0zfAzDfAvJWW6AfYlDBQyDV5DClI2m5sAmK+OIO7s59XfsRsWHp02jAJrRadPRGTt6SQ=="], - "playwright/playwright-core": ["playwright-core@1.60.0", "", { "bin": { "playwright-core": "cli.js" } }, "sha512-9bW6zvX/m0lEbgTKJ6YppOKx8H3VOPBMOCFh2irXFOT4BbHgrx5hPjwJYLT40Lu+4qtD36qKc/Hn56StUW57IA=="], + "playwright/playwright-core": ["playwright-core@1.62.1", "", { "bin": { "playwright-core": "cli.js" } }, "sha512-wPYSwEBJY9GHraISXqyqtx0na0LpO3XEX7jNDhntbex7tzUS7kLnZsOlFruFJB4Hi/rhDMjXGqHewDZ68nYZVw=="], "pretty-format/ansi-regex": ["ansi-regex@5.0.1", "", {}, "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ=="], diff --git a/docs/superpowers/reviews/2026-08-06-prima-vs-playwright-cli.md b/docs/superpowers/reviews/2026-08-06-prima-vs-playwright-cli.md new file mode 100644 index 00000000..c5baecb1 --- /dev/null +++ b/docs/superpowers/reviews/2026-08-06-prima-vs-playwright-cli.md @@ -0,0 +1,268 @@ +# Prima boat vs playwright-cli — field review + +**Date:** 2026-08-06 +**Target:** Testeiya Agent (`localhost:3050`) — sidebar Workflows section + Skill editor +**Setup:** global config (`~/.explorbot/config.js`), no project config +**Models:** `groq/openai/gpt-oss-20b` (model), `openrouter/gpt-5.6-luna` (vision + agentic) +**Comparator:** `@playwright/cli` 0.1.13 +**Method:** drive the feature with prima, fall back to playwright-cli whenever prima stalled + +--- + +## Verdict up front + +Prima is not a replacement for playwright-cli today. It is a strong *complement*: two of its +commands (`verify`, `ask`) do work playwright-cli cannot do at all, and four defects stop it +from owning the driving loop. Recommended split as it stands: + +- **playwright-cli** — attach, navigate, read page state, drive verified locators. +- **prima** — `verify` for assertions, `ask` for visual judgement, `--no-heal` for a failure + report you can retarget from. + +--- + +## The feature under test — it works + +Everything the sidebar Workflow and Skill editor promises checked out. + +**Workflows section** + +- The rail button opens a Workflows panel with five categories: Analysis & Planning, + Test Design & Management, Test Execution & Automation, Reporting/CI-CD & Quality Gates, + Metrics/Release & Analytics. +- Accordion is single-open; the first category is expanded by default and lists its four + prompts (Review requirements, Risk-based focus, Analyze PR requirements, Analyze PR diff). +- "Workflow overview" opens the diagram dialog. Vision check and my own screenshot agree: + a clean left-to-right five-stage pipeline, arrows between stages, nothing clipped or + overlapping. + +**Skill editor** + +- Skills popover lists 42 skills with category filter chips. +- Row actions menu offers "Change skill globally", "Edit for this project", "Disable". +- The editor opens with the skill markdown loaded, Save disabled. +- Typing flips the header to `● unsaved` and enables Save. +- "Close editor" discards: no file appeared under `~/.testeiya/skills/`, and the agent + repo's `git status` stayed clean. + +One cosmetic thing worth a look: the filter chips include single-item categories named +after the skill itself — "Playwright Best Practices Skill 1", "Playwright Cli 1" — next to +real groupings like "Test Management 12". Category derivation looks like it falls back to +the skill name when a skill has no category. + +--- + +## Where prima was better + +**1. `verify` — the standout.** One command turns a sentence into a pass/fail *and* the +assertion code that proved it. + +``` +prima verify "the sidebar Workflows section lists five workflow categories, and the + Analysis & Planning category is expanded showing individual workflow prompts" +→ 14.8s, passed: true + I.seeElement({"role":"button","text":"Analysis & Planning"}); + ... all five categories ... + I.see('Review requirements', '#base-ui-_r_2al_'); ... all four prompts ... +``` + +With playwright-cli the same check is snapshot → read the tree myself → write five +assertions by hand. Prima did it in one call, and the output is reusable test material. +This is the command that justifies the boat. + +**2. `ask` — visual judgement with no image in my context.** + +``` +prima ask "Is the workflow diagram laid out as a readable left-to-right pipeline, + or is anything visually broken, overlapping or cut off?" +→ 8.7s, correct answer +``` + +I checked it against the PNG afterwards and it was right. playwright-cli can only hand me +a file; *I* have to look at it, and the image lands in my context permanently. + +**3. Descriptions outlive refs.** playwright-cli refs (`e13`, `e26`, `e647`) are snapshot-scoped. +This app re-renders constantly, so every interaction meant re-snapshotting for fresh refs — +8k bytes each time. `prima click "the Workflows button in the left vertical sidebar rail"` +needs no prior read at all. + +**4. Artifacts stay on disk.** Every envelope wrote `aria.yml` / `page.html` and cited absolute +paths. `page.html` was 376K and never entered my context. That is the design goal, delivered. + +**5. The `--no-heal` failure envelope is exactly right.** + +``` +prima pw '({page}) => page.click("[data-test=nonexistent-thing]", {timeout:3000})' --no-heal +→ 8.6s, ok: false + error: page.click: Timeout 3000ms exceeded. Call log: waiting for locator(...) + ### Current page (compact ARIA) ← enough to retarget, without the full tree +``` + +Error plus a compact ARIA snapshot in one response is better than what playwright-cli gives +on a failed click, which is the error alone. + +--- + +## Where playwright-cli was better + +**1. Speed — one to two orders of magnitude.** `snapshot` returned in **0.2–0.3s**. Prima's +cheapest command was 6.1s; the median was ~12s; `go` took **3m00s** and one owned-browser +`verify` took **2m32s**. Across ~17 prima commands I spent roughly ten minutes waiting. + +**2. It does exactly what you asked — nothing else.** `prima do` given two instructions +("close the dialog", "open the Skills menu") performed both and then kept going: searched the +skill list for "playwright-cli" and inserted the skill, leaving `/playwright-cli` typed into a +live chat box. Two instructions in, four actions out, `ok: true`. Prima's `go` did the same +during recovery — it clicked "Workspace", "Show panel", pressed F5 and clicked "Cancel" while +merely trying to navigate. playwright-cli has never once done something I did not type. + +**3. Attach just works.** playwright-cli opened and reattached to its own browser with no +ceremony. Prima could not attach at all without hand-patching a file (details below). + +**4. `eval` reaches state prima cannot express.** The one assertion prima got wrong — +"is Save enabled now?" — was a one-liner for playwright-cli: + +``` +$PW eval '() => [...document.querySelectorAll("button")].filter(b=>/Save/.test(b.textContent)) + .map(b=>({t:b.textContent.trim(), disabled:b.disabled}))' +→ [{ "t": "saveSave", "disabled": false }] +``` + +playwright-cli's snapshot also carries `[disabled]`, `[pressed]`, `[active]` inline. Prima's +UI map carries none of that. + +**5. Honest failures.** playwright-cli never told me an action succeeded when it hadn't. +Prima did (see #1 below). + +**6. `research` costs more context than a snapshot, not less.** The one `research` call +returned ~5k tokens of UI-map markdown — versus 8,110 bytes (~2k tokens) for a full +playwright-cli snapshot of the same page. Its locators were also worse: generated Base-UI ids +(`button#base-ui-_R_qcmpbmulb_`) and positional CSS chains +(`div:nth-of-type(1) > div > div:nth-of-type(1) > div`), neither of which survives a re-render. +Prima's own compact-ARIA failure block is the cheaper, better read. + +--- + +## Defects found in prima, worst first + +**1. Heal converts a failed action into a different action and reports success.** + +``` +prima pw '({page}) => page.click("[data-test=nonexistent-thing]", {timeout:3000})' +→ ok: true + healed: true (recovered after 1 attempt) + used: // Click the "New agent" button + I.click({ role: "button", text: "New agent" }); +``` + +The requested element does not exist. Heal did not find another route to the same intent — +there was no such intent to reach. It picked an unrelated control, clicked it, and returned +`ok: true`. An orchestrator that trusts the envelope believes its own action landed. Heal +should only re-route toward the *same* target; when the target cannot be identified, the +correct answer is the `--no-heal` envelope. + +**2. `verify` false-negatives with no diagnostic.** + +``` +prima verify "the Save button in the skill editor is now enabled because the skill + content has unsaved changes" +→ passed: false + evidence: no assertion held on the current page + code: (empty) +``` + +The app was working — `● unsaved` was displayed and Save was enabled. + +The cause is the vocabulary, not the model. `rules/navigator/verification-actions.md` offers +exactly nine assertions — `see`, `seeElement`, `seeInField`, `seeInTitle`, `seeInSource` and +their `dontSee*` counterparts — none of which express enabled/disabled/checked/selected, and +the rules additionally steer away from attribute selectors (`NEVER use ':has-text(...)'`, +"never check value via CSS attribute selectors"). So a claim about interactive state is +unprovable by construction, and prima reports that as the *feature* failing rather than as a +check it could not express. Empty `code:` and a generic evidence line leave no way to tell +"the app is broken" from "I could not phrase this". + +**3. Auto-discovery of a playwright-cli session cannot work; `--endpoint` needs playwright 1.62.** + +> **Corrected 2026-08-07 after re-testing.** The original run was on explorbot's pinned +> playwright **1.60.0** against a 1.61.0-alpha daemon, and concluded attach was broken +> outright. That was a stale-dependency artifact. Re-tested with `playwright@1.62.1` and +> `@playwright/cli@0.1.17` (playwright 1.62.0-alpha): `chromium.connect()` succeeds with +> **our own** playwright, and `prima verify --endpoint <sock>` returns `ok: true` in 16.6s. +> Upgrading the pin fixes the connect half. What remains is discovery. + +- **Version skew (fixed by upgrading).** `playwright@^1.60` could not `connect()` to a + 1.61/1.62 browser server — it timed out. `playwright@1.62.1` connects to both its own + minor and the 1.62.0-alpha daemon. The pin should move to `^1.62`. +- **Missing `workspaceDir` (still broken).** No `@playwright/cli` release writes a + `workspaceDir` field into `~/.cache/ms-playwright/b/browser@<guid>` — verified on 0.1.13 + and 0.1.17. `parseDescriptor` (`boat/prima/src/pw-registry.ts:56`) requires it, so every + descriptor is dropped, `selectDescriptor` filters on a field that never exists, and + auto-discovery finds nothing no matter what version is installed. Prima cannot attach + without `--endpoint` unless it stops keying on `workspaceDir`. +- **The failure message sends you in a circle.** With a session open, discovery fails with + "No browser to drive... Open one first: `playwright-cli open <url>`" — advising exactly + what the user already did. `prima browser list` likewise reports "no browser instances + running" while a session is live. + +**4. A redirect that appends query params is treated as failed navigation.** The app sends +`/` → `/?session=<uuid>&ws=1`. `prima go http://localhost:3050` burned **3 minutes** and eight +attempts before healing; against prima's own browser the same navigation ended in a hard tool +error after **2m32s** — + +``` +error: tool: Navigation to / failed: redirected to /?session=...&ws=1 and could not resolve +``` + +— in an envelope whose own inlined ARIA proves the page had loaded correctly. Session-param +redirects are common enough that this alone blocks unattended use. + +### Smaller issues + +- **`used:` is often not runnable.** `click` concatenated all five ladder attempts, including + invalid JS: `I.click(".sidebar button:has-text("Workflows")")`. `do` emitted a + `// 1. Open dialog...` comment line inside the code. The spec promises "the exact code that + worked" — it should be the winning line only. +- **`### Changes` / ariaDiff never appeared** in any successful envelope. That block is the + envelope's core evidence promise; without it, a successful `click` proves nothing and I had + to spend a playwright-cli snapshot to confirm every action. +- **`network.jsonl` is advertised in every envelope and was 0 bytes in all 18 runs.** +- **`click` reports itself as `do`.** Every `click` envelope printed `command: do "..."`. +- **Every command requires a URL** even when attached to a browser already sitting on the page, + and even when the site is registered. `EXPLORBOT_URL` satisfies config loading but not page + opening on an empty owned browser, which needs `--url` as well. +- **Prima pollutes a shared browser.** After `research`, the visual-annotation overlays + (`Legend`, `e8`, `e10`, …) were still in the live DOM and showed up in the next + playwright-cli snapshot. + +### Environment friction (not prima's design) + +- **`node_modules` is a committed, self-referential symlink.** `git ls-files -s node_modules` + shows mode `120000` pointing at `/home/davert/projects/explorbot/node_modules` — itself. + It was added in `7cc52eb` ("Let EXPLORBOT_* variables win over the global config"). Being + tracked, it overrides the `node_modules/` line in `.gitignore`. Every Node resolution fails + with `ELOOP` / "Too many levels of symbolic links", so `npx tsc`, the build, and the CLI are + all dead on a fresh checkout of this branch. Removing it and running `bun install` fixes it. +- `bun run build:npm` fails with `env: unknown error: execvp failed`; `bash scripts/build-npm.sh` + works. +- The `prima` bin is not exposed by the existing global npm link — `explorbot prima ...` only. +- Prima needs the Node build for browser-server endpoints, so `dist/` must exist before any of + this runs. + +--- + +## What would make prima a replacement + +In priority order: + +1. Heal must never substitute a different target; unresolvable intent → failure envelope. +2. `verify` must distinguish "assertion failed" from "cannot express this assertion", and + `rules/navigator/verification-actions.md` needs state assertions + (enabled/disabled/checked/selected) alongside the nine text/presence ones it has now. +3. Treat a redirect that preserves origin and path as navigation success. +4. Move the playwright pin to `^1.62` (fixes `--endpoint`), and stop keying discovery on + `workspaceDir` — no `@playwright/cli` release emits it. +5. Emit `### Changes` on every action, and reduce `used:` to the winning line. + +With 1–3 fixed, prima could own the assertion and inspection half of a session outright. +Driving would still belong to playwright-cli until the per-command latency comes down. diff --git a/docs/superpowers/specs/2026-08-07-prima-fixes-design.md b/docs/superpowers/specs/2026-08-07-prima-fixes-design.md new file mode 100644 index 00000000..992d4021 --- /dev/null +++ b/docs/superpowers/specs/2026-08-07-prima-fixes-design.md @@ -0,0 +1,394 @@ +# Prima Fixes — Perception Ladder, No Heal, Proof-Carrying Envelopes + +**Date:** 2026-08-07 +**Status:** Draft for review +**Supersedes parts of:** `2026-08-01-prima-boat-design.md` +**Evidence:** `docs/superpowers/reviews/2026-08-06-prima-vs-playwright-cli.md` + +## Problem + +A field run of prima against a live app (Testeiya, `localhost:3050`) found the boat working but +not trustworthy as an executor. Four defects matter: + +- A `pw` call on a selector that does not exist returned `ok: true`, `healed: true`, having + clicked an unrelated control ("New agent"). Heal substituted the target and reported success. +- `### Changes` never appeared in any successful envelope, so a successful action proved + nothing and every step had to be confirmed with a playwright-cli snapshot. +- `do` given two instructions performed four, typing into a live chat box actions nobody asked + for. +- Auto-discovery of a playwright-cli session never matched, while the failure message advised + opening the session the user already had open. + +Underneath the third defect is a perception problem. `action.ts:139` captures +`page.locator('body').ariaSnapshot()` **without** `mode: 'ai'`, so `do` sees roles and text but +no element handles and must invent locators. That is why the `click` tool's schema demands +"multiple commands targeting the SAME element" and why one click produced five ladder attempts, +one of them invalid JavaScript. + +## Goals + +- Prima is the executor for an expensive orchestrator: page data never enters that context on + the happy path, intent is never guessed, and every action carries proof of what changed. +- `do` acts on element handles it was given, not on locators it invented. +- Failures are failures. No code path may reach a different element than the one asked for. +- Attachment to an existing playwright-cli session works without hand-editing files. + +## Non-Goals + +- Replacing playwright-cli. It remains the fallback for direct driving and the tool prima + points back to when it cannot help. +- Reducing per-command latency, and improving `research` locator quality. Separate concerns. +- Changing Tester, Navigator, or Researcher behaviour outside the shared pieces named below. + +--- + +## 1. Perception ladder for `do` + +Four tiers, tried in order. Each tier answers "what can I act on here?" at a different cost. + +| Tier | Source | When | +|---|---|---| +| 1 | Research UI map | a stored map exists for this state hash and the state is well-visited | +| 2 | **ARIA snapshot with refs** | default | +| 3 | Compact HTML tree | a ref action failed, or the control is absent from the ARIA tree | +| 4 | Vision (`visualClick`) | click only, when the target is identifiable solely by appearance | + +### Tier 1 — research map + +Read through `getPreviousResearch(hash)` — a TTL-free disk read. Not `getCachedResearch`, whose +six-hour TTL carries session-scoped semantics that must not be stretched for cross-invocation +reuse. The trigger is the StateManager visit count for the current state hash, the same number +the envelope already prints as `visit #N`: use the map from the **third visit onward**, so a +state has to prove it is worth the map before prima prefers one. Below that, skip to tier 2. +The threshold is a single config field under `ai.agents.prima`, per the one-knob convention. + +Research cost is real — 41s on the vision model in the field run — so tier 1 only pays off +amortized across repeat visits to one state. Prima does not run research on the caller's behalf +inside `do`; it consumes a map that already exists. + +### Tier 2 — ARIA snapshot with refs (the default) + +`page.locator('body').ariaSnapshot({ mode: 'ai' })` emits `[ref=eN]` handles. Playwright resolves +them natively through the `aria-ref=` selector engine, verified live against the running app: + +``` +aria-ref=e1 → 1 match +``` + +No DOM mutation, no attribute stamping. The model is given the ref-bearing tree and names a ref; +resolution to a CodeceptJS command is §1.1. One ref becomes one command and the +multiple-locator fallback ladder collapses. + +**Ref lifetime is one context injection.** Refs are snapshot-scoped and shift between calls — +in the field run the same sidebar button was `ref=e13` in one snapshot and `ref=e284` in a later +one on the same page. Prima re-snapshots whenever the `do` loop re-injects context on state-hash +change (the hook exists at `prima.ts:104`), and the prompt states that refs from an earlier +injection are dead. + +**Ref shape.** Refs are frame-qualified: every ref this app emits is `f1e13`-shaped, not `e13`. +A validator that accepts only `e\d+` rejects every real ref and silently disables the whole tier, +so the accepted shape is `(f\d+)?e\d+`. Refs are never adapted or invented — a ref that does not +parse, or that resolves to nothing, is a failure with its own message. + +### Tier 3 — compact HTML tree + +The current `simplifiedHtml` path. Entered when a ref action fails or the target is not +represented in the accessibility tree. + +### Descending the ladder — `context()` + +The tiers above are only reachable if something can move between them mid-run. The `do` loop +re-injects context between iterations when the state hash changes, which leaves no way to +recover from a ref that died *within* an iteration — the model's only remaining moves would be +to guess a locator, which the prompt forbids, or to stop. + +`do` therefore carries a `context` tool. First call returns the page as it is now with fresh +refs, which replace every ref the model was holding; a later call on the same page drops to +capped markup for elements the accessibility tree does not describe. It is the tier descent, not +a page dump. + +This is deliberately **not** the `context` tool from `createAgentTools`. That one returns +`getInteractiveARIA()` — `compactAriaSnapshot` over `ActionResult.ariaSnapshot`, captured without +`mode: 'ai'`, so it carries **no refs** — plus a 6k-char HTML dump beside it. Offering it here +would hand the model a ref-less tree and push it straight back to guessed locators, undoing +tier 2. + +### Tier 4 — vision + +`visualClick` (`src/ai/tools.ts:791`), already implemented. Click only — a coordinate is not a +handle, so it cannot serve fills, selects, or assertions. + +### 1.1 Acting on a ref — `clickRef` / `hoverRef` in `boat/prima/src/tools.ts` + +CodeceptJS has no `aria-ref` locator, so a ref cannot be handed to `I.click()` directly. It is +resolved to a locator CodeceptJS does understand, using machinery that already exists: + +``` +page.locator(`aria-ref=${ref}`) + → WebElement.fromPlaywrightLocator(...) (web-element.ts:112) + → I.click(<xpath>) +``` + +**Which XPath, and the one that is not available.** `WebElement` declares two — `clickXPath`, +built attribute-first by `buildClickableXPath` (`utils/xpath.ts`), and `xpath`, the absolute +positional path. On this path only `clickXPath` exists: `fromPlaywrightLocator` builds through +`fromRawData`, which hardcodes `xpath: ''` (`web-element.ts:89`). Only `fromXPathMatch`, the +static-HTML path, populates the absolute form. So there is no positional fallback to reach for, +and that is the right outcome anyway — an absolute path is deduplication machinery, fragile the +moment the DOM shifts, and the ref resolution and the click happen on separate round-trips. + +Verified live, ref → `clickXPath` → match count: + +``` +e15 link "Checking the proxy…" //*[self::a and contains(.,"Checking the proxy and the firewall")] → 1 +e19 button "Reload" //*[@id="reload-button"] → 1 +e20 button "Details" //*[@id="details-button"] → 1 +``` + +**Require a unique match.** Because there is no fallback, the resolved `clickXPath` is checked +to match exactly one element before it is used. Zero or many is a failure, reported as such — +never a click on an ambiguous match and never a retreat to guessed locators. + +Ref acting lives in **separate tools in the boat** — `clickRef` and `hoverRef`, from a +`createRefTools` factory in `boat/prima/src/tools.ts`. `click` and `hover` in `src/ai/tools.ts` +are left byte-identical. A prima-only tool belongs to prima; core keeps only what every caller +uses, and lends the boat its result-shaping helpers (`successToolResult`, `failedToolResult`, +`commitNote`) rather than having them copied. + +Adding an optional `ref` to the existing tools looks cheaper and is wrong. A tool's schema and +description are shared with every caller, and Tester never receives ref-bearing snapshots, so it +would be shown a parameter it can only fill by inventing one. Making `commands` optional to +accommodate the new field weakens the contract for Tester too. "Prima enables ref mode, Tester is +untouched" is not achievable through one definition: there is one definition, and Tester sees it. + +Each ref tool takes a ref and nothing else, resolves it, executes one command, and reports that +command as `used:` — real CodeceptJS a generated test can keep. A ref that does not resolve is a +failure, not a cue to fall back to a guessed locator: it means the context is stale. + +`hoverRef` exists for the same reason as `hover` — revealing hover-only controls is a +prerequisite for clicking them. + +Prima keeps the locator tools alongside the ref tools: tier 3 works from markup that carries no +refs, and needs them. Tier 4 keeps the coordinate input. + +--- + +## 2. Heal is deleted + +`heal()` (`prima.ts:428-460`) is removed, along with `--no-heal`, `PrimaOptions.heal`, the +`healed:` / `healNote` envelope fields, the `HealAttempt` type, and the +`### Healing attempts` section. The three call sites — `pw` (`:79`), `do` (`:135`), `go` (`:211`) +— go straight to `failureEnvelope`. + +The failure envelope already does the right thing and becomes the only failure path: `ok: false`, +the exact error with its call log, and compact ARIA inline. Measured at 8.6s in the field run +against 34.8s for the heal path that got the answer wrong. + +This removes the substitution defect at the root. No remaining code path can select an element +other than the one asked for, so no envelope can report success for an action the caller did not +request. Routine obstructions — an overlay covering a button, an element not yet visible — now +return the failure envelope, and the orchestrating model decides. That is the accepted cost: the +compact ARIA block is the one place page data deliberately enters the expensive context. + +--- + +## 3. Proof-carrying envelopes + +Two defects with one cause. `renderOutcome` (`envelope.ts:76`) returns the **first** of +changes / answer / research / verdict, so `### Changes` is structurally impossible alongside +`### Verdict`, `### Answer`, or `### Research`. And `pageChanges` returns `ariaChanges ?? null`, +which renders nothing when the diff is empty or `previousState` is null. + +- `### Changes` renders on every action envelope, showing `no change` explicitly when the tree is + identical. A caller can then tell "nothing happened" from "prima did not say". +- `renderOutcome` stops being mutually exclusive: `### Changes` renders alongside the command's + own outcome section. +- Prima captures an explicit before-snapshot rather than relying on whatever `stateManager` + holds at process start. +- New `### Steps` block for `do`: one line per instruction, each naming the ariaDiff entry that + proves it, or marked `unproven`. + +### Refs must never reach the diff or hash pipeline + +The ref-bearing snapshot is a **context artifact only**. Refs are stripped before hashing and +diffing. Storing the `mode: 'ai'` output in `ActionResult.ariaSnapshot` would poison every +`### Changes` block, because Playwright renumbers refs on each call. Measured on an identical +page whose refs merely shifted: + +``` +ref churn only, page identical → diff count = 6 + added: button "Cancel" [ref=e21], button "Save" [ref=e20], textbox "Name" [ref=e22] + removed: button "Cancel" [ref=e11], button "Save" [ref=e10], textbox "Name" [ref=e12] +``` + +Six phantom entries for a page that did not change. Capture ref-bearing and ref-free variants, +and feed only the ref-free one to `diffAriaSnapshots` and `getStateHash`. + +--- + +## 4. Executor prompt + +`instructionSystemPrompt` (`prima.ts:493`) already says to stop when every instruction is done, +but nothing tracks per-instruction completion, so the loop runs until the model stops calling +tools. Two rules are added, stated as general principles rather than as counter-examples from +any debugging session: + +- Act only on the listed instructions. An adjacent action that appears helpful is out of scope; + report it as an observation instead of performing it. +- For each instruction, cite the observed page change that proves it. An instruction that cannot + be tied to an observed change is reported unproven rather than claimed as done. + +The prompt also states the ref contract: act on refs from the current context injection; refs +from an earlier injection are dead. + +--- + +## 5. Attachment and discovery + +Three separate faults, all confirmed empirically. + +**Connect with the daemon's own build when attached.** `connectDescriptor` (`prima.ts:374`) +tries prima's own playwright first and only falls back to `descriptor.playwrightLib` if connect +fails. Connect *succeeds* across builds, so the fallback is never reached — and then the tier-2 +snapshot breaks: + +| client lib | `connect()` | `ariaSnapshot({mode:'ai'})` | +|---|---|---| +| own playwright 1.62.1 | ok | `locator.ariaSnapshot: timeout: expected float, got undefined` | +| daemon playwright-core 1.62.0-alpha | ok | ok, 3376 bytes, `aria-ref=e1` → 1 match | + +In attached mode, prefer `descriptor.playwrightLib` when present and fall back to our own. Carry +`playwrightLib` through the `--endpoint` path too, which currently hardcodes `''`. + +**Stop keying discovery on `workspaceDir`.** No `@playwright/cli` release writes that field — +verified on 0.1.13 and 0.1.17. `parseDescriptor` (`pw-registry.ts:56`) requires it and +`selectDescriptor` filters on it, so every descriptor is dropped and discovery finds nothing no +matter what is installed. Resolution becomes: `--endpoint` → `--pw-session <title>` → +`PLAYWRIGHT_CLI_SESSION` → live descriptor titled `default` → the single live descriptor → +tool error listing candidate titles. Liveness-probe before selecting. + +**Bump the playwright pin to `^1.62`.** `playwright@^1.60` could not `connect()` to a 1.61/1.62 +browser server at all; 1.62.1 connects. The pin is the reason the original review concluded +attach was broken outright. + +Consequences: `browser list` shows attachable sessions rather than reporting none while one is +live, and the no-browser error stops advising `playwright-cli open <url>` to someone who already +ran it. + +--- + +## 6. `verify` honesty + +`rules/navigator/verification-actions.md` offers nine assertions — `see`, `seeElement`, +`seeInField`, `seeInTitle`, `seeInSource` and their `dontSee*` counterparts — none of which +express enabled, disabled, checked, selected, or expanded. A claim about interactive state is +therefore unprovable by construction, and the field run reported a working feature as failing: + +``` +verify "the Save button in the skill editor is now enabled ..." +→ passed: false, code: (empty), evidence: no assertion held on the current page +``` + +The app was correct — `● unsaved` was displayed and the button was enabled. + +- Add state assertions to the rule. +- `verify` distinguishes **assertion failed** from **could not express this assertion**. The + second is not a test failure and must not be reported as one. +- With §7's compaction fix, `verify` reads state from the same ref-bearing snapshot the tier-2 + ladder produces. + +--- + +## 7. Prerequisite fix — ARIA compaction drops refs and state + +`compactAriaSnapshot` keeps only the first bracket group on a line. Playwright emits state +attributes before `[ref=]`, so every stateful control loses its handle. Measured: + +``` +button "Plain" [ref=e10] → ref KEPT +button "Active" [active] [ref=e13] → ref LOST +button "Disabled" [disabled] [ref=e14] → ref LOST +button "Pressed" [pressed] [ref=e15] → ref LOST +button "Expanded" [expanded] [ref=e16] → ref LOST +checkbox "Checked" [checked] [ref=e17] → ref LOST +button "Cursor" [ref=e18] [cursor=pointer] → ref KEPT +``` + +The controls most worth acting on and asserting about are exactly the ones stripped of their +handle. **Keep every bracket group on a line.** No allow-list, no drop-list — parsing one group +and discarding the rest is the whole bug, and any rule about which groups survive re-creates it +the next time Playwright adds an attribute. This fix is a prerequisite for §1 tier 2 and enables +§6. + +--- + +## 8. Envelope hygiene + +- `used:` carries the winning line only — no concatenated ladder attempts, no comment lines. The + field run produced `I.click(".sidebar button:has-text("Workflows")")`, which is not valid + JavaScript, and a `// 1. Open dialog...` comment inside the code block. +- `click` and `fill` stop labelling themselves `do` in `command:`. +- `network.jsonl` is written and advertised only when requests were actually captured. It was + 0 bytes across all 18 runs while being advertised in every envelope; an artifact line that + points at an empty file is worse than no line. +- Research annotation overlays are removed from the DOM after use. The browser is shared with + playwright-cli, and leftover `Legend` / `e8` / `e10` nodes appeared in its next snapshot. +- Commands stop requiring a URL when attached to a browser already on a page. +- A redirect that preserves origin and path counts as navigation success. The app's + `/` → `/?session=<uuid>&ws=1` redirect cost 3m00s and eight attempts under the old heal path, + and a hard tool failure against a prima-owned browser — in an envelope whose own inlined ARIA + proved the page had loaded. + +--- + +## Testing + +- **Unit** — envelope rendering with `### Changes` always present, including the `no change` + form and the combination with `### Verdict` / `### Answer` / `### Research`; compaction + preserving every bracket group across the §7 combinations and any order of them; ref-free + diffing (the six-phantom case must yield zero); descriptor selection without `workspaceDir`, + including the ambiguous multi-session error. +- **Ref resolution** — a ref resolves to an attribute-based `clickXPath` matching exactly one + element, and still matching after unrelated siblings are added or removed; a ref that resolves + to zero or many fails rather than falling back to a guessed locator. +- **Integration** — `do` prompt behaviour through the existing `@copilotkit/aimock` harness per + `docs/contributing/ai-integration-tests.md`: instructions performed and nothing beyond them, + per-instruction proof citation, unproven reporting. Fictional fixture data only. +- **End-to-end** — against a local fixture: a ref named from the snapshot clicks the element it + names and no other; a failing `pw` returns `ok: false` with compact ARIA and never a + substituted action; attach to a live playwright-cli session with no descriptor editing; a + session-param redirect resolves as success. + +--- + +## Decisions Log + +- Heal is deleted outright rather than constrained to same-target recovery. No flag, no opt-in. +- `do` perception is a four-tier ladder: research map, ARIA with refs, compact HTML, vision for + click. +- Refs come from Playwright's native `aria-ref=` engine. No eidx attribute stamping, so the + shared browser's DOM is not mutated. +- Ref acting lives in `boat/prima/src/tools.ts`, not in core; `click` and `hover` are untouched. + Extending a shared tool would show Tester a `ref` parameter it can only fill by inventing one, + and would weaken `commands` for every caller. +- A ref resolves through `WebElement` to an attribute-based `clickXPath`. The absolute positional + `xpath` is not a fallback — `fromRawData` never populates it on the live-locator path, and it + would be the fragile choice regardless. +- A resolved `clickXPath` must match exactly one element. A dead or ambiguous ref is a failure, + never a fallback to guessed locators. +- ARIA compaction keeps every bracket group. No allow-list of attributes to preserve. +- Refs are a context artifact only, stripped before hashing and diffing. +- Ref lifetime is one context injection; the loop re-snapshots on state-hash change. +- Tier 1 reads maps via `getPreviousResearch`; the trigger is StateManager visit count. +- Discovery matches on title plus liveness; `workspaceDir` is abandoned as a key. +- Attached mode connects with `descriptor.playwrightLib` first, because connect succeeds across + builds but `ariaSnapshot` does not. +- The playwright pin moves to `^1.62`. +- `verify` reports inexpressible assertions as inexpressible, never as failures, and does not + record them as verifications — a claim that could not be checked must not be remembered as + one that failed. +- `do` carries a prima-shaped `context` tool that descends the ladder — fresh refs first, capped + markup on a repeat call. The shared `context` from `createAgentTools` is not reused: it returns + a ref-less tree and an HTML dump. +- The baseline snapshot is captured lazily when a command needs a diff, not in `start()`, which + runs before a page is loaded. diff --git a/package.json b/package.json index 0affdfea..b18b25bc 100644 --- a/package.json +++ b/package.json @@ -113,7 +113,7 @@ "ora-classic": "^5.4.2", "parse5": "^8.0.0", "pixelmatch": "^7.2.0", - "playwright": "^1.60", + "playwright": "^1.62", "pngjs": "^7.0.0", "react": "^19.1.1", "sambanova-ai-provider": "^1.2.2", diff --git a/rules/navigator/verification-actions.md b/rules/navigator/verification-actions.md index e11d7445..b786e490 100644 --- a/rules/navigator/verification-actions.md +++ b/rules/navigator/verification-actions.md @@ -105,9 +105,26 @@ Checks that page source does NOT contain expected text. I.dontSeeInSource('error-class'); </example> +### I.seeAttributesOnElements + +I.seeAttributesOnElements(<locator>, <attributes>) + +Checks the interactive state of a control: enabled, disabled, checked, selected, expanded, required, read-only. +This is the ONLY way to assert state — presence assertions cannot express it. +Assert the state you expect; to assert the opposite state, assert that value explicitly rather than negating. + +<example> + I.seeAttributesOnElements('#submit', { disabled: null }); + I.seeAttributesOnElements({"role":"button","text":"Save"}, { 'aria-disabled': 'false' }); + I.seeAttributesOnElements({"role":"checkbox","text":"Accept"}, { checked: true }); + I.seeAttributesOnElements({"role":"button","text":"Details"}, { 'aria-expanded': 'true' }); +</example> + <verification_rules> Be strict in assertions to avoid false positives. Prefer I.seeElement() with ARIA locators - most reliable. +For a claim about a control's state, use I.seeAttributesOnElements() — presence of the element is not evidence of its state. +If no assertion above can express the claim, say so instead of proposing an assertion that checks something weaker. I.see() and I.dontSee() MUST include context parameter. For input field values, ALWAYS use I.seeInField() — never check value via CSS attribute selectors or I.seeInSource. Prefer text locators (label, name, placeholder) for form fields: I.seeInField('Search', 'value') over I.seeInField('input[name="search"]', 'value'). diff --git a/src/ai/navigator.ts b/src/ai/navigator.ts index a596f585..0028cc2d 100644 --- a/src/ai/navigator.ts +++ b/src/ai/navigator.ts @@ -620,7 +620,7 @@ class Navigator implements Agent { return suggestion; } - async verifyState(message: string, actionResult: ActionResult): Promise<{ verified: boolean; successfulCodes: string[]; assertionSteps: Array<{ name: string; args: any[] }>; totalAttempted: number }> { + async verifyState(message: string, actionResult: ActionResult): Promise<{ verified: boolean; inexpressible: boolean; successfulCodes: string[]; assertionSteps: Array<{ name: string; args: any[] }>; totalAttempted: number }> { tag('info').log('AI Navigator verifying state at', actionResult.url); debugLog('Verification message:', message); @@ -773,10 +773,16 @@ class Navigator implements Agent { let verified = successfulCodes.length >= majorityNeeded; if (alreadyVerified) verified = true; + const inexpressible = !alreadyVerified && totalAttempted === 0; + if (inexpressible) { + tag('warning').log('No assertion could express this claim'); + return { verified: false, inexpressible, successfulCodes, assertionSteps, totalAttempted }; + } + actionResult.addVerification(message, verified); this.stateManager.updateState(actionResult); - return { verified, successfulCodes, assertionSteps, totalAttempted }; + return { verified, inexpressible, successfulCodes, assertionSteps, totalAttempted }; } private checkAlreadyVerified(aiResponse: string, actionResult: ActionResult): boolean { diff --git a/src/ai/tools.ts b/src/ai/tools.ts index 526cd2d0..75d19556 100644 --- a/src/ai/tools.ts +++ b/src/ai/tools.ts @@ -676,6 +676,13 @@ export function createAgentTools({ explorer, stateManager, ai, researcher, navig ); } + if (result.inexpressible) { + return failedToolResult('verify', `No assertion could express this claim: ${assertion}`, { + inexpressible: true, + suggestion: 'This is not evidence the page is wrong — the claim could not be turned into an assertion. Restate it in terms of what is visible or of a control state, or check it with see().', + }); + } + return failedToolResult('verify', `Verification failed: ${assertion}`, { suggestion: 'The assertion could not be verified. Check if the condition is actually present on the page or try a different assertion.', }); @@ -1112,14 +1119,14 @@ function errorText(error: unknown): string { return 'Unknown error occurred'; } -async function commitNote(activeNote: any, result: TestResult, toolResult: any, action: any): Promise<void> { +export async function commitNote(activeNote: any, result: TestResult, toolResult: any, action: any): Promise<void> { if (toolResult?.pageDiff?.ariaChanges || toolResult?.pageDiff?.urlChanged) { activeNote.screenshot = await action.saveScreenshot(); } activeNote.commit(result); } -function successToolResult(action: string, data?: Record<string, any>, source?: { playwrightGroupId?: string | null; assertionSteps?: any[] }) { +export function successToolResult(action: string, data?: Record<string, any>, source?: { playwrightGroupId?: string | null; assertionSteps?: any[] }) { const result: Record<string, any> = { success: true, action, ...data }; if (source?.playwrightGroupId) { result.playwrightGroupId = source.playwrightGroupId; @@ -1155,7 +1162,7 @@ function hasObservablePageChange(data?: Record<string, any>): boolean { return Array.isArray(data.pageDiff.htmlParts) && data.pageDiff.htmlParts.length > 0; } -async function failedToolResult(action: string, message: string, data?: Record<string, any>, error?: Error | null) { +export async function failedToolResult(action: string, message: string, data?: Record<string, any>, error?: Error | null) { const result: Record<string, any> = { success: false, action, message, ...data }; if (data?.pageDiff) { result.suggestion = data.suggestion ? `${data.suggestion} ${PAGE_DIFF_SUGGESTION}` : PAGE_DIFF_SUGGESTION; diff --git a/src/utils/aria.ts b/src/utils/aria.ts index 6d984999..36a11f68 100644 --- a/src/utils/aria.ts +++ b/src/utils/aria.ts @@ -85,8 +85,7 @@ const parseLabel = (label: string): { role: string; name?: string; attributes: R } const attributes: Record<string, string | boolean | null> = {}; - const attrMatch = rest.match(/\[([^\]]*)\]/); - if (attrMatch) { + for (const attrMatch of rest.matchAll(/\[([^\]]*)\]/g)) { for (const tok of attrMatch[1].split(/[\s,]+/).filter(Boolean)) { const eq = tok.indexOf('='); if (eq === -1) { diff --git a/src/utils/url-matcher.ts b/src/utils/url-matcher.ts index ca33146e..1ee6db67 100644 --- a/src/utils/url-matcher.ts +++ b/src/utils/url-matcher.ts @@ -100,6 +100,9 @@ export function matchesNavigationUrl(expected: string, current: string): boolean if (!expectedPath.includes('#')) { currentPath = currentPath.split('#')[0]; } + if (!expectedPath.includes('?')) { + currentPath = currentPath.split('?')[0]; + } const normalize = (value: string) => value.replace(/^\/+|\/+$/g, '').toLowerCase(); return normalize(expectedPath) === normalize(currentPath); } diff --git a/src/utils/web-element.ts b/src/utils/web-element.ts index fc2ce338..32db94e8 100644 --- a/src/utils/web-element.ts +++ b/src/utils/web-element.ts @@ -125,6 +125,15 @@ export class WebElement { return WebElement.fromPlaywrightLocator(page.locator(`[${EXPLORBOT_ATTRS.eidx}="${eidx}"]`)); } + static isAriaRef(ref: string): boolean { + return /^(f\d+)?e\d+$/i.test(ref); + } + + static async fromAriaRef(page: any, ref: string): Promise<WebElement | null> { + if (!WebElement.isAriaRef(ref)) return null; + return WebElement.fromPlaywrightLocator(page.locator(`aria-ref=${ref}`)); + } + static async fromEidxList(page: any, eidxList: string[]): Promise<WebElement[]> { const validEidxList = eidxList.filter((eidx) => /^e\d+$/i.test(eidx)); if (validEidxList.length === 0) return []; diff --git a/tests/integration/prima-heal.test.ts b/tests/integration/prima-heal.test.ts deleted file mode 100644 index 02e29801..00000000 --- a/tests/integration/prima-heal.test.ts +++ /dev/null @@ -1,160 +0,0 @@ -import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'bun:test'; -import { mkdtempSync, rmSync } from 'node:fs'; -import { tmpdir } from 'node:os'; -import path from 'node:path'; -import { createOpenAI } from '@ai-sdk/openai'; -import { LLMock } from '@copilotkit/aimock'; -import { Prima } from '../../boat/prima/src/prima.ts'; -import { ActionResult } from '../../src/action-result.ts'; -import { Navigator } from '../../src/ai/navigator.ts'; -import { Provider } from '../../src/ai/provider.ts'; -import { ConfigParser } from '../../src/config.ts'; - -const checkoutState = { - url: '/checkout', - title: 'Checkout - Widget Depot', - hash: 'checkout_h1_checkout', - html: '<html><body><h1>Checkout</h1><button id="place-order">Place order</button></body></html>', - ariaSnapshot: '- heading "Checkout" [level=1]\n- button "Place order"', -}; - -const recovery = ['Both routes below reach the same outcome.', '', "```js\nI.click('Confirm purchase')\n```", '', "```js\nI.click('#place-order')\n```"].join('\n'); - -function extractPromptText(entry: any): string { - if (!entry?.body?.messages) return ''; - return entry.body.messages - .map((message: any) => { - if (typeof message.content === 'string') return message.content; - if (Array.isArray(message.content)) { - return message.content - .filter((part: any) => part.type === 'text') - .map((part: any) => part.text || '') - .join('\n'); - } - return ''; - }) - .join('\n'); -} - -describe('Prima heal with aimock', () => { - let mock: LLMock; - let provider: Provider; - let prima: Prima; - let attempted: string[]; - let artifacts: string; - - function buildNavigator(succeedsWith: string): Navigator { - const action: any = { - lastError: null, - actionResult: null, - exitIframe: async () => {}, - attempt: async (code: string) => { - attempted.push(code); - if (code.includes(succeedsWith)) { - action.lastError = null; - return true; - } - action.lastError = new Error('element not visible'); - return false; - }, - getActor: () => ({ wait: async () => {} }), - stateManager: { getCurrentState: () => checkoutState }, - }; - - const experienceTracker = { getSuccessfulExperience: () => [], writeFlow: () => {} }; - return new Navigator({ - explorer: { action: () => action, capture: async () => ActionResult.fromState(checkoutState as any) } as any, - ai: provider, - config: ConfigParser.getInstance().getConfig(), - stateManager: { getCurrentState: () => checkoutState, getVisitCount: () => 1, getExperienceTracker: () => experienceTracker } as any, - knowledgeTracker: { renderRelevantKnowledge: () => '', renderRelevantContext: () => '', getRelevantKnowledge: () => [] } as any, - requestStore: {} as any, - playwrightRecorder: {} as any, - }); - } - - beforeAll(async () => { - mock = new LLMock({ port: 0, logLevel: 'silent' }); - await mock.start(); - - const openai = createOpenAI({ - baseURL: `${mock.url}/v1`, - apiKey: 'test-key', - compatibility: 'compatible', - }); - - ConfigParser.resetForTesting(); - ConfigParser.setupTestConfig(); - provider = new Provider({ model: openai.chat('test-model'), config: {} }); - artifacts = mkdtempSync(path.join(tmpdir(), 'prima-heal-')); - }); - - beforeEach(() => { - mock.clearRequests(); - mock.resetMatchCounts(); - mock.clearFixtures(); - - attempted = []; - prima = new Prima({ instance: 'default' }); - (prima as any).artifactsDir = artifacts; - (prima as any).bot = { - getExplorer: () => ({ - action: () => ({ - execute: async () => { - throw new Error("locator 'text=Confirm' not found"); - }, - }), - capture: async () => ActionResult.fromState(checkoutState as any), - }), - stateManager: () => ({ getCurrentState: () => checkoutState, getVisitCount: () => 1 }), - getCurrentState: () => checkoutState, - requestStore: () => ({ getRequests: () => [] }), - getProvider: () => provider, - }; - }); - - afterAll(async () => { - await mock.stop(); - rmSync(artifacts, { recursive: true, force: true }); - ConfigParser.cleanupAllTestDirectories(); - }); - - it('recovers a failed pw along another route and reports the code that worked', async () => { - mock.on({}, { content: recovery }); - (prima as any).bot.agentNavigator = () => buildNavigator('#place-order'); - - const envelope = await prima.pw("({ page }) => page.click('text=Confirm')"); - - expect(envelope.ok).toBe(true); - expect(envelope.healed).toBe(true); - expect(envelope.healNote).toContain('2 attempts'); - expect(envelope.used).toEqual(["I.click('#place-order')"]); - expect(attempted).toEqual(["I.click('Confirm purchase')", "I.click('#place-order')"]); - }); - - it('asks the model to reach the same outcome another way', async () => { - mock.on({}, { content: recovery }); - (prima as any).bot.agentNavigator = () => buildNavigator('#place-order'); - - await prima.pw("({ page }) => page.click('text=Confirm')"); - - const prompt = extractPromptText(mock.getLastRequest()); - expect(prompt).toContain("({ page }) => page.click('text=Confirm')"); - expect(prompt).toContain("locator 'text=Confirm' not found"); - expect(prompt).toContain('Reach the same outcome on the current page in a different way.'); - expect(prompt).toContain('button "Place order"'); - }); - - it('reports every attempt and its outcome when recovery does not work', async () => { - mock.on({}, { content: recovery }); - (prima as any).bot.agentNavigator = () => buildNavigator('nothing-matches-this'); - - const envelope = await prima.pw("({ page }) => page.click('text=Confirm')"); - - expect(envelope.ok).toBe(false); - expect(envelope.failure?.error).toContain("locator 'text=Confirm' not found"); - expect(envelope.failure?.attempts.map((attempt) => attempt.code)).toContain("I.click('#place-order')"); - expect(envelope.failure?.attempts.every((attempt) => attempt.outcome.includes('element not visible'))).toBe(true); - expect(envelope.failure?.compactAria).toContain('button "Place order"'); - }); -}); diff --git a/tests/integration/prima-smoke.test.ts b/tests/integration/prima-smoke.test.ts index cc35ba8b..c0cf911e 100644 --- a/tests/integration/prima-smoke.test.ts +++ b/tests/integration/prima-smoke.test.ts @@ -127,12 +127,11 @@ describe('Prima drives a real page', () => { expect(envelope.page.url).toContain('note=the+hinge+arrived+bent'); }); - test('pw writes the aria, html and network artifacts to disk', async () => { + test('pw writes the aria and html artifacts to disk', async () => { const envelope = await prima.pw("({ page }) => page.click('text=Submit')"); expect(existsSync(envelope.artifacts!.aria)).toBe(true); expect(existsSync(envelope.artifacts!.html)).toBe(true); - expect(existsSync(envelope.artifacts!.network)).toBe(true); expect(await Bun.file(envelope.artifacts!.aria).text()).toContain('Thanks for the note'); expect(await Bun.file(envelope.artifacts!.html).text()).toContain('Thanks for the note'); }); @@ -149,12 +148,11 @@ describe('Prima drives a real page', () => { expect(await page.title()).toBe('Widget Depot Feedback'); }); - test('a failing pw without a usable AI model reports healing as unavailable', async () => { + test('a failing pw without a usable AI model fails with the page inlined', async () => { const envelope = await prima.pw("({ page }) => page.click('text=Ship the order')"); expect(envelope.ok).toBe(false); - expect(envelope.healed).toBe(false); - expect(envelope.healNote).toContain('ai unavailable'); + expect(envelope).not.toHaveProperty('healed'); expect(envelope.failure?.compactAria).toContain('button "Submit"'); }); }); diff --git a/tests/unit/aria.test.ts b/tests/unit/aria.test.ts index 9b14a675..4a0fda6b 100644 --- a/tests/unit/aria.test.ts +++ b/tests/unit/aria.test.ts @@ -148,4 +148,25 @@ describe('aria', () => { expect(result).toContain('link "Keep 9"'); expect(result).not.toContain('omitted'); }); + + it.each([ + ['- button "Plain" [ref=e10]', 'ref=e10'], + ['- button "Active" [active] [ref=e13]', 'active ref=e13'], + ['- button "Disabled" [disabled] [ref=e14]', 'disabled ref=e14'], + ['- button "Pressed" [pressed] [ref=e15]', 'pressed ref=e15'], + ['- button "Expanded" [expanded] [ref=e16]', 'expanded ref=e16'], + ['- checkbox "Checked" [checked] [ref=e17]', 'checked ref=e17'], + ['- button "Cursor" [ref=e18] [cursor=pointer]', 'cursor=pointer ref=e18'], + ])('keeps every bracket group of %p', (snapshot, expected) => { + expect(compactAriaSnapshot(snapshot, true)).toContain(`[${expected}]`); + }); + + it('keeps refs on stateful nodes nested in a tree', () => { + const snapshot = ['- navigation "Panel sections" [ref=e6]:', ' - button "Workspace" [ref=e10]', ' - button "Workflows" [active] [ref=e13]'].join('\n'); + + const result = compactAriaSnapshot(snapshot, true); + + expect(result).toContain('ref=e10'); + expect(result).toContain('ref=e13'); + }); }); diff --git a/tests/unit/url-matcher.test.ts b/tests/unit/url-matcher.test.ts index ad477133..26aa25f4 100644 --- a/tests/unit/url-matcher.test.ts +++ b/tests/unit/url-matcher.test.ts @@ -196,6 +196,19 @@ describe('url-matcher', () => { it('compares absolute and relative URLs by path', () => { expect(matchesNavigationUrl('/users', 'https://example.test/users#details')).toBe(true); }); + + it('accepts a query the app appended when none was requested', () => { + expect(matchesNavigationUrl('/', '/?session=8f2c1e&ws=1')).toBe(true); + expect(matchesNavigationUrl('http://localhost:3050', 'http://localhost:3050/?session=8f2c1e&ws=1')).toBe(true); + }); + + it('still requires the path to match when a query was appended', () => { + expect(matchesNavigationUrl('/billing', '/login?redirect=/billing')).toBe(false); + }); + + it('requires an explicitly requested query', () => { + expect(matchesNavigationUrl('/search?q=shoes', '/search?q=hats')).toBe(false); + }); }); describe('normalizeUrl', () => { From 7fcf7bf51e09fad28d10c634ee207faeb72045f5 Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Sun, 9 Aug 2026 12:06:32 +0300 Subject: [PATCH 02/39] prima check: run a scenario end to end and report what it proved MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `prima check <scenario>` spawns a full test run — it drives the page, verifies the outcome itself and reports every step with its proof. Each --expected outcome comes back as PASSED, FAILED or not verified, so a caller sees the mapping instead of a bare ok flag. `prima do` now ends where the instructions end. A done() step closes the sequence, names the instructions it could not carry out, and fails the command for them; it used to keep acting past the last step and could report success while a check it was asked to make never held. A test that stops making progress is handed to final review rather than marked failed on the spot, and console/network errors are recorded as page problems instead of failed steps — both used to sink runs that had already done their work. Also: pw returns its value, verify lists each assertion with its own result, aria diffs report typed values, long values are offloaded to an excerpt, pages are ready when the DOM goes quiet, and prima prints the envelope and nothing else. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- CHANGELOG.md | 79 +++++++- bin/explorbot-cli.ts | 2 +- boat/prima/src/cli.ts | 68 +++++-- boat/prima/src/envelope.ts | 79 +++++--- boat/prima/src/prima.ts | 245 +++++++++++++++++++----- boat/prima/src/pw-parser.ts | 12 +- boat/prima/src/tools.ts | 84 -------- boat/prima/tests/envelope.test.ts | 94 +++++++-- boat/prima/tests/prima.test.ts | 198 ++++++++++++++----- boat/prima/tests/pw-parser.test.ts | 8 +- boat/prima/tests/tools.test.ts | 118 ------------ rules/navigator/verification-actions.md | 25 +-- src/action.ts | 4 +- src/ai/navigator.ts | 18 +- src/ai/tester.ts | 30 +-- src/explorer.ts | 4 +- src/playwright-recorder.ts | 23 +++ src/test-plan.ts | 9 + src/utils/aria.ts | 76 ++++++-- src/utils/logger.ts | 11 ++ src/utils/page-readiness.ts | 25 ++- src/utils/web-sandbox.ts | 7 +- tests/integration/prima-smoke.test.ts | 24 ++- tests/unit/aria.test.ts | 84 ++++++++ tests/unit/page-readiness.test.ts | 21 +- tests/unit/playwright-recorder.test.ts | 14 +- tests/unit/tester-focus-scope.test.ts | 17 +- 27 files changed, 938 insertions(+), 441 deletions(-) delete mode 100644 boat/prima/src/tools.ts delete mode 100644 boat/prima/tests/tools.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 6aeef7c2..a4895336 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,8 +1,73 @@ # Changelog +## 2026-08-09 + +### `prima check` runs a whole scenario and reports what it proved + +`prima check` takes a behaviour, not a click path, and runs it the way a tester would: it drives +the page, verifies the outcome itself, and reports every step with the proof for each. Use it when +you want a verdict; use `prima do` when you already know the steps. + +```bash +prima check "a workflow can be created and appears in the list" --url http://app.test +prima check "signup rejects a duplicate email" \ + --expected "an error names the email as taken" \ + --expected "no second account is created" +``` + +- **`--expected <outcome>`** — an outcome the run must reach; repeat the flag for several. Without + it the scenario text is the single expected outcome. Each one comes back under + `### Expected outcomes` as `PASSED`, `FAILED` or `not verified` — "not verified" means the run + never checked it, which is not the same as false. +- **`--url <url>`** — open that page before starting, when the browser is not already on it. + +### `prima status <hash>` reopens an earlier command + +Every envelope prints a hash on its `### Instance` line. `prima status <hash>` returns the page +detail and artifact paths recorded for that command, so envelopes stay short and the full ARIA +tree, HTML and network log are one command away instead of inline. + +```bash +prima status 66800d6d2c8c553 +``` + +### Changes + +- [Prima] `do` now ends at the last step you gave it. It used to keep acting after the sequence + was finished, and could report success while a step it was asked to check never held. A step it + could not carry out is named under `### Failure` and fails the command. +- [Prima] `do` carries the same tools a test run does, so a single call can act, look and assert + across a long sequence rather than being split into one command per step. +- [Prima] `pw` returns the value of the expression under `### Value`. A `page.title()` or + `locator.count()` used to run and have its answer thrown away. +- [Prima] `verify` lists every assertion it ran with its own `PASSED` or `FAILED`, plus the + Playwright form of the ones that held, and gives no overall verdict — read the lines and decide. +- [Prima] Research output drops CSS selectors, XPaths and coordinates, which were the bulk of the + map and are not what you act on. +- [Prima] Prima commands print the envelope and nothing else. The banner, config line, browser + startup and disconnect chatter are hidden unless `--verbose` or `--debug` is passed. +- [Prima] `click` and `fill` are removed — describe the whole sequence to `do` instead, which + attaches once and carries all of it. +- [Tester] A test that stops making progress is handed to final review instead of being marked + failed on the spot. A run that had already done its work and gone quiet was reported as a + failure; the verdict now comes from reviewing the result. +- [Tester] Console and network errors seen during a run are reported as page problems rather than + as failed steps, so they no longer sink a test that otherwise passed. +- [Tester] An expected outcome counts as settled only when it is recorded back word for word, + and the prompt now says so — outcomes phrased differently were silently left unaccounted for. +- Page changes report values typed into fields, under a `typed:` section, alongside what was added, + removed and toggled. Filling a form previously showed as no change at all. +- Long field values in page snapshots are cut to an excerpt with a pointer to the full text, which + keeps a page holding a large document readable. +- Pages are considered ready as soon as the DOM goes quiet, instead of waiting on network idle. + Applications with a live websocket never reached network idle, so every snapshot paid the full + timeout. +- Generated tests assert element visibility, hidden state and field values through real Playwright + locators instead of leaving a TODO comment. + ## 2026-08-07 -### Prima drives by element refs, and never substitutes your target +### Prima never substitutes your target A failed action now fails. Previously a `pw` call on a selector that did not exist could be "healed" into clicking a different element and still report `ok: true` — so `ok: true` did not @@ -10,12 +75,6 @@ mean your own action landed. Automatic retry along a different route is gone ent with the `--no-heal` flag that used to switch it off, and the `healed:` line and `### Healing attempts` block in the envelope. -`prima do` now works from the refs Playwright puts in the page snapshot. New `clickRef` and -`hoverRef` steps take the ref of the element they want instead of composing a locator and hoping, -so one instruction becomes one command rather than a list of fallbacks. A ref that no longer -resolves is reported as stale instead of being replaced by a guess. The existing `click` and -`hover` steps are unchanged and still available for pages without refs. - ### Configuration - **`ai.agents.prima.researchAfterVisits`** — how many visits to a page before `prima do` works @@ -32,9 +91,9 @@ resolves is reported as stale instead of being replaced by a guess. The existing and told you to open the session you already had. - [Prima] Attached sessions are driven through the browser's own Playwright build, which is what makes reading the page work across versions. -- [Prima] New `context()` step for `do`: when a ref goes stale mid-run it returns the page again - with fresh refs, and drops to raw markup if asked a second time on the same page. -- [Prima] `click` and `fill` report themselves in the envelope instead of appearing as `do`. +- [Prima] New `context()` step for `do`: when the element an instruction needs is missing from the + context it holds, it returns the page again, and drops to raw markup if asked a second time on + the same page. - [Prima] `network.jsonl` is listed only when requests were actually recorded, instead of always pointing at an empty file. - [Navigator] `verify` now separates "this claim is false" from "no assertion can express this diff --git a/bin/explorbot-cli.ts b/bin/explorbot-cli.ts index 02bb38e4..fdd06b66 100755 --- a/bin/explorbot-cli.ts +++ b/bin/explorbot-cli.ts @@ -27,7 +27,7 @@ const pkgVersion = JSON.parse(fs.readFileSync(pkgPath, 'utf-8')).version as stri program.name(cli).description('AI-powered web exploration tool').version(pkgVersion, '-V, --version'); -if (!process.env.EXPLORBOT_NO_BANNER) { +if (!process.env.EXPLORBOT_NO_BANNER && !process.argv.includes('prima')) { console.log(`⛵ ${chalk.yellow.bold(`Explorbot v${pkgVersion}`)} ${chalk.dim('Autonomous Testing Agent')}`); } diff --git a/boat/prima/src/cli.ts b/boat/prima/src/cli.ts index ee8247ae..47673ae3 100644 --- a/boat/prima/src/cli.ts +++ b/boat/prima/src/cli.ts @@ -2,7 +2,7 @@ import { Command } from 'commander'; import dedent from 'dedent'; import { keepServerRunning } from '../../../src/browser-server.ts'; import { browserErrorMessage } from '../../../src/utils/browser-errors.ts'; -import { setPreserveConsoleLogs } from '../../../src/utils/logger.ts'; +import { setQuietMode } from '../../../src/utils/logger.ts'; import { type EnvelopeData, renderEnvelope } from './envelope.ts'; import { Prima, type PrimaOptions } from './prima.ts'; @@ -14,27 +14,57 @@ const helpContract = dedent` pw <fn> Precise. A Playwright function expression built from a locator you already verified. No AI on the happy path. prima pw "({ page }) => page.click('[data-test=submit]')" - click / fill One action described in words; AI resolves it on the current page. - prima click "the primary action button in the header" - prima fill "the search box" "a search term" - do <steps...> Several described steps, run tester-style in one process. - prima do "open the account menu" "choose the settings entry" - Never pass a locator or a function expression to click/fill/do - describe the target. + check <scenario> + A whole scenario run as a test: it drives the page, verifies the outcome + itself, and reports every step it took with the proof for each. + prima check "the workflow editor opens and a workflow can be saved" + do <steps...> Several described steps, run tester-style in one process. This is the + tier that pays: one process attaches once and carries the whole + sequence, so a run of six steps costs a fraction of six commands. + Reach for it whenever the next few steps are already known. + prima do "open the account menu" "choose the settings entry" \\ + "switch the theme to dark" "check the change took effect" + Never pass a locator or a function expression to do - describe the target. Never pass a description to pw - it takes executable code only. + do stops at the last step you gave it and never carries on past it. A step it could + not carry out is named under ### Failure and makes the command fail. LOOP prima go <url|path|words> reach the page you want to work on prima research once per new page; returns verified locators prima pw "..." drive the page with those locators prima verify "..." assert the outcome (prima ask "..." to inspect instead) - Fall back to click/fill/do whenever research left you no locator to hold. + Fall back to do whenever research left you no locator to hold. + + CHECK + check runs a whole scenario the way a tester would: it plans, drives the page, and + verifies the outcome itself, then reports every step with the proof for each. + Give it an outcome, not a click path - it decides how to get there. + prima check "a workflow can be created and appears in the list" --url http://app.test + prima check "signup rejects a duplicate email" --expected "an error names the email as taken" --expected "no second account is created" + --url opens that page first, when the browser is not already on it. + --expected one outcome the run must reach; repeat it for several. Without it the + scenario text is the single expected outcome. + ### Expected outcomes echoes each one back as PASSED, FAILED or not verified - + "not verified" means the run never checked it, which is not the same as false. + ### Steps lists what ran; page problems seen on the way are reported separately + under ### Answer, because they are not step failures. + Prefer check over do when you want a verdict on a behaviour; prefer do when you + already know the steps and want them carried out. + + VERIFY + verify runs the assertions it can express and reports each one with PASSED or + FAILED, plus the playwright form of the ones that held. It does not decide whether + your claim is true - read the lines and decide. Assertions that ran are evidence; + "none ran" means the claim could not be expressed, which is not the same as false. ENVELOPE ### Result ok, command, used ### Page url, title, state hash, visit count - ### Changes what the accessibility tree gained or lost, or 'no change' + ### Changes what the accessibility tree gained, lost, toggled or had typed into it ### Steps per-instruction outcome of do, each with the change that proves it - ### Answer | ### Research | ### Verdict output of ask, research, verify + ### Expected outcomes each --expected of check, as PASSED, FAILED or not verified + ### Answer | ### Research | ### Assertions output of ask, research, verify ### Failure error, compact ARIA of the page ### Instance the browser you are on and the other instances running ### Artifacts paths to the full aria.yml, page.html and network.jsonl @@ -107,12 +137,12 @@ function addCommonOptions(cmd: Command): Command { } function primaFor(options: any): Prima { - setPreserveConsoleLogs(true); if (options.ephemeral) process.env.EXPLORBOT_EPHEMERAL = '1'; return new Prima(buildOptions(options)); } async function runPrima(options: any, command: string, run: (prima: Prima) => Promise<EnvelopeData>): Promise<void> { + setQuietMode(!options.verbose && !options.debug); const prima = primaFor(options); let envelope: EnvelopeData; @@ -153,13 +183,11 @@ export function createPrimaCommands(name = 'prima'): Command { await runPrima(options, `do ${instructions.join(' ')}`, (prima) => prima.do(instructions)); }); - addCommonOptions(cmd.command('click <target>').description('Click an element described in plain words')).action(async (target, options) => { - await runPrima(options, `click ${target}`, (prima) => prima.click(target)); - }); - - addCommonOptions(cmd.command('fill <field> <value>').description('Fill a field described in plain words')).action(async (field, value, options) => { - await runPrima(options, `fill ${field} ${value}`, (prima) => prima.fill(field, value)); - }); + addCommonOptions(cmd.command('check <scenario>').description('Run a scenario end to end as a test, with its own verification, and report the steps it took')) + .option('--expected <outcome>', 'An outcome the run must reach; repeat the flag for several', (value: string, all: string[]) => [...all, value], []) + .action(async (scenario, options) => { + await runPrima(options, `check ${scenario}`, (prima) => prima.check(scenario, options.expected)); + }); addCommonOptions(cmd.command('ask <question>').description('Answer a question about the current page').option('--no-vision', 'Answer from page structure only, without a screenshot')).action(async (question, options) => { await runPrima(options, `ask ${question}`, (prima) => prima.ask(question)); @@ -180,6 +208,10 @@ export function createPrimaCommands(name = 'prima'): Command { await runPrima(options, `go ${target}`, (prima) => prima.go(target)); }); + addCommonOptions(cmd.command('status <hash>').description('Show the artifacts and page detail recorded for an earlier command')).action(async (hash, options) => { + await runPrima(options, `status ${hash}`, (prima) => prima.status(hash)); + }); + const browser = cmd.command('browser').description('Manage the browsers prima drives'); addCommonOptions(browser.command('start').description('Start a prima-owned browser and hold it open until Ctrl+C')) diff --git a/boat/prima/src/envelope.ts b/boat/prima/src/envelope.ts index 1368be0b..61368c1f 100644 --- a/boat/prima/src/envelope.ts +++ b/boat/prima/src/envelope.ts @@ -1,6 +1,12 @@ import { mkdirSync, writeFileSync } from 'node:fs'; import path from 'node:path'; +const EXPECTATION_LABELS = { + passed: 'PASSED ', + failed: 'FAILED ', + unverified: 'not verified', +}; + export interface InstanceInfo { name: string; tabs: number; @@ -15,17 +21,20 @@ export interface EnvelopeData { used?: string[]; page: { url: string; previousUrl?: string; title: string; state: string; visits: number }; changes?: string | null; - steps?: Array<{ instruction: string; proof: string | null }>; + steps?: Array<{ label: string; ok: boolean; proof: string }>; + expectations?: Array<{ text: string; status: 'passed' | 'failed' | 'unverified' }>; + value?: string; answer?: string; research?: string; - verdict?: { passed: boolean; evidence: string; code: string }; + assertions?: Array<{ code: string; passed: boolean; proof: string[] }>; failure?: { error: string; compactAria?: string }; instance: InstanceInfo; + status?: string; artifacts?: { aria: string; html: string; network?: string }; } export function renderEnvelope(data: EnvelopeData): string { - const sections = [renderResult(data), renderPage(data), renderChanges(data), renderSteps(data), renderOutcome(data), ...renderFailure(data), renderInstance(data.instance), renderArtifacts(data)]; + const sections = [renderResult(data), renderPage(data), renderValue(data), renderChanges(data), renderSteps(data), renderExpectations(data), renderOutcome(data), ...renderFailure(data), renderInstance(data), renderArtifacts(data)]; return sections.filter((section) => section).join('\n\n'); } @@ -62,6 +71,11 @@ function renderPage(data: EnvelopeData): string { return section('Page', lines.join('\n')); } +function renderValue(data: EnvelopeData): string | null { + if (data.value === undefined) return null; + return section('Value', data.value); +} + function renderChanges(data: EnvelopeData): string | null { if (data.changes === undefined || data.changes === null) return null; return section('Changes', data.changes); @@ -69,19 +83,41 @@ function renderChanges(data: EnvelopeData): string | null { function renderSteps(data: EnvelopeData): string | null { if (!data.steps?.length) return null; - const lines = data.steps.map((step, index) => { - if (!step.proof) return `${index + 1}. ${step.instruction} — unproven`; - return `${index + 1}. ${step.instruction} — proven by ${step.proof}`; + + const lines: string[] = []; + data.steps.forEach((step, index) => { + lines.push(`${index + 1}. ${step.ok ? 'ok ' : 'FAIL'} ${step.label}`); + for (const line of (step.proof || '').split('\n').filter(Boolean)) lines.push(` ${line}`); }); return section('Steps', lines.join('\n')); } +function renderExpectations(data: EnvelopeData): string | null { + if (!data.expectations?.length) return null; + const lines = data.expectations.map((expectation, index) => `${index + 1}. ${EXPECTATION_LABELS[expectation.status]} ${expectation.text}`); + return section('Expected outcomes', lines.join('\n')); +} + function renderOutcome(data: EnvelopeData): string | null { if (data.answer) return section('Answer', data.answer); if (data.research) return section('Research', data.research); - if (!data.verdict) return null; - const lines = [`passed: ${data.verdict.passed}`, `evidence: ${data.verdict.evidence}`, `code: ${data.verdict.code}`]; - return section('Verdict', lines.join('\n')); + if (!data.assertions) return null; + + if (!data.assertions.length) return section('Assertions', 'none ran — no assertion could express this claim, so nothing was checked against the page'); + + const lines = data.assertions.map((assertion) => { + const code = assertion.code + .split('\n') + .map((line) => line.trim()) + .filter((line) => line && !line.startsWith('//')) + .join(' '); + return `${code} => ${assertion.passed ? 'PASSED' : 'FAILED'}`; + }); + + const proof = data.assertions.flatMap((assertion) => assertion.proof); + if (proof.length) lines.push('', 'playwright:', ...proof); + + return section('Assertions', lines.join('\n')); } function renderFailure(data: EnvelopeData): Array<string | null> { @@ -94,22 +130,19 @@ function renderCompactAria(compactAria?: string): string | null { return section('Current page (compact ARIA)', compactAria); } -function renderInstance(instance: InstanceInfo): string { - const others = instance.others.map((other) => `${other.name} (${tabsLabel(other.tabs)})`); - const lines = [`instance: ${instance.name} (${tabsLabel(instance.tabs)}) | other instances: ${otherInstances(others)}`, browserLine(instance)]; - return section('Instance', lines.join('\n')); -} - -function otherInstances(others: string[]): string { - if (!others.length) return 'none'; - return others.join(', '); +function renderInstance(data: EnvelopeData): string | null { + const instance = data.instance; + const parts = [`${instance.name} (${tabsLabel(instance.tabs)})`, browserLine(instance)]; + if (instance.others.length) parts.push(`other instances: ${instance.others.map((other) => `${other.name} (${tabsLabel(other.tabs)})`).join(', ')}`); + if (data.status) parts.push(`details: prima status ${data.status}`); + return section('Instance', parts.join(' | ')); } function browserLine(instance: InstanceInfo): string { - if (instance.attached) return `browser: attached (${instance.attached})`; - if (instance.startedAgo) return `browser: running, started ${instance.startedAgo} ago`; - if (instance.tabs > 0) return 'browser: running'; - return 'browser: not running'; + if (instance.attached) return `attached to ${instance.attached}`; + if (instance.startedAgo) return `running, started ${instance.startedAgo} ago`; + if (instance.tabs > 0) return 'running'; + return 'not running'; } function tabsLabel(tabs: number): string { @@ -117,7 +150,7 @@ function tabsLabel(tabs: number): string { return `${tabs} tabs`; } -function renderArtifacts(data: EnvelopeData): string | null { +export function renderArtifacts(data: EnvelopeData): string | null { if (!data.artifacts) return null; const lines = [`aria: ${data.artifacts.aria}`, `html: ${data.artifacts.html}`]; if (data.artifacts.network) lines.push(`network: ${data.artifacts.network}`); diff --git a/boat/prima/src/prima.ts b/boat/prima/src/prima.ts index acab8ca7..922f3fc1 100644 --- a/boat/prima/src/prima.ts +++ b/boat/prima/src/prima.ts @@ -1,3 +1,5 @@ +import { createHash } from 'node:crypto'; +import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'; import { createRequire } from 'node:module'; import path from 'node:path'; import { tool } from 'ai'; @@ -7,22 +9,24 @@ import * as playwright from 'playwright'; import type { Browser } from 'playwright'; import { ActionResult } from '../../../src/action-result.ts'; import { actionRule, locatorRule } from '../../../src/ai/rules.ts'; -import { createCodeceptJSTools } from '../../../src/ai/tools.ts'; +import { createAgentTools, createCodeceptJSTools } from '../../../src/ai/tools.ts'; import { getAliveEndpoint, launchServer, listInstances, stopServer } from '../../../src/browser-server.ts'; import { ConfigMissingError, ConfigParser, type ExplorbotConfig, outputPath } from '../../../src/config.ts'; import { ExplorBot } from '../../../src/explorbot.ts'; import type { WebPageState } from '../../../src/state-manager.ts'; -import { Task } from '../../../src/test-plan.ts'; +import { Task, Test, TestResult } from '../../../src/test-plan.ts'; import { getPreviousResearch } from '../../../src/ai/researcher/cache.ts'; import { compactAriaSnapshot } from '../../../src/utils/aria.ts'; +import { mdq } from '../../../src/utils/markdown-query.ts'; import { browserErrorMessage } from '../../../src/utils/browser-errors.ts'; import { pluralize } from '../../../src/utils/logger.ts'; import { type EnvelopeData, type InstanceInfo, writeArtifacts } from './envelope.ts'; -import { isFunctionExpression, toCodeceptWrapper } from './pw-parser.ts'; -import { createRefTools } from './tools.ts'; +import { isFunctionExpression, takePwValue, toCodeceptWrapper } from './pw-parser.ts'; import { type PwServerDescriptor, readDescriptors, selectDescriptor } from './pw-registry.ts'; -const MAX_INSTRUCTION_ITERATIONS = 6; +const TESTER_ONLY_TOOLS = ['learnExperience', 'askUser']; +const ITERATIONS_PER_INSTRUCTION = 2; +const MAX_INSTRUCTION_ITERATIONS = 24; const DEFAULT_RESEARCH_AFTER_VISITS = 3; const CONTEXT_HTML_CAP = 6000; const MAX_TOOL_ROUNDTRIPS = 5; @@ -30,6 +34,29 @@ const AI_AGENT_NAME = 'prima'; const CONNECT_TIMEOUT = 3000; const requireLib = createRequire(import.meta.url); +const VOLATILE_COLUMNS = ['CSS', 'XPath', 'Coordinates', 'eidx']; + +function dropVolatileColumns(markdown: string): string { + return mdq(markdown) + .query('table') + .replaceEach((table) => { + const rows = table.toJson(); + if (!rows.length) return table.text(); + + const columns = Object.keys(rows[0]).filter((name) => !VOLATILE_COLUMNS.includes(name)); + if (!columns.length) return table.text(); + + const header = `| ${columns.join(' | ')} |`; + const divider = `|${columns.map(() => '------').join('|')}|`; + const body = rows.map((row) => `| ${columns.map((name) => row[name] || '-').join(' | ')} |`); + return [header, divider, ...body, ''].join('\n'); + }); +} + +function stripRefs(tree: string): string { + return tree.replace(/\s*\[ref=[^\]]+\]/g, ''); +} + function cap(text: string, max: number): string { if (text.length <= max) return text; return `${text.slice(0, max)}\n[...truncated; ${text.length - max} chars omitted...]`; @@ -39,6 +66,7 @@ export class Prima { private options: PrimaOptions; private bot: ExplorBot; private artifactsDir?: string; + private hash?: string; private server: { close: () => Promise<void> } | null = null; private attached: string | null = null; @@ -77,11 +105,13 @@ export class Prima { const previousState = await this.baselineState(); let result: ActionResult | null = null; + let returnedValue: unknown; let executionError: unknown = null; try { const executed = await this.bot.getExplorer().action().execute(toCodeceptWrapper(expression), { verbatim: true }); result = executed.actionResult; + returnedValue = executed.lastValue; } catch (error) { executionError = error; } @@ -89,7 +119,9 @@ export class Prima { if (executionError) return this.failureEnvelope(command, executionError, previousState); result ||= await this.capturedResult(previousState); - return this.successEnvelope(command, [expression], result, previousState); + const envelope = await this.successEnvelope(command, [expression], result, previousState); + envelope.value = takePwValue(returnedValue); + return envelope; } async do(instructions: string[], label?: string): Promise<EnvelopeData> { @@ -102,16 +134,19 @@ export class Prima { const conversation = provider.startConversation(this.instructionSystemPrompt(), AI_AGENT_NAME); const task = new Task(instructions.join('; '), previousState?.url || ''); const deps = { explorer: this.bot.getExplorer(), stateManager: this.bot.stateManager(), ai: provider }; - const tools = { ...createCodeceptJSTools(deps, task), ...createRefTools(deps, task), context: this.contextTool() }; + const report: { called: boolean; summary: string; unmet: string[] } = { called: false, summary: '', unmet: [] }; + const tools = { ...createCodeceptJSTools(deps, task), ...this.testerTools(deps), context: this.contextTool(), done: this.doneTool(report) }; conversation.addUserText(await this.instructionPrompt(instructions, await this.capturedResult(previousState))); const used: string[] = []; + const checks = new Map<string, boolean>(); + const steps: Array<{ label: string; ok: boolean; proof: string }> = []; let failure: { code: string; message: string } | null = null; let aiError: unknown = null; let narration = ''; let contextHash = this.bot.stateManager().getCurrentState()?.hash; - for (let iteration = 1; iteration <= Math.min(instructions.length + 2, MAX_INSTRUCTION_ITERATIONS); iteration++) { + for (let iteration = 1; iteration <= Math.min(instructions.length * ITERATIONS_PER_INSTRUCTION + 2, MAX_INSTRUCTION_ITERATIONS); iteration++) { const state = this.bot.stateManager().getCurrentState(); if (iteration > 1 && state && state.hash !== contextHash) { contextHash = state.hash; @@ -131,13 +166,32 @@ export class Prima { } for (const execution of executions) { + const output = execution.output || {}; + + if (output.action === 'done') continue; + + if (output.action === 'verify' && !output.inexpressible) { + const claim = execution.input?.assertion || 'verification'; + let passed = execution.wasSuccessful; + if (output.alreadyVerified) passed = output.verifications?.[claim] === true; + checks.set(claim, passed); + steps.push({ label: `verify: ${claim}`, ok: passed, proof: output.code || output.message || '' }); + continue; + } + if (!execution.wasSuccessful) { - failure = { code: execution.output?.code || '', message: execution.output?.message || 'action failed' }; + failure = { code: output.code || '', message: output.message || 'action failed' }; + steps.push({ label: output.code || execution.toolName || 'action', ok: false, proof: output.message || '' }); continue; } - used.push(...this.executedCodes(execution.output?.code)); + + const codes = this.executedCodes(output.code); + used.push(...codes); + steps.push({ label: codes.join('; ') || execution.toolName || 'action', ok: true, proof: output.pageDiff?.ariaChanges || '' }); failure = null; } + + if (report.called) break; } if (aiError) return this.failureEnvelope(command, aiError, previousState); @@ -145,6 +199,7 @@ export class Prima { if (failure) { const envelope = await this.failureEnvelope(command, failure.message, previousState); envelope.used = used; + envelope.steps = steps; return envelope; } @@ -154,15 +209,47 @@ export class Prima { } const result = await this.capturedResult(this.bot.stateManager().getCurrentState()); - return this.successEnvelope(command, used, result, previousState); - } + const envelope = await this.successEnvelope(command, used, result, previousState); + envelope.steps = steps; + if (report.summary) envelope.answer = report.summary; - async click(target: string): Promise<EnvelopeData> { - return this.do([`click ${target}`], `click ${target}`); + const unmet = [...report.unmet]; + for (const [claim, passed] of checks) { + if (!passed) unmet.push(`unproven: ${claim}`); + } + if (unmet.length) { + envelope.ok = false; + envelope.failure = { error: unmet.join('\n') }; + } + return envelope; } - async fill(field: string, value: string): Promise<EnvelopeData> { - return this.do([`fill ${field} with value: ${value}`], `fill ${field} ${value}`); + async check(scenario: string, expected: string[] = []): Promise<EnvelopeData> { + const command = `check ${scenario}`; + const guard = await this.aiGuard(command); + if (guard) return guard; + + const previousState = await this.baselineState(); + const outcomes = expected.length ? expected : [scenario]; + const test = new Test(scenario, 'normal', outcomes, previousState?.url || this.options.url || ''); + const tester = this.bot.agentTester(); + + const outcome = await tester.test(test); + + const notes = Object.values(test.notes || {}) as Array<{ message: string; status?: string; log?: string; observation?: boolean }>; + const result = await this.capturedResult(this.bot.stateManager().getCurrentState()); + const envelope = await this.reportEnvelope(command, result, previousState, { ok: outcome.success }); + envelope.steps = notes.filter((note) => !note.observation).map((note) => ({ label: note.message, ok: note.status !== TestResult.FAILED, proof: note.log || '' })); + envelope.expectations = outcomes.map((text) => { + const checked = notes.findLast((note) => note.message === text && !!note.status); + if (checked?.status === TestResult.PASSED) return { text, status: 'passed' as const }; + if (checked?.status === TestResult.FAILED) return { text, status: 'failed' as const }; + return { text, status: 'unverified' as const }; + }); + + const observations = notes.filter((note) => note.observation).map((note) => note.message); + if (observations.length) envelope.answer = ['Page problems noticed while running, not step failures:', ...observations.map((line) => `- ${line}`)].join('\n'); + return envelope; } async ask(question: string): Promise<EnvelopeData> { @@ -183,13 +270,7 @@ export class Prima { const previousState = this.bot.stateManager().getCurrentState(); const result = await this.capturedResult(previousState); const verification = await this.bot.agentNavigator().verifyState(assertion, result); - const codes = verification.successfulCodes || []; - if (verification.inexpressible) { - return this.reportEnvelope(command, result, previousState, { ok: false, verdict: { passed: false, evidence: 'not checked: no assertion can express this claim, so this is not a statement about the page', code: '' } }); - } - - const verdict = { passed: verification.verified, evidence: this.verdictEvidence(verification.verified, codes), code: codes.join('\n') }; - return this.reportEnvelope(command, result, previousState, { ok: verification.verified, verdict }); + return this.reportEnvelope(command, result, previousState, { assertions: verification.results || [] }); } async research(opts: { data?: boolean; deep?: boolean; fresh?: boolean } = {}): Promise<EnvelopeData> { @@ -201,7 +282,7 @@ export class Prima { const previousState = this.bot.stateManager().getCurrentState(); const result = await this.capturedResult(previousState); const uiMap = await this.bot.agentResearcher().research(result, { screenshot: true, data: opts.data, deep: opts.deep, force: opts.fresh }); - return this.reportEnvelope(command, result, previousState, { research: uiMap }); + return this.reportEnvelope(command, result, previousState, { research: dropVolatileColumns(uiMap) }); } async go(target: string): Promise<EnvelopeData> { @@ -465,26 +546,31 @@ export class Prima { 2. Interact with the page only through the provided tools. 3. Pick the smallest interaction that fulfills an instruction, then move to the next one. 4. After the page changes, work from the updated context you are given, not from the earlier one. - 5. Stop calling tools when every instruction is done, or when an instruction cannot be performed on this page — say what is missing instead. + 5. Call done() once the last instruction is carried out, or once it is clear it cannot be on this page — list the ones you could not carry out there. </approach> <scope> Do only what the instructions ask. An action that looks helpful but was not asked for is out of scope — report it as something you noticed, never perform it. Continuing past the last instruction is a failure, even when the next step seems obvious. + An instruction worded as a condition — do X if Y appears — is satisfied the moment you can see Y is absent. Say so and move on. Never search for something the page does not show. </scope> + <pace> + Work in as few turns as you can. When the next actions are already determined by what you can see, ask for them together in one turn rather than one at a time — each turn costs a full round trip. + Only stop to look again when what you find changes what you would do next. + A batch may not run past an instruction that inspects the page — settle that one first, because the actions after it destroy the state it would have read. + </pace> + <proof> An instruction is done only when a change on the page shows it. After each action read the reported change and decide which part of it proves the instruction. When nothing observable followed, say the instruction is unproven and why. Do not restate the action as if it were the outcome. </proof> - <refs> - Page context lists refs for the elements you can act on. When your target has one, act on it with the ref tools rather than composing a locator — the ref names that exact element. - Fall back to the locator tools only for a target the context describes without a ref. - Refs belong to the context they came in. Once new context arrives the earlier refs are dead — use only the ones in the newest context you were given. - When a ref no longer resolves, or the element an instruction needs is missing from your newest context, call context() to look again and act on what it returns. - Never adapt a ref from an earlier context, invent one, or fall back to a locator you guessed. - </refs> + <targets> + The page context lists every element by role and name. Target what you act on that way — a role with its name is stable while the page re-renders, so it keeps working after your own actions change the page. + Narrow with the container it sits in when a name appears more than once, rather than guessing at an id or a class. + When the element an instruction needs is missing from your newest context, call context() to look again and act on what it returns. Never invent a locator the context does not support. + </targets> ${locatorRule} @@ -503,6 +589,35 @@ export class Prima { `; } + private testerTools(deps: any): any { + const researcher = this.bot.agentResearcher?.(); + const navigator = this.bot.agentNavigator?.(); + if (!researcher || !navigator) return {}; + + const tools = createAgentTools({ ...deps, researcher, navigator, withExperience: false }); + for (const name of TESTER_ONLY_TOOLS) delete tools[name]; + return tools; + } + + private doneTool(report: { called: boolean; summary: string; unmet: string[] }): any { + return tool({ + description: dedent` + End the sequence. Call it once every instruction is either carried out or established as impossible on this page. + Nothing runs after this call, so do not call it while an instruction is still worth another attempt. + `, + inputSchema: z.object({ + summary: z.string().describe('One sentence on what the page shows now that proves the instructions were carried out'), + unmet: z.array(z.string()).describe('Instructions that could not be carried out, each with what blocked it. Empty when all of them were carried out.'), + }), + execute: async ({ summary, unmet }) => { + report.called = true; + report.summary = summary; + report.unmet = unmet || []; + return { success: true, action: 'done' }; + }, + }); + } + private contextTool(): any { let refreshed = false; return tool({ @@ -541,7 +656,7 @@ export class Prima { return dedent` <page url="${result.url}" title="${result.title}"> - ${compactAriaSnapshot(await this.refAriaSnapshot(result), true)} + ${stripRefs(compactAriaSnapshot(await this.refAriaSnapshot(result), true, (value) => this.offloadValue(value)))} </page> ${experience} @@ -559,6 +674,18 @@ export class Prima { return DEFAULT_RESEARCH_AFTER_VISITS; } + private offloadValue(value: string): string | undefined { + const dir = this.statusDir(); + const name = `value-${createHash('sha1').update(value).digest('hex').slice(0, 8)}.txt`; + try { + mkdirSync(dir, { recursive: true }); + writeFileSync(path.join(dir, name), value, 'utf-8'); + } catch { + return undefined; + } + return path.join(path.basename(dir), name); + } + private async refAriaSnapshot(result: ActionResult): Promise<string | null> { const snapshot = await Promise.resolve(this.bot.getExplorer()?.withPage?.((page: any) => page.locator('body').ariaSnapshot({ mode: 'ai' }))).catch(() => null); return snapshot || result.ariaSnapshot; @@ -617,12 +744,6 @@ export class Prima { return response?.text || ''; } - private verdictEvidence(verified: boolean, codes: string[]): string { - if (!verified) return 'no assertion held on the current page'; - if (!codes.length) return 'already verified on this page'; - return `${codes[0]} passed`; - } - private async successEnvelope(command: string, used: string[], result: ActionResult, previousState: WebPageState | null): Promise<EnvelopeData> { return { ok: true, @@ -631,7 +752,7 @@ export class Prima { page: this.pageBlock(result, previousState), changes: await this.pageChanges(result, previousState, used[0]), instance: await this.instanceInfo(), - artifacts: await this.writeSnapshot(result), + status: await this.saveStatus(result), }; } @@ -646,7 +767,7 @@ export class Prima { page: this.pageBlock(result, previousState), failure, instance: await this.instanceInfo(), - artifacts: await this.writeSnapshot(result), + status: await this.saveStatus(result), }; } @@ -657,7 +778,7 @@ export class Prima { page: this.pageBlock(result, previousState), ...outcome, instance: await this.instanceInfo(), - artifacts: await this.writeSnapshot(result), + status: await this.saveStatus(result), }; } @@ -696,17 +817,49 @@ export class Prima { return toolResult.pageDiff?.ariaChanges || 'no change'; } - private async writeSnapshot(result: ActionResult): Promise<EnvelopeData['artifacts']> { - return writeArtifacts(this.nextArtifactDir(), { + async status(hash: string): Promise<EnvelopeData> { + const dir = this.statusDir(hash); + const statusFile = path.join(dir, 'status.json'); + if (!existsSync(statusFile)) return this.toolFailureEnvelope(`status ${hash}`, `No command was recorded under ${hash}. Every envelope prints its own hash on the Instance line.`); + + const saved = JSON.parse(readFileSync(statusFile, 'utf-8')); + return { + ok: true, + command: `status ${hash}`, + page: saved.page, + changes: saved.changes, + instance: await this.instanceInfo(), + artifacts: { aria: path.join(dir, 'aria.yml'), html: path.join(dir, 'page.html') }, + }; + } + + private async saveStatus(result: ActionResult): Promise<string> { + const hash = this.statusHash(); + await this.writeSnapshot(result); + writeFileSync(path.join(this.statusDir(hash), 'status.json'), JSON.stringify({ page: this.pageBlock(result, null), changes: compactAriaSnapshot(result.ariaSnapshot, true) }), 'utf-8'); + return hash; + } + + private async writeSnapshot(result: ActionResult): Promise<undefined> { + writeArtifacts(this.statusDir(), { aria: result.ariaSnapshot, html: await result.combinedHtml(), requests: this.bot.requestStore().getRequests(), }); + return undefined; + } + + private statusHash(): string { + this.hash ||= createHash('sha1') + .update(`${this.options.path || process.cwd()}-${Date.now()}`) + .digest('hex') + .slice(0, 15); + return this.hash; } - private nextArtifactDir(): string { + private statusDir(hash = this.statusHash()): string { this.artifactsDir ||= outputPath('prima'); - return path.join(this.artifactsDir, new Date().toISOString().replace(/[:.]/g, '-')); + return path.join(this.artifactsDir, hash); } private tabCount(): number { diff --git a/boat/prima/src/pw-parser.ts b/boat/prima/src/pw-parser.ts index 50f942ed..7fa9ccf3 100644 --- a/boat/prima/src/pw-parser.ts +++ b/boat/prima/src/pw-parser.ts @@ -13,5 +13,15 @@ export function isFunctionExpression(expr: string): { valid: boolean; error?: st } export function toCodeceptWrapper(expr: string): string { - return `I.usePlaywrightTo('pw', async (playwright) => (${expr.trim()})(playwright))`; + return `return I.usePlaywrightTo('pw', async (playwright) => (${expr.trim()})(playwright))`; +} + +export function takePwValue(held: unknown): string | undefined { + if (held === undefined || held === null) return undefined; + if (typeof held === 'string') return held; + try { + return JSON.stringify(held, null, 2); + } catch { + return String(held); + } } diff --git a/boat/prima/src/tools.ts b/boat/prima/src/tools.ts deleted file mode 100644 index 86082227..00000000 --- a/boat/prima/src/tools.ts +++ /dev/null @@ -1,84 +0,0 @@ -import { tool } from 'ai'; -import dedent from 'dedent'; -import { z } from 'zod'; -import { ActionResult } from '../../../src/action-result.ts'; -import type { ToolDeps } from '../../../src/ai/agent.ts'; -import { commitNote, failedToolResult, successToolResult } from '../../../src/ai/tools.ts'; -import { type Task, TestResult } from '../../../src/test-plan.ts'; -import { WebElement } from '../../../src/utils/web-element.ts'; - -const REF_INPUT_DESCRIPTION = 'Ref of the target element, copied exactly as the current page snapshot writes it inside [ref=...]. Use it whenever the snapshot lists one for your target. Only refs from the latest snapshot resolve; never adapt or invent one.'; - -export async function refToXPath(explorer: any, ref: string): Promise<{ xpath?: string; error?: string }> { - if (!WebElement.isAriaRef(ref)) return { error: `"${ref}" is not a ref. Pass a ref exactly as the snapshot writes it inside [ref=...], or use commands instead.` }; - - const element = await explorer.withPage((page: any) => WebElement.fromAriaRef(page, ref)); - if (!element) return { error: `Ref ${ref} matches no element. It belongs to an older snapshot of this page.` }; - - const xpath = element.clickXPath; - if (!xpath) return { error: `Ref ${ref} resolved to an element with no distinctive attributes to target.` }; - - const matches = await explorer.withPage((page: any) => page.locator(`xpath=${xpath}`).count()); - if (matches !== 1) return { error: `Ref ${ref} resolved to ${xpath}, which matches ${matches} elements instead of exactly one.` }; - - return { xpath }; -} - -export function createRefTools({ explorer, stateManager }: ToolDeps, task: Task) { - const runRef = async (action: 'clickRef' | 'hoverRef', ref: string, explanation: string, toCommand: (xpath: string) => string) => { - const activeNote = task.startNote(explanation); - const resolved = await refToXPath(explorer, ref); - - if (resolved.error) { - activeNote.commit(TestResult.FAILED); - return failedToolResult(action, resolved.error, { - suggestion: 'Do not retry with a guessed locator. Take a fresh look at the page and use a ref it lists, or report that the element is not there.', - }); - } - - const previousState = ActionResult.fromState(stateManager.getCurrentState()!); - const runner = explorer.action(); - const command = toCommand(resolved.xpath!); - const success = await runner.attempt(command, explanation); - const toolResult = await ActionResult.fromState(stateManager.getCurrentState()!).toToolResult(previousState, command); - - if (!success) { - await commitNote(activeNote, TestResult.FAILED, toolResult, runner); - return failedToolResult(action, `${command} failed on the element behind ${ref}`, { ...toolResult, code: command }, runner.lastError); - } - - await commitNote(activeNote, TestResult.PASSED, toolResult, runner); - return successToolResult(action, { ...toolResult, code: command }, runner); - }; - - return { - clickRef: tool({ - description: dedent` - Click the element a page snapshot gave you a ref for. - - Use this whenever the snapshot lists a ref for your target — the ref names that exact - element, so no locator and no fallback list is needed. - Use click() instead when you are working from markup that carries no refs. - `, - inputSchema: z.object({ - ref: z.string().describe(REF_INPUT_DESCRIPTION), - explanation: z.string().describe('Why you are clicking this element'), - }), - execute: async ({ ref, explanation }) => runRef('clickRef', ref, explanation, (xpath) => `I.click(${JSON.stringify(xpath)})`), - }), - - hoverRef: tool({ - description: dedent` - Move the mouse to the element a page snapshot gave you a ref for, to reveal hover-only controls. - - Use this whenever the snapshot lists a ref for your target. It does not click. - Use hover() instead when you are working from markup that carries no refs. - `, - inputSchema: z.object({ - ref: z.string().describe(REF_INPUT_DESCRIPTION), - explanation: z.string().describe('Why you are hovering this element'), - }), - execute: async ({ ref, explanation }) => runRef('hoverRef', ref, explanation, (xpath) => `I.moveCursorTo(${JSON.stringify(xpath)})`), - }), - }; -} diff --git a/boat/prima/tests/envelope.test.ts b/boat/prima/tests/envelope.test.ts index 0c3dbad2..4f475195 100644 --- a/boat/prima/tests/envelope.test.ts +++ b/boat/prima/tests/envelope.test.ts @@ -17,14 +17,15 @@ const base: EnvelopeData = { describe('renderEnvelope', () => { test('success envelope contains all sections in order', () => { const out = renderEnvelope(base); - const sections = ['### Result', '### Page', '### Changes', '### Instance', '### Artifacts']; + const sections = ['### Result', '### Page', '### Changes', '### Instance']; const positions = sections.map((s) => out.indexOf(s)); expect(positions.every((p) => p >= 0)).toBe(true); expect([...positions].sort((a, b) => a - b)).toEqual(positions); expect(out).toContain('ok: true'); expect(out).toContain("used: I.click('Login')"); expect(out).toContain('(changed: https://app.example.com/login → https://app.example.com/dashboard)'); - expect(out).toContain('instance: default (3 tabs) | other instances: auth-test (1 tab)'); + expect(out).toContain('default (3 tabs)'); + expect(out).toContain('other instances: auth-test (1 tab)'); }); test('unchanged url renders without changed marker', () => { @@ -63,49 +64,106 @@ describe('renderEnvelope', () => { expect(out).not.toContain('### Answer'); }); - test('verdict replaces changes for verify', () => { - const out = renderEnvelope({ ...base, changes: undefined, verdict: { passed: true, evidence: 'heading "Dashboard" present', code: "I.see('Dashboard')" } }); - expect(out).toContain('### Verdict'); - expect(out).toContain('passed: true'); - expect(out).toContain("I.see('Dashboard')"); + test('assertions render one line per check with its own result, and no overall verdict', () => { + const out = renderEnvelope({ + ...base, + changes: undefined, + assertions: [ + { code: "I.see('Dashboard')", passed: true, proof: ['await expect(page).toContainText("Dashboard");'] }, + { code: "I.seeElement('.chart')", passed: false, proof: [] }, + ], + }); + + expect(out).toContain('### Assertions'); + expect(out).toContain("I.see('Dashboard') => PASSED"); + expect(out).toContain("I.seeElement('.chart') => FAILED"); + expect(out).toContain('playwright:'); + expect(out).toContain('await expect(page).toContainText("Dashboard");'); + expect(out).not.toContain('passed: '); + }); + + test('a multi-line assertion gets one result, not one per line', () => { + const out = renderEnvelope({ + ...base, + changes: undefined, + assertions: [{ code: '// check the box\nI.seeElement({\n role: "button",\n text: "Submit"\n});', passed: false, proof: [] }], + }); + + expect(out).toContain('I.seeElement({ role: "button", text: "Submit" }); => FAILED'); + expect(out.match(/=> FAILED/g)).toHaveLength(1); + expect(out).not.toContain('// check the box'); + }); + + test('no expressible assertion is stated as such, not as a failure', () => { + const out = renderEnvelope({ ...base, changes: undefined, assertions: [] }); + expect(out).toContain('none ran'); + expect(out).not.toContain('FAILED'); }); test('changes render on every action envelope, including when nothing moved', () => { expect(renderEnvelope({ ...base, changes: 'no change' })).toContain('### Changes\nno change'); }); - test('changes render alongside a verdict rather than replacing it', () => { - const out = renderEnvelope({ ...base, changes: 'no change', verdict: { passed: true, evidence: 'I.seeElement()', code: 'I.seeElement()' } }); + test('changes render alongside assertions rather than replacing them', () => { + const out = renderEnvelope({ ...base, changes: 'no change', assertions: [{ code: 'I.seeElement()', passed: true, proof: [] }] }); expect(out).toContain('### Changes'); - expect(out).toContain('### Verdict'); + expect(out).toContain('### Assertions'); }); - test('steps report per-instruction proof and name what stayed unproven', () => { + test('steps report every action and check in order, with its proof', () => { const out = renderEnvelope({ ...base, steps: [ - { instruction: 'open the account menu', proof: 'added menu "Account"' }, - { instruction: 'choose the settings entry', proof: null }, + { label: "I.click('Add workflow')", ok: true, proof: 'ariaDiff:\n added:\n - textbox "Title"' }, + { label: 'I.seeAttributesOnElements({"role":"button"}, { disabled: true })', ok: true, proof: 'await expect(page.getByRole("button")).toBeDisabled();' }, + { label: "I.fillField('Title', 'x')", ok: false, proof: 'element not found' }, ], }); - expect(out).toContain('1. open the account menu — proven by added menu "Account"'); - expect(out).toContain('2. choose the settings entry — unproven'); + + expect(out).toContain("1. ok I.click('Add workflow')"); + expect(out).toContain('2. ok I.seeAttributesOnElements'); + expect(out).toContain("3. FAIL I.fillField('Title', 'x')"); + expect(out).toContain(' element not found'); + expect(out).toContain(' await expect(page.getByRole("button")).toBeDisabled();'); + }); + + test('every expected outcome is echoed with its own result, including the ones nothing checked', () => { + const out = renderEnvelope({ + ...base, + changes: undefined, + expectations: [ + { text: 'the editor opens', status: 'passed' }, + { text: 'the draft is saved', status: 'failed' }, + { text: 'the list refreshes', status: 'unverified' }, + ], + }); + + expect(out).toContain('### Expected outcomes'); + expect(out).toContain('1. PASSED the editor opens'); + expect(out).toContain('2. FAILED the draft is saved'); + expect(out).toContain('3. not verified the list refreshes'); }); test('attached instance renders attached browser line', () => { const out = renderEnvelope({ ...base, instance: { ...base.instance, attached: 'playwright-cli session "default", workspace /w' } }); - expect(out).toContain('browser: attached (playwright-cli session "default"'); + expect(out).toContain('attached to playwright-cli session "default"'); expect(out).not.toContain('started 12m ago'); }); test('instance without evidence of a live browser reports it as not running', () => { const out = renderEnvelope({ ...base, instance: { name: 'default', tabs: 0, others: [] } }); - expect(out).toContain('browser: not running'); + expect(out).toContain('not running'); + }); + + test('the status hash is offered as the way to reach details', () => { + const out = renderEnvelope({ ...base, status: 'abc123def456789', artifacts: undefined }); + expect(out).toContain('details: prima status abc123def456789'); + expect(out).not.toContain('### Artifacts'); }); test('open tabs alone are evidence enough for a running browser', () => { const out = renderEnvelope({ ...base, instance: { name: 'default', tabs: 2, others: [] } }); - expect(out).toContain('browser: running'); + expect(out).toContain('| running'); expect(out).not.toContain('browser: not running'); }); }); diff --git a/boat/prima/tests/prima.test.ts b/boat/prima/tests/prima.test.ts index 8a507a83..59faf9fc 100644 --- a/boat/prima/tests/prima.test.ts +++ b/boat/prima/tests/prima.test.ts @@ -6,6 +6,7 @@ import { Navigator } from '../../../src/ai/navigator.ts'; import { getEndpointFilePath, listInstances } from '../../../src/browser-server.ts'; import { ConfigParser } from '../../../src/config.ts'; import { ExplorBot } from '../../../src/explorbot.ts'; +import { TestResult } from '../../../src/test-plan.ts'; import { Prima } from '../src/prima.ts'; let artifactsRoot: string; @@ -122,9 +123,10 @@ describe('Prima.pw', () => { const { prima } = fakePrima(); const envelope = await prima.pw("({ page }) => page.click('text=Login')"); expect(envelope.changes).toContain('heading "Dashboard"'); - expect(existsSync(envelope.artifacts!.aria)).toBe(true); - expect(existsSync(envelope.artifacts!.html)).toBe(true); - expect(envelope.artifacts!.network).toBeUndefined(); + expect(envelope.status).toMatch(/^[0-9a-f]{15}$/); + expect(existsSync(path.join(artifactsRoot, envelope.status!, 'aria.yml'))).toBe(true); + expect(existsSync(path.join(artifactsRoot, envelope.status!, 'page.html'))).toBe(true); + expect(existsSync(path.join(artifactsRoot, envelope.status!, 'network.jsonl'))).toBe(false); expect(envelope.instance.name).toBe('default'); }); @@ -142,7 +144,7 @@ describe('Prima.pw', () => { expect(envelope.ok).toBe(false); expect(envelope.failure?.error).toContain("locator 'text=Login' not found"); expect(envelope.page.url).toBe('https://app.example.com/login'); - expect(envelope.artifacts).toBeTruthy(); + expect(envelope.status).toBeTruthy(); }); test('unrecoverable browser still returns a failure envelope when capture fails too', async () => { @@ -161,7 +163,7 @@ describe('Prima.pw', () => { expect(envelope.ok).toBe(false); expect(envelope.failure?.error).toContain('Browser page is unavailable'); expect(envelope.page.url).toContain('/login'); - expect(envelope.artifacts).toBeTruthy(); + expect(envelope.status).toBeTruthy(); }); test('missing explorer returns a failure envelope instead of rejecting', async () => { @@ -191,7 +193,7 @@ describe('Prima failure never substitutes a target', () => { expect(envelope.used).toBeUndefined(); expect(envelope.failure?.error).toContain("locator 'text=Login' not found"); expect(envelope.failure?.compactAria).toContain('button'); - expect(envelope.artifacts).toBeTruthy(); + expect(envelope.status).toBeTruthy(); }); test('a failed pw carries no healed marker', async () => { @@ -444,25 +446,31 @@ describe('Prima.do', () => { }); await prima.do(['first step', 'second step']); - expect(calls).toBe(4); + expect(calls).toBe(6); calls = 0; await prima.do(['a', 'b', 'c', 'd', 'e', 'f', 'g']); - expect(calls).toBe(6); + expect(calls).toBe(16); + + calls = 0; + await prima.do(Array.from({ length: 20 }, (_, i) => `step ${i}`)); + expect(calls).toBe(24); }); test('a failed tool execution fails the command instead of reaching for another element', async () => { const { prima } = fakePrima(); - let called = false; + let resolved = false; (prima as any).bot.getProvider = () => fakeProvider(async () => ({ toolExecutions: [toolExecution("I.click('Login')", false, 'element not found')] })); - (prima as any).bot.agentNavigator = () => { - called = true; - return { resolveState: async () => true }; - }; + (prima as any).bot.agentNavigator = () => ({ + resolveState: async () => { + resolved = true; + return true; + }, + }); const envelope = await prima.do(['click the login link']); - expect(called).toBe(false); + expect(resolved).toBe(false); expect(envelope.ok).toBe(false); expect(envelope.failure?.error).toContain('element not found'); }); @@ -514,7 +522,79 @@ describe('Prima.do', () => { expect(prompts.length).toBe(1); }); - test('context() hands back refs first and drops to markup only when asked again', async () => { + test('done() ends the sequence instead of burning the remaining iteration budget', async () => { + const { prima } = fakePrima(); + let calls = 0; + (prima as any).bot.getProvider = () => + fakeProvider(async (_conversation: unknown, tools: any) => { + calls++; + if (calls === 1) return { toolExecutions: [toolExecution("I.click('Invoices')")] }; + await tools.done.execute({ summary: 'the invoice list is open', unmet: [] }); + return { toolExecutions: [{ toolName: 'done', input: {}, output: { success: true, action: 'done' }, wasSuccessful: true }] }; + }); + + const envelope = await prima.do(['open the invoices page', 'read the first invoice']); + + expect(calls).toBe(2); + expect(envelope.ok).toBe(true); + expect(envelope.answer).toBe('the invoice list is open'); + expect(envelope.steps?.map((step) => step.label)).toEqual(["I.click('Invoices')"]); + }); + + test('instructions the model could not carry out fail the command', async () => { + const { prima } = fakePrima(); + (prima as any).bot.getProvider = () => + fakeProvider(async (_conversation: unknown, tools: any) => { + await tools.done.execute({ summary: 'the list is open', unmet: ['no PDF link exists on this page'] }); + return { toolExecutions: [toolExecution("I.click('Invoices')"), { toolName: 'done', input: {}, output: { success: true, action: 'done' }, wasSuccessful: true }] }; + }); + + const envelope = await prima.do(['open the invoices page', 'download the PDF']); + + expect(envelope.ok).toBe(false); + expect(envelope.failure?.error).toContain('no PDF link exists on this page'); + }); + + test('a check that never passed fails the command even after later actions succeed', async () => { + const { prima } = fakePrima(); + let calls = 0; + (prima as any).bot.getProvider = () => + fakeProvider(async (_conversation: unknown, tools: any) => { + calls++; + if (calls === 1) { + return { + toolExecutions: [{ toolName: 'verify', input: { assertion: 'unsaved indicator is visible' }, output: { success: false, action: 'verify', message: 'Verification failed' }, wasSuccessful: false }, toolExecution("I.click('Close')")], + }; + } + await tools.done.execute({ summary: 'the editor is closed', unmet: [] }); + return { toolExecutions: [{ toolName: 'done', input: {}, output: { success: true, action: 'done' }, wasSuccessful: true }] }; + }); + + const envelope = await prima.do(['confirm the unsaved indicator', 'close the editor']); + + expect(envelope.ok).toBe(false); + expect(envelope.failure?.error).toContain('unsaved indicator is visible'); + expect(envelope.steps?.[0]).toMatchObject({ label: 'verify: unsaved indicator is visible', ok: false }); + }); + + test('a check that passes on a retry does not fail the command', async () => { + const { prima } = fakePrima(); + let calls = 0; + (prima as any).bot.getProvider = () => + fakeProvider(async (_conversation: unknown, tools: any) => { + calls++; + if (calls === 1) return { toolExecutions: [toolExecution("I.click('Delete')"), { toolName: 'verify', input: { assertion: 'the row is gone' }, output: { success: false, action: 'verify', message: 'Verification failed' }, wasSuccessful: false }] }; + if (calls === 2) return { toolExecutions: [{ toolName: 'verify', input: { assertion: 'the row is gone' }, output: { success: true, action: 'verify', code: "I.dontSee('Item')" }, wasSuccessful: true }] }; + await tools.done.execute({ summary: 'the row is gone', unmet: [] }); + return { toolExecutions: [{ toolName: 'done', input: {}, output: { success: true, action: 'done' }, wasSuccessful: true }] }; + }); + + const envelope = await prima.do(['delete the row', 'confirm it is gone']); + + expect(envelope.ok).toBe(true); + }); + + test('context() hands back the page tree first and drops to markup only when asked again', async () => { const { prima } = fakePrima(); let captured: any; (prima as any).bot.getProvider = () => @@ -525,39 +605,48 @@ describe('Prima.do', () => { await prima.do(['open the invoices page']); - const first = await captured.execute({ reason: 'the ref no longer resolves' }); - expect(first.context).toContain('ref=e7'); + const first = await captured.execute({ reason: 'the control is not in my context' }); + expect(first.context).toContain('button "Refreshed"'); + expect(first.context).not.toContain('ref='); const second = await captured.execute({ reason: 'still cannot reach it' }); - expect(second.context).not.toContain('ref=e7'); expect(second.context).toContain('<form>'); }); +}); - test('click is a single-instruction alias over do', async () => { +describe('Prima.check', () => { + test('every --expected outcome is reported with its own result', async () => { const { prima } = fakePrima(); - const received: string[][] = []; - (prima as any).do = async (instructions: string[]) => { - received.push(instructions); - return { ok: true }; - }; + (prima as any).bot.agentTester = () => ({ + test: async (test: any) => { + test.addNote('the editor opens', TestResult.PASSED); + test.addNote('the draft is saved', TestResult.FAILED); + return { success: false }; + }, + }); + + const envelope = await prima.check('edit and save a skill', ['the editor opens', 'the draft is saved', 'the list refreshes']); - await prima.click('the login link'); - expect(received[0].length).toBe(1); - expect(received[0][0]).toContain('the login link'); + expect(envelope.expectations).toEqual([ + { text: 'the editor opens', status: 'passed' }, + { text: 'the draft is saved', status: 'failed' }, + { text: 'the list refreshes', status: 'unverified' }, + ]); }); - test('fill is a single-instruction alias carrying field and value', async () => { + test('the scenario stands in as the only outcome when none was given', async () => { const { prima } = fakePrima(); - const received: string[][] = []; - (prima as any).do = async (instructions: string[]) => { - received.push(instructions); - return { ok: true }; - }; + (prima as any).bot.agentTester = () => ({ + test: async (test: any) => { + test.addNote('edit and save a skill', TestResult.PASSED); + return { success: true }; + }, + }); + + const envelope = await prima.check('edit and save a skill'); - await prima.fill('the email field', 'user@example.com'); - expect(received[0].length).toBe(1); - expect(received[0][0]).toContain('the email field'); - expect(received[0][0]).toContain('user@example.com'); + expect(envelope.ok).toBe(true); + expect(envelope.expectations).toEqual([{ text: 'edit and save a skill', status: 'passed' }]); }); }); @@ -570,7 +659,7 @@ describe('Prima.ask, verify, research', () => { const envelope = await prima.ask('what do I see?'); expect(envelope.answer).toContain('login'); expect(envelope.changes).toBeUndefined(); - expect(envelope.artifacts).toBeTruthy(); + expect(envelope.status).toBeTruthy(); }); test('ask with noVision answers from researcher summary', async () => { @@ -604,28 +693,41 @@ describe('Prima.ask, verify, research', () => { expect(envelope.answer).toContain('vision'); }); - test('verify returns verdict with assertion code', async () => { + test('verify reports each assertion with its own result and gives no verdict', async () => { const { prima } = fakePrima(); (prima as any).bot.agentNavigator = () => ({ - verifyState: async () => ({ verified: true, successfulCodes: ["I.see('Dashboard')"], assertionSteps: [], totalAttempted: 1 }), + verifyState: async () => ({ + verified: false, + results: [ + { code: "I.see('Dashboard')", passed: true, proof: ['await expect(page).toContainText("Dashboard");'] }, + { code: "I.seeElement('.chart')", passed: false, proof: [] }, + ], + successfulCodes: ["I.see('Dashboard')"], + assertionSteps: [], + totalAttempted: 2, + }), }); const envelope = await prima.verify('user sees the dashboard'); - expect(envelope.verdict?.passed).toBe(true); - expect(envelope.verdict?.code).toBe("I.see('Dashboard')"); + + expect(envelope).not.toHaveProperty('verdict'); + expect(envelope.assertions).toEqual([ + { code: "I.see('Dashboard')", passed: true, proof: ['await expect(page).toContainText("Dashboard");'] }, + { code: "I.seeElement('.chart')", passed: false, proof: [] }, + ]); expect(envelope.ok).toBe(true); }); - test('failed verification is reported as a failed verdict', async () => { + test('verify with nothing expressible reports no assertions rather than a failure', async () => { const { prima } = fakePrima(); (prima as any).bot.agentNavigator = () => ({ - verifyState: async () => ({ verified: false, successfulCodes: [], assertionSteps: [], totalAttempted: 3 }), + verifyState: async () => ({ verified: false, inexpressible: true, results: [], successfulCodes: [], assertionSteps: [], totalAttempted: 0 }), }); - const envelope = await prima.verify('user sees the dashboard'); - expect(envelope.verdict?.passed).toBe(false); - expect(envelope.verdict?.evidence).toBeTruthy(); - expect(envelope.ok).toBe(false); + const envelope = await prima.verify('the save button is disabled'); + + expect(envelope.assertions).toEqual([]); + expect(envelope.ok).toBe(true); }); test('research returns UI map in envelope', async () => { diff --git a/boat/prima/tests/pw-parser.test.ts b/boat/prima/tests/pw-parser.test.ts index db6f9608..f29f0320 100644 --- a/boat/prima/tests/pw-parser.test.ts +++ b/boat/prima/tests/pw-parser.test.ts @@ -22,12 +22,16 @@ describe('isFunctionExpression', () => { describe('toCodeceptWrapper', () => { test('calls the function from an async wrapper the native helper API accepts', () => { const code = toCodeceptWrapper("({ page }) => page.click('text=Login')"); - expect(code).toBe("I.usePlaywrightTo('pw', async (playwright) => (({ page }) => page.click('text=Login'))(playwright))"); + expect(code).toBe("return I.usePlaywrightTo('pw', async (playwright) => (({ page }) => page.click('text=Login'))(playwright))"); + }); + + test('returns the expression result so the caller can read it', () => { + expect(toCodeceptWrapper('({ page }) => page.title()')).toStartWith('return '); }); test('keeps an already async function callable', () => { const code = toCodeceptWrapper("async ({ page }) => { await page.click('text=Login') }"); expect(code).toContain("async ({ page }) => { await page.click('text=Login') }"); - expect(code).toStartWith("I.usePlaywrightTo('pw', async (playwright) =>"); + expect(code).toStartWith("return I.usePlaywrightTo('pw', async (playwright) =>"); }); }); diff --git a/boat/prima/tests/tools.test.ts b/boat/prima/tests/tools.test.ts deleted file mode 100644 index 4f17d289..00000000 --- a/boat/prima/tests/tools.test.ts +++ /dev/null @@ -1,118 +0,0 @@ -import { beforeEach, describe, expect, it } from 'bun:test'; -import { ActionResult } from '../../../src/action-result.ts'; -import { createCodeceptJSTools } from '../../../src/ai/tools.ts'; -import { createRefTools } from '../src/tools.ts'; -import { ConfigParser } from '../../../src/config.ts'; -import { Task } from '../../../src/test-plan.ts'; - -beforeEach(() => { - ConfigParser.resetForTesting(); - ConfigParser.setupTestConfig(); -}); - -const state = new ActionResult({ url: '/page', title: 'Page', html: '<html></html>', ariaSnapshot: '' }); - -function buildTools(page: any, attempted: string[]) { - const action = { - lastError: null as Error | null, - attempt: async (command: string) => { - attempted.push(command); - return true; - }, - saveScreenshot: async () => null, - }; - const explorer = { - action: () => action, - withPage: (fn: (page: any) => any) => fn(page), - } as any; - const stateManager = { getCurrentState: () => state } as any; - return createRefTools({ explorer, stateManager, ai: {} as any }, new Task('t', '/page')); -} - -function pageWith(element: any, matches: number) { - return { - locator: (selector: string) => { - if (selector.startsWith('xpath=')) return { count: async () => matches }; - return { - count: async () => (element ? 1 : 0), - first: () => ({ evaluate: async () => element }), - }; - }, - }; -} - -const saveButton = { tag: 'button', allAttrs: { id: 'save-button' }, text: 'Save', outerHTML: '<button id="save-button">Save</button>', x: 1, y: 1 }; - -describe('clickRef', () => { - it('resolves a ref to an attribute-based xpath and clicks it', async () => { - const attempted: string[] = []; - const tools = buildTools(pageWith(saveButton, 1), attempted); - - const result = await tools.clickRef.execute({ ref: 'f1e13', explanation: 'click save' }); - - expect(result.success).toBe(true); - expect(attempted).toEqual(['I.click("//*[@id=\\"save-button\\"]")']); - expect(result.code).toContain('save-button'); - }); - - it('fails without guessing when the ref matches nothing', async () => { - const attempted: string[] = []; - const tools = buildTools(pageWith(null, 0), attempted); - - const result = await tools.clickRef.execute({ ref: 'f1e99', explanation: 'click save' }); - - expect(result.success).toBe(false); - expect(result.message).toContain('matches no element'); - expect(attempted).toEqual([]); - }); - - it('fails when the resolved xpath is ambiguous', async () => { - const attempted: string[] = []; - const tools = buildTools(pageWith(saveButton, 3), attempted); - - const result = await tools.clickRef.execute({ ref: 'f1e13', explanation: 'click save' }); - - expect(result.success).toBe(false); - expect(result.message).toContain('matches 3 elements'); - expect(attempted).toEqual([]); - }); - - it.each(['e13', 'f1e13', 'f12e345'])('accepts the ref shape %p that playwright emits', async (ref) => { - const attempted: string[] = []; - const tools = buildTools(pageWith(saveButton, 1), attempted); - - const result = await tools.clickRef.execute({ ref, explanation: 'click save' }); - - expect(result.success).toBe(true); - }); - - it('rejects a ref that is not snapshot-shaped', async () => { - const attempted: string[] = []; - const tools = buildTools(pageWith(saveButton, 1), attempted); - - const result = await tools.clickRef.execute({ ref: 'button.save', explanation: 'click save' }); - - expect(result.success).toBe(false); - expect(result.message).toContain('is not a ref'); - expect(attempted).toEqual([]); - }); - - it('leaves the locator tools alone — they take no ref', async () => { - const ladder = createCodeceptJSTools({ explorer: {} as any, stateManager: {} as any, ai: {} as any }, new Task('t', '/page')); - - expect(Object.keys(ladder.click.inputSchema.shape)).toEqual(['commands', 'explanation']); - expect(Object.keys(ladder.hover.inputSchema.shape)).toEqual(['commands', 'explanation']); - }); -}); - -describe('hoverRef', () => { - it('resolves a ref to a moveCursorTo command', async () => { - const attempted: string[] = []; - const tools = buildTools(pageWith(saveButton, 1), attempted); - - const result = await tools.hoverRef.execute({ ref: 'f1e13', explanation: 'reveal row actions' }); - - expect(result.success).toBe(true); - expect(attempted).toEqual(['I.moveCursorTo("//*[@id=\\"save-button\\"]")']); - }); -}); diff --git a/rules/navigator/verification-actions.md b/rules/navigator/verification-actions.md index b786e490..91c59847 100644 --- a/rules/navigator/verification-actions.md +++ b/rules/navigator/verification-actions.md @@ -105,25 +105,28 @@ Checks that page source does NOT contain expected text. I.dontSeeInSource('error-class'); </example> -### I.seeAttributesOnElements +### Asserting the state of a control -I.seeAttributesOnElements(<locator>, <attributes>) - -Checks the interactive state of a control: enabled, disabled, checked, selected, expanded, required, read-only. -This is the ONLY way to assert state — presence assertions cannot express it. -Assert the state you expect; to assert the opposite state, assert that value explicitly rather than negating. +State means disabled, checked, readonly, required, selected, expanded. A presence assertion says +nothing about it, so assert it with an attribute selector: <example> - I.seeAttributesOnElements('#submit', { disabled: null }); - I.seeAttributesOnElements({"role":"button","text":"Save"}, { 'aria-disabled': 'false' }); - I.seeAttributesOnElements({"role":"checkbox","text":"Accept"}, { checked: true }); - I.seeAttributesOnElements({"role":"button","text":"Details"}, { 'aria-expanded': 'true' }); + I.seeElement('button[aria-label="Submit"][disabled]'); + I.dontSeeElement('button[aria-label="Submit"][disabled]'); + I.seeElement('input[name="accept"][checked]'); + I.seeElement('button[aria-label="Details"][aria-expanded="true"]'); </example> +Use I.seeElement for the state you expect and I.dontSeeElement for the state you expect to be +absent — that pair expresses both directions. + +I.seeAttributesOnElements(<locator>, { disabled: true }) also exists, but only takes a state that +must be PRESENT and does not resolve reliably against a role/text locator. Prefer the selector form above. + <verification_rules> Be strict in assertions to avoid false positives. Prefer I.seeElement() with ARIA locators - most reliable. -For a claim about a control's state, use I.seeAttributesOnElements() — presence of the element is not evidence of its state. +For a claim about a control's state, assert it with an attribute selector — presence of the element is not evidence of its state. If no assertion above can express the claim, say so instead of proposing an assertion that checks something weaker. I.see() and I.dontSee() MUST include context parameter. For input field values, ALWAYS use I.seeInField() — never check value via CSS attribute selectors or I.seeInSource. diff --git a/src/action.ts b/src/action.ts index 6725ad2b..6625ce1f 100644 --- a/src/action.ts +++ b/src/action.ts @@ -32,6 +32,7 @@ class Action { public playwrightHelper: any; public playwrightGroupId: string | null = null; public assertionSteps: Array<{ name: string; args: any[] }> = []; + public lastValue: unknown; private recorder?: PlaywrightRecorder; private recovery: RecoveryRunner; private mainDocumentStatus: number | undefined = undefined; @@ -302,9 +303,10 @@ class Action { await playwrightSandbox(page, sanitizedCode); await sleep(this.config.action?.delay || 500); } else { - codeceptJSSandbox(this.actor, sanitizedCode); + const returned = codeceptJSSandbox(this.actor, sanitizedCode); await recorder.add(() => sleep(this.config.action?.delay || 500)); await recorder.promise(); + this.lastValue = await returned; } if (executedSteps.length > 0) { diff --git a/src/ai/navigator.ts b/src/ai/navigator.ts index 0028cc2d..94bd1979 100644 --- a/src/ai/navigator.ts +++ b/src/ai/navigator.ts @@ -8,6 +8,7 @@ import type { ExperienceTracker } from '../experience-tracker.js'; import Explorer from '../explorer.ts'; import type { KnowledgeTracker } from '../knowledge-tracker.js'; import { type StateManager, normalizeUrl } from '../state-manager.js'; +import { renderAssertion } from '../playwright-recorder.ts'; import { extractCodeBlocks } from '../utils/code-extractor.js'; import { HooksRunner } from '../utils/hooks-runner.ts'; import { createDebug, pluralize, tag } from '../utils/logger.js'; @@ -620,7 +621,7 @@ class Navigator implements Agent { return suggestion; } - async verifyState(message: string, actionResult: ActionResult): Promise<{ verified: boolean; inexpressible: boolean; successfulCodes: string[]; assertionSteps: Array<{ name: string; args: any[] }>; totalAttempted: number }> { + async verifyState(message: string, actionResult: ActionResult): Promise<{ verified: boolean; inexpressible: boolean; results: AssertionResult[]; successfulCodes: string[]; assertionSteps: Array<{ name: string; args: any[] }>; totalAttempted: number }> { tag('info').log('AI Navigator verifying state at', actionResult.url); debugLog('Verification message:', message); @@ -698,6 +699,7 @@ class Navigator implements Agent { let codeBlocks: string[] = []; const successfulCodes: string[] = []; + const results: AssertionResult[] = []; const assertionSteps: Array<{ name: string; args: any[] }> = []; const action = this.explorer.action(); @@ -739,6 +741,8 @@ class Navigator implements Agent { await action.exitIframe(); const verified = await action.attempt(codeBlock, message); + const proof = action.assertionSteps.map(renderAssertion).filter(Boolean); + results.push({ code: codeBlock, passed: verified, proof }); if (verified) { tag('success').log('Verification passed'); @@ -747,12 +751,6 @@ class Navigator implements Agent { } else { failures++; } - - const target = Math.min(codeBlocks.length, this.verifyAttempts); - const majorityNeeded = Math.floor(target / 2) + 1; - if (successfulCodes.length >= majorityNeeded || failures > target - majorityNeeded) { - stop(); - } }, { maxAttempts: this.verifyAttempts, @@ -776,13 +774,13 @@ class Navigator implements Agent { const inexpressible = !alreadyVerified && totalAttempted === 0; if (inexpressible) { tag('warning').log('No assertion could express this claim'); - return { verified: false, inexpressible, successfulCodes, assertionSteps, totalAttempted }; + return { verified: false, inexpressible, results, successfulCodes, assertionSteps, totalAttempted }; } actionResult.addVerification(message, verified); this.stateManager.updateState(actionResult); - return { verified, inexpressible, successfulCodes, assertionSteps, totalAttempted }; + return { verified, inexpressible, results, successfulCodes, assertionSteps, totalAttempted }; } private checkAlreadyVerified(aiResponse: string, actionResult: ActionResult): boolean { @@ -793,4 +791,6 @@ class Navigator implements Agent { } } +export type AssertionResult = { code: string; passed: boolean; proof: string[] }; + export { Navigator }; diff --git a/src/ai/tester.ts b/src/ai/tester.ts index 73758421..4cc0f705 100644 --- a/src/ai/tester.ts +++ b/src/ai/tester.ts @@ -62,7 +62,6 @@ export class Tester extends TaskAgent implements Agent { private seenUiMapUrls = new Set<string>(); private lastAnalyzedStateHash: string | null = null; private stalledIterations = 0; - private hasSuccessfulAssertion = false; private readonly MAX_STALLED_ITERATIONS = 3; constructor(deps: AgentDeps, researcher: Researcher, navigator: Navigator, agentTools?: any) { @@ -111,7 +110,6 @@ export class Tester extends TaskAgent implements Agent { this.seenUiMapUrls.clear(); this.lastAnalyzedStateHash = null; this.stalledIterations = 0; - this.hasSuccessfulAssertion = false; this.stateManager.clearHistory(); this.resetFailureCount(); this.pilot?.reset(); @@ -119,7 +117,7 @@ export class Tester extends TaskAgent implements Agent { const requestStore = this.requestStore; requestStore.clear(); const offFailedRequest = requestStore.onFailedRequest((r) => { - task.addNote(`Network error: ${r.method} ${r.path} → ${r.status}`, TestResult.FAILED); + task.addObservation(`Network error: ${r.method} ${r.path} → ${r.status}`); }); const initialState = ActionResult.fromState(state); @@ -314,17 +312,9 @@ export class Tester extends TaskAgent implements Agent { const allToolNames = result?.toolExecutions?.map((execution: any) => execution.toolName) || []; const successfulToolNames = result?.toolExecutions?.filter((execution: any) => execution.wasSuccessful)?.map((execution: any) => execution.toolName) || []; const actionPerformed = !!allToolNames.find((toolName: string) => this.ACTION_TOOLS.includes(toolName)); - const successfulActionPerformed = !!successfulToolNames.find((toolName: string) => this.ACTION_TOOLS.includes(toolName)); assertionPerformed = !!successfulToolNames.find((toolName: string) => this.ASSERTION_TOOLS.includes(toolName)); const wasSuccessful = result?.toolExecutions?.every((execution: any) => execution.wasSuccessful); - if (successfulActionPerformed) { - this.hasSuccessfulAssertion = false; - } - if (assertionPerformed) { - this.hasSuccessfulAssertion = true; - } - this.trackToolExecutions(result?.toolExecutions || []); if (this.consecutiveEmptyResults >= 5) { @@ -416,6 +406,7 @@ export class Tester extends TaskAgent implements Agent { if (extensions >= this.MAX_EXTENSIONS) break; extensions++; + this.stalledIterations = 0; tag('info').log(`Pilot extending test (${extensions}/${this.MAX_EXTENSIONS})`); conversation.cleanupTag('page_aria', '...trimmed...', 1); conversation.cleanupTag('page_html', '...trimmed...', 0); @@ -474,13 +465,7 @@ export class Tester extends TaskAgent implements Agent { this.stalledIterations++; if (this.stalledIterations < this.MAX_STALLED_ITERATIONS) return false; - if (this.hasSuccessfulAssertion) { - task.addNote('No further browser progress after successful verification; requesting final review'); - return true; - } - - task.addNote('No browser progress after repeated attempts on unchanged page', TestResult.FAILED); - task.finish(TestResult.FAILED); + task.addNote('No further browser progress on unchanged page; requesting final review'); return true; } @@ -810,6 +795,9 @@ export class Tester extends TaskAgent implements Agent { ${task.expected.map((e) => `- ${e}`).join('\n')} </expected_results> + An expected result counts as settled only when you record it back word for word as it is written above. + A note in your own wording is a general note and leaves that result unsettled. + Your goal is to perform actions on the web page and verify the expected outcomes. Try to achieve as many goals as possible. If goal is not achievable, log that and skip to next one. @@ -1059,6 +1047,9 @@ export class Tester extends TaskAgent implements Agent { - You unsuccessfully tried multiple iterations and failed - If the expected result was expected to fail, use status="success" instead + When a note settles one of the expected results, that note must repeat the expected result word + for word. Paraphrasing it leaves the expected result unsettled and it is reported as unverified. + Example: - record({ notes: ["clicked login button", "login form appeared", "fill credentials"], status: "success" }) `, @@ -1143,8 +1134,7 @@ export class Tester extends TaskAgent implements Agent { this.stalledIterations++; if (this.stalledIterations < this.MAX_STALLED_ITERATIONS) return false; - task.addNote('No browser progress after repeated execution errors', TestResult.FAILED); - task.finish(TestResult.FAILED); + task.addNote('No browser progress after repeated execution errors; requesting final review'); return true; } diff --git a/src/explorer.ts b/src/explorer.ts index cf490d1e..2190e8ee 100644 --- a/src/explorer.ts +++ b/src/explorer.ts @@ -721,11 +721,11 @@ class Explorer { if (this.observedTestPages.has(page)) return; this.testPageErrorHandler ||= (err: Error) => { - this._activeTest?.addNote(`Console error: ${err.message}`, TestResult.FAILED); + this._activeTest?.addObservation(`Console error: ${err.message}`); }; this.testConsoleHandler ||= (msg: any) => { if (msg.type() !== 'error') return; - this._activeTest?.addNote(`Console error: ${msg.text()}`, TestResult.FAILED); + this._activeTest?.addObservation(`Console error: ${msg.text()}`); }; this.testDialogHandler ||= (dialog: any) => { const dialogType = dialog.type(); diff --git a/src/playwright-recorder.ts b/src/playwright-recorder.ts index 7f925b2b..71a18364 100644 --- a/src/playwright-recorder.ts +++ b/src/playwright-recorder.ts @@ -287,8 +287,31 @@ function formatSelectOption(options: any): string { return `[${values.map((v) => quote(v)).join(', ')}]`; } +function assertionLocator(target: any): string | null { + if (typeof target === 'string') return `page.locator(${JSON.stringify(target)})`; + if (!target || typeof target !== 'object') return null; + + const role = target.role || target.aria; + const name = target.text ?? target.name ?? target.label; + if (role && name) return `page.getByRole(${JSON.stringify(String(role))}, { name: ${JSON.stringify(String(name))} })`; + if (role) return `page.getByRole(${JSON.stringify(String(role))})`; + if (name) return `page.getByText(${JSON.stringify(String(name))})`; + if (target.css) return `page.locator(${JSON.stringify(String(target.css))})`; + if (target.xpath) return `page.locator(${JSON.stringify(`xpath=${target.xpath}`)})`; + return null; +} + export function renderAssertion(assertion: { name: string; args: any[] }): string { const args = assertion.args; + const target = assertionLocator(args[0]); + + if (target) { + if (assertion.name === 'seeElement') return `await expect(${target}).toBeVisible();`; + if (assertion.name === 'dontSeeElement') return `await expect(${target}).toBeHidden();`; + if (assertion.name === 'seeInField' && args[1] !== undefined) return `await expect(${target}).toHaveValue(${JSON.stringify(String(args[1]))});`; + if (assertion.name === 'dontSeeInField' && args[1] !== undefined) return `await expect(${target}).not.toHaveValue(${JSON.stringify(String(args[1]))});`; + } + if (assertion.name === 'see' && typeof args[0] === 'string') { return `await expect(page).toContainText(${JSON.stringify(args[0])});`; } diff --git a/src/test-plan.ts b/src/test-plan.ts index f9f83d2a..98f18eac 100644 --- a/src/test-plan.ts +++ b/src/test-plan.ts @@ -27,6 +27,7 @@ export interface Note { endTime: number; screenshot?: string; log?: string; + observation?: boolean; } export class ActiveNote { @@ -131,6 +132,14 @@ export class Task { this.notes[timestamp] = { message, status, startTime: now, endTime: now, screenshot, log }; } + addObservation(message: string): void { + const isDuplicate = Object.values(this.notes).some((note) => note.message === message); + if (isDuplicate) return; + + const now = performance.now(); + this.notes[`${now}_${this.timestampCounter++}`] = { message, status: TestResult.FAILED, startTime: now, endTime: now, observation: true }; + } + addUrlNote(state: UrlNoteState, prevState?: { title?: string; h1?: string; h2?: string }): void { const fullUrl = state.fullUrl || state.url; if (!fullUrl) return; diff --git a/src/utils/aria.ts b/src/utils/aria.ts index 36a11f68..fbe02a31 100644 --- a/src/utils/aria.ts +++ b/src/utils/aria.ts @@ -191,7 +191,9 @@ const dropEmpty = (nodes: AriaNode[], opts: { keepNamed?: boolean } = {}): AriaN // ───────────────────────────────────────────────────────────────── // One-line representation of a node. Stable attr order so diff comparisons are deterministic. -const formatNode = (node: AriaNode): string => { +const MAX_INLINE_VALUE = 400; + +const formatNode = (node: AriaNode, offload?: ValueOffload): string => { let line = node.role; if (node.name?.trim()) line += ` "${node.name.trim()}"`; const attrStr = Object.keys(node.attributes) @@ -207,11 +209,19 @@ const formatNode = (node: AriaNode): string => { if (attrStr) line += ` [${attrStr}]`; if (node.value !== undefined && node.value !== null) { const text = String(node.value).trim(); - if (text) line += `: ${text}`; + if (text && text.length <= MAX_INLINE_VALUE) line += `: ${text}`; + if (text && text.length > MAX_INLINE_VALUE) line += `: ${offloadValue(text, offload)}`; } return line; }; +const offloadValue = (text: string, offload?: ValueOffload): string => { + const head = `${text.slice(0, MAX_INLINE_VALUE)}…`; + const reference = offload?.(text); + if (!reference) return `${head} (${text.length} chars, truncated)`; + return `${head} (${text.length} chars) [Full Text: ${reference}]`; +}; + // Group consecutive same-role siblings. [a,a,b,a,a,a] → [[a,a],[b],[a,a,a]] const groupByConsecutiveRole = (nodes: AriaNode[]): AriaNode[][] => nodes.reduce<AriaNode[][]>((groups, node) => { @@ -238,15 +248,15 @@ const collapseGroup = (group: AriaNode[], depth: number): RenderEntry[] => { const collapseSiblingGroups = (nodes: AriaNode[], depth: number): RenderEntry[] => groupByConsecutiveRole(nodes).flatMap((group) => collapseGroup(group, depth)); // Tree → indented YAML text. -const renderTree = (nodes: AriaNode[], depth = 0): string => +const renderTree = (nodes: AriaNode[], depth = 0, offload?: ValueOffload): string => collapseSiblingGroups(nodes, depth) .map((entry) => { if ('placeholder' in entry) return entry.placeholder; const { node } = entry; const indent = ' '.repeat(depth); - const head = `${indent}- ${formatNode(node)}`; + const head = `${indent}- ${formatNode(node, offload)}`; if (node.children.length === 0) return head; - return `${head}:\n${renderTree(node.children, depth + 1)}`; + return `${head}:\n${renderTree(node.children, depth + 1, offload)}`; }) .join('\n'); @@ -381,6 +391,40 @@ const detectToggles = (prev: FlatEntry[], curr: FlatEntry[]): { toggled: string[ return { toggled, togglePaths }; }; +const detectValueChanges = (prev: FlatEntry[], curr: FlatEntry[]): { typed: string[]; typedPaths: Set<string> } => { + const typed: string[] = []; + const typedPaths = new Set<string>(); + const currByPath = new Map(curr.map((e) => [e.path, e])); + + for (const before of prev) { + const after = currByPath.get(before.path); + if (!after) continue; + if (before.entry.role !== after.entry.role) continue; + if (before.entry.name !== after.entry.name) continue; + + const was = valueWord(before.entry.value); + const now = valueWord(after.entry.value); + if (was === now) continue; + + typedPaths.add(before.path); + let label = String(after.entry.role); + const name = after.entry.name; + if (typeof name === 'string' && name.trim()) label += ` "${name.trim()}"`; + typed.push(`${label}: ${was} -> ${now}`); + } + return { typed, typedPaths }; +}; + +const VALUE_EXCERPT = 60; + +const valueWord = (value: unknown): string => { + if (value === undefined || value === null) return 'empty'; + const text = String(value).trim(); + if (!text) return 'empty'; + if (text.length <= VALUE_EXCERPT) return JSON.stringify(text); + return `${JSON.stringify(text.slice(0, VALUE_EXCERPT))}… (${text.length} chars)`; +}; + const TOP_DIFF_ITEMS = 10; const formatDiffSection = (label: string, items: string[]): string[] => { @@ -405,9 +449,13 @@ const formatDiffSection = (label: string, items: string[]): string[] => { return lines; }; -const formatDiff = (added: string[], removed: string[], toggled: string[]): string | null => { - if (added.length === 0 && removed.length === 0 && toggled.length === 0) return null; +const formatDiff = (added: string[], removed: string[], toggled: string[], typed: string[] = []): string | null => { + if (added.length === 0 && removed.length === 0 && toggled.length === 0 && typed.length === 0) return null; const sections = ['ariaDiff:']; + if (typed.length > 0) { + sections.push(' typed:'); + for (const line of typed) sections.push(` - ${line}`); + } if (toggled.length > 0) { sections.push(' toggled:'); for (const line of toggled) sections.push(` - ${line}`); @@ -473,13 +521,15 @@ const findDialogOrModal = (nodes: AriaNode[]): FocusAreaResult | null => { // Public API — pipelines composed visibly, top-to-bottom // ───────────────────────────────────────────────────────────────── -export const compactAriaSnapshot = (snapshot: string | null, keepNamed = false): string => { +export type ValueOffload = (value: string) => string | undefined; + +export const compactAriaSnapshot = (snapshot: string | null, keepNamed = false, offload?: ValueOffload): string => { if (!snapshot) return ''; let tree = parseSnapshot(snapshot); tree = unwrapIgnored(tree); tree = nameIconButtons(tree); tree = dropEmpty(tree, { keepNamed }); - return renderTree(tree); + return renderTree(tree, 0, offload); }; export const diffAriaSnapshots = (previous: string | null, current: string | null): AriaDiff => { @@ -493,15 +543,17 @@ export const diffAriaSnapshots = (previous: string | null, current: string | nul const prevAll = flat(previous); const currAll = flat(current); const { toggled, togglePaths } = detectToggles(prevAll, currAll); - const prev = prevAll.filter((e) => !togglePaths.has(e.path)); - const curr = currAll.filter((e) => !togglePaths.has(e.path)); + const { typed, typedPaths } = detectValueChanges(prevAll, currAll); + const skip = (entry: FlatEntry) => togglePaths.has(entry.path) || typedPaths.has(entry.path); + const prev = prevAll.filter((e) => !skip(e)); + const curr = currAll.filter((e) => !skip(e)); const prevTotals = countBy(prev.map((e) => e.summary)); const currTotals = countBy(curr.map((e) => e.summary)); const byCount = diffByCount(prevTotals, currTotals); const renames = detectRenames(prev, curr, prevTotals, currTotals); const added = [...byCount.added, ...renames.added]; const removed = [...byCount.removed, ...renames.removed]; - return { text: formatDiff(added, removed, toggled), count: added.length + removed.length + toggled.length }; + return { text: formatDiff(added, removed, toggled, typed), count: added.length + removed.length + toggled.length + typed.length }; }; export const detectFocusArea = (snapshot: string | null): FocusAreaResult => { diff --git a/src/utils/logger.ts b/src/utils/logger.ts index e7e4173e..ac3319e2 100644 --- a/src/utils/logger.ts +++ b/src/utils/logger.ts @@ -79,9 +79,11 @@ const debugFilter = new DebugFilter(); class ConsoleDestination implements LogDestination { private verboseMode = false; private forceEnabled = false; + private quiet = false; private recentSteps = new RecentStepFilter(); isEnabled(): boolean { + if (this.quiet) return false; return this.forceEnabled || !process.env.INK_RUNNING; } @@ -89,6 +91,10 @@ class ConsoleDestination implements LogDestination { this.forceEnabled = enabled; } + setQuiet(enabled: boolean): void { + this.quiet = enabled; + } + setVerboseMode(enabled: boolean): void { this.verboseMode = enabled; } @@ -360,6 +366,10 @@ class Logger { this.console.forceEnable(enabled); } + setQuietMode(enabled: boolean): void { + this.console.setQuiet(enabled); + } + isVerboseMode(): boolean { return this.debugDestination.isEnabled(); } @@ -537,6 +547,7 @@ export const startLogCapture = () => logger.captain.startCapture(); export const stopLogCapture = () => logger.captain.stopCapture(); export const setVerboseMode = (enabled: boolean) => logger.setVerboseMode(enabled); export const setPreserveConsoleLogs = (enabled: boolean) => logger.setPreserveConsoleLogs(enabled); +export const setQuietMode = (enabled: boolean) => logger.setQuietMode(enabled); export const isVerboseMode = () => logger.isVerboseMode(); export const setDebugMode = (enabled: boolean) => logger.setDebugMode(enabled); export const isDebugMode = () => logger.isDebugMode(); diff --git a/src/utils/page-readiness.ts b/src/utils/page-readiness.ts index 2a30b830..2f8276d2 100644 --- a/src/utils/page-readiness.ts +++ b/src/utils/page-readiness.ts @@ -4,10 +4,33 @@ export async function waitForPageReadiness(page: any, options: PageReadinessOpti const timeout = options.timeout ?? 6000; await page.waitForLoadState?.('domcontentloaded', { timeout })?.catch(() => {}); - await Promise.race([waitForNetworkIdle(page, timeout), waitForVisibleSpinnersHidden(page, options.spinnerSelectors || [], timeout), sleep(timeout)]).catch(() => {}); + await Promise.race([waitForNetworkIdle(page, timeout), waitForDomQuiet(page, timeout), waitForVisibleSpinnersHidden(page, options.spinnerSelectors || [], timeout), sleep(timeout)]).catch(() => {}); await waitForPageBodyContent(page, timeout); } +const DOM_QUIET_MS = 350; + +function waitForDomQuiet(page: any, timeout: number): Promise<void> { + if (!page?.waitForFunction) return new Promise(() => {}); + + return page + .waitForFunction( + (quiet: number) => { + const store = window as any; + if (!store.__explorbotDomQuiet) { + store.__explorbotDomQuiet = { last: Date.now() }; + new MutationObserver(() => { + store.__explorbotDomQuiet.last = Date.now(); + }).observe(document, { subtree: true, childList: true, attributes: true, characterData: true }); + } + return Date.now() - store.__explorbotDomQuiet.last >= quiet; + }, + DOM_QUIET_MS, + { timeout, polling: 100 } + ) + .catch(() => {}); +} + function waitForNetworkIdle(page: any, timeout: number): Promise<void> { if (!page?.waitForLoadState) return Promise.resolve(); return page.waitForLoadState('networkidle', { timeout }).catch(() => {}); diff --git a/src/utils/web-sandbox.ts b/src/utils/web-sandbox.ts index 06382ed7..00a02e35 100644 --- a/src/utils/web-sandbox.ts +++ b/src/utils/web-sandbox.ts @@ -28,13 +28,12 @@ export function playwrightSandbox(page: any, code: string): Promise<any> { return run(page); } -export function codeceptJSSandbox(actor: any, codeOrFn: string | ((...args: any[]) => void)): void { +export function codeceptJSSandbox(actor: any, codeOrFn: string | ((...args: any[]) => void)): any { if (typeof codeOrFn === 'function') { - codeOrFn(actor, tryTo, retryTo, within, hopeThat, step, faker); - return; + return codeOrFn(actor, tryTo, retryTo, within, hopeThat, step, faker); } const run = createSandbox(CODECEPT_ARG_NAMES, codeOrFn); - run(actor, tryTo, retryTo, within, hopeThat, step, faker); + return run(actor, tryTo, retryTo, within, hopeThat, step, faker); } function createSandbox(argNames: string[], body: string): (...args: any[]) => any { diff --git a/tests/integration/prima-smoke.test.ts b/tests/integration/prima-smoke.test.ts index c0cf911e..8ab7d669 100644 --- a/tests/integration/prima-smoke.test.ts +++ b/tests/integration/prima-smoke.test.ts @@ -127,13 +127,27 @@ describe('Prima drives a real page', () => { expect(envelope.page.url).toContain('note=the+hinge+arrived+bent'); }); - test('pw writes the aria and html artifacts to disk', async () => { + test('an action reports a status hash instead of artifact paths, and status resolves it', async () => { const envelope = await prima.pw("({ page }) => page.click('text=Submit')"); - expect(existsSync(envelope.artifacts!.aria)).toBe(true); - expect(existsSync(envelope.artifacts!.html)).toBe(true); - expect(await Bun.file(envelope.artifacts!.aria).text()).toContain('Thanks for the note'); - expect(await Bun.file(envelope.artifacts!.html).text()).toContain('Thanks for the note'); + expect(envelope.status).toMatch(/^[0-9a-f]{15}$/); + expect(envelope.artifacts).toBeUndefined(); + expect(renderEnvelope(envelope)).toContain(`prima status ${envelope.status}`); + + const status = await prima.status(envelope.status!); + + expect(status.ok).toBe(true); + expect(existsSync(status.artifacts!.aria)).toBe(true); + expect(existsSync(status.artifacts!.html)).toBe(true); + expect(await Bun.file(status.artifacts!.aria).text()).toContain('Thanks for the note'); + expect(await Bun.file(status.artifacts!.html).text()).toContain('Thanks for the note'); + }); + + test('status on an unknown hash is a tool error, not a crash', async () => { + const envelope = await prima.status('000000000000000'); + + expect(envelope.ok).toBe(false); + expect(envelope.failure?.error).toContain('No command was recorded'); }); test('a non-function argument is a tool error and leaves the page alone', async () => { diff --git a/tests/unit/aria.test.ts b/tests/unit/aria.test.ts index 4a0fda6b..da6e00f5 100644 --- a/tests/unit/aria.test.ts +++ b/tests/unit/aria.test.ts @@ -161,6 +161,90 @@ describe('aria', () => { expect(compactAriaSnapshot(snapshot, true)).toContain(`[${expected}]`); }); + it('reports a typed value as a typed change, not as add/remove churn', () => { + const diff = diffAriaSnapshots('- textbox "Title"', '- textbox "Title": Bench probe'); + + expect(diff.text).toBe(['ariaDiff:', ' typed:', ' - textbox "Title": empty -> "Bench probe"', ' added: []', ' removed: []'].join('\n')); + expect(diff.count).toBe(1); + }); + + it('reports a value being replaced', () => { + const diff = diffAriaSnapshots('- textbox "Title": Bench probe', '- textbox "Title": Bench probe renamed'); + + expect(diff.text).toContain('- textbox "Title": "Bench probe" -> "Bench probe renamed"'); + expect(diff.count).toBe(1); + }); + + it('reports a value being cleared', () => { + const diff = diffAriaSnapshots('- textbox "Search": shoes', '- textbox "Search"'); + + expect(diff.text).toContain('- textbox "Search": "shoes" -> empty'); + }); + + it('treats a whitespace-only value as empty', () => { + expect(diffAriaSnapshots('- textbox "Title"', '- textbox "Title": ').text).toBeNull(); + }); + + it('reports an unchanged value as no diff at all', () => { + expect(diffAriaSnapshots('- textbox "Title": same', '- textbox "Title": same').text).toBeNull(); + }); + + it('reports a typed value alongside a toggle in the same diff', () => { + const before = ['- textbox "Title"', '- checkbox "Agree"'].join('\n'); + const after = ['- textbox "Title": Bench probe', '- checkbox "Agree" [checked]'].join('\n'); + + const diff = diffAriaSnapshots(before, after); + + expect(diff.text).toContain('typed:'); + expect(diff.text).toContain('- textbox "Title": empty -> "Bench probe"'); + expect(diff.text).toContain('toggled:'); + expect(diff.text).toContain('- checkbox "Agree": unchecked -> checked'); + expect(diff.count).toBe(2); + }); + + it('keeps a genuinely new field as added rather than typed', () => { + const diff = diffAriaSnapshots('- textbox "Title"', ['- textbox "Title"', '- textbox "Emoji": x'].join('\n')); + + expect(diff.text).toContain('added:'); + expect(diff.text).toContain('textbox "Emoji": x'); + expect(diff.text).not.toContain('typed:'); + }); + + it('excerpts a long typed value instead of printing it whole', () => { + const long = 'x'.repeat(5000); + const diff = diffAriaSnapshots(`- textbox "Editor": ${long}`, `- textbox "Editor": ${long}y`); + + expect(diff.text).toContain('(5000 chars)'); + expect(diff.text).toContain('(5001 chars)'); + expect(diff.text!.length).toBeLessThan(400); + }); + + it('truncates a long value and references the file the caller stored it in', () => { + const long = 'x'.repeat(5000); + const stored: string[] = []; + + const out = compactAriaSnapshot(`- textbox "Editor": ${long}`, true, (value) => { + stored.push(value); + return 'abc123/value-deadbeef.txt'; + }); + + expect(out).toContain('(5000 chars) [Full Text: abc123/value-deadbeef.txt]'); + expect(out.length).toBeLessThan(600); + expect(stored[0]).toBe(long); + }); + + it('truncates a long value with a char count when no store is given', () => { + const out = compactAriaSnapshot(`- textbox "Editor": ${'y'.repeat(5000)}`, true); + + expect(out).toContain('(5000 chars, truncated)'); + expect(out).not.toContain('[Full Text:'); + expect(out.length).toBeLessThan(600); + }); + + it('leaves a short value inline', () => { + expect(compactAriaSnapshot('- textbox "Name": Bench probe', true)).toContain(': Bench probe'); + }); + it('keeps refs on stateful nodes nested in a tree', () => { const snapshot = ['- navigation "Panel sections" [ref=e6]:', ' - button "Workspace" [ref=e10]', ' - button "Workflows" [active] [ref=e13]'].join('\n'); diff --git a/tests/unit/page-readiness.test.ts b/tests/unit/page-readiness.test.ts index 51bf8ae5..c46eeccb 100644 --- a/tests/unit/page-readiness.test.ts +++ b/tests/unit/page-readiness.test.ts @@ -25,6 +25,16 @@ describe('page readiness', () => { expect(page.waitedSelectors).toEqual(['.spinner']); }); + it('finishes as soon as the dom goes quiet, without waiting for network idle', async () => { + const page = new FakePage([], 5_000, 5); + + const started = Date.now(); + await waitForPageReadiness(page, { timeout: 5_000 }); + + expect(page.waitedForDomQuiet).toBe(true); + expect(Date.now() - started).toBeLessThan(1_000); + }); + it('does not finish from spinner selectors that are not visible', async () => { const page = new FakePage([], 40); let ready = false; @@ -51,10 +61,12 @@ class FakePage { loadStates: string[] = []; waitedSelectors: string[] = []; waitedForBodyContent = false; + waitedForDomQuiet = false; constructor( private visibleSelectors: string[] = [], - private networkIdleDelay = 0 + private networkIdleDelay = 0, + private domQuietDelay = 10_000 ) {} async waitForLoadState(state: string): Promise<void> { @@ -76,7 +88,12 @@ class FakePage { }; } - async waitForFunction(): Promise<void> { + async waitForFunction(_fn: unknown, arg?: unknown): Promise<void> { + if (typeof arg === 'number') { + await sleep(this.domQuietDelay); + this.waitedForDomQuiet = true; + return; + } this.waitedForBodyContent = true; } } diff --git a/tests/unit/playwright-recorder.test.ts b/tests/unit/playwright-recorder.test.ts index 3352cf4c..f8d9a841 100644 --- a/tests/unit/playwright-recorder.test.ts +++ b/tests/unit/playwright-recorder.test.ts @@ -182,8 +182,18 @@ describe('renderAssertion', () => { expect(renderAssertion(assertion('seeHttpHeader', 'X-Api', 'v1'))).toBe(`// TODO(playwright): seeHttpHeader("X-Api", "v1")`); }); - it('falls back to TODO when seeElement arg is not a string', () => { - expect(renderAssertion(assertion('seeElement', { css: '.x' }))).toMatch(/^\/\/ TODO\(playwright\)/); + it('renders an ARIA locator as getByRole rather than a TODO', () => { + expect(renderAssertion(assertion('seeElement', { role: 'button', text: 'Add workflow' }))).toBe(`await expect(page.getByRole("button", { name: "Add workflow" })).toBeVisible();`); + expect(renderAssertion(assertion('dontSeeElement', { role: 'alert', text: 'Error' }))).toBe(`await expect(page.getByRole("alert", { name: "Error" })).toBeHidden();`); + expect(renderAssertion(assertion('seeInField', { role: 'textbox', text: 'Email' }, 'a@b.c'))).toBe(`await expect(page.getByRole("textbox", { name: "Email" })).toHaveValue("a@b.c");`); + }); + + it('renders a css-keyed object locator without a TODO', () => { + expect(renderAssertion(assertion('seeElement', { css: '.x' }))).toBe(`await expect(page.locator(".x")).toBeVisible();`); + }); + + it('still falls back to TODO when the target cannot be expressed', () => { + expect(renderAssertion(assertion('seeElement', { weird: 1 }))).toMatch(/^\/\/ TODO\(playwright\)/); }); it('falls back to TODO when seeInField is missing the value', () => { diff --git a/tests/unit/tester-focus-scope.test.ts b/tests/unit/tester-focus-scope.test.ts index 6adfeb4e..3b89af59 100644 --- a/tests/unit/tester-focus-scope.test.ts +++ b/tests/unit/tester-focus-scope.test.ts @@ -148,17 +148,28 @@ describe('Tester experience context', () => { }); describe('Tester stalled execution', () => { - it('hands a verified scenario to final review without marking it failed', () => { + it('hands a stalled scenario to final review without deciding the verdict itself', () => { const tester = buildTester(); const task = new Test('filter items', 'normal', 'filtered items appear', '/page'); const state = buildState('- main:', '/page'); (tester as any).stateManager.getCurrentState = () => state; - (tester as any).hasSuccessfulAssertion = true; expect((tester as any).shouldStopForStalledExecution(task, state, [])).toBe(false); expect((tester as any).shouldStopForStalledExecution(task, state, [])).toBe(false); expect((tester as any).shouldStopForStalledExecution(task, state, [])).toBe(true); expect(task.hasFinished).toBe(false); - expect(task.getPrintableNotes()).toContain('No further browser progress after successful verification; requesting final review'); + expect(task.result).toBe(null); + expect(task.getPrintableNotes()).toContain('No further browser progress on unchanged page; requesting final review'); + }); + + it('hands repeated execution errors to final review without marking the test failed', () => { + const tester = buildTester(); + const task = new Test('filter items', 'normal', 'filtered items appear', '/page'); + + expect((tester as any).shouldStopAfterStalledLoopError(task)).toBe(false); + expect((tester as any).shouldStopAfterStalledLoopError(task)).toBe(false); + expect((tester as any).shouldStopAfterStalledLoopError(task)).toBe(true); + expect(task.hasFinished).toBe(false); + expect(task.result).toBe(null); }); }); From 6a098d902e4e3c0cbb00768c21277249bcda9573 Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Sun, 9 Aug 2026 12:12:59 +0300 Subject: [PATCH 03/39] Let do succeed when an instruction needs no action MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An instruction satisfied by absence — "dismiss the banner if one appeared" — or one that only verifies leaves no executed command behind. The "no action was performed" guard fired on both and reported failure for a sequence the model had closed cleanly. It now applies only when the model ended without calling done(). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- boat/prima/src/prima.ts | 2 +- boat/prima/tests/prima.test.ts | 14 ++++++++++++++ 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/boat/prima/src/prima.ts b/boat/prima/src/prima.ts index 922f3fc1..ad5bb9e4 100644 --- a/boat/prima/src/prima.ts +++ b/boat/prima/src/prima.ts @@ -203,7 +203,7 @@ export class Prima { return envelope; } - if (!used.length) { + if (!used.length && !report.called) { const reason = ['No action was performed for these instructions on the current page.', narration].filter(Boolean).join(' '); return this.failureEnvelope(command, reason, previousState); } diff --git a/boat/prima/tests/prima.test.ts b/boat/prima/tests/prima.test.ts index 59faf9fc..b7e195ca 100644 --- a/boat/prima/tests/prima.test.ts +++ b/boat/prima/tests/prima.test.ts @@ -577,6 +577,20 @@ describe('Prima.do', () => { expect(envelope.steps?.[0]).toMatchObject({ label: 'verify: unsaved indicator is visible', ok: false }); }); + test('an instruction satisfied without acting still succeeds when the model closes the sequence', async () => { + const { prima } = fakePrima(); + (prima as any).bot.getProvider = () => + fakeProvider(async (_conversation: unknown, tools: any) => { + await tools.done.execute({ summary: 'no cookie banner is present on this page', unmet: [] }); + return { toolExecutions: [{ toolName: 'done', input: {}, output: { success: true, action: 'done' }, wasSuccessful: true }] }; + }); + + const envelope = await prima.do(['dismiss the cookie banner if one appeared']); + + expect(envelope.ok).toBe(true); + expect(envelope.answer).toBe('no cookie banner is present on this page'); + }); + test('a check that passes on a retry does not fail the command', async () => { const { prima } = fakePrima(); let calls = 0; From e82707d74d89c04326b697b0d224c79d7b91bc95 Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Sun, 9 Aug 2026 12:50:04 +0300 Subject: [PATCH 04/39] do accounts for every instruction by number MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit do had no idea which instruction it was on. It re-attempted satisfied steps, kept acting past the end of the sequence, and its one self-report at the end could claim a success it never earned — measured against a live app, four do calls spent 227s of a 245s run thrashing. Instructions now carry numbers that never change. completed() reports the ones just satisfied, blocked() reports one the page cannot do, and what is still open is re-stated after each turn. The loop ends when nothing is open. Prima staples the actions it actually executed to each report, so a completed instruction with an empty trail is visible as such — an instruction satisfied by absence still passes, since conditionals legitimately complete without acting. ### Steps is now one line per instruction with its proof, and an instruction nobody reported is named as unaccounted for rather than silently passing. A stray failed action no longer fails a sequence whose instructions all closed. Also: prima reads the attached session's own URL, so --url is no longer required on every command; --pw-session and --url parse before the subcommand too; cached research is returned without a banner telling the model to refresh it; help leads with check and orders the tiers by what they cost. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- boat/prima/src/cli.ts | 82 ++++++------ boat/prima/src/prima.ts | 153 ++++++++++++++++++----- boat/prima/tests/prima.test.ts | 179 ++++++++++++++++++++------- src/ai/researcher.ts | 2 +- tests/integration/prima-do.test.ts | 12 +- tests/integration/researcher.test.ts | 5 +- 6 files changed, 314 insertions(+), 119 deletions(-) diff --git a/boat/prima/src/cli.ts b/boat/prima/src/cli.ts index 47673ae3..b86170dd 100644 --- a/boat/prima/src/cli.ts +++ b/boat/prima/src/cli.ts @@ -10,53 +10,42 @@ const helpContract = dedent` Prima drives a browser that is already open. One command per process; every command prints a plain-text envelope on stdout and exits 0 when ok, 1 when not. - TIERS - choose by what you hold, not by how hard the step looks - pw <fn> Precise. A Playwright function expression built from a locator you - already verified. No AI on the happy path. - prima pw "({ page }) => page.click('[data-test=submit]')" + TIERS - start at the top and come down only when the tier above cannot hold the work check <scenario> - A whole scenario run as a test: it drives the page, verifies the outcome - itself, and reports every step it took with the proof for each. - prima check "the workflow editor opens and a workflow can be saved" - do <steps...> Several described steps, run tester-style in one process. This is the - tier that pays: one process attaches once and carries the whole - sequence, so a run of six steps costs a fraction of six commands. - Reach for it whenever the next few steps are already known. + A whole behaviour, run as a test. It drives the page, verifies the + outcome itself, and reports each expected outcome with its proof. + One command for something you want a verdict on. Start here. + prima check "a workflow can be created and appears in the list" \\ + --expected "the new workflow is listed" + do <steps...> Several described steps, carried out in one process. Pass the WHOLE + remaining sequence - a run of eight steps costs a fraction of eight + commands, and that is the whole point of this tier. prima do "open the account menu" "choose the settings entry" \\ "switch the theme to dark" "check the change took effect" - Never pass a locator or a function expression to do - describe the target. - Never pass a description to pw - it takes executable code only. - do stops at the last step you gave it and never carries on past it. A step it could - not carry out is named under ### Failure and makes the command fail. - - LOOP - prima go <url|path|words> reach the page you want to work on - prima research once per new page; returns verified locators - prima pw "..." drive the page with those locators - prima verify "..." assert the outcome (prima ask "..." to inspect instead) - Fall back to do whenever research left you no locator to hold. + pw <fn> One Playwright function expression, from a locator you already + verified. No AI. For when you know exactly what to run. + prima pw "({ page }) => page.click('[data-test=submit]')" + Describe targets to check and do; give pw executable code only. Never mix the two. + Coming down a tier to run steps one at a time costs more than the tier above, in + both time and what you have to read - it is a fallback, not a default. CHECK - check runs a whole scenario the way a tester would: it plans, drives the page, and - verifies the outcome itself, then reports every step with the proof for each. Give it an outcome, not a click path - it decides how to get there. - prima check "a workflow can be created and appears in the list" --url http://app.test - prima check "signup rejects a duplicate email" --expected "an error names the email as taken" --expected "no second account is created" - --url opens that page first, when the browser is not already on it. --expected one outcome the run must reach; repeat it for several. Without it the - scenario text is the single expected outcome. - ### Expected outcomes echoes each one back as PASSED, FAILED or not verified - - "not verified" means the run never checked it, which is not the same as false. - ### Steps lists what ran; page problems seen on the way are reported separately - under ### Answer, because they are not step failures. - Prefer check over do when you want a verdict on a behaviour; prefer do when you - already know the steps and want them carried out. + scenario text is the single expected outcome. Each comes back under + ### Expected outcomes as PASSED, FAILED or not verified - "not verified" + means the run never checked it, which is not the same as false. + Page problems seen on the way are reported under ### Answer, not as step failures. + + DO + Each instruction is numbered and accounted for: ### Steps reports every one as ok or + FAIL with what proved it. One that could not be carried out fails the command and + says why. Nothing runs past the last instruction you gave. VERIFY - verify runs the assertions it can express and reports each one with PASSED or - FAILED, plus the playwright form of the ones that held. It does not decide whether - your claim is true - read the lines and decide. Assertions that ran are evidence; - "none ran" means the claim could not be expressed, which is not the same as false. + Reports each assertion it could express as PASSED or FAILED with its playwright form, + and gives no overall verdict - read the lines and decide. "none ran" means the claim + could not be expressed, which is not the same as false. ENVELOPE ### Result ok, command, used @@ -102,7 +91,10 @@ const helpContract = dedent` you pass. `; -function buildOptions(options: any): PrimaOptions { +let rootOptions: () => any = () => ({}); + +function buildOptions(subcommand: any): PrimaOptions { + const options = { ...rootOptions(), ...stripEmpty(subcommand) }; return { verbose: options.verbose || options.debug, config: options.config, @@ -121,6 +113,15 @@ function buildOptions(options: any): PrimaOptions { }; } +function stripEmpty(options: any): any { + const present: any = {}; + for (const [key, value] of Object.entries(options || {})) { + if (value === undefined) continue; + present[key] = value; + } + return present; +} + function addCommonOptions(cmd: Command): Command { return cmd .option('-v, --verbose', 'Enable verbose logging') @@ -173,7 +174,10 @@ async function runBrowser(options: any, run: (prima: Prima) => Promise<boolean>) export function createPrimaCommands(name = 'prima'): Command { const cmd = new Command(name); cmd.description('Drive an already-open browser one command at a time and report back in a plain-text envelope'); + cmd.option('--pw-session <title>', 'Title of the playwright-cli session to attach to'); + cmd.option('--url <url>', 'Page to open when the session has no page yet'); cmd.addHelpText('after', `\n${helpContract}`); + rootOptions = () => cmd.opts(); addCommonOptions(cmd.command('pw <fn>').description('Run a Playwright function expression against the open page')).action(async (fn, options) => { await runPrima(options, `pw ${fn}`, (prima) => prima.pw(fn)); diff --git a/boat/prima/src/prima.ts b/boat/prima/src/prima.ts index ad5bb9e4..795af281 100644 --- a/boat/prima/src/prima.ts +++ b/boat/prima/src/prima.ts @@ -35,6 +35,7 @@ const CONNECT_TIMEOUT = 3000; const requireLib = createRequire(import.meta.url); const VOLATILE_COLUMNS = ['CSS', 'XPath', 'Coordinates', 'eidx']; +const UNACCOUNTED: Record<string, string> = { open: 'never reported — the run ended with this instruction still open' }; function dropVolatileColumns(markdown: string): string { return mdq(markdown) @@ -67,6 +68,7 @@ export class Prima { private bot: ExplorBot; private artifactsDir?: string; private hash?: string; + private sessionUrl?: string; private server: { close: () => Promise<void> } | null = null; private attached: string | null = null; @@ -85,8 +87,14 @@ export class Prima { } async start(): Promise<void> { + let discovery: Discovery | undefined; + if (!this.options.endpoint) { + discovery = await this.discover(); + this.adoptSessionUrl(discovery); + } + const config = await this.loadConfig(); - await this.resolveBrowser(config); + await this.resolveBrowser(config, discovery); await this.bot.start(); if (!this.options.url) return; @@ -134,17 +142,19 @@ export class Prima { const conversation = provider.startConversation(this.instructionSystemPrompt(), AI_AGENT_NAME); const task = new Task(instructions.join('; '), previousState?.url || ''); const deps = { explorer: this.bot.getExplorer(), stateManager: this.bot.stateManager(), ai: provider }; - const report: { called: boolean; summary: string; unmet: string[] } = { called: false, summary: '', unmet: [] }; - const tools = { ...createCodeceptJSTools(deps, task), ...this.testerTools(deps), context: this.contextTool(), done: this.doneTool(report) }; + const ledger: LedgerEntry[] = instructions.map((text) => ({ text, status: 'open', proof: '', evidence: [] })); + const tools = { ...createCodeceptJSTools(deps, task), ...this.testerTools(deps), context: this.contextTool(), completed: this.completedTool(), blocked: this.blockedTool() }; conversation.addUserText(await this.instructionPrompt(instructions, await this.capturedResult(previousState))); const used: string[] = []; const checks = new Map<string, boolean>(); - const steps: Array<{ label: string; ok: boolean; proof: string }> = []; + const evidence: string[] = []; let failure: { code: string; message: string } | null = null; let aiError: unknown = null; let narration = ''; + let nudged = false; let contextHash = this.bot.stateManager().getCurrentState()?.hash; + let openList = this.openInstructions(ledger); for (let iteration = 1; iteration <= Math.min(instructions.length * ITERATIONS_PER_INSTRUCTION + 2, MAX_INSTRUCTION_ITERATIONS); iteration++) { const state = this.bot.stateManager().getCurrentState(); @@ -153,6 +163,12 @@ export class Prima { conversation.addUserText(await this.pageContext(ActionResult.fromState(state))); } + const remaining = this.openInstructions(ledger); + if (iteration > 1 && remaining !== openList) { + openList = remaining; + conversation.addUserText(`<remaining>\n${remaining}\n</remaining>`); + } + const invoked = await provider.invokeConversation(conversation, tools, { maxToolRoundtrips: MAX_TOOL_ROUNDTRIPS, agentName: AI_AGENT_NAME }).catch((error: unknown) => { aiError = error; return null; @@ -162,48 +178,82 @@ export class Prima { const executions = invoked.toolExecutions || []; if (!executions.length) { narration = invoked.response?.text?.trim() || ''; - break; + const unreported = this.openInstructions(ledger); + if (!unreported || nudged) break; + nudged = true; + conversation.addUserText(dedent` + These instructions are still unreported: + ${unreported} + + Report each one with completed() or blocked(). Do not act again on anything you have already carried out. + `); + continue; } for (const execution of executions) { const output = execution.output || {}; - if (output.action === 'done') continue; + if (output.action === 'completed') { + for (const number of execution.input?.numbers || []) { + const entry = ledger[number - 1]; + if (entry?.status !== 'open') continue; + entry.status = 'done'; + entry.proof = execution.input?.proof || ''; + entry.evidence = [...evidence]; + } + evidence.length = 0; + continue; + } + + if (output.action === 'blocked') { + const entry = ledger[(execution.input?.instruction || 0) - 1]; + if (entry?.status === 'open') { + entry.status = 'blocked'; + entry.proof = execution.input?.reason || ''; + entry.evidence = [...evidence]; + } + evidence.length = 0; + continue; + } if (output.action === 'verify' && !output.inexpressible) { const claim = execution.input?.assertion || 'verification'; let passed = execution.wasSuccessful; if (output.alreadyVerified) passed = output.verifications?.[claim] === true; checks.set(claim, passed); - steps.push({ label: `verify: ${claim}`, ok: passed, proof: output.code || output.message || '' }); + evidence.push(`verify: ${claim} => ${passed ? 'PASSED' : 'FAILED'}`); continue; } if (!execution.wasSuccessful) { failure = { code: output.code || '', message: output.message || 'action failed' }; - steps.push({ label: output.code || execution.toolName || 'action', ok: false, proof: output.message || '' }); + evidence.push(`FAILED ${output.code || execution.toolName || 'action'}: ${output.message || ''}`); continue; } const codes = this.executedCodes(output.code); used.push(...codes); - steps.push({ label: codes.join('; ') || execution.toolName || 'action', ok: true, proof: output.pageDiff?.ariaChanges || '' }); + evidence.push(...codes); + if (output.pageDiff?.ariaChanges) evidence.push(output.pageDiff.ariaChanges); failure = null; } - if (report.called) break; + if (ledger.every((entry) => entry.status !== 'open')) break; } if (aiError) return this.failureEnvelope(command, aiError, previousState); - if (failure) { + const steps = ledger.map((entry) => ({ label: entry.text, ok: entry.status === 'done', proof: [entry.proof || UNACCOUNTED[entry.status], ...entry.evidence].filter(Boolean).join('\n') })); + const unfinished = ledger.filter((entry) => entry.status !== 'done'); + + if (failure && unfinished.length) { const envelope = await this.failureEnvelope(command, failure.message, previousState); envelope.used = used; envelope.steps = steps; return envelope; } - if (!used.length && !report.called) { + if (!used.length && unfinished.length === ledger.length) { const reason = ['No action was performed for these instructions on the current page.', narration].filter(Boolean).join(' '); return this.failureEnvelope(command, reason, previousState); } @@ -211,9 +261,8 @@ export class Prima { const result = await this.capturedResult(this.bot.stateManager().getCurrentState()); const envelope = await this.successEnvelope(command, used, result, previousState); envelope.steps = steps; - if (report.summary) envelope.answer = report.summary; - const unmet = [...report.unmet]; + const unmet = unfinished.map((entry) => `${entry.status}: ${entry.text}${entry.proof ? ` — ${entry.proof}` : ''}`); for (const [claim, passed] of checks) { if (!passed) unmet.push(`unproven: ${claim}`); } @@ -224,6 +273,14 @@ export class Prima { return envelope; } + private openInstructions(ledger: LedgerEntry[]): string { + return ledger + .map((entry, index) => ({ entry, number: index + 1 })) + .filter(({ entry }) => entry.status === 'open') + .map(({ entry, number }) => `${number}. ${entry.text}`) + .join('\n'); + } + async check(scenario: string, expected: string[] = []): Promise<EnvelopeData> { const command = `check ${scenario}`; const guard = await this.aiGuard(command); @@ -396,13 +453,22 @@ export class Prima { } private configBaseUrl(): string | undefined { - const url = this.options.baseUrl || this.options.url; + const url = this.options.baseUrl || this.options.url || this.sessionUrl; if (!url) return undefined; if (!URL.canParse(url)) return undefined; return url; } - private async resolveBrowser(config: ExplorbotConfig): Promise<void> { + private adoptSessionUrl(discovery: Discovery): void { + if (this.options.baseUrl || this.options.url) return; + + const url = discovery.browser?.contexts()[0]?.pages()[0]?.url(); + if (!url?.startsWith('http')) return; + this.sessionUrl = new URL(url).origin; + this.bot.getOptions().baseUrl = this.sessionUrl; + } + + private async resolveBrowser(config: ExplorbotConfig, discovered?: Discovery): Promise<void> { if (this.options.endpoint) { const endpoint = this.options.endpoint; const browserName = config.playwright.browser || 'chromium'; @@ -415,7 +481,7 @@ export class Prima { `); } - const { match, candidates, browser } = await this.discover(); + const { match, candidates, browser } = discovered || (await this.discover()); if (match && (await this.attachToEndpoint(match, browser))) return; if (!match && candidates.length) { @@ -542,13 +608,21 @@ export class Prima { </role> <approach> - 1. Read the page context and perform the instructions in the order they are listed. + 1. Read the page context and carry out the instructions in the order they are listed. 2. Interact with the page only through the provided tools. 3. Pick the smallest interaction that fulfills an instruction, then move to the next one. 4. After the page changes, work from the updated context you are given, not from the earlier one. - 5. Call done() once the last instruction is carried out, or once it is clear it cannot be on this page — list the ones you could not carry out there. + 5. Account for every instruction: completed() as soon as one is satisfied, blocked() when the page cannot do what it asks. </approach> + <ledger> + Instructions are numbered and those numbers never change. Report by number. + Saying in your reply that something is done does not report it — only completed() does. Nothing you write is read as a report. + An instruction you have reported is finished. Never act on it again, and never report it twice. + You are shown what is still open after each turn. When nothing is open the run is over. + Reaching for blocked() after a couple of honest attempts costs less than a third attempt that fails the same way. + </ledger> + <scope> Do only what the instructions ask. An action that looks helpful but was not asked for is out of scope — report it as something you noticed, never perform it. Continuing past the last instruction is a failure, even when the next step seems obvious. @@ -563,7 +637,8 @@ export class Prima { <proof> An instruction is done only when a change on the page shows it. After each action read the reported change and decide which part of it proves the instruction. - When nothing observable followed, say the instruction is unproven and why. Do not restate the action as if it were the outcome. + That part is what completed() takes as its proof. Do not restate the action as if it were the outcome. + An instruction that only inspects the page is satisfied by what you can see, including seeing that something is absent — those need no action at all. </proof> <targets> @@ -599,22 +674,31 @@ export class Prima { return tools; } - private doneTool(report: { called: boolean; summary: string; unmet: string[] }): any { + private completedTool(): any { return tool({ description: dedent` - End the sequence. Call it once every instruction is either carried out or established as impossible on this page. - Nothing runs after this call, so do not call it while an instruction is still worth another attempt. + Report the instructions you have just satisfied, by their number. Report several together when one turn satisfied several. + A reported instruction is finished — you will not be asked for it again and must not act on it again. `, inputSchema: z.object({ - summary: z.string().describe('One sentence on what the page shows now that proves the instructions were carried out'), - unmet: z.array(z.string()).describe('Instructions that could not be carried out, each with what blocked it. Empty when all of them were carried out.'), + numbers: z.array(z.number()).describe('Numbers of the instructions now satisfied, as they are numbered in the instruction list'), + proof: z.string().describe('What on the page shows they are satisfied'), }), - execute: async ({ summary, unmet }) => { - report.called = true; - report.summary = summary; - report.unmet = unmet || []; - return { success: true, action: 'done' }; - }, + execute: async () => ({ success: true, action: 'completed' }), + }); + } + + private blockedTool(): any { + return tool({ + description: dedent` + Report one instruction that cannot be carried out on this page, by its number. Reach for this instead of trying the same thing again. + The rest of the sequence continues without it. + `, + inputSchema: z.object({ + instruction: z.number().describe('Number of the instruction that cannot be carried out'), + reason: z.string().describe('What stopped it — what you looked for and what the page showed instead'), + }), + execute: async () => ({ success: true, action: 'blocked' }), }); } @@ -879,6 +963,13 @@ interface Discovery { browser?: Browser; } +interface LedgerEntry { + text: string; + status: 'open' | 'done' | 'blocked'; + proof: string; + evidence: string[]; +} + export interface PrimaOptions { verbose?: boolean; config?: string; diff --git a/boat/prima/tests/prima.test.ts b/boat/prima/tests/prima.test.ts index b7e195ca..b007cdd7 100644 --- a/boat/prima/tests/prima.test.ts +++ b/boat/prima/tests/prima.test.ts @@ -51,6 +51,14 @@ function toolExecution(code: string, success = true, message?: string) { return { toolName: 'click', input: { commands: [code] }, output: { success, code, message }, wasSuccessful: success }; } +function completedExecution(numbers: number[], proof: string) { + return { toolName: 'completed', input: { numbers, proof }, output: { success: true, action: 'completed' }, wasSuccessful: true }; +} + +function blockedExecution(instruction: number, reason: string) { + return { toolName: 'blocked', input: { instruction, reason }, output: { success: true, action: 'blocked' }, wasSuccessful: true }; +} + function fakeProvider(invokeConversation: (...args: any[]) => Promise<any>, prompts: string[] = []) { return { startConversation: () => ({ addUserText: (text: string) => prompts.push(text) }), @@ -423,7 +431,7 @@ describe('Prima.do', () => { fakeProvider(async () => { calls++; if (calls > 1) return { toolExecutions: [] }; - return { toolExecutions: [toolExecution("I.click('Login')")] }; + return { toolExecutions: [toolExecution("I.click('Login')"), completedExecution([1, 2], 'the invoice PDF is open')] }; }, prompts); const envelope = await prima.do(['open the first invoice', 'download its PDF']); @@ -433,7 +441,113 @@ describe('Prima.do', () => { expect(prompts.join('\n')).toContain('button "Sign in"'); expect(envelope.used).toEqual(["I.click('Login')"]); expect(envelope.ok).toBe(true); - expect(calls).toBe(2); + expect(calls).toBe(1); + }); + + test('every instruction is reported in order with the actions that proved it', async () => { + const { prima } = fakePrima(); + let calls = 0; + (prima as any).bot.getProvider = () => + fakeProvider(async () => { + calls++; + if (calls === 1) return { toolExecutions: [toolExecution("I.click('Invoices')"), completedExecution([1], 'the invoice list is open')] }; + return { toolExecutions: [toolExecution("I.click('Download')"), completedExecution([2], 'the PDF opened in a new tab')] }; + }); + + const envelope = await prima.do(['open the invoices page', 'download the PDF']); + + expect(envelope.ok).toBe(true); + expect(envelope.steps).toEqual([ + { label: 'open the invoices page', ok: true, proof: "the invoice list is open\nI.click('Invoices')" }, + { label: 'download the PDF', ok: true, proof: "the PDF opened in a new tab\nI.click('Download')" }, + ]); + }); + + test('an instruction reported without any action behind it says so rather than being rejected', async () => { + const { prima } = fakePrima(); + (prima as any).bot.getProvider = () => fakeProvider(async () => ({ toolExecutions: [completedExecution([1], 'no cookie banner is present on this page')] })); + + const envelope = await prima.do(['dismiss the cookie banner if one appeared']); + + expect(envelope.ok).toBe(true); + expect(envelope.steps).toEqual([{ label: 'dismiss the cookie banner if one appeared', ok: true, proof: 'no cookie banner is present on this page' }]); + }); + + test('the remaining instructions are re-stated as the ledger closes, and finished ones are not', async () => { + const { prima } = fakePrima(); + const prompts: string[] = []; + let calls = 0; + (prima as any).bot.getProvider = () => + fakeProvider(async () => { + calls++; + if (calls === 1) return { toolExecutions: [toolExecution("I.click('Invoices')"), completedExecution([1], 'list is open')] }; + return { toolExecutions: [toolExecution("I.click('Download')"), completedExecution([2], 'pdf is open')] }; + }, prompts); + + await prima.do(['open the invoices page', 'download the PDF']); + + const remaining = prompts.find((prompt) => prompt.startsWith('<remaining>')); + expect(remaining).toContain('2. download the PDF'); + expect(remaining).not.toContain('open the invoices page'); + }); + + test('a model that narrates instead of reporting is asked once for the ledger', async () => { + const { prima } = fakePrima(); + const prompts: string[] = []; + let calls = 0; + (prima as any).bot.getProvider = () => + fakeProvider(async () => { + calls++; + if (calls === 1) return { toolExecutions: [toolExecution("I.click('Invoices')")] }; + if (calls === 2) return { toolExecutions: [], response: { text: 'I opened the invoices page.' } }; + return { toolExecutions: [completedExecution([1], 'the invoice list is open')] }; + }, prompts); + + const envelope = await prima.do(['open the invoices page']); + + expect(prompts.join('\n')).toContain('still unreported'); + expect(envelope.ok).toBe(true); + expect(calls).toBe(3); + }); + + test('a model that will not report even when asked stops rather than looping', async () => { + const { prima } = fakePrima(); + let calls = 0; + (prima as any).bot.getProvider = () => + fakeProvider(async () => { + calls++; + if (calls === 1) return { toolExecutions: [toolExecution("I.click('Invoices')")] }; + return { toolExecutions: [], response: { text: 'I opened the invoices page.' } }; + }); + + const envelope = await prima.do(['open the invoices page']); + + expect(envelope.ok).toBe(false); + expect(calls).toBe(3); + }); + + test('an instruction the model never reported is named as unaccounted for', async () => { + const { prima } = fakePrima(); + (prima as any).bot.getProvider = () => fakeProvider(async () => ({ toolExecutions: [toolExecution("I.click('Invoices')"), completedExecution([1], 'list is open')] })); + + const envelope = await prima.do(['open the invoices page', 'download the PDF']); + + expect(envelope.ok).toBe(false); + expect(envelope.failure?.error).toContain('open: download the PDF'); + expect(envelope.steps?.[1]).toMatchObject({ label: 'download the PDF', ok: false }); + expect(envelope.steps?.[1].proof).toContain('never reported'); + }); + + test('a stray failed action does not fail a sequence whose instructions all closed', async () => { + const { prima } = fakePrima(); + (prima as any).bot.getProvider = () => + fakeProvider(async () => ({ + toolExecutions: [toolExecution("I.click('Invoices')"), completedExecution([1], 'list is open'), toolExecution("I.pressKey('End')", false, 'No element is focused')], + })); + + const envelope = await prima.do(['open the invoices page']); + + expect(envelope.ok).toBe(true); }); test('caps iterations at the instruction budget', async () => { @@ -502,9 +616,10 @@ describe('Prima.do', () => { await prima.do(['open the invoices page', 'read the first invoice']); - expect(prompts.length).toBe(2); - expect(prompts[0]).toContain('button "Sign in"'); - expect(prompts[1]).toContain('heading "Dashboard"'); + const contexts = prompts.filter((prompt) => prompt.includes('<page url=')); + expect(contexts.length).toBe(2); + expect(contexts[0]).toContain('button "Sign in"'); + expect(contexts[1]).toContain('heading "Dashboard"'); }); test('keeps the context when the state did not change between iterations', async () => { @@ -519,95 +634,73 @@ describe('Prima.do', () => { }, prompts); await prima.do(['open the invoices page', 'read the first invoice']); - expect(prompts.length).toBe(1); + expect(prompts.filter((prompt) => prompt.includes('<page url=')).length).toBe(1); }); - test('done() ends the sequence instead of burning the remaining iteration budget', async () => { + test('a closed ledger ends the sequence instead of burning the remaining iteration budget', async () => { const { prima } = fakePrima(); let calls = 0; (prima as any).bot.getProvider = () => - fakeProvider(async (_conversation: unknown, tools: any) => { + fakeProvider(async () => { calls++; - if (calls === 1) return { toolExecutions: [toolExecution("I.click('Invoices')")] }; - await tools.done.execute({ summary: 'the invoice list is open', unmet: [] }); - return { toolExecutions: [{ toolName: 'done', input: {}, output: { success: true, action: 'done' }, wasSuccessful: true }] }; + if (calls === 1) return { toolExecutions: [toolExecution("I.click('Invoices')"), completedExecution([1], 'the invoice list is open')] }; + return { toolExecutions: [completedExecution([2], 'the first invoice is on screen')] }; }); const envelope = await prima.do(['open the invoices page', 'read the first invoice']); expect(calls).toBe(2); expect(envelope.ok).toBe(true); - expect(envelope.answer).toBe('the invoice list is open'); - expect(envelope.steps?.map((step) => step.label)).toEqual(["I.click('Invoices')"]); }); - test('instructions the model could not carry out fail the command', async () => { + test('an instruction the model reports as blocked fails the command and keeps its reason', async () => { const { prima } = fakePrima(); (prima as any).bot.getProvider = () => - fakeProvider(async (_conversation: unknown, tools: any) => { - await tools.done.execute({ summary: 'the list is open', unmet: ['no PDF link exists on this page'] }); - return { toolExecutions: [toolExecution("I.click('Invoices')"), { toolName: 'done', input: {}, output: { success: true, action: 'done' }, wasSuccessful: true }] }; - }); + fakeProvider(async () => ({ + toolExecutions: [toolExecution("I.click('Invoices')"), completedExecution([1], 'the list is open'), blockedExecution(2, 'no PDF link exists on this page')], + })); const envelope = await prima.do(['open the invoices page', 'download the PDF']); expect(envelope.ok).toBe(false); - expect(envelope.failure?.error).toContain('no PDF link exists on this page'); + expect(envelope.failure?.error).toContain('blocked: download the PDF — no PDF link exists on this page'); }); test('a check that never passed fails the command even after later actions succeed', async () => { const { prima } = fakePrima(); let calls = 0; (prima as any).bot.getProvider = () => - fakeProvider(async (_conversation: unknown, tools: any) => { + fakeProvider(async () => { calls++; if (calls === 1) { return { toolExecutions: [{ toolName: 'verify', input: { assertion: 'unsaved indicator is visible' }, output: { success: false, action: 'verify', message: 'Verification failed' }, wasSuccessful: false }, toolExecution("I.click('Close')")], }; } - await tools.done.execute({ summary: 'the editor is closed', unmet: [] }); - return { toolExecutions: [{ toolName: 'done', input: {}, output: { success: true, action: 'done' }, wasSuccessful: true }] }; + return { toolExecutions: [completedExecution([1, 2], 'the editor is closed')] }; }); const envelope = await prima.do(['confirm the unsaved indicator', 'close the editor']); expect(envelope.ok).toBe(false); - expect(envelope.failure?.error).toContain('unsaved indicator is visible'); - expect(envelope.steps?.[0]).toMatchObject({ label: 'verify: unsaved indicator is visible', ok: false }); - }); - - test('an instruction satisfied without acting still succeeds when the model closes the sequence', async () => { - const { prima } = fakePrima(); - (prima as any).bot.getProvider = () => - fakeProvider(async (_conversation: unknown, tools: any) => { - await tools.done.execute({ summary: 'no cookie banner is present on this page', unmet: [] }); - return { toolExecutions: [{ toolName: 'done', input: {}, output: { success: true, action: 'done' }, wasSuccessful: true }] }; - }); - - const envelope = await prima.do(['dismiss the cookie banner if one appeared']); - - expect(envelope.ok).toBe(true); - expect(envelope.answer).toBe('no cookie banner is present on this page'); + expect(envelope.failure?.error).toContain('unproven: unsaved indicator is visible'); + expect(envelope.steps?.[0].proof).toContain('verify: unsaved indicator is visible => FAILED'); }); test('a check that passes on a retry does not fail the command', async () => { const { prima } = fakePrima(); let calls = 0; (prima as any).bot.getProvider = () => - fakeProvider(async (_conversation: unknown, tools: any) => { + fakeProvider(async () => { calls++; if (calls === 1) return { toolExecutions: [toolExecution("I.click('Delete')"), { toolName: 'verify', input: { assertion: 'the row is gone' }, output: { success: false, action: 'verify', message: 'Verification failed' }, wasSuccessful: false }] }; - if (calls === 2) return { toolExecutions: [{ toolName: 'verify', input: { assertion: 'the row is gone' }, output: { success: true, action: 'verify', code: "I.dontSee('Item')" }, wasSuccessful: true }] }; - await tools.done.execute({ summary: 'the row is gone', unmet: [] }); - return { toolExecutions: [{ toolName: 'done', input: {}, output: { success: true, action: 'done' }, wasSuccessful: true }] }; + return { toolExecutions: [{ toolName: 'verify', input: { assertion: 'the row is gone' }, output: { success: true, action: 'verify', code: "I.dontSee('Item')" }, wasSuccessful: true }, completedExecution([1, 2], 'the row is gone')] }; }); const envelope = await prima.do(['delete the row', 'confirm it is gone']); expect(envelope.ok).toBe(true); }); - test('context() hands back the page tree first and drops to markup only when asked again', async () => { const { prima } = fakePrima(); let captured: any; diff --git a/src/ai/researcher.ts b/src/ai/researcher.ts index 2dc5e441..a0a321a6 100644 --- a/src/ai/researcher.ts +++ b/src/ai/researcher.ts @@ -102,7 +102,7 @@ export class Researcher extends ResearcherBase implements Agent { const cached = getCachedResearch(stateHash); if (cached) { debugLog('Previous research result found'); - return `!! UI MAP IS CACHED AND MAY NOT REPRESENT CURRENT STATE; REFRESH RESEARCH IF YOU NOTICE ISSUES !!\n\n${cached}`; + return cached; } } diff --git a/tests/integration/prima-do.test.ts b/tests/integration/prima-do.test.ts index c5f7c1bc..fe59c7fa 100644 --- a/tests/integration/prima-do.test.ts +++ b/tests/integration/prima-do.test.ts @@ -20,6 +20,10 @@ function clickCall(id: string, commands: string[], explanation: string) { return { id, name: 'click', arguments: JSON.stringify({ commands, explanation }) }; } +function completedCall(id: string, numbers: number[], proof: string) { + return { id, name: 'completed', arguments: JSON.stringify({ numbers, proof }) }; +} + function extractPromptText(entry: any): string { if (!entry?.body?.messages) return ''; return entry.body.messages @@ -95,8 +99,8 @@ describe('Prima.do with aimock', () => { }); it('runs both instructions and collects the executed code', async () => { - mock.on({ sequenceIndex: 0 }, { toolCalls: [clickCall('call-1', ['I.click("Account")'], 'open the account menu')] }); - mock.on({ sequenceIndex: 1 }, { toolCalls: [clickCall('call-2', ['I.click("Settings")'], 'choose the settings entry')] }); + mock.on({ sequenceIndex: 0 }, { toolCalls: [clickCall('call-1', ['I.click("Account")'], 'open the account menu'), completedCall('call-2', [1], 'the account menu is open')] }); + mock.on({ sequenceIndex: 1 }, { toolCalls: [clickCall('call-3', ['I.click("Settings")'], 'choose the settings entry'), completedCall('call-4', [2], 'the settings page is shown')] }); mock.on({}, { content: 'Both instructions are done.' }); const envelope = await prima.do(['open the account menu', 'choose the settings entry']); @@ -105,6 +109,10 @@ describe('Prima.do with aimock', () => { expect(envelope.used).toEqual(['I.click("Account")', 'I.click("Settings")']); expect(executed).toEqual(['I.click("Account")', 'I.click("Settings")']); expect(envelope.command).toContain('open the account menu'); + expect(envelope.steps).toEqual([ + { label: 'open the account menu', ok: true, proof: 'the account menu is open\nI.click("Account")' }, + { label: 'choose the settings entry', ok: true, proof: 'the settings page is shown\nI.click("Settings")' }, + ]); }); it('sends every instruction and the page context in one prompt', async () => { diff --git a/tests/integration/researcher.test.ts b/tests/integration/researcher.test.ts index 77356532..e534fa7b 100644 --- a/tests/integration/researcher.test.ts +++ b/tests/integration/researcher.test.ts @@ -181,13 +181,12 @@ describe('Researcher with aimock', () => { expect(prompt).toContain('/tasks/board'); }); - it('returns cached research without AI call', async () => { + it('returns cached research verbatim and without an AI call', async () => { saveResearch(fakeState.hash!, '## Cached Research\n\nPreviously analyzed page.'); const result = await researcher.research(fakeState, { fix: false }); - expect(result).toContain('Cached Research'); - expect(result).toContain('CACHED AND MAY NOT REPRESENT CURRENT STATE'); + expect(result).toBe('## Cached Research\n\nPreviously analyzed page.'); expect(mock.getRequests().length).toBe(0); }); From d4198a72d62e63cb2544941f6dcc3e60f5227d73 Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Sun, 9 Aug 2026 12:59:38 +0300 Subject: [PATCH 05/39] Give every finished test a verdict MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A test that achieved all its expectations was never given a result. Test .hasFinished is true once every expectation carries a status, so the loop broke, finishTest saw a finished task and never called finish(), and the run reported success: false on a scenario that fully passed. The dead "Test with no result" log branch existed because this state was reachable. That is what made prima check print ok: false over six PASSED outcomes, and it is the same null status that crashed the HTML reporter on test.status.toLowerCase() after every check run. finishTest now decides from what the run achieved: all expectations met passes, anything short fails. do re-states what is still open on every turn rather than only when the list changes — a model that acted without reporting was otherwise never reminded, and its instruction came back as unaccounted for despite the envelope proving it landed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- boat/prima/src/prima.ts | 7 ++----- src/ai/tester.ts | 15 ++++++++------- tests/unit/tester-focus-scope.test.ts | 26 +++++++++++++++++++++++++- 3 files changed, 35 insertions(+), 13 deletions(-) diff --git a/boat/prima/src/prima.ts b/boat/prima/src/prima.ts index 795af281..165910a1 100644 --- a/boat/prima/src/prima.ts +++ b/boat/prima/src/prima.ts @@ -154,7 +154,6 @@ export class Prima { let narration = ''; let nudged = false; let contextHash = this.bot.stateManager().getCurrentState()?.hash; - let openList = this.openInstructions(ledger); for (let iteration = 1; iteration <= Math.min(instructions.length * ITERATIONS_PER_INSTRUCTION + 2, MAX_INSTRUCTION_ITERATIONS); iteration++) { const state = this.bot.stateManager().getCurrentState(); @@ -163,10 +162,8 @@ export class Prima { conversation.addUserText(await this.pageContext(ActionResult.fromState(state))); } - const remaining = this.openInstructions(ledger); - if (iteration > 1 && remaining !== openList) { - openList = remaining; - conversation.addUserText(`<remaining>\n${remaining}\n</remaining>`); + if (iteration > 1) { + conversation.addUserText(`<remaining>\n${this.openInstructions(ledger)}\n</remaining>`); } const invoked = await provider.invokeConversation(conversation, tools, { maxToolRoundtrips: MAX_TOOL_ROUNDTRIPS, agentName: AI_AGENT_NAME }).catch((error: unknown) => { diff --git a/src/ai/tester.ts b/src/ai/tester.ts index 4cc0f705..f76eb67e 100644 --- a/src/ai/tester.ts +++ b/src/ai/tester.ts @@ -639,19 +639,20 @@ export class Tester extends TaskAgent implements Agent { } private finishTest(task: Test): void { - if (!task.hasFinished) { - task.finish(TestResult.FAILED); + if (!task.result) { + if (task.hasAchievedAll()) task.finish(TestResult.PASSED); + else task.finish(TestResult.FAILED); } if (task.isSuccessful) { tag('success').log(`Successful test: ${task.scenario}`); - } else if (task.isSkipped) { + return; + } + if (task.isSkipped) { tag('warning').log(`Skipped test: ${task.scenario}`); - } else if (task.hasFailed) { - tag('error').log(`Failed test: ${task.scenario}`); - } else { - tag('warning').log(`Test with no result: ${task.scenario}`); + return; } + tag('error').log(`Failed test: ${task.scenario}`); } private async abortStartedTestOnErrorPage(task: Test, actionResult: ActionResult): Promise<{ success: boolean }> { diff --git a/tests/unit/tester-focus-scope.test.ts b/tests/unit/tester-focus-scope.test.ts index 3b89af59..a63144bd 100644 --- a/tests/unit/tester-focus-scope.test.ts +++ b/tests/unit/tester-focus-scope.test.ts @@ -2,7 +2,7 @@ import { describe, expect, it } from 'bun:test'; import { ActionResult } from '../../src/action-result.ts'; import { Tester } from '../../src/ai/tester.ts'; import { renderExperienceToc } from '../../src/experience-tracker.ts'; -import { Test } from '../../src/test-plan.ts'; +import { Test, TestResult } from '../../src/test-plan.ts'; function buildTester(): Tester { const provider: any = { @@ -173,3 +173,27 @@ describe('Tester stalled execution', () => { expect(task.result).toBe(null); }); }); + +describe('Tester verdict', () => { + it('passes a test whose expectations were all achieved, instead of leaving it without a result', () => { + const tester = buildTester(); + const task = new Test('filter items', 'normal', ['filtered items appear', 'the count updates'], '/page'); + task.addNote('filtered items appear', TestResult.PASSED); + task.addNote('the count updates', TestResult.PASSED); + + (tester as any).finishTest(task); + + expect(task.result).toBe(TestResult.PASSED); + expect(task.isSuccessful).toBe(true); + }); + + it('fails a test that stopped without achieving every expectation', () => { + const tester = buildTester(); + const task = new Test('filter items', 'normal', ['filtered items appear', 'the count updates'], '/page'); + task.addNote('filtered items appear', TestResult.PASSED); + + (tester as any).finishTest(task); + + expect(task.result).toBe(TestResult.FAILED); + }); +}); From aa697575849d224e79409e0c93253cb41ac9d845 Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Sun, 9 Aug 2026 13:13:36 +0300 Subject: [PATCH 06/39] Cut prima's top-level help by a third MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The help was the single largest thing prima put in a caller's context — 6,718 bytes, 56% of everything an agent read across a whole benchmark run, for one call. Per-command semantics now live under each subcommand's own --help, leaving the top level with the tiers, the envelope shape and the session default: 4,439 bytes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- boat/prima/src/cli.ts | 124 +++++++++++++++++++++--------------------- 1 file changed, 61 insertions(+), 63 deletions(-) diff --git a/boat/prima/src/cli.ts b/boat/prima/src/cli.ts index b86170dd..fab4e9cd 100644 --- a/boat/prima/src/cli.ts +++ b/boat/prima/src/cli.ts @@ -29,66 +29,58 @@ const helpContract = dedent` Coming down a tier to run steps one at a time costs more than the tier above, in both time and what you have to read - it is a fallback, not a default. - CHECK - Give it an outcome, not a click path - it decides how to get there. - --expected one outcome the run must reach; repeat it for several. Without it the - scenario text is the single expected outcome. Each comes back under - ### Expected outcomes as PASSED, FAILED or not verified - "not verified" - means the run never checked it, which is not the same as false. - Page problems seen on the way are reported under ### Answer, not as step failures. - - DO - Each instruction is numbered and accounted for: ### Steps reports every one as ok or - FAIL with what proved it. One that could not be carried out fails the command and - says why. Nothing runs past the last instruction you gave. - - VERIFY - Reports each assertion it could express as PASSED or FAILED with its playwright form, - and gives no overall verdict - read the lines and decide. "none ran" means the claim - could not be expressed, which is not the same as false. + Also: ask, verify, research, go, status, browser. Run <command> --help for what + each of them reports. ENVELOPE - ### Result ok, command, used - ### Page url, title, state hash, visit count - ### Changes what the accessibility tree gained, lost, toggled or had typed into it - ### Steps per-instruction outcome of do, each with the change that proves it - ### Expected outcomes each --expected of check, as PASSED, FAILED or not verified - ### Answer | ### Research | ### Assertions output of ask, research, verify - ### Failure error, compact ARIA of the page - ### Instance the browser you are on and the other instances running - ### Artifacts paths to the full aria.yml, page.html and network.jsonl - used: is code that already executed - CodeceptJS steps to copy as they are, except - for pw, whose Playwright expression a test needs inside I.usePlaywrightTo(...). - Log lines can precede the envelope; start parsing at the first ### line. - - FAILURE - A failed action fails. Nothing is retried along a different route and no other - element is ever substituted for the one you asked for, so ok: true always means - your own action landed. - Failures print compact ARIA inline, so retarget from the envelope itself and open - the artifact files only when the inline snapshot is not enough. + ### Result ok/command/used · ### Page url/title/state · ### Changes what the + accessibility tree gained, lost, toggled or had typed · ### Steps per-instruction + outcome with its proof · ### Expected outcomes each --expected as PASSED, FAILED or + not verified · ### Answer|Research|Assertions · ### Failure error and compact ARIA · + ### Instance · ### Artifacts. Start at the first ### line; logs can precede it. + used: is code that already executed. A failed action fails - nothing is retried along + a different route, so ok: true means your own action landed. SESSIONS - By default prima attaches to the playwright-cli browser of this workspace and works - on the tabs it already has open; driving the same session from both tools is the - intended usage. - playwright-cli open <url> the session prima attaches to - --pw-session <title> which playwright-cli session, when several are open - --endpoint <ep> attach to a browser server endpoint directly - prima browser start a prima-owned browser instead, when no session is open - --instance <name> which prima-owned browser you talk to; parallel work - needs one each - --session [file] cookies and storage persisted across processes; ignored - while attached, the attached session keeps its own - Prima never launches a browser implicitly and never closes an attached one - it - disconnects. browser list shows both kinds; ### Instance names the one you are on. - Every browser is reached over a Playwright browser-server endpoint, which needs the - Node build - run prima as "npx explorbot prima ..." or through the published prima - bin; from source under Bun the connection does not open. - When no AI model is usable pw still works; for everything else drive - playwright-cli directly. - Parsed but not active yet: --framework, so reported code is CodeceptJS whatever - you pass. + Prima attaches to the playwright-cli browser of this workspace and drives the tabs it + already has open; using the same session from both tools is the intended usage. + playwright-cli open <url> the session prima attaches to + --pw-session <title> which session, when several are open + prima browser start a prima-owned browser instead, when none is open + Prima never launches a browser implicitly and never closes an attached one. + Run it as "npx explorbot prima ..." or through the published prima bin - the + browser-server connection needs the Node build. +`; + +const checkHelp = dedent` + Give it an outcome, not a click path - it decides how to get there. + --expected one outcome the run must reach; repeat it for several. Without it the + scenario text is the single expected outcome. Each comes back under + ### Expected outcomes as PASSED, FAILED or not verified - "not verified" + means the run never checked it, which is not the same as false. + Page problems seen on the way are reported under ### Answer, not as step failures. +`; + +const doHelp = dedent` + Every instruction is numbered and accounted for: ### Steps reports each as ok or FAIL + with what proved it. One that could not be carried out fails the command and says why. + Nothing runs past the last instruction you gave. + Pass the whole remaining sequence in one call - that is what makes this tier cheap. +`; + +const verifyHelp = dedent` + Reports each assertion it could express as PASSED or FAILED with its playwright form, + and gives no overall verdict - read the lines and decide. "none ran" means the claim + could not be expressed, which is not the same as false. +`; + +const sessionHelp = dedent` + --endpoint <ep> attach to a browser server endpoint directly, skipping discovery + --instance <name> which prima-owned browser you talk to; parallel work needs one each + --session [file] cookies and storage persisted across processes; ignored while + attached, since the attached session keeps its own + --framework parsed but not active yet; reported code is CodeceptJS either way + When no AI model is usable pw still works; for everything else drive playwright-cli. `; let rootOptions: () => any = () => ({}); @@ -134,7 +126,8 @@ function addCommonOptions(cmd: Command): Command { .option('--framework <name>', 'Not active yet: framework the reported code targets, codeceptjs or playwright') .option('--url <url>', 'Page to open when the session has no page yet') .option('--endpoint <ep>', 'Websocket endpoint of a browser server to attach to, skipping discovery') - .option('--pw-session <title>', 'Title of the playwright-cli session to attach to'); + .option('--pw-session <title>', 'Title of the playwright-cli session to attach to') + .addHelpText('after', `\n${sessionHelp}`); } function primaFor(options: any): Prima { @@ -183,12 +176,15 @@ export function createPrimaCommands(name = 'prima'): Command { await runPrima(options, `pw ${fn}`, (prima) => prima.pw(fn)); }); - addCommonOptions(cmd.command('do <instructions...>').description('Run high-level instructions tester-style, one argument per instruction')).action(async (instructions, options) => { - await runPrima(options, `do ${instructions.join(' ')}`, (prima) => prima.do(instructions)); - }); + addCommonOptions(cmd.command('do <instructions...>').description('Run high-level instructions tester-style, one argument per instruction')) + .addHelpText('after', `\n${doHelp}`) + .action(async (instructions, options) => { + await runPrima(options, `do ${instructions.join(' ')}`, (prima) => prima.do(instructions)); + }); addCommonOptions(cmd.command('check <scenario>').description('Run a scenario end to end as a test, with its own verification, and report the steps it took')) .option('--expected <outcome>', 'An outcome the run must reach; repeat the flag for several', (value: string, all: string[]) => [...all, value], []) + .addHelpText('after', `\n${checkHelp}`) .action(async (scenario, options) => { await runPrima(options, `check ${scenario}`, (prima) => prima.check(scenario, options.expected)); }); @@ -197,9 +193,11 @@ export function createPrimaCommands(name = 'prima'): Command { await runPrima(options, `ask ${question}`, (prima) => prima.ask(question)); }); - addCommonOptions(cmd.command('verify <assertion>').alias('assert').description('Assert a statement about the current page')).action(async (assertion, options) => { - await runPrima(options, `verify ${assertion}`, (prima) => prima.verify(assertion)); - }); + addCommonOptions(cmd.command('verify <assertion>').alias('assert').description('Assert a statement about the current page')) + .addHelpText('after', `\n${verifyHelp}`) + .action(async (assertion, options) => { + await runPrima(options, `verify ${assertion}`, (prima) => prima.verify(assertion)); + }); addCommonOptions( cmd.command('research').description('Map the current page and return verified locators').option('--data', 'Include data extraction in the map').option('--deep', 'Expand hidden elements for a deeper map').option('--fresh', 'Ignore the cached map and research the page again') From b88fb0bf1f04e41b5f500786d5dda444f9bcc663 Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Sun, 9 Aug 2026 13:44:07 +0300 Subject: [PATCH 07/39] Read focus from the page, not from an ARIA snapshot MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit pressKey refused every key that needs focus — Enter, End, Home, arrows, Backspace, Delete, single characters — because it decided focus by regexing [focused] out of the cached ARIA snapshot, and Playwright's ariaSnapshot never emits [focused] in either plain or ai mode. Probed against a live page: a button that IS document.activeElement produces zero lines containing "focus". The guard could therefore never pass, and only the focus-free keys (Escape, Tab, F-keys) ever ran. That is what cost the last benchmark three editor close/reopen cycles and four failed attempts to move a caret to end of content. Focus is now read from document.activeElement at the moment of the press, and an unreadable page lets the press through rather than blocking it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- src/ai/tools.ts | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/src/ai/tools.ts b/src/ai/tools.ts index 75d19556..ac9e06a4 100644 --- a/src/ai/tools.ts +++ b/src/ai/tools.ts @@ -4,7 +4,7 @@ import { z } from 'zod'; import { ActionResult, type PageDiff, type ToolResultMetadata } from '../action-result.ts'; import type { ExperienceTracker } from '../experience-tracker.ts'; import { type Task, TestResult } from '../test-plan.js'; -import { LARGE_ARIA_CHANGE_THRESHOLD, extractFocusedElement } from '../utils/aria.ts'; +import { LARGE_ARIA_CHANGE_THRESHOLD } from '../utils/aria.ts'; import { isFatalBrowserError } from '../utils/browser-errors.ts'; import { createDebug, tag } from '../utils/logger.js'; import { pause } from '../utils/loop.js'; @@ -299,15 +299,11 @@ export function createCodeceptJSTools({ explorer, stateManager, ai }: ToolDeps, const focusFreeKeys = new Set(['Escape', 'Esc', 'Tab', 'F1', 'F2', 'F3', 'F4', 'F5', 'F6', 'F7', 'F8', 'F9', 'F10', 'F11', 'F12']); const needsFocus = !focusFreeKeys.has(keyToUse) && !modifier; - if (needsFocus) { - const currentAriaState = stateManager.getCurrentState()?.ariaSnapshot; - const focused = extractFocusedElement(currentAriaState ?? null); - if (!focused) { - activeNote.commit(TestResult.FAILED); - return failedToolResult('pressKey', `No element is focused. Key '${keyToUse}' requires a focused element.`, { - suggestion: 'Click the target element first, then press the key.', - }); - } + if (needsFocus && !(await hasFocusedElement(explorer))) { + activeNote.commit(TestResult.FAILED); + return failedToolResult('pressKey', `No element is focused. Key '${keyToUse}' requires a focused element.`, { + suggestion: 'Click the target element first, then press the key.', + }); } const previousState = ActionResult.fromState(stateManager.getCurrentState()!); @@ -1126,6 +1122,10 @@ export async function commitNote(activeNote: any, result: TestResult, toolResult activeNote.commit(result); } +async function hasFocusedElement(explorer: any): Promise<boolean> { + return explorer.withPage((page: any) => page.evaluate(() => !!document.activeElement && document.activeElement !== document.body)).catch(() => true); +} + export function successToolResult(action: string, data?: Record<string, any>, source?: { playwrightGroupId?: string | null; assertionSteps?: any[] }) { const result: Record<string, any> = { success: true, action, ...data }; if (source?.playwrightGroupId) { From f85f24a24895fed7b7c9c5e0590f72c8c4bb16c7 Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Sun, 9 Aug 2026 13:50:35 +0300 Subject: [PATCH 08/39] Capture the focused element with the page state MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Tester's <current_focus> block and Pilot's focused: line have never rendered. Both read focus by regexing [focused] out of the ARIA snapshot, and Playwright does not put it there — so both silently omitted focus on every single turn, and the tester never knew it was sitting in a text field. Focus is now read from document.activeElement alongside the ARIA snapshot and carried on ActionResult and WebPageState, with the element's tag mapped to its ARIA role so the existing textbox/combobox checks work. The ARIA parser it replaces is deleted rather than left as a decoy. Verified live: a focused search input reads back as {role: textbox, name: "Search skills", value: "explorbot"}. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- src/action-result.ts | 11 +++++++++++ src/action.ts | 25 ++++++++++++++++++++++++- src/ai/pilot.ts | 4 ++-- src/ai/tester.ts | 4 ++-- src/state-manager.ts | 3 ++- src/utils/aria.ts | 31 ------------------------------- 6 files changed, 41 insertions(+), 37 deletions(-) diff --git a/src/action-result.ts b/src/action-result.ts index 3f688c6c..e5ca7d59 100644 --- a/src/action-result.ts +++ b/src/action-result.ts @@ -30,6 +30,7 @@ interface ActionResultData extends WebPageState { iframeSnapshots?: Array<{ src: string; html: string; id?: string }>; ariaSnapshot?: string | null; ariaSnapshotFile?: string; + focusedElement?: FocusedElement | null; iframeURL?: string; links?: Link[]; } @@ -74,6 +75,7 @@ export class ActionResult implements ActionResultData { private snapshotCache = new TTLCache<string>(); readonly logFile: string | undefined = undefined; readonly ariaSnapshotFile: string | undefined = undefined; + readonly focusedElement: FocusedElement | null = null; private _ariaSnapshot: string | null | undefined = undefined; private _lastExtractedHtml: string | undefined = undefined; notes: string[] = []; @@ -107,6 +109,9 @@ export class ActionResult implements ActionResultData { if (data.ariaSnapshotFile !== undefined) { this.ariaSnapshotFile = data.ariaSnapshotFile; } + if (data.focusedElement !== undefined) { + this.focusedElement = data.focusedElement; + } // Store HTML in a private property if provided if (data.html !== undefined) { @@ -643,3 +648,9 @@ export class Diff { this._ariaChangeCount = ariaDiff.count; } } + +export interface FocusedElement { + role: string; + name: string; + value?: string; +} diff --git a/src/action.ts b/src/action.ts index 6625ce1f..76d303be 100644 --- a/src/action.ts +++ b/src/action.ts @@ -3,7 +3,7 @@ import { join } from 'node:path'; import { context, trace } from '@opentelemetry/api'; import { container, recorder } from 'codeceptjs'; import * as codeceptjs from 'codeceptjs'; -import { ActionResult } from './action-result.js'; +import { ActionResult, type FocusedElement } from './action-result.js'; import { clearActivity, setActivity } from './activity.ts'; import { ConfigParser, outputPath } from './config.js'; import type { ExplorbotConfig } from './config.js'; @@ -134,10 +134,12 @@ class Action { let ariaSnapshot: string | null = null; let ariaSnapshotFile: string | undefined = undefined; + let focusedElement: FocusedElement | null = null; try { const page = this.playwrightHelper.page; ariaSnapshot = await page.locator('body').ariaSnapshot(); + focusedElement = await page.evaluate(readFocusedElement); } catch (err) { debugLog('ARIA snapshot failed:', err instanceof Error ? `${err.message}\n${err.stack}` : err); } @@ -161,6 +163,7 @@ class Action { iframeSnapshots, ariaSnapshot, ariaSnapshotFile, + focusedElement, iframeURL: frame ? frame.url?.() || 'iframe' : undefined, }); this.stateManager.updateState(result, codeBlock); @@ -444,3 +447,23 @@ const detachStepLogger = (listener: StepListener) => { codeceptjs.event.dispatcher.off(codeceptjs.event.step.passed, listener); codeceptjs.event.dispatcher.off(codeceptjs.event.step.failed, listener); }; + +const readFocusedElement = () => { + const el = document.activeElement as any; + if (!el || el === document.body) return null; + + const tag = el.tagName.toLowerCase(); + const textish = new Set(['text', 'search', 'email', 'password', 'url', 'tel', 'number']); + let role = el.getAttribute('role') || tag; + if (tag === 'textarea' || el.isContentEditable) role = 'textbox'; + if (tag === 'input' && textish.has(el.type)) role = 'textbox'; + if (tag === 'select') role = 'combobox'; + if (tag === 'a') role = 'link'; + + const label = el.getAttribute('aria-label') || el.getAttribute('placeholder') || el.labels?.[0]?.textContent || el.textContent || ''; + const focused: { role: string; name: string; value?: string } = { role, name: label.trim().slice(0, 80) }; + + const value = el.value ?? el.textContent; + if (typeof value === 'string' && value) focused.value = value.slice(0, 200); + return focused; +}; diff --git a/src/ai/pilot.ts b/src/ai/pilot.ts index 1aa6e09c..b701e97d 100644 --- a/src/ai/pilot.ts +++ b/src/ai/pilot.ts @@ -8,7 +8,7 @@ import type Explorer from '../explorer.ts'; import type { PlaywrightRecorder } from '../playwright-recorder.ts'; import type { StateManager } from '../state-manager.ts'; import { type Test, TestResult } from '../test-plan.ts'; -import { collectInteractiveNodes, detectFocusArea, extractFocusedElement } from '../utils/aria.ts'; +import { collectInteractiveNodes, detectFocusArea } from '../utils/aria.ts'; import { ErrorPageError } from '../utils/error-page.ts'; import { createDebug, tag } from '../utils/logger.ts'; @@ -703,7 +703,7 @@ export class Pilot implements Agent { lines.push(`url: ${state.url}`); lines.push(`title: ${state.title || 'unknown'}`); - const focused = extractFocusedElement(state.ariaSnapshot); + const focused = state.focusedElement; if (focused) { const valuePart = focused.value ? ` (value: "${focused.value}")` : ''; lines.push(`focused: ${focused.role} "${focused.name}"${valuePart}`); diff --git a/src/ai/tester.ts b/src/ai/tester.ts index f76eb67e..88a5f762 100644 --- a/src/ai/tester.ts +++ b/src/ai/tester.ts @@ -11,7 +11,7 @@ import { Observability } from '../observability.ts'; import type { StateTransition } from '../state-manager.ts'; import { Stats } from '../stats.ts'; import { type Test, TestResult, type TestResultType } from '../test-plan.ts'; -import { detectFocusArea, extractFocusedElement } from '../utils/aria.ts'; +import { detectFocusArea } from '../utils/aria.ts'; import { ErrorPageError, isErrorPage } from '../utils/error-page.ts'; import { createDebug, tag } from '../utils/logger.ts'; import { loop } from '../utils/loop.ts'; @@ -514,7 +514,7 @@ export class Tester extends TaskAgent implements Agent { const focusArea = detectFocusArea(currentState.ariaSnapshot); - const focusedElement = extractFocusedElement(currentState.ariaSnapshot); + const focusedElement = currentState.focusedElement; if (focusedElement) { const isTextInput = ['textbox', 'combobox', 'searchbox'].includes(focusedElement.role); context += dedent` diff --git a/src/state-manager.ts b/src/state-manager.ts index 8918feec..7d9f5e84 100644 --- a/src/state-manager.ts +++ b/src/state-manager.ts @@ -1,4 +1,4 @@ -import { ActionResult } from './action-result.js'; +import { type FocusedElement, ActionResult } from './action-result.js'; import type { ExperienceTracker } from './experience-tracker.js'; import type { Knowledge, KnowledgeTracker } from './knowledge-tracker.js'; import { detectFocusArea } from './utils/aria.js'; @@ -46,6 +46,7 @@ export interface WebPageState { h4?: string; ariaSnapshot?: string | null; ariaSnapshotFile?: string; + focusedElement?: FocusedElement | null; links?: Link[]; verifications?: Record<string, boolean>; } diff --git a/src/utils/aria.ts b/src/utils/aria.ts index fbe02a31..90f10305 100644 --- a/src/utils/aria.ts +++ b/src/utils/aria.ts @@ -582,37 +582,6 @@ export const collectInteractiveNodes = (snapshot: string | null): Array<Record<s // Standalone helpers (regex on raw strings — not part of the pipeline) // ───────────────────────────────────────────────────────────────── -export interface FocusedElementInfo { - role: string; - name: string; - value?: string; - attributes?: string[]; -} - -export function extractFocusedElement(ariaSnapshot: string | null): FocusedElementInfo | null { - if (!ariaSnapshot) return null; - - const focusedMatch = ariaSnapshot.match(/-\s*(\w+)\s+"([^"]*)"([^:\n]*)\[focused\](?::\s*(.*))?/); - if (!focusedMatch) return null; - - const [, role, name, attributesStr, value] = focusedMatch; - - const attributes: string[] = []; - if (attributesStr) { - const attrMatches = attributesStr.matchAll(/\[([^\]]+)\]/g); - for (const match of attrMatches) { - if (match[1] !== 'focused') { - attributes.push(match[1]); - } - } - } - - const result: FocusedElementInfo = { role, name }; - if (value) result.value = value.trim(); - if (attributes.length > 0) result.attributes = attributes; - return result; -} - export function parseAriaLocator(ariaStr: string): { role: string; text: string } | null { const trimmed = ariaStr.trim(); if (trimmed === '-' || trimmed === '' || trimmed === '"-"') return null; From 538f62e0a1357e1dbf4323d397325abb76136dbc Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Sun, 9 Aug 2026 13:52:54 +0300 Subject: [PATCH 09/39] Report check's routine steps as a count, not a list A check of an eight-point goal printed 41 numbered steps, most of them successful intermediate actions with near-duplicate labels. The verdict already lives in ### Expected outcomes, so listing every step that worked spends the caller's context on what it will not read. ### Steps now carries the failures and the expectation outcomes, and closes with how many routine steps ran and the status hash that has the full log. Nothing is dropped silently. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- boat/prima/src/prima.ts | 8 +++++++- boat/prima/tests/prima.test.ts | 22 ++++++++++++++++++++++ 2 files changed, 29 insertions(+), 1 deletion(-) diff --git a/boat/prima/src/prima.ts b/boat/prima/src/prima.ts index 165910a1..2408aa8c 100644 --- a/boat/prima/src/prima.ts +++ b/boat/prima/src/prima.ts @@ -293,7 +293,13 @@ export class Prima { const notes = Object.values(test.notes || {}) as Array<{ message: string; status?: string; log?: string; observation?: boolean }>; const result = await this.capturedResult(this.bot.stateManager().getCurrentState()); const envelope = await this.reportEnvelope(command, result, previousState, { ok: outcome.success }); - envelope.steps = notes.filter((note) => !note.observation).map((note) => ({ label: note.message, ok: note.status !== TestResult.FAILED, proof: note.log || '' })); + const recorded = notes.filter((note) => !note.observation); + const worthReporting = recorded.filter((note) => note.status === TestResult.FAILED || outcomes.includes(note.message)); + envelope.steps = worthReporting.map((note) => ({ label: note.message, ok: note.status !== TestResult.FAILED, proof: note.log || '' })); + + const routine = recorded.length - worthReporting.length; + if (routine) envelope.steps.push({ label: `${routine} more ${pluralize(routine, 'step')} ran without failing — prima status ${envelope.status} for the full log`, ok: true, proof: '' }); + envelope.expectations = outcomes.map((text) => { const checked = notes.findLast((note) => note.message === text && !!note.status); if (checked?.status === TestResult.PASSED) return { text, status: 'passed' as const }; diff --git a/boat/prima/tests/prima.test.ts b/boat/prima/tests/prima.test.ts index b007cdd7..1d366722 100644 --- a/boat/prima/tests/prima.test.ts +++ b/boat/prima/tests/prima.test.ts @@ -741,6 +741,28 @@ describe('Prima.check', () => { ]); }); + test('routine steps are counted rather than listed, and failures are kept', async () => { + const { prima } = fakePrima(); + (prima as any).bot.agentTester = () => ({ + test: async (test: any) => { + test.addNote('opened the panel', TestResult.PASSED); + test.addNote('clicked the row', TestResult.PASSED); + test.addNote('Failed to open the editor (click)', TestResult.FAILED); + test.addNote('the editor opens', TestResult.PASSED); + return { success: false }; + }, + }); + + const envelope = await prima.check('edit a skill', ['the editor opens']); + const labels = envelope.steps?.map((step) => step.label) || []; + + expect(labels).toContain('Failed to open the editor (click)'); + expect(labels).toContain('the editor opens'); + expect(labels).not.toContain('opened the panel'); + expect(labels.at(-1)).toContain('2 more steps ran without failing'); + expect(labels.at(-1)).toContain('prima status'); + }); + test('the scenario stands in as the only outcome when none was given', async () => { const { prima } = fakePrima(); (prima as any).bot.agentTester = () => ({ From c5a2e5dd7d9c584b01d18a9807bdb7ad1e3c8385 Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Sun, 9 Aug 2026 15:06:21 +0300 Subject: [PATCH 10/39] Show do what it has already finished, not just what is left MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit do told the model which instructions were still open but never which were closed, so nothing stopped it redoing satisfied work — measured on a live page, it re-clicked a panel toggle it had already activated, closing what it had just opened, and repeated the fill behind it. Each turn now carries the whole ledger with each instruction's state and the proof that closed it, and asks for completed() on anything the page already shows is satisfied before acting again. Same instructions against the same page: 10 commands and both instructions unaccounted for, down to 2 commands with both closed and proven. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- boat/prima/src/prima.ts | 22 ++++++++++++++++++++-- boat/prima/tests/prima.test.ts | 6 +++--- 2 files changed, 23 insertions(+), 5 deletions(-) diff --git a/boat/prima/src/prima.ts b/boat/prima/src/prima.ts index 2408aa8c..5f022dce 100644 --- a/boat/prima/src/prima.ts +++ b/boat/prima/src/prima.ts @@ -163,7 +163,13 @@ export class Prima { } if (iteration > 1) { - conversation.addUserText(`<remaining>\n${this.openInstructions(ledger)}\n</remaining>`); + conversation.addUserText(dedent` + <progress> + ${this.ledgerProgress(ledger)} + </progress> + + Call completed() now for every open instruction the page already shows is satisfied, before you act again. + `); } const invoked = await provider.invokeConversation(conversation, tools, { maxToolRoundtrips: MAX_TOOL_ROUNDTRIPS, agentName: AI_AGENT_NAME }).catch((error: unknown) => { @@ -278,6 +284,16 @@ export class Prima { .join('\n'); } + private ledgerProgress(ledger: LedgerEntry[]): string { + return ledger + .map((entry, index) => { + const head = `${index + 1}. ${entry.status} — ${entry.text}`; + if (entry.status === 'open') return head; + return `${head} (${entry.proof})`; + }) + .join('\n'); + } + async check(scenario: string, expected: string[] = []): Promise<EnvelopeData> { const command = `check ${scenario}`; const guard = await this.aiGuard(command); @@ -621,8 +637,10 @@ export class Prima { <ledger> Instructions are numbered and those numbers never change. Report by number. Saying in your reply that something is done does not report it — only completed() does. Nothing you write is read as a report. + Report an instruction the moment the page shows it is satisfied, before moving on. Waiting until later is how work gets repeated. An instruction you have reported is finished. Never act on it again, and never report it twice. - You are shown what is still open after each turn. When nothing is open the run is over. + After each turn you are shown every instruction with its state. Act only on the ones still open — repeating an action that already + landed can undo it, since a control that opened something will close it again. Reaching for blocked() after a couple of honest attempts costs less than a third attempt that fails the same way. </ledger> diff --git a/boat/prima/tests/prima.test.ts b/boat/prima/tests/prima.test.ts index 1d366722..f6de0af8 100644 --- a/boat/prima/tests/prima.test.ts +++ b/boat/prima/tests/prima.test.ts @@ -486,9 +486,9 @@ describe('Prima.do', () => { await prima.do(['open the invoices page', 'download the PDF']); - const remaining = prompts.find((prompt) => prompt.startsWith('<remaining>')); - expect(remaining).toContain('2. download the PDF'); - expect(remaining).not.toContain('open the invoices page'); + const progress = prompts.find((prompt) => prompt.startsWith('<progress>')); + expect(progress).toContain('1. done — open the invoices page (list is open)'); + expect(progress).toContain('2. open — download the PDF'); }); test('a model that narrates instead of reporting is asked once for the ledger', async () => { From b8369d856cbe5c5fda534b45a3a8a89b11453412 Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Sun, 9 Aug 2026 15:07:56 +0300 Subject: [PATCH 11/39] Stop dropping the tester's rules as soon as it takes a note MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit prepareInstructionsForNextStep reassigned its message instead of appending, so from the second step onward the tester lost the whole <rules> block along with the scenario reminder — including "do not do unsuccessful clicks again" and "do not run same tool calls with same parameters again", the two rules written to stop exactly the repetition that fills its step logs. The log now appends to the rules rather than replacing them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- src/ai/tester.ts | 3 ++- tests/unit/tester-focus-scope.test.ts | 14 ++++++++++++++ 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/src/ai/tester.ts b/src/ai/tester.ts index 88a5f762..e59414d3 100644 --- a/src/ai/tester.ts +++ b/src/ai/tester.ts @@ -485,7 +485,8 @@ export class Tester extends TaskAgent implements Agent { `; if (task.getPrintableNotes()) { - outcomeStatus = dedent` + outcomeStatus += dedent` + Your current log: <notes> ${task.notesToString()} diff --git a/tests/unit/tester-focus-scope.test.ts b/tests/unit/tester-focus-scope.test.ts index a63144bd..6f5664fa 100644 --- a/tests/unit/tester-focus-scope.test.ts +++ b/tests/unit/tester-focus-scope.test.ts @@ -197,3 +197,17 @@ describe('Tester verdict', () => { expect(task.result).toBe(TestResult.FAILED); }); }); + +describe('Tester step instructions', () => { + it('keeps the rules once the log has entries, instead of replacing them with it', async () => { + const tester = buildTester(); + const task = new Test('filter items', 'normal', ['filtered items appear'], '/page'); + task.addNote('clicked the filter button'); + + const instructions = await (tester as any).prepareInstructionsForNextStep(task); + + expect(instructions).toContain('Do not run same tool calls with same parameters again'); + expect(instructions).toContain('clicked the filter button'); + expect(instructions).toContain('filtered items appear'); + }); +}); From 1ec5b87423d38a85fd92cf634800ded1c3b5f25c Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Sun, 9 Aug 2026 15:34:56 +0300 Subject: [PATCH 12/39] Settle expected outcomes on evidence, not on an exact echo MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An expected outcome counted as met only when the tester recorded a note whose text matched it byte for byte. Paraphrase once and a run that genuinely verified everything reported "not verified" across the board — which is what happened in the last benchmark, and what pushed the caller down into do and tripled the cost of the run. Pilot now settles the outcomes nothing checked off by name: it reads the run log and judges each one on what the steps show happened, with "unverified" reserved for outcomes the run never established either way. Outcomes the run did check off by name skip the call entirely. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- boat/prima/src/prima.ts | 7 +- boat/prima/tests/prima.test.ts | 9 +++ src/ai/pilot.ts | 49 ++++++++++++ tests/integration/pilot-expectations.test.ts | 82 ++++++++++++++++++++ 4 files changed, 141 insertions(+), 6 deletions(-) create mode 100644 tests/integration/pilot-expectations.test.ts diff --git a/boat/prima/src/prima.ts b/boat/prima/src/prima.ts index 5f022dce..4eaf5b21 100644 --- a/boat/prima/src/prima.ts +++ b/boat/prima/src/prima.ts @@ -316,12 +316,7 @@ export class Prima { const routine = recorded.length - worthReporting.length; if (routine) envelope.steps.push({ label: `${routine} more ${pluralize(routine, 'step')} ran without failing — prima status ${envelope.status} for the full log`, ok: true, proof: '' }); - envelope.expectations = outcomes.map((text) => { - const checked = notes.findLast((note) => note.message === text && !!note.status); - if (checked?.status === TestResult.PASSED) return { text, status: 'passed' as const }; - if (checked?.status === TestResult.FAILED) return { text, status: 'failed' as const }; - return { text, status: 'unverified' as const }; - }); + envelope.expectations = await this.bot.agentPilot().settleExpectations(test); const observations = notes.filter((note) => note.observation).map((note) => note.message); if (observations.length) envelope.answer = ['Page problems noticed while running, not step failures:', ...observations.map((line) => `- ${line}`)].join('\n'); diff --git a/boat/prima/tests/prima.test.ts b/boat/prima/tests/prima.test.ts index f6de0af8..6d37b672 100644 --- a/boat/prima/tests/prima.test.ts +++ b/boat/prima/tests/prima.test.ts @@ -731,6 +731,13 @@ describe('Prima.check', () => { return { success: false }; }, }); + (prima as any).bot.agentPilot = () => ({ + settleExpectations: async () => [ + { text: 'the editor opens', status: 'passed' }, + { text: 'the draft is saved', status: 'failed' }, + { text: 'the list refreshes', status: 'unverified' }, + ], + }); const envelope = await prima.check('edit and save a skill', ['the editor opens', 'the draft is saved', 'the list refreshes']); @@ -752,6 +759,7 @@ describe('Prima.check', () => { return { success: false }; }, }); + (prima as any).bot.agentPilot = () => ({ settleExpectations: async () => [{ text: 'the editor opens', status: 'passed' }] }); const envelope = await prima.check('edit a skill', ['the editor opens']); const labels = envelope.steps?.map((step) => step.label) || []; @@ -771,6 +779,7 @@ describe('Prima.check', () => { return { success: true }; }, }); + (prima as any).bot.agentPilot = () => ({ settleExpectations: async () => [{ text: 'edit and save a skill', status: 'passed' }] }); const envelope = await prima.check('edit and save a skill'); diff --git a/src/ai/pilot.ts b/src/ai/pilot.ts index b701e97d..a0f8aa1b 100644 --- a/src/ai/pilot.ts +++ b/src/ai/pilot.ts @@ -561,6 +561,55 @@ export class Pilot implements Agent { return text; } + async settleExpectations(task: Test): Promise<Array<{ text: string; status: 'passed' | 'failed' | 'unverified' }>> { + const undecided = task.expected.filter((text) => !task.getCheckedExpectations().includes(text)); + const decided = (text: string): 'passed' | 'failed' => { + if (task.hasAchievedAny() && !task.getRemainingExpectations().includes(text)) return 'passed'; + return 'failed'; + }; + + if (!undecided.length) return task.expected.map((text) => ({ text, status: decided(text) })); + + const schema = z.object({ + outcomes: z.array( + z.object({ + expectation: z.string().describe('The expected outcome, repeated exactly as it was given'), + status: z.enum(['passed', 'failed', 'unverified']).describe('passed = the log shows it happened, failed = the log shows it did not, unverified = the run never established either way'), + }) + ), + }); + + const userContent = dedent` + A test run has finished. Decide, for each expected outcome, what the run established about it. + + <expected_outcomes> + ${undecided.map((text) => `- ${text}`).join('\n')} + </expected_outcomes> + + <run_log> + ${task.notesToString() || 'No steps recorded.'} + </run_log> + + The log is written in the tester's own words, so an outcome can be satisfied by a step that describes it + differently. Judge by what the steps show happened, not by whether the wording matches. + Choose "unverified" only when the log neither shows the outcome happening nor shows it failing — + that is a statement about the run, not about the application. + `; + + const response = await this.provider + .generateObject([{ role: 'user' as const, content: userContent }], schema, this.provider.getAgenticModel('pilot'), { + agentName: 'pilot', + telemetry: { functionId: 'pilot.settleExpectations' }, + }) + .catch(() => null); + + const judged = new Map((response?.object?.outcomes || []).map((outcome: any) => [outcome.expectation, outcome.status])); + return task.expected.map((text) => { + if (!undecided.includes(text)) return { text, status: decided(text) }; + return { text, status: (judged.get(text) as 'passed' | 'failed' | 'unverified') || 'unverified' }; + }); + } + private formatExpectations(task: Test): string { const checked = task.getCheckedExpectations(); const remaining = task.getRemainingExpectations(); diff --git a/tests/integration/pilot-expectations.test.ts b/tests/integration/pilot-expectations.test.ts new file mode 100644 index 00000000..8708b3e1 --- /dev/null +++ b/tests/integration/pilot-expectations.test.ts @@ -0,0 +1,82 @@ +import { LLMock } from '@copilotkit/aimock'; +import { createOpenAI } from '@ai-sdk/openai'; +import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'bun:test'; +import { Pilot } from '../../src/ai/pilot.ts'; +import { Provider } from '../../src/ai/provider.ts'; +import { ConfigParser } from '../../src/config.ts'; +import { Test, TestResult } from '../../src/test-plan.ts'; + +function extractPromptText(entry: any): string { + if (!entry?.body?.messages) return ''; + return entry.body.messages + .map((message: any) => { + if (typeof message.content === 'string') return message.content; + if (Array.isArray(message.content)) { + return message.content + .filter((part: any) => part.type === 'text') + .map((part: any) => part.text || '') + .join('\n'); + } + return ''; + }) + .join('\n'); +} + +describe('Pilot settling expected outcomes', () => { + let mock: LLMock; + let pilot: Pilot; + + beforeAll(async () => { + mock = new LLMock({ port: 0, logLevel: 'silent' }); + await mock.start(); + + const openai = createOpenAI({ baseURL: `${mock.url}/v1`, apiKey: 'test-key', compatibility: 'compatible' }); + ConfigParser.setupTestConfig(); + const provider = new Provider({ model: openai.chat('test-model'), config: {} }); + + pilot = new Pilot({ ai: provider, config: ConfigParser.getInstance().getConfig(), explorer: {}, stateManager: {}, requestStore: {}, playwrightRecorder: {} } as any, {} as any, {} as any); + }); + + beforeEach(() => { + mock.clearRequests(); + mock.resetMatchCounts(); + mock.clearFixtures(); + }); + + afterAll(async () => { + await mock.stop(); + }); + + it('settles an outcome the tester described in its own words', async () => { + const task = new Test('open the editor', 'normal', ['the global editor opens with its markdown loaded'], '/skills'); + task.addNote('Clicked Change skill globally; editor panel appeared showing SKILL.md content', TestResult.PASSED); + + mock.on({}, { content: JSON.stringify({ outcomes: [{ expectation: 'the global editor opens with its markdown loaded', status: 'passed' }] }) }); + + const settled = await pilot.settleExpectations(task); + + expect(settled).toEqual([{ text: 'the global editor opens with its markdown loaded', status: 'passed' }]); + expect(extractPromptText(mock.getRequests()[0])).toContain('editor panel appeared showing SKILL.md content'); + }); + + it('asks nothing when the run already checked every outcome off by name', async () => { + const task = new Test('open the editor', 'normal', ['the editor opens'], '/skills'); + task.addNote('the editor opens', TestResult.PASSED); + + const settled = await pilot.settleExpectations(task); + + expect(settled).toEqual([{ text: 'the editor opens', status: 'passed' }]); + expect(mock.getRequests().length).toBe(0); + }); + + it('leaves an outcome unverified when the model cannot tell from the log', async () => { + const task = new Test('open the editor', 'normal', ['the list refreshes'], '/skills'); + task.addNote('Clicked around the sidebar', TestResult.PASSED); + + mock.on({}, { content: JSON.stringify({ outcomes: [{ expectation: 'the list refreshes', status: 'unverified' }] }) }); + + const settled = await pilot.settleExpectations(task); + + expect(settled).toEqual([{ text: 'the list refreshes', status: 'unverified' }]); + }); +}); From 3a2349d79021c891f29ffb75697f85befce9a027 Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Sun, 9 Aug 2026 15:51:56 +0300 Subject: [PATCH 13/39] Keep check's step log to what failed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ### Steps mixed failed attempts with the notes that named expected outcomes. Those notes are written when the tester settles an outcome, not when the action happened, so an outcome reached first appeared last — the log read as out of order, and listed the same behaviour as FAIL among the attempts and PASSED among the outcomes. Expected outcomes now live only in ### Expected outcomes, where Pilot gives them a real verdict. ### Steps is the failure log plus a count of what ran cleanly, and the two sections no longer contradict each other. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- boat/prima/src/prima.ts | 10 +++++----- boat/prima/tests/prima.test.ts | 6 +++--- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/boat/prima/src/prima.ts b/boat/prima/src/prima.ts index 4eaf5b21..fd64705c 100644 --- a/boat/prima/src/prima.ts +++ b/boat/prima/src/prima.ts @@ -309,12 +309,12 @@ export class Prima { const notes = Object.values(test.notes || {}) as Array<{ message: string; status?: string; log?: string; observation?: boolean }>; const result = await this.capturedResult(this.bot.stateManager().getCurrentState()); const envelope = await this.reportEnvelope(command, result, previousState, { ok: outcome.success }); - const recorded = notes.filter((note) => !note.observation); - const worthReporting = recorded.filter((note) => note.status === TestResult.FAILED || outcomes.includes(note.message)); - envelope.steps = worthReporting.map((note) => ({ label: note.message, ok: note.status !== TestResult.FAILED, proof: note.log || '' })); + const recorded = notes.filter((note) => !note.observation && !outcomes.includes(note.message)); + const failed = recorded.filter((note) => note.status === TestResult.FAILED); + envelope.steps = failed.map((note) => ({ label: note.message, ok: false, proof: note.log || '' })); - const routine = recorded.length - worthReporting.length; - if (routine) envelope.steps.push({ label: `${routine} more ${pluralize(routine, 'step')} ran without failing — prima status ${envelope.status} for the full log`, ok: true, proof: '' }); + const routine = recorded.length - failed.length; + if (routine) envelope.steps.push({ label: `${routine} further ${pluralize(routine, 'step')} ran without failing — prima status ${envelope.status} for the full log`, ok: true, proof: '' }); envelope.expectations = await this.bot.agentPilot().settleExpectations(test); diff --git a/boat/prima/tests/prima.test.ts b/boat/prima/tests/prima.test.ts index 6d37b672..7a13fa30 100644 --- a/boat/prima/tests/prima.test.ts +++ b/boat/prima/tests/prima.test.ts @@ -748,7 +748,7 @@ describe('Prima.check', () => { ]); }); - test('routine steps are counted rather than listed, and failures are kept', async () => { + test('steps report the failures and count the rest, leaving verdicts to the outcomes section', async () => { const { prima } = fakePrima(); (prima as any).bot.agentTester = () => ({ test: async (test: any) => { @@ -765,9 +765,9 @@ describe('Prima.check', () => { const labels = envelope.steps?.map((step) => step.label) || []; expect(labels).toContain('Failed to open the editor (click)'); - expect(labels).toContain('the editor opens'); + expect(labels).not.toContain('the editor opens'); expect(labels).not.toContain('opened the panel'); - expect(labels.at(-1)).toContain('2 more steps ran without failing'); + expect(labels.at(-1)).toContain('2 further steps ran without failing'); expect(labels.at(-1)).toContain('prima status'); }); From 900d92620ee6d5b68e5b343272b7a0861c063067 Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Sun, 9 Aug 2026 15:54:56 +0300 Subject: [PATCH 14/39] Settle do's unreported instructions instead of writing them off MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit do reported instructions as "never reported — still open" while used: showed the command had run and the ariaDiff showed the page change it produced. A caller who can see both stops trusting the envelope and re-derives the truth by hand, which is what turned one benchmark run into sixteen commands. When the loop ends with instructions unreported but actions executed, do now asks once more for an account of them, with only completed() and blocked() available. Same shape as settling expected outcomes: judge at the end from what happened, rather than demanding bookkeeping mid-flight that the model does not reliably keep. The report handling both paths share is extracted rather than copied. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- boat/prima/src/prima.ts | 71 +++++++++++++++++++++++----------- boat/prima/tests/prima.test.ts | 29 ++++++++++++-- 2 files changed, 74 insertions(+), 26 deletions(-) diff --git a/boat/prima/src/prima.ts b/boat/prima/src/prima.ts index fd64705c..71f24f20 100644 --- a/boat/prima/src/prima.ts +++ b/boat/prima/src/prima.ts @@ -196,28 +196,7 @@ export class Prima { for (const execution of executions) { const output = execution.output || {}; - if (output.action === 'completed') { - for (const number of execution.input?.numbers || []) { - const entry = ledger[number - 1]; - if (entry?.status !== 'open') continue; - entry.status = 'done'; - entry.proof = execution.input?.proof || ''; - entry.evidence = [...evidence]; - } - evidence.length = 0; - continue; - } - - if (output.action === 'blocked') { - const entry = ledger[(execution.input?.instruction || 0) - 1]; - if (entry?.status === 'open') { - entry.status = 'blocked'; - entry.proof = execution.input?.reason || ''; - entry.evidence = [...evidence]; - } - evidence.length = 0; - continue; - } + if (this.applyLedgerReport(execution, ledger, evidence)) continue; if (output.action === 'verify' && !output.inexpressible) { const claim = execution.input?.assertion || 'verification'; @@ -246,6 +225,10 @@ export class Prima { if (aiError) return this.failureEnvelope(command, aiError, previousState); + if (used.length && ledger.some((entry) => entry.status === 'open')) { + await this.settleLedger(conversation, provider, ledger, evidence); + } + const steps = ledger.map((entry) => ({ label: entry.text, ok: entry.status === 'done', proof: [entry.proof || UNACCOUNTED[entry.status], ...entry.evidence].filter(Boolean).join('\n') })); const unfinished = ledger.filter((entry) => entry.status !== 'done'); @@ -284,6 +267,50 @@ export class Prima { .join('\n'); } + private applyLedgerReport(execution: any, ledger: LedgerEntry[], evidence: string[]): boolean { + const action = execution.output?.action; + + if (action === 'completed') { + for (const number of execution.input?.numbers || []) { + const entry = ledger[number - 1]; + if (entry?.status !== 'open') continue; + entry.status = 'done'; + entry.proof = execution.input?.proof || ''; + entry.evidence = [...evidence]; + } + evidence.length = 0; + return true; + } + + if (action !== 'blocked') return false; + + const entry = ledger[(execution.input?.instruction || 0) - 1]; + if (entry?.status === 'open') { + entry.status = 'blocked'; + entry.proof = execution.input?.reason || ''; + entry.evidence = [...evidence]; + } + evidence.length = 0; + return true; + } + + private async settleLedger(conversation: any, provider: any, ledger: LedgerEntry[], evidence: string[]): Promise<void> { + conversation.addUserText(dedent` + The run is over and these instructions were never reported: + + ${this.openInstructions(ledger)} + + Account for each one from what you actually did. completed() for the ones the page ended up showing were + carried out, blocked() for the ones it could not do. Report every one — nothing else runs after this. + `); + + const invoked = await provider.invokeConversation(conversation, { completed: this.completedTool(), blocked: this.blockedTool() }, { maxToolRoundtrips: 2, toolChoice: 'required', agentName: AI_AGENT_NAME }).catch(() => null); + + for (const execution of invoked?.toolExecutions || []) { + this.applyLedgerReport(execution, ledger, evidence); + } + } + private ledgerProgress(ledger: LedgerEntry[]): string { return ledger .map((entry, index) => { diff --git a/boat/prima/tests/prima.test.ts b/boat/prima/tests/prima.test.ts index 7a13fa30..8eb7a556 100644 --- a/boat/prima/tests/prima.test.ts +++ b/boat/prima/tests/prima.test.ts @@ -523,7 +523,27 @@ describe('Prima.do', () => { const envelope = await prima.do(['open the invoices page']); expect(envelope.ok).toBe(false); - expect(calls).toBe(3); + expect(calls).toBe(4); + }); + + test('instructions left unreported are settled from what the run actually did', async () => { + const { prima } = fakePrima(); + const prompts: string[] = []; + let calls = 0; + (prima as any).bot.getProvider = () => + fakeProvider(async () => { + calls++; + if (calls === 1) return { toolExecutions: [toolExecution("I.click('Invoices')"), toolExecution("I.click('Download')")] }; + if (calls === 2) return { toolExecutions: [], response: { text: 'Both done.' } }; + if (calls === 3) return { toolExecutions: [], response: { text: 'Still both done.' } }; + return { toolExecutions: [completedExecution([1, 2], 'the invoice list opened and the PDF downloaded')] }; + }, prompts); + + const envelope = await prima.do(['open the invoices page', 'download the PDF']); + + expect(envelope.ok).toBe(true); + expect(prompts.join('\n')).toContain('The run is over and these instructions were never reported'); + expect(envelope.steps?.every((step) => step.ok)).toBe(true); }); test('an instruction the model never reported is named as unaccounted for', async () => { @@ -559,16 +579,17 @@ describe('Prima.do', () => { return { toolExecutions: [toolExecution("I.click('Next')")] }; }); + // one extra call past each cap is the closing pass that asks for the unreported instructions await prima.do(['first step', 'second step']); - expect(calls).toBe(6); + expect(calls).toBe(6 + 1); calls = 0; await prima.do(['a', 'b', 'c', 'd', 'e', 'f', 'g']); - expect(calls).toBe(16); + expect(calls).toBe(16 + 1); calls = 0; await prima.do(Array.from({ length: 20 }, (_, i) => `step ${i}`)); - expect(calls).toBe(24); + expect(calls).toBe(24 + 1); }); test('a failed tool execution fails the command instead of reaching for another element', async () => { From 9e1f04cd96450103dd050f5b4825fdbc0f81572a Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Sun, 9 Aug 2026 15:59:35 +0300 Subject: [PATCH 15/39] Judge unreported instructions against when they were due MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Settling at the end of a run asked the model about the page as it stands, but later instructions have moved it on — an instruction to confirm the editor's content cannot be judged from a page whose next instruction closed that editor. Live, the model reported nothing for it and the instruction came back unaccounted for. The closing pass now says to judge each instruction against what was seen when it was due, and that something confirmed earlier stays confirmed. Same five instructions that left one unaccounted for now close all five. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- boat/prima/src/prima.ts | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/boat/prima/src/prima.ts b/boat/prima/src/prima.ts index 71f24f20..9641eb87 100644 --- a/boat/prima/src/prima.ts +++ b/boat/prima/src/prima.ts @@ -300,8 +300,9 @@ export class Prima { ${this.openInstructions(ledger)} - Account for each one from what you actually did. completed() for the ones the page ended up showing were - carried out, blocked() for the ones it could not do. Report every one — nothing else runs after this. + Judge each one against what you saw at the time it was due, not against the page as it stands now — later + instructions have moved it on, and something you confirmed earlier stays confirmed even if it is gone. + completed() for those, blocked() for the ones the page could not do. Report every one — nothing else runs after this. `); const invoked = await provider.invokeConversation(conversation, { completed: this.completedTool(), blocked: this.blockedTool() }, { maxToolRoundtrips: 2, toolChoice: 'required', agentName: AI_AGENT_NAME }).catch(() => null); From 01a100dbf73f10373ad6a0a29b4c3a32b8fe3502 Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Sun, 9 Aug 2026 17:32:27 +0300 Subject: [PATCH 16/39] Describe see() by what only a screenshot can answer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The tool told the model to "use it to verify the actions were performed correctly and the page is in the expected state", so it ran a second, slower model after almost every action — 18 vision calls in one measured scenario, 23% of that run's wall clock. Every action already reports what changed on the page, so those calls asked a question that had just been answered. It now describes what a screenshot uniquely gives — layout, what an image or canvas depicts, colour, whether something is covered — and says not to use it to confirm an action landed. The example follows the description instead of contradicting it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- src/ai/tools.ts | 11 ++++++----- testomatio.debug.json | 1 + 2 files changed, 7 insertions(+), 5 deletions(-) create mode 120000 testomatio.debug.json diff --git a/src/ai/tools.ts b/src/ai/tools.ts index ac9e06a4..abb13005 100644 --- a/src/ai/tools.ts +++ b/src/ai/tools.ts @@ -540,14 +540,15 @@ export function createAgentTools({ explorer, stateManager, ai, researcher, navig const tools: Record<string, any> = { see: tool({ description: dedent` - Check the page contents based on current page state and screenshot. - This tool will trigger visual research to check the page contents on request. - Use it to verify the actions were performed correctly and the page is in the expected state. + Answer a question about the page from a screenshot, for things its structure cannot express: + layout and position, what an image or canvas depicts, colour, and whether something is covered or cut off. + This runs a second model and is the slowest tool here, so reach for it only when the question is genuinely visual. + Do NOT use it to confirm an action landed — every action already reports what changed on the page. Input schema has exactly one field: request. Do not pass text, reason, assertion, or other fields. <example> - request: "Check current state of the Login form" - result: "Login form is visible with username and password fields, username is filled with 'testuser' and password is empty' + request: "Is the save button covered by anything, and does the chart show any plotted data?" + result: "The save button is partly behind a cookie banner at the bottom. The chart area is empty apart from its axes." </example> `, inputSchema: z.object({ diff --git a/testomatio.debug.json b/testomatio.debug.json new file mode 120000 index 00000000..3c3f2de6 --- /dev/null +++ b/testomatio.debug.json @@ -0,0 +1 @@ +/tmp/testomatio.debug.2026-08-09T14-20-32.json \ No newline at end of file From 97c7cf5cb9d47d7738e346714c05053e8dc01543 Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Sun, 9 Aug 2026 17:56:45 +0300 Subject: [PATCH 17/39] Give do the refs it was already fetching, and a tool to click them MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit do asked the page for a ref-bearing snapshot and then stripped every ref back out before showing it to the model. Refs are what make a click unambiguous: aria-ref resolves to exactly one element in ~2ms, so it can raise no strict-mode violation, produce no "multiple elements" error, and never reach disambiguateElements — the model call measured at ~10s per ambiguous click, and 21% of one check's wall clock. clickRef takes a ref from the context and clicks it through Playwright's own selector engine, so CodeceptJS locator resolution never runs. It sits beside click rather than changing it, so nothing that composes locators is affected. click now says to prefer it when the element carries a ref. Verified: 100 refs reach do's context, a ref resolves to one element in 2ms, and refs survive a DOM change caused by our own click. Adoption by the model is not yet consistent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- boat/prima/src/prima.ts | 21 ++++++++++++-------- boat/prima/tests/prima.test.ts | 2 +- src/ai/tools.ts | 36 ++++++++++++++++++++++++++++++++++ 3 files changed, 50 insertions(+), 9 deletions(-) diff --git a/boat/prima/src/prima.ts b/boat/prima/src/prima.ts index 9641eb87..5bec94bd 100644 --- a/boat/prima/src/prima.ts +++ b/boat/prima/src/prima.ts @@ -54,10 +54,6 @@ function dropVolatileColumns(markdown: string): string { }); } -function stripRefs(tree: string): string { - return tree.replace(/\s*\[ref=[^\]]+\]/g, ''); -} - function cap(text: string, max: number): string { if (text.length <= max) return text; return `${text.slice(0, max)}\n[...truncated; ${text.length - max} chars omitted...]`; @@ -686,14 +682,23 @@ export class Prima { </proof> <targets> - The page context lists every element by role and name. Target what you act on that way — a role with its name is stable while the page re-renders, so it keeps working after your own actions change the page. - Narrow with the container it sits in when a name appears more than once, rather than guessing at an id or a class. - When the element an instruction needs is missing from your newest context, call context() to look again and act on what it returns. Never invent a locator the context does not support. + The page context lists every element with a ref, like [ref=e14]. To click one, pass that ref to clickRef — a ref names one + exact element, so it cannot match several by mistake and costs nothing to resolve. This is the cheapest way to act. + Use click() with a role and name for anything clickRef cannot take, and narrow with the container it sits in when a name + appears more than once, rather than guessing at an id or a class. + Refs belong to the context you were given. Use the ones in your newest context, never one you invented or remembered from + an older page. When the element an instruction needs is missing from that context, call context() and act on what it returns. </targets> ${locatorRule} ${actionRule} + + <targets_first> + Everything above about composing locators applies to click() and the other locator tools. It does not apply when the + element carries a ref: pass that ref to clickRef instead and compose nothing. Reach for a locator only for elements + that have no ref, or when a ref has stopped resolving. + </targets_first> `; } @@ -784,7 +789,7 @@ export class Prima { return dedent` <page url="${result.url}" title="${result.title}"> - ${stripRefs(compactAriaSnapshot(await this.refAriaSnapshot(result), true, (value) => this.offloadValue(value)))} + ${compactAriaSnapshot(await this.refAriaSnapshot(result), true, (value) => this.offloadValue(value))} </page> ${experience} diff --git a/boat/prima/tests/prima.test.ts b/boat/prima/tests/prima.test.ts index 8eb7a556..630bda4c 100644 --- a/boat/prima/tests/prima.test.ts +++ b/boat/prima/tests/prima.test.ts @@ -735,7 +735,7 @@ describe('Prima.do', () => { const first = await captured.execute({ reason: 'the control is not in my context' }); expect(first.context).toContain('button "Refreshed"'); - expect(first.context).not.toContain('ref='); + expect(first.context).toContain('ref=e7'); const second = await captured.execute({ reason: 'still cannot reach it' }); expect(second.context).toContain('<form>'); diff --git a/src/ai/tools.ts b/src/ai/tools.ts index abb13005..cff4cbc3 100644 --- a/src/ai/tools.ts +++ b/src/ai/tools.ts @@ -33,6 +33,10 @@ export function createCodeceptJSTools({ explorer, stateManager, ai }: ToolDeps, description: dedent` Click an element by trying multiple CodeceptJS commands in order until one succeeds. + Use this only for elements the page context gives you no ref for. When the element shows a ref such as [ref=e14], + call clickRef with that ref instead — composing a locator for an element that already has a ref is wasted work, + and a locator can match several elements where a ref cannot. + Follow <locator_priority> from system prompt for locator selection. I.click(locator) - click element matching locator @@ -792,6 +796,38 @@ export function createAgentTools({ explorer, stateManager, ai, researcher, navig }, }), + clickRef: tool({ + description: dedent` + Click an element by the ref the page context gave it, e.g. [ref=e14]. + + Prefer this over click() whenever the element you want carries a ref. A ref names one exact element, so it + cannot match several by mistake and never needs disambiguating — it is the fastest way to click. + Only pass a ref that appears in the page context you were given. Never invent or guess one. + If it reports the ref is gone, the page has been rebuilt: get fresh context and use the new ref. + `, + inputSchema: z.object({ + ref: z.string().describe('The ref exactly as it appears in the page context, e.g. "e14"'), + element: z.string().describe('Role and name of the element you are clicking, for the record'), + }), + execute: async ({ ref, element }) => { + const activeNote = task.startNote(`Click ${element}`); + const previousState = ActionResult.fromState(stateManager.getCurrentState()!); + const action = explorer.action(); + const code = `I.usePlaywrightTo(${JSON.stringify(`click ${element}`)}, async ({ page }) => page.locator(${JSON.stringify(`aria-ref=${ref}`)}).click())`; + + if (!(await action.attempt(code, `Click ${element}`))) { + activeNote.commit(TestResult.FAILED); + return failedToolResult('clickRef', `Ref ${ref} could not be clicked: ${errorText(action.lastError)}`, { + suggestion: 'The ref may belong to an older version of the page. Get fresh context and use the ref it gives, or fall back to click() with a locator.', + }); + } + + const toolResult = await ActionResult.fromState(stateManager.getCurrentState()!).toToolResult(previousState, code); + await commitNote(activeNote, TestResult.PASSED, toolResult, action); + return successToolResult('clickRef', { ...toolResult, code }, action); + }, + }), + visualClick: tool({ description: dedent` Click an element by visual identification when locator-based click() fails. From b5c023eae59fdfac923b8d69f9511c43a3b3f6cf Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Sun, 9 Aug 2026 17:58:47 +0300 Subject: [PATCH 18/39] Report a ref click as a locator a later test can replay MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit clickRef executes through aria-ref, which names an element only for the session that produced the snapshot. Reporting that as the executed code put a selector into generated tests that cannot resolve when they run again. The click still goes through the ref — that is what makes it unambiguous and fast. What gets reported is the element's own role and accessible name, read back from the page, so the generated test has something that survives the session. If the element cannot be described that way, the ref form is reported unchanged rather than a guess. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- src/ai/tools.ts | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/src/ai/tools.ts b/src/ai/tools.ts index cff4cbc3..9f863f46 100644 --- a/src/ai/tools.ts +++ b/src/ai/tools.ts @@ -813,15 +813,18 @@ export function createAgentTools({ explorer, stateManager, ai, researcher, navig const activeNote = task.startNote(`Click ${element}`); const previousState = ActionResult.fromState(stateManager.getCurrentState()!); const action = explorer.action(); - const code = `I.usePlaywrightTo(${JSON.stringify(`click ${element}`)}, async ({ page }) => page.locator(${JSON.stringify(`aria-ref=${ref}`)}).click())`; + const named = await describeRef(explorer, ref); + const run = `I.usePlaywrightTo(${JSON.stringify(`click ${element}`)}, async ({ page }) => page.locator(${JSON.stringify(`aria-ref=${ref}`)}).click())`; - if (!(await action.attempt(code, `Click ${element}`))) { + if (!(await action.attempt(run, `Click ${element}`))) { activeNote.commit(TestResult.FAILED); return failedToolResult('clickRef', `Ref ${ref} could not be clicked: ${errorText(action.lastError)}`, { suggestion: 'The ref may belong to an older version of the page. Get fresh context and use the ref it gives, or fall back to click() with a locator.', }); } + // a ref belongs to this session only, so the run is reported as the locator a later test can replay + const code = named ? `I.click(${JSON.stringify(named)})` : run; const toolResult = await ActionResult.fromState(stateManager.getCurrentState()!).toToolResult(previousState, code); await commitNote(activeNote, TestResult.PASSED, toolResult, action); return successToolResult('clickRef', { ...toolResult, code }, action); @@ -1159,6 +1162,19 @@ export async function commitNote(activeNote: any, result: TestResult, toolResult activeNote.commit(result); } +async function describeRef(explorer: any, ref: string): Promise<{ role: string; text: string } | null> { + return explorer + .withPage(async (page: any) => { + const handle = page.locator(`aria-ref=${ref}`); + const role = await handle.getAttribute('role'); + const label = await handle.getAttribute('aria-label'); + const text = (label || (await handle.innerText()) || '').trim().split('\n')[0]; + if (!role || !text) return null; + return { role, text }; + }) + .catch(() => null); +} + async function hasFocusedElement(explorer: any): Promise<boolean> { return explorer.withPage((page: any) => page.evaluate(() => !!document.activeElement && document.activeElement !== document.body)).catch(() => true); } From 68e36c246fb4cfc6e20c159b96422da4775a3bd9 Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Sun, 9 Aug 2026 18:02:29 +0300 Subject: [PATCH 19/39] Give the tester refs in its page context MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit check runs through the tester, which had clickRef available but no refs to pass it — its ARIA block came from the stored snapshot, captured without mode: 'ai'. The tool was unreachable in the tier where clicks cost most. The block is now built from a ref-bearing snapshot taken at the moment the context is written. The stored snapshot is untouched, so state hashes and page diffs stay ref-free and an identical page whose refs merely renumber still reads as unchanged. A page that cannot be re-snapshotted falls back to the stored one rather than losing its context. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- src/ai/tester.ts | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/src/ai/tester.ts b/src/ai/tester.ts index e59414d3..b160cc6e 100644 --- a/src/ai/tester.ts +++ b/src/ai/tester.ts @@ -11,7 +11,7 @@ import { Observability } from '../observability.ts'; import type { StateTransition } from '../state-manager.ts'; import { Stats } from '../stats.ts'; import { type Test, TestResult, type TestResultType } from '../test-plan.ts'; -import { detectFocusArea } from '../utils/aria.ts'; +import { compactAriaSnapshot, detectFocusArea } from '../utils/aria.ts'; import { ErrorPageError, isErrorPage } from '../utils/error-page.ts'; import { createDebug, tag } from '../utils/logger.ts'; import { loop } from '../utils/loop.ts'; @@ -597,7 +597,7 @@ export class Tester extends TaskAgent implements Agent { </page> <page_aria> - ${currentState.getInteractiveARIA()} + ${await this.interactiveAriaWithRefs(currentState)} </page_aria> ${uiMapSection} @@ -634,11 +634,17 @@ export class Tester extends TaskAgent implements Agent { </page> <page_aria> - ${currentState.getInteractiveARIA()} + ${await this.interactiveAriaWithRefs(currentState)} </page_aria> `; } + private async interactiveAriaWithRefs(state: ActionResult): Promise<string> { + const withRefs = await Promise.resolve(this.explorer?.withPage?.((page: any) => page.locator('body').ariaSnapshot({ mode: 'ai' }))).catch(() => null); + if (!withRefs) return state.getInteractiveARIA(); + return compactAriaSnapshot(withRefs, false); + } + private finishTest(task: Test): void { if (!task.result) { if (task.hasAchievedAll()) task.finish(TestResult.PASSED); From 9207e63c3d05c1d184d68b73e0b1b71ba9430fbf Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Sun, 9 Aug 2026 18:12:03 +0300 Subject: [PATCH 20/39] Make clickRef actually run, and offer it where refs exist MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit clickRef was defined in createAgentTools, which has no task in scope, so its first line threw ReferenceError on every call. It had never executed once — which is why no run ever produced a ref click, and why the model looked like it was ignoring the tool. It lives beside click now, where the task it records into is in scope. do offers clickRef while the accessibility tree is the context and brings click back only once the model has dropped to markup, where there are no refs to use. That follows the shape of the context rather than asking the model to choose between two tools for the same job. The recorded locator derives the element's implicit role, since a plain button carries no role attribute and the description came back empty. Verified against a live page: a ref click succeeds in ~800ms and reports I.click({"role":"button","text":"..."}). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- boat/prima/src/prima.ts | 11 +++- boat/prima/tests/prima.test.ts | 27 +++++++++ src/ai/tools.ts | 92 +++++++++++++++--------------- tests/integration/prima-do.test.ts | 22 ++++--- 4 files changed, 96 insertions(+), 56 deletions(-) diff --git a/boat/prima/src/prima.ts b/boat/prima/src/prima.ts index 5bec94bd..c353d477 100644 --- a/boat/prima/src/prima.ts +++ b/boat/prima/src/prima.ts @@ -139,7 +139,12 @@ export class Prima { const task = new Task(instructions.join('; '), previousState?.url || ''); const deps = { explorer: this.bot.getExplorer(), stateManager: this.bot.stateManager(), ai: provider }; const ledger: LedgerEntry[] = instructions.map((text) => ({ text, status: 'open', proof: '', evidence: [] })); - const tools = { ...createCodeceptJSTools(deps, task), ...this.testerTools(deps), context: this.contextTool(), completed: this.completedTool(), blocked: this.blockedTool() }; + const descent = { markup: false }; + const codeceptTools = createCodeceptJSTools(deps, task); + const baseTools = { ...codeceptTools, ...this.testerTools(deps), context: this.contextTool(descent), completed: this.completedTool(), blocked: this.blockedTool() }; + // the accessibility tree carries a ref for every element, so clickRef is the only click it needs; + // click returns once the model has dropped to markup, where there are no refs to use + const { click, ...refTools } = baseTools; conversation.addUserText(await this.instructionPrompt(instructions, await this.capturedResult(previousState))); const used: string[] = []; @@ -168,6 +173,7 @@ export class Prima { `); } + const tools = descent.markup ? baseTools : refTools; const invoked = await provider.invokeConversation(conversation, tools, { maxToolRoundtrips: MAX_TOOL_ROUNDTRIPS, agentName: AI_AGENT_NAME }).catch((error: unknown) => { aiError = error; return null; @@ -751,7 +757,7 @@ export class Prima { }); } - private contextTool(): any { + private contextTool(descent: { markup: boolean }): any { let refreshed = false; return tool({ description: dedent` @@ -769,6 +775,7 @@ export class Prima { refreshed = true; return { success: true, context: await this.pageContext(result) }; } + descent.markup = true; return { success: true, context: cap(await result.simplifiedHtml(), CONTEXT_HTML_CAP) }; }, }); diff --git a/boat/prima/tests/prima.test.ts b/boat/prima/tests/prima.test.ts index 630bda4c..95f35af0 100644 --- a/boat/prima/tests/prima.test.ts +++ b/boat/prima/tests/prima.test.ts @@ -722,6 +722,33 @@ describe('Prima.do', () => { expect(envelope.ok).toBe(true); }); + test('clickRef replaces click while the tree is the context, and click returns on markup', async () => { + const { prima } = fakePrima(); + (prima as any).bot.agentResearcher = () => ({}); + (prima as any).bot.agentNavigator = () => ({}); + const offered: string[][] = []; + let contextTool: any; + let calls = 0; + (prima as any).bot.getProvider = () => + fakeProvider(async (_conversation: unknown, tools: any) => { + offered.push(Object.keys(tools)); + contextTool = tools.context; + calls++; + if (calls === 1) { + await contextTool.execute({ reason: 'tree' }); + await contextTool.execute({ reason: 'markup please' }); + return { toolExecutions: [toolExecution("I.click('Invoices')")] }; + } + return { toolExecutions: [completedExecution([1], 'done')] }; + }); + + await prima.do(['open the invoices page']); + + expect(offered[0]).toContain('clickRef'); + expect(offered[0]).not.toContain('click'); + expect(offered[1]).toContain('click'); + }); + test('context() hands back the page tree first and drops to markup only when asked again', async () => { const { prima } = fakePrima(); let captured: any; diff --git a/src/ai/tools.ts b/src/ai/tools.ts index 9f863f46..d818f7c5 100644 --- a/src/ai/tools.ts +++ b/src/ai/tools.ts @@ -157,6 +157,41 @@ export function createCodeceptJSTools({ explorer, stateManager, ai }: ToolDeps, }, }), + clickRef: tool({ + description: dedent` + Click an element by the ref the page context gave it, e.g. [ref=e14]. + + Prefer this over click() whenever the element you want carries a ref. A ref names one exact element, so it + cannot match several by mistake and never needs disambiguating — it is the fastest way to click. + Only pass a ref that appears in the page context you were given. Never invent or guess one. + If it reports the ref is gone, the page has been rebuilt: get fresh context and use the new ref. + `, + inputSchema: z.object({ + ref: z.string().describe('The ref exactly as it appears in the page context, e.g. "e14"'), + element: z.string().describe('Role and name of the element you are clicking, for the record'), + }), + execute: async ({ ref, element }) => { + const activeNote = task.startNote(`Click ${element}`); + const previousState = ActionResult.fromState(stateManager.getCurrentState()!); + const action = explorer.action(); + const named = await describeRef(explorer, ref); + const run = `I.usePlaywrightTo(${JSON.stringify(`click ${element}`)}, async ({ page }) => page.locator(${JSON.stringify(`aria-ref=${ref}`)}).click())`; + + if (!(await action.attempt(run, `Click ${element}`))) { + activeNote.commit(TestResult.FAILED); + return failedToolResult('clickRef', `Ref ${ref} could not be clicked: ${errorText(action.lastError)}`, { + suggestion: 'The ref may belong to an older version of the page. Get fresh context and use the ref it gives, or fall back to click() with a locator.', + }); + } + + // a ref belongs to this session only, so the run is reported as the locator a later test can replay + const code = named ? `I.click(${JSON.stringify(named)})` : run; + const toolResult = await ActionResult.fromState(stateManager.getCurrentState()!).toToolResult(previousState, code); + await commitNote(activeNote, TestResult.PASSED, toolResult, action); + return successToolResult('clickRef', { ...toolResult, code }, action); + }, + }), + hover: tool({ description: dedent` Move the mouse cursor to an element to reveal hover-only controls. @@ -796,41 +831,6 @@ export function createAgentTools({ explorer, stateManager, ai, researcher, navig }, }), - clickRef: tool({ - description: dedent` - Click an element by the ref the page context gave it, e.g. [ref=e14]. - - Prefer this over click() whenever the element you want carries a ref. A ref names one exact element, so it - cannot match several by mistake and never needs disambiguating — it is the fastest way to click. - Only pass a ref that appears in the page context you were given. Never invent or guess one. - If it reports the ref is gone, the page has been rebuilt: get fresh context and use the new ref. - `, - inputSchema: z.object({ - ref: z.string().describe('The ref exactly as it appears in the page context, e.g. "e14"'), - element: z.string().describe('Role and name of the element you are clicking, for the record'), - }), - execute: async ({ ref, element }) => { - const activeNote = task.startNote(`Click ${element}`); - const previousState = ActionResult.fromState(stateManager.getCurrentState()!); - const action = explorer.action(); - const named = await describeRef(explorer, ref); - const run = `I.usePlaywrightTo(${JSON.stringify(`click ${element}`)}, async ({ page }) => page.locator(${JSON.stringify(`aria-ref=${ref}`)}).click())`; - - if (!(await action.attempt(run, `Click ${element}`))) { - activeNote.commit(TestResult.FAILED); - return failedToolResult('clickRef', `Ref ${ref} could not be clicked: ${errorText(action.lastError)}`, { - suggestion: 'The ref may belong to an older version of the page. Get fresh context and use the ref it gives, or fall back to click() with a locator.', - }); - } - - // a ref belongs to this session only, so the run is reported as the locator a later test can replay - const code = named ? `I.click(${JSON.stringify(named)})` : run; - const toolResult = await ActionResult.fromState(stateManager.getCurrentState()!).toToolResult(previousState, code); - await commitNote(activeNote, TestResult.PASSED, toolResult, action); - return successToolResult('clickRef', { ...toolResult, code }, action); - }, - }), - visualClick: tool({ description: dedent` Click an element by visual identification when locator-based click() fails. @@ -1163,16 +1163,18 @@ export async function commitNote(activeNote: any, result: TestResult, toolResult } async function describeRef(explorer: any, ref: string): Promise<{ role: string; text: string } | null> { - return explorer - .withPage(async (page: any) => { - const handle = page.locator(`aria-ref=${ref}`); - const role = await handle.getAttribute('role'); - const label = await handle.getAttribute('aria-label'); - const text = (label || (await handle.innerText()) || '').trim().split('\n')[0]; - if (!role || !text) return null; - return { role, text }; - }) - .catch(() => null); + return Promise.resolve( + explorer?.withPage?.((page: any) => + page.locator(`aria-ref=${ref}`).evaluate((el: any) => { + const tag = el.tagName.toLowerCase(); + const roles: Record<string, string> = { a: 'link', button: 'button', select: 'combobox', textarea: 'textbox' }; + const role = el.getAttribute('role') || roles[tag] || tag; + const text = (el.getAttribute('aria-label') || el.innerText || el.value || '').trim().split('\n')[0]; + if (!text) return null; + return { role, text }; + }) + ) + ).catch(() => null); } async function hasFocusedElement(explorer: any): Promise<boolean> { diff --git a/tests/integration/prima-do.test.ts b/tests/integration/prima-do.test.ts index fe59c7fa..a2fb8a76 100644 --- a/tests/integration/prima-do.test.ts +++ b/tests/integration/prima-do.test.ts @@ -20,6 +20,10 @@ function clickCall(id: string, commands: string[], explanation: string) { return { id, name: 'click', arguments: JSON.stringify({ commands, explanation }) }; } +function clickRefCall(id: string, ref: string, element: string) { + return { id, name: 'clickRef', arguments: JSON.stringify({ ref, element }) }; +} + function completedCall(id: string, numbers: number[], proof: string) { return { id, name: 'completed', arguments: JSON.stringify({ numbers, proof }) }; } @@ -89,6 +93,8 @@ describe('Prima.do with aimock', () => { requestStore: () => ({ getRequests: () => [] }), getProvider: () => provider, experienceTracker: () => ({ renderExperienceTocFor: () => '' }), + agentResearcher: () => ({}), + agentNavigator: () => ({}), }; }); @@ -99,20 +105,17 @@ describe('Prima.do with aimock', () => { }); it('runs both instructions and collects the executed code', async () => { - mock.on({ sequenceIndex: 0 }, { toolCalls: [clickCall('call-1', ['I.click("Account")'], 'open the account menu'), completedCall('call-2', [1], 'the account menu is open')] }); - mock.on({ sequenceIndex: 1 }, { toolCalls: [clickCall('call-3', ['I.click("Settings")'], 'choose the settings entry'), completedCall('call-4', [2], 'the settings page is shown')] }); + mock.on({ sequenceIndex: 0 }, { toolCalls: [clickRefCall('call-1', 'e5', 'button "Account"'), completedCall('call-2', [1], 'the account menu is open')] }); + mock.on({ sequenceIndex: 1 }, { toolCalls: [clickRefCall('call-3', 'e9', 'link "Settings"'), completedCall('call-4', [2], 'the settings page is shown')] }); mock.on({}, { content: 'Both instructions are done.' }); const envelope = await prima.do(['open the account menu', 'choose the settings entry']); expect(envelope.ok).toBe(true); - expect(envelope.used).toEqual(['I.click("Account")', 'I.click("Settings")']); - expect(executed).toEqual(['I.click("Account")', 'I.click("Settings")']); + expect(executed.every((code) => code.includes('aria-ref='))).toBe(true); + expect(executed).toHaveLength(2); expect(envelope.command).toContain('open the account menu'); - expect(envelope.steps).toEqual([ - { label: 'open the account menu', ok: true, proof: 'the account menu is open\nI.click("Account")' }, - { label: 'choose the settings entry', ok: true, proof: 'the settings page is shown\nI.click("Settings")' }, - ]); + expect(envelope.steps?.map((step) => step.ok)).toEqual([true, true]); }); it('sends every instruction and the page context in one prompt', async () => { @@ -134,7 +137,8 @@ describe('Prima.do with aimock', () => { await prima.do(['open the account menu']); const tools = (mock.getRequests()[0] as any).body.tools.map((entry: any) => entry.function.name); - expect(tools).toContain('click'); + expect(tools).toContain('clickRef'); + expect(tools).not.toContain('click'); expect(tools).toContain('form'); }); From a6bc34398ccc7f81185356549426f1f3b0bc29e7 Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Sun, 9 Aug 2026 18:14:32 +0300 Subject: [PATCH 21/39] Keep click alongside clickRef MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Withholding click while the accessibility tree was the context did not push the model onto refs — it routed the same locator clicks through form, which takes arbitrary CodeceptJS commands — and the run came back with every instruction unaccounted for where the same three had closed before. Both tools are offered again. clickRef stays available and working; which one the model reaches for is a separate problem from whether the ref path exists, and starving it of the familiar tool is not what solves it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- boat/prima/src/prima.ts | 7 +------ boat/prima/tests/prima.test.ts | 27 --------------------------- tests/integration/prima-do.test.ts | 2 +- 3 files changed, 2 insertions(+), 34 deletions(-) diff --git a/boat/prima/src/prima.ts b/boat/prima/src/prima.ts index c353d477..8bf3d068 100644 --- a/boat/prima/src/prima.ts +++ b/boat/prima/src/prima.ts @@ -140,11 +140,7 @@ export class Prima { const deps = { explorer: this.bot.getExplorer(), stateManager: this.bot.stateManager(), ai: provider }; const ledger: LedgerEntry[] = instructions.map((text) => ({ text, status: 'open', proof: '', evidence: [] })); const descent = { markup: false }; - const codeceptTools = createCodeceptJSTools(deps, task); - const baseTools = { ...codeceptTools, ...this.testerTools(deps), context: this.contextTool(descent), completed: this.completedTool(), blocked: this.blockedTool() }; - // the accessibility tree carries a ref for every element, so clickRef is the only click it needs; - // click returns once the model has dropped to markup, where there are no refs to use - const { click, ...refTools } = baseTools; + const tools = { ...createCodeceptJSTools(deps, task), ...this.testerTools(deps), context: this.contextTool(descent), completed: this.completedTool(), blocked: this.blockedTool() }; conversation.addUserText(await this.instructionPrompt(instructions, await this.capturedResult(previousState))); const used: string[] = []; @@ -173,7 +169,6 @@ export class Prima { `); } - const tools = descent.markup ? baseTools : refTools; const invoked = await provider.invokeConversation(conversation, tools, { maxToolRoundtrips: MAX_TOOL_ROUNDTRIPS, agentName: AI_AGENT_NAME }).catch((error: unknown) => { aiError = error; return null; diff --git a/boat/prima/tests/prima.test.ts b/boat/prima/tests/prima.test.ts index 95f35af0..630bda4c 100644 --- a/boat/prima/tests/prima.test.ts +++ b/boat/prima/tests/prima.test.ts @@ -722,33 +722,6 @@ describe('Prima.do', () => { expect(envelope.ok).toBe(true); }); - test('clickRef replaces click while the tree is the context, and click returns on markup', async () => { - const { prima } = fakePrima(); - (prima as any).bot.agentResearcher = () => ({}); - (prima as any).bot.agentNavigator = () => ({}); - const offered: string[][] = []; - let contextTool: any; - let calls = 0; - (prima as any).bot.getProvider = () => - fakeProvider(async (_conversation: unknown, tools: any) => { - offered.push(Object.keys(tools)); - contextTool = tools.context; - calls++; - if (calls === 1) { - await contextTool.execute({ reason: 'tree' }); - await contextTool.execute({ reason: 'markup please' }); - return { toolExecutions: [toolExecution("I.click('Invoices')")] }; - } - return { toolExecutions: [completedExecution([1], 'done')] }; - }); - - await prima.do(['open the invoices page']); - - expect(offered[0]).toContain('clickRef'); - expect(offered[0]).not.toContain('click'); - expect(offered[1]).toContain('click'); - }); - test('context() hands back the page tree first and drops to markup only when asked again', async () => { const { prima } = fakePrima(); let captured: any; diff --git a/tests/integration/prima-do.test.ts b/tests/integration/prima-do.test.ts index a2fb8a76..7f07473f 100644 --- a/tests/integration/prima-do.test.ts +++ b/tests/integration/prima-do.test.ts @@ -138,7 +138,7 @@ describe('Prima.do with aimock', () => { const tools = (mock.getRequests()[0] as any).body.tools.map((entry: any) => entry.function.name); expect(tools).toContain('clickRef'); - expect(tools).not.toContain('click'); + expect(tools).toContain('click'); expect(tools).toContain('form'); }); From 3f1cc223ebb92a08765d4bf24fa468337b43f445 Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Sun, 9 Aug 2026 20:58:55 +0300 Subject: [PATCH 22/39] Say that a ref-bearing element needs no locator MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The locator priority rule opened straight into how to compose a locator, and form invited any CodeceptJS command, so both taught locator composition as the way to click even for elements the context had already named with a ref. Each now says what it does not cover: an element carrying a ref is clicked with clickRef and that ref, and the priority list is for elements the context gives no ref for. The clause is self-limiting — an agent whose context has no refs never meets the case — so navigator and tester read the same rule they did before. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- src/ai/rules.ts | 4 ++++ src/ai/tools.ts | 2 ++ 2 files changed, 6 insertions(+) diff --git a/src/ai/rules.ts b/src/ai/rules.ts index ab8b6c94..05fac8b7 100644 --- a/src/ai/rules.ts +++ b/src/ai/rules.ts @@ -4,6 +4,10 @@ export const recommendedCodeceptCommands = ['I.click', 'I.type', 'I.fillField', const locatorPriorityRule = dedent` <locator_priority> + When the page context shows the element a ref, such as [ref=e14], there is no locator to select: click it with clickRef + and that ref. A ref names one exact element, so it never matches the wrong one and never has to be narrowed. Everything + below is for elements the context gives no ref for. + Use the following priority when selecting locators: 1. ARIA locators (first choice) - target browser's accessibility tree, most reliable diff --git a/src/ai/tools.ts b/src/ai/tools.ts index d818f7c5..172cee15 100644 --- a/src/ai/tools.ts +++ b/src/ai/tools.ts @@ -393,6 +393,8 @@ export function createCodeceptJSTools({ explorer, stateManager, ai }: ToolDeps, Execute raw CodeceptJS code block with multiple commands. USE THIS TOOL for typing text into fields: I.fillField, I.type + Do not put a click on a ref-bearing element in here — clickRef with its ref is cheaper and cannot mis-target. + Follow <actions> from system prompt for available commands. Follow <locator_priority> from system prompt for locator selection. From 83166d5a84c87d0d9ee23d35e48a37c200270677 Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Mon, 10 Aug 2026 00:06:01 +0300 Subject: [PATCH 23/39] Cut prima's help to the commands and how to call them MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The help was 6.7k, then 4.4k, of prose about envelope sections and session management. Every agent reads it before its first command, and in one benchmark it was over half of everything prima put into the caller's context — to explain output the agent is about to see anyway and flags the option list already documents. What is left says what prima is, how it pairs with playwright-cli, and shows check, do and pw carrying a whole job rather than one click. 2.4k, most of it the command list commander generates. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- boat/prima/bin/prima-cli.ts | 0 boat/prima/src/cli.ts | 58 ++++++++++--------------------------- 2 files changed, 15 insertions(+), 43 deletions(-) mode change 100644 => 100755 boat/prima/bin/prima-cli.ts diff --git a/boat/prima/bin/prima-cli.ts b/boat/prima/bin/prima-cli.ts old mode 100644 new mode 100755 diff --git a/boat/prima/src/cli.ts b/boat/prima/src/cli.ts index fab4e9cd..5e810ef0 100644 --- a/boat/prima/src/cli.ts +++ b/boat/prima/src/cli.ts @@ -7,49 +7,21 @@ import { type EnvelopeData, renderEnvelope } from './envelope.ts'; import { Prima, type PrimaOptions } from './prima.ts'; const helpContract = dedent` - Prima drives a browser that is already open. One command per process; every command - prints a plain-text envelope on stdout and exits 0 when ok, 1 when not. - - TIERS - start at the top and come down only when the tier above cannot hold the work - check <scenario> - A whole behaviour, run as a test. It drives the page, verifies the - outcome itself, and reports each expected outcome with its proof. - One command for something you want a verdict on. Start here. - prima check "a workflow can be created and appears in the list" \\ - --expected "the new workflow is listed" - do <steps...> Several described steps, carried out in one process. Pass the WHOLE - remaining sequence - a run of eight steps costs a fraction of eight - commands, and that is the whole point of this tier. - prima do "open the account menu" "choose the settings entry" \\ - "switch the theme to dark" "check the change took effect" - pw <fn> One Playwright function expression, from a locator you already - verified. No AI. For when you know exactly what to run. - prima pw "({ page }) => page.click('[data-test=submit]')" - Describe targets to check and do; give pw executable code only. Never mix the two. - Coming down a tier to run steps one at a time costs more than the tier above, in - both time and what you have to read - it is a fallback, not a default. - - Also: ask, verify, research, go, status, browser. Run <command> --help for what - each of them reports. - - ENVELOPE - ### Result ok/command/used · ### Page url/title/state · ### Changes what the - accessibility tree gained, lost, toggled or had typed · ### Steps per-instruction - outcome with its proof · ### Expected outcomes each --expected as PASSED, FAILED or - not verified · ### Answer|Research|Assertions · ### Failure error and compact ARIA · - ### Instance · ### Artifacts. Start at the first ### line; logs can precede it. - used: is code that already executed. A failed action fails - nothing is retried along - a different route, so ok: true means your own action landed. - - SESSIONS - Prima attaches to the playwright-cli browser of this workspace and drives the tabs it - already has open; using the same session from both tools is the intended usage. - playwright-cli open <url> the session prima attaches to - --pw-session <title> which session, when several are open - prima browser start a prima-owned browser instead, when none is open - Prima never launches a browser implicitly and never closes an attached one. - Run it as "npx explorbot prima ..." or through the published prima bin - the - browser-server connection needs the Node build. + Prima is a high-level AI extension to playwright-cli. It drives the browser that + playwright-cli already has open. + + playwright-cli open <url> start the session + prima <command> ... drive it + playwright-cli close end it + + Give check and do a whole job, never a single click - one call carries the sequence, + and that is what makes them cheaper than driving playwright-cli step by step. + + prima check "a workflow can be created and appears in the list" --expected "the new workflow is listed" + prima do "open the account menu" "choose the settings entry" "switch the theme to dark" "check it took effect" + prima pw "({ page }) => page.click('[data-test=submit]')" + + Describe targets to check and do; give pw executable code only. `; const checkHelp = dedent` From fcfec112fe6a28a43a2e8b95868ddb34b25faf5d Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Mon, 10 Aug 2026 00:09:22 +0300 Subject: [PATCH 24/39] State what prima's commands are, rather than instructing the model MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The help told the model what to do — give check a whole job, describe targets, give pw code only. Instructions compete with everything else in the context for authority. A description of what each command takes and what it costs carries the same information without spending that. The same rewrite applies to the check and do help. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- boat/prima/src/cli.ts | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/boat/prima/src/cli.ts b/boat/prima/src/cli.ts index 5e810ef0..59265587 100644 --- a/boat/prima/src/cli.ts +++ b/boat/prima/src/cli.ts @@ -10,34 +10,35 @@ const helpContract = dedent` Prima is a high-level AI extension to playwright-cli. It drives the browser that playwright-cli already has open. - playwright-cli open <url> start the session - prima <command> ... drive it - playwright-cli close end it + playwright-cli open <url> starts the session + prima <command> ... drives it + playwright-cli close ends it - Give check and do a whole job, never a single click - one call carries the sequence, - and that is what makes them cheaper than driving playwright-cli step by step. + check and do each carry a whole job in one call - a behaviour to verify, or a sequence + of steps to carry out. That is where they cost less than the same work driven through + playwright-cli one command at a time. prima check "a workflow can be created and appears in the list" --expected "the new workflow is listed" prima do "open the account menu" "choose the settings entry" "switch the theme to dark" "check it took effect" prima pw "({ page }) => page.click('[data-test=submit]')" - Describe targets to check and do; give pw executable code only. + check and do take targets described in words. pw takes executable code. `; const checkHelp = dedent` - Give it an outcome, not a click path - it decides how to get there. - --expected one outcome the run must reach; repeat it for several. Without it the + check takes an outcome rather than a click path, and works out how to reach it. + --expected one outcome the run must reach, repeatable for several. Without it the scenario text is the single expected outcome. Each comes back under ### Expected outcomes as PASSED, FAILED or not verified - "not verified" means the run never checked it, which is not the same as false. - Page problems seen on the way are reported under ### Answer, not as step failures. + Page problems seen on the way appear under ### Answer, not as step failures. `; const doHelp = dedent` - Every instruction is numbered and accounted for: ### Steps reports each as ok or FAIL + Each instruction is numbered and accounted for: ### Steps reports each as ok or FAIL with what proved it. One that could not be carried out fails the command and says why. - Nothing runs past the last instruction you gave. - Pass the whole remaining sequence in one call - that is what makes this tier cheap. + Nothing runs past the last instruction given. A whole remaining sequence in one call is + what makes this tier cheap. `; const verifyHelp = dedent` From 28a4f7e18318a95812956c34e7383f793b490599 Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Mon, 10 Aug 2026 00:16:25 +0300 Subject: [PATCH 25/39] Drop the explanatory paragraph from prima's help The examples show what check, do and pw take. The paragraph above them restated it in longer words. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- boat/prima/src/cli.ts | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/boat/prima/src/cli.ts b/boat/prima/src/cli.ts index 59265587..e5b6c91c 100644 --- a/boat/prima/src/cli.ts +++ b/boat/prima/src/cli.ts @@ -7,22 +7,17 @@ import { type EnvelopeData, renderEnvelope } from './envelope.ts'; import { Prima, type PrimaOptions } from './prima.ts'; const helpContract = dedent` - Prima is a high-level AI extension to playwright-cli. It drives the browser that - playwright-cli already has open. + Prima is a high-level AI extension to playwright-cli, driving the browser it has open. playwright-cli open <url> starts the session prima <command> ... drives it playwright-cli close ends it - check and do each carry a whole job in one call - a behaviour to verify, or a sequence - of steps to carry out. That is where they cost less than the same work driven through - playwright-cli one command at a time. + One call takes a whole job: prima check "a workflow can be created and appears in the list" --expected "the new workflow is listed" prima do "open the account menu" "choose the settings entry" "switch the theme to dark" "check it took effect" prima pw "({ page }) => page.click('[data-test=submit]')" - - check and do take targets described in words. pw takes executable code. `; const checkHelp = dedent` From baf80002ddc45bc166a1e5972decf6a2fc38245c Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Mon, 10 Aug 2026 00:18:46 +0300 Subject: [PATCH 26/39] Say what prima is for in its one-line description "Drive an already-open browser one command at a time" describes the plumbing, which an agent reading help does not need to decide anything. What decides is that prima takes described behaviour rather than locators, and returns a verdict with its proof. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- boat/prima/src/cli.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/boat/prima/src/cli.ts b/boat/prima/src/cli.ts index e5b6c91c..e3513cde 100644 --- a/boat/prima/src/cli.ts +++ b/boat/prima/src/cli.ts @@ -134,7 +134,7 @@ async function runBrowser(options: any, run: (prima: Prima) => Promise<boolean>) export function createPrimaCommands(name = 'prima'): Command { const cmd = new Command(name); - cmd.description('Drive an already-open browser one command at a time and report back in a plain-text envelope'); + cmd.description('Tests and drives a web app through described behaviour instead of locators: one command carries a whole scenario, verifies it, and reports the proof'); cmd.option('--pw-session <title>', 'Title of the playwright-cli session to attach to'); cmd.option('--url <url>', 'Page to open when the session has no page yet'); cmd.addHelpText('after', `\n${helpContract}`); From f4017c9120bc56edbe00b9f83883c25f960785a6 Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Mon, 10 Aug 2026 00:26:42 +0300 Subject: [PATCH 27/39] Add prima models MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Which model answers for which role was only discoverable by reading ~/.explorbot/config.js. prima models prints the resolved roles and the config file they came from. It reads configuration only — no browser, no session, and any registered site will do — so it answers before anything is running. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- boat/prima/src/cli.ts | 8 ++++++++ boat/prima/src/prima.ts | 27 +++++++++++++++++++++++++++ 2 files changed, 35 insertions(+) diff --git a/boat/prima/src/cli.ts b/boat/prima/src/cli.ts index e3513cde..c27d0c46 100644 --- a/boat/prima/src/cli.ts +++ b/boat/prima/src/cli.ts @@ -178,6 +178,14 @@ export function createPrimaCommands(name = 'prima'): Command { await runPrima(options, `go ${target}`, (prima) => prima.go(target)); }); + addCommonOptions(cmd.command('models').description('Show which AI model answers for each role')).action(async (options) => { + setQuietMode(!options.verbose && !options.debug); + const prima = primaFor(options); + console.log(await prima.models().catch((error: unknown) => browserErrorMessage(error))); + await prima.stop().catch(() => {}); + process.exit(0); + }); + addCommonOptions(cmd.command('status <hash>').description('Show the artifacts and page detail recorded for an earlier command')).action(async (hash, options) => { await runPrima(options, `status ${hash}`, (prima) => prima.status(hash)); }); diff --git a/boat/prima/src/prima.ts b/boat/prima/src/prima.ts index 8bf3d068..27723dd6 100644 --- a/boat/prima/src/prima.ts +++ b/boat/prima/src/prima.ts @@ -11,6 +11,7 @@ import { ActionResult } from '../../../src/action-result.ts'; import { actionRule, locatorRule } from '../../../src/ai/rules.ts'; import { createAgentTools, createCodeceptJSTools } from '../../../src/ai/tools.ts'; import { getAliveEndpoint, launchServer, listInstances, stopServer } from '../../../src/browser-server.ts'; +import { listSites } from '../../../src/global-config.ts'; import { ConfigMissingError, ConfigParser, type ExplorbotConfig, outputPath } from '../../../src/config.ts'; import { ExplorBot } from '../../../src/explorbot.ts'; import type { WebPageState } from '../../../src/state-manager.ts'; @@ -429,6 +430,32 @@ export class Prima { return stopped; } + async models(): Promise<string> { + const [site] = listSites(); + if (site && !this.configBaseUrl()) this.sessionUrl = site.url; + const config = await this.loadConfig(); + + const named = (model: unknown): string => { + if (typeof model === 'string') return model; + return (model as any)?.modelId || (model as any)?.model || 'unknown'; + }; + + const ai = config.ai || ({} as any); + const roles: Array<[string, unknown]> = [ + ['model', ai.model], + ['agenticModel', ai.agenticModel], + ['visionModel', ai.visionModel], + ]; + for (const [agent, agentConfig] of Object.entries(ai.agents || {})) { + if ((agentConfig as any)?.model) roles.push([agent, (agentConfig as any).model]); + } + + const lines = roles.filter(([, model]) => model).map(([role, model]) => `${role.padEnd(14)} ${named(model)}`); + lines.push(`config ${ConfigParser.getInstance().getConfigPath() || 'built-in defaults'}`); + if (ai.langfuse?.enabled) lines.push('telemetry langfuse'); + return lines.join('\n'); + } + async browserStatus(): Promise<string> { await this.loadConfig(); const info = await this.instanceInfo(); From 57fd97d5311b5b4c5faad0c0199b485044f34d01 Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Mon, 10 Aug 2026 01:41:56 +0300 Subject: [PATCH 28/39] Report do as an ordered log of what happened MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The envelope described the run three ways and none of them was a trace. used: listed only the commands that succeeded, so a run that clicked, failed, then escaped reported "used: I.pressKey(Escape)" as if that were the whole job. ### Changes repeated the page diff that already appeared under every step. And evidence only reached the envelope when the model reported an instruction — when it did not, everything it had done vanished behind "never reported". ### Steps is now the log: every action, check and report in the order it happened, each with its own outcome. One report closing several instructions is one line carrying one proof. Instructions nobody reported are appended as their own lines rather than swallowing the trace. used and ### Changes are gone from do, since the log carries both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- boat/prima/src/prima.ts | 37 +++++++++++++++--------------- boat/prima/tests/prima.test.ts | 37 ++++++++++++++++++++++++------ tests/integration/prima-do.test.ts | 2 +- 3 files changed, 50 insertions(+), 26 deletions(-) diff --git a/boat/prima/src/prima.ts b/boat/prima/src/prima.ts index 27723dd6..4112af0f 100644 --- a/boat/prima/src/prima.ts +++ b/boat/prima/src/prima.ts @@ -139,14 +139,14 @@ export class Prima { const conversation = provider.startConversation(this.instructionSystemPrompt(), AI_AGENT_NAME); const task = new Task(instructions.join('; '), previousState?.url || ''); const deps = { explorer: this.bot.getExplorer(), stateManager: this.bot.stateManager(), ai: provider }; - const ledger: LedgerEntry[] = instructions.map((text) => ({ text, status: 'open', proof: '', evidence: [] })); + const ledger: LedgerEntry[] = instructions.map((text) => ({ text, status: 'open', proof: '' })); const descent = { markup: false }; const tools = { ...createCodeceptJSTools(deps, task), ...this.testerTools(deps), context: this.contextTool(descent), completed: this.completedTool(), blocked: this.blockedTool() }; conversation.addUserText(await this.instructionPrompt(instructions, await this.capturedResult(previousState))); const used: string[] = []; const checks = new Map<string, boolean>(); - const evidence: string[] = []; + const trace: Array<{ label: string; ok: boolean; proof: string }> = []; let failure: { code: string; message: string } | null = null; let aiError: unknown = null; let narration = ''; @@ -194,27 +194,26 @@ export class Prima { for (const execution of executions) { const output = execution.output || {}; - if (this.applyLedgerReport(execution, ledger, evidence)) continue; + if (this.applyLedgerReport(execution, ledger, trace)) continue; if (output.action === 'verify' && !output.inexpressible) { const claim = execution.input?.assertion || 'verification'; let passed = execution.wasSuccessful; if (output.alreadyVerified) passed = output.verifications?.[claim] === true; checks.set(claim, passed); - evidence.push(`verify: ${claim} => ${passed ? 'PASSED' : 'FAILED'}`); + trace.push({ label: `verify: ${claim}`, ok: passed, proof: output.code || '' }); continue; } if (!execution.wasSuccessful) { failure = { code: output.code || '', message: output.message || 'action failed' }; - evidence.push(`FAILED ${output.code || execution.toolName || 'action'}: ${output.message || ''}`); + trace.push({ label: output.code || execution.toolName || 'action', ok: false, proof: output.message || '' }); continue; } const codes = this.executedCodes(output.code); used.push(...codes); - evidence.push(...codes); - if (output.pageDiff?.ariaChanges) evidence.push(output.pageDiff.ariaChanges); + trace.push({ label: codes.join('; ') || execution.toolName || 'action', ok: true, proof: '' }); failure = null; } @@ -224,15 +223,14 @@ export class Prima { if (aiError) return this.failureEnvelope(command, aiError, previousState); if (used.length && ledger.some((entry) => entry.status === 'open')) { - await this.settleLedger(conversation, provider, ledger, evidence); + await this.settleLedger(conversation, provider, ledger, trace); } - const steps = ledger.map((entry) => ({ label: entry.text, ok: entry.status === 'done', proof: [entry.proof || UNACCOUNTED[entry.status], ...entry.evidence].filter(Boolean).join('\n') })); const unfinished = ledger.filter((entry) => entry.status !== 'done'); + const steps = [...trace, ...ledger.filter((entry) => entry.status === 'open').map((entry) => ({ label: `unreported: ${entry.text}`, ok: false, proof: UNACCOUNTED.open }))]; if (failure && unfinished.length) { const envelope = await this.failureEnvelope(command, failure.message, previousState); - envelope.used = used; envelope.steps = steps; return envelope; } @@ -245,6 +243,9 @@ export class Prima { const result = await this.capturedResult(this.bot.stateManager().getCurrentState()); const envelope = await this.successEnvelope(command, used, result, previousState); envelope.steps = steps; + // the step log already reports every action and what it changed + envelope.used = undefined; + envelope.changes = undefined; const unmet = unfinished.map((entry) => `${entry.status}: ${entry.text}${entry.proof ? ` — ${entry.proof}` : ''}`); for (const [claim, passed] of checks) { @@ -265,18 +266,20 @@ export class Prima { .join('\n'); } - private applyLedgerReport(execution: any, ledger: LedgerEntry[], evidence: string[]): boolean { + private applyLedgerReport(execution: any, ledger: LedgerEntry[], trace: Array<{ label: string; ok: boolean; proof: string }>): boolean { const action = execution.output?.action; if (action === 'completed') { + const closed: string[] = []; for (const number of execution.input?.numbers || []) { const entry = ledger[number - 1]; if (entry?.status !== 'open') continue; entry.status = 'done'; entry.proof = execution.input?.proof || ''; - entry.evidence = [...evidence]; + closed.push(entry.text); } - evidence.length = 0; + // one report carries one proof, however many instructions it closed + if (closed.length) trace.push({ label: `done: ${closed.join('; ')}`, ok: true, proof: execution.input?.proof || '' }); return true; } @@ -286,13 +289,12 @@ export class Prima { if (entry?.status === 'open') { entry.status = 'blocked'; entry.proof = execution.input?.reason || ''; - entry.evidence = [...evidence]; + trace.push({ label: `blocked: ${entry.text}`, ok: false, proof: entry.proof }); } - evidence.length = 0; return true; } - private async settleLedger(conversation: any, provider: any, ledger: LedgerEntry[], evidence: string[]): Promise<void> { + private async settleLedger(conversation: any, provider: any, ledger: LedgerEntry[], trace: Array<{ label: string; ok: boolean; proof: string }>): Promise<void> { conversation.addUserText(dedent` The run is over and these instructions were never reported: @@ -306,7 +308,7 @@ export class Prima { const invoked = await provider.invokeConversation(conversation, { completed: this.completedTool(), blocked: this.blockedTool() }, { maxToolRoundtrips: 2, toolChoice: 'required', agentName: AI_AGENT_NAME }).catch(() => null); for (const execution of invoked?.toolExecutions || []) { - this.applyLedgerReport(execution, ledger, evidence); + this.applyLedgerReport(execution, ledger, trace); } } @@ -1045,7 +1047,6 @@ interface LedgerEntry { text: string; status: 'open' | 'done' | 'blocked'; proof: string; - evidence: string[]; } export interface PrimaOptions { diff --git a/boat/prima/tests/prima.test.ts b/boat/prima/tests/prima.test.ts index 630bda4c..c052d21c 100644 --- a/boat/prima/tests/prima.test.ts +++ b/boat/prima/tests/prima.test.ts @@ -439,7 +439,7 @@ describe('Prima.do', () => { expect(prompts.join('\n')).toContain('open the first invoice'); expect(prompts.join('\n')).toContain('download its PDF'); expect(prompts.join('\n')).toContain('button "Sign in"'); - expect(envelope.used).toEqual(["I.click('Login')"]); + expect(envelope.steps?.[0]).toMatchObject({ label: "I.click('Login')", ok: true }); expect(envelope.ok).toBe(true); expect(calls).toBe(1); }); @@ -458,11 +458,34 @@ describe('Prima.do', () => { expect(envelope.ok).toBe(true); expect(envelope.steps).toEqual([ - { label: 'open the invoices page', ok: true, proof: "the invoice list is open\nI.click('Invoices')" }, - { label: 'download the PDF', ok: true, proof: "the PDF opened in a new tab\nI.click('Download')" }, + { label: "I.click('Invoices')", ok: true, proof: '' }, + { label: 'done: open the invoices page', ok: true, proof: 'the invoice list is open' }, + { label: "I.click('Download')", ok: true, proof: '' }, + { label: 'done: download the PDF', ok: true, proof: 'the PDF opened in a new tab' }, ]); }); + test('one report closing several instructions is logged once, not once per instruction', async () => { + const { prima } = fakePrima(); + (prima as any).bot.getProvider = () => fakeProvider(async () => ({ toolExecutions: [toolExecution("I.click('Overview')"), completedExecution([1, 2], 'the workflow list is shown')] })); + + const envelope = await prima.do(['click the overview button', 'confirm the workflow list appears']); + + expect(envelope.steps?.map((step) => step.label)).toEqual(["I.click('Overview')", 'done: click the overview button; confirm the workflow list appears']); + expect(envelope.steps?.filter((step) => step.proof === 'the workflow list is shown')).toHaveLength(1); + }); + + test('the page diff is reported once, under changes, not again under every step', async () => { + const { prima } = fakePrima(); + (prima as any).bot.getProvider = () => fakeProvider(async () => ({ toolExecutions: [toolExecution("I.click('Overview')"), completedExecution([1], 'the list is shown')] })); + + const envelope = await prima.do(['click the overview button']); + + expect(envelope.changes).toBeUndefined(); + expect(envelope.used).toBeUndefined(); + expect(envelope.steps?.some((step) => step.label === "I.click('Overview')")).toBe(true); + }); + test('an instruction reported without any action behind it says so rather than being rejected', async () => { const { prima } = fakePrima(); (prima as any).bot.getProvider = () => fakeProvider(async () => ({ toolExecutions: [completedExecution([1], 'no cookie banner is present on this page')] })); @@ -470,7 +493,7 @@ describe('Prima.do', () => { const envelope = await prima.do(['dismiss the cookie banner if one appeared']); expect(envelope.ok).toBe(true); - expect(envelope.steps).toEqual([{ label: 'dismiss the cookie banner if one appeared', ok: true, proof: 'no cookie banner is present on this page' }]); + expect(envelope.steps).toEqual([{ label: 'done: dismiss the cookie banner if one appeared', ok: true, proof: 'no cookie banner is present on this page' }]); }); test('the remaining instructions are re-stated as the ledger closes, and finished ones are not', async () => { @@ -554,8 +577,8 @@ describe('Prima.do', () => { expect(envelope.ok).toBe(false); expect(envelope.failure?.error).toContain('open: download the PDF'); - expect(envelope.steps?.[1]).toMatchObject({ label: 'download the PDF', ok: false }); - expect(envelope.steps?.[1].proof).toContain('never reported'); + expect(envelope.steps?.at(-1)).toMatchObject({ label: 'unreported: download the PDF', ok: false }); + expect(envelope.steps?.at(-1)?.proof).toContain('never reported'); }); test('a stray failed action does not fail a sequence whose instructions all closed', async () => { @@ -705,7 +728,7 @@ describe('Prima.do', () => { expect(envelope.ok).toBe(false); expect(envelope.failure?.error).toContain('unproven: unsaved indicator is visible'); - expect(envelope.steps?.[0].proof).toContain('verify: unsaved indicator is visible => FAILED'); + expect(envelope.steps?.[0]).toMatchObject({ label: 'verify: unsaved indicator is visible', ok: false }); }); test('a check that passes on a retry does not fail the command', async () => { diff --git a/tests/integration/prima-do.test.ts b/tests/integration/prima-do.test.ts index 7f07473f..3de1b9ac 100644 --- a/tests/integration/prima-do.test.ts +++ b/tests/integration/prima-do.test.ts @@ -115,7 +115,7 @@ describe('Prima.do with aimock', () => { expect(executed.every((code) => code.includes('aria-ref='))).toBe(true); expect(executed).toHaveLength(2); expect(envelope.command).toContain('open the account menu'); - expect(envelope.steps?.map((step) => step.ok)).toEqual([true, true]); + expect(envelope.steps?.map((step) => step.label).filter((label) => label.startsWith('done:'))).toEqual(['done: open the account menu', 'done: choose the settings entry']); }); it('sends every instruction and the page context in one prompt', async () => { From 3f91fe1f888ddbf703820d2658ace6add3413caa Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Mon, 10 Aug 2026 01:48:24 +0300 Subject: [PATCH 29/39] Keep the page each step produced next to the log The step log said what ran but nothing showed what the page looked like when it ran, so a claim in it could not be checked against anything. Each action now writes the page it produced into the run's folder as <n>-<step>.aria.yaml, .html and, when the action changed something, .diff.yaml. The envelope prints the folder once at the end of ### Steps. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- boat/prima/src/envelope.ts | 2 ++ boat/prima/src/prima.ts | 19 +++++++++++++++++++ boat/prima/tests/prima.test.ts | 15 ++++++++++++++- 3 files changed, 35 insertions(+), 1 deletion(-) diff --git a/boat/prima/src/envelope.ts b/boat/prima/src/envelope.ts index 61368c1f..78411c5d 100644 --- a/boat/prima/src/envelope.ts +++ b/boat/prima/src/envelope.ts @@ -23,6 +23,7 @@ export interface EnvelopeData { changes?: string | null; steps?: Array<{ label: string; ok: boolean; proof: string }>; expectations?: Array<{ text: string; status: 'passed' | 'failed' | 'unverified' }>; + stepFiles?: string; value?: string; answer?: string; research?: string; @@ -89,6 +90,7 @@ function renderSteps(data: EnvelopeData): string | null { lines.push(`${index + 1}. ${step.ok ? 'ok ' : 'FAIL'} ${step.label}`); for (const line of (step.proof || '').split('\n').filter(Boolean)) lines.push(` ${line}`); }); + if (data.stepFiles) lines.push('', `page after each step: ${data.stepFiles}`); return section('Steps', lines.join('\n')); } diff --git a/boat/prima/src/prima.ts b/boat/prima/src/prima.ts index 4112af0f..451bbee7 100644 --- a/boat/prima/src/prima.ts +++ b/boat/prima/src/prima.ts @@ -21,6 +21,7 @@ import { compactAriaSnapshot } from '../../../src/utils/aria.ts'; import { mdq } from '../../../src/utils/markdown-query.ts'; import { browserErrorMessage } from '../../../src/utils/browser-errors.ts'; import { pluralize } from '../../../src/utils/logger.ts'; +import { safeFilename } from '../../../src/utils/strings.ts'; import { type EnvelopeData, type InstanceInfo, writeArtifacts } from './envelope.ts'; import { isFunctionExpression, takePwValue, toCodeceptWrapper } from './pw-parser.ts'; import { type PwServerDescriptor, readDescriptors, selectDescriptor } from './pw-registry.ts'; @@ -208,12 +209,14 @@ export class Prima { if (!execution.wasSuccessful) { failure = { code: output.code || '', message: output.message || 'action failed' }; trace.push({ label: output.code || execution.toolName || 'action', ok: false, proof: output.message || '' }); + await this.writeStepFiles(trace.length, output.code || execution.toolName || 'action', ''); continue; } const codes = this.executedCodes(output.code); used.push(...codes); trace.push({ label: codes.join('; ') || execution.toolName || 'action', ok: true, proof: '' }); + await this.writeStepFiles(trace.length, codes.join(' ') || execution.toolName || 'action', output.pageDiff?.ariaChanges || ''); failure = null; } @@ -232,6 +235,7 @@ export class Prima { if (failure && unfinished.length) { const envelope = await this.failureEnvelope(command, failure.message, previousState); envelope.steps = steps; + envelope.stepFiles = this.statusDir(); return envelope; } @@ -243,6 +247,7 @@ export class Prima { const result = await this.capturedResult(this.bot.stateManager().getCurrentState()); const envelope = await this.successEnvelope(command, used, result, previousState); envelope.steps = steps; + envelope.stepFiles = this.statusDir(); // the step log already reports every action and what it changed envelope.used = undefined; envelope.changes = undefined; @@ -1004,6 +1009,20 @@ export class Prima { return hash; } + private async writeStepFiles(index: number, label: string, diff: string): Promise<void> { + const state = this.bot.stateManager().getCurrentState(); + if (!state) return; + + const dir = this.statusDir(); + mkdirSync(dir, { recursive: true }); + const stem = path.join(dir, `${index}-${safeFilename(label.slice(0, 60))}`); + const result = ActionResult.fromState(state); + + writeFileSync(`${stem}.aria.yaml`, result.ariaSnapshot ?? '', 'utf-8'); + writeFileSync(`${stem}.html`, await result.combinedHtml(), 'utf-8'); + if (diff) writeFileSync(`${stem}.diff.yaml`, diff, 'utf-8'); + } + private async writeSnapshot(result: ActionResult): Promise<undefined> { writeArtifacts(this.statusDir(), { aria: result.ariaSnapshot, diff --git a/boat/prima/tests/prima.test.ts b/boat/prima/tests/prima.test.ts index c052d21c..b1cd5cd6 100644 --- a/boat/prima/tests/prima.test.ts +++ b/boat/prima/tests/prima.test.ts @@ -1,5 +1,5 @@ import { afterAll, afterEach, beforeAll, describe, expect, spyOn, test } from 'bun:test'; -import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; import { tmpdir } from 'node:os'; import path from 'node:path'; import { Navigator } from '../../../src/ai/navigator.ts'; @@ -569,6 +569,19 @@ describe('Prima.do', () => { expect(envelope.steps?.every((step) => step.ok)).toBe(true); }); + test('each action leaves the page it produced beside the log', async () => { + const { prima } = fakePrima(); + const clicked = { ...toolExecution("I.click('Invoices')"), output: { success: true, code: "I.click('Invoices')", pageDiff: { ariaChanges: 'added:\n - heading "Dashboard"' } } }; + (prima as any).bot.getProvider = () => fakeProvider(async () => ({ toolExecutions: [clicked, completedExecution([1], 'the list is open')] })); + + const envelope = await prima.do(['open the invoices page']); + const dir = envelope.stepFiles!; + + expect(existsSync(path.join(dir, '1-i_click__invoices__.aria.yaml'))).toBe(true); + expect(existsSync(path.join(dir, '1-i_click__invoices__.html'))).toBe(true); + expect(readFileSync(path.join(dir, '1-i_click__invoices__.diff.yaml'), 'utf-8')).toContain('heading "Dashboard"'); + }); + test('an instruction the model never reported is named as unaccounted for', async () => { const { prima } = fakePrima(); (prima as any).bot.getProvider = () => fakeProvider(async () => ({ toolExecutions: [toolExecution("I.click('Invoices')"), completedExecution([1], 'list is open')] })); From 943e5e2d634d5953f4367b11f1573528d35ac7cb Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Mon, 10 Aug 2026 01:55:40 +0300 Subject: [PATCH 30/39] Let the model, not a failed check, decide whether do succeeded MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A check that failed before the action which satisfied it vetoed the whole command. Verifying "debug panel is enabled", then enabling it, then reporting the instruction done came back ok: false with "unproven: Debug panel is enabled" — the check was never re-run, and its stale result outranked what the model had proven. do no longer draws a verdict from verify results. Each check is logged with its own outcome, as verify itself does, and whether the run succeeded follows from the instructions the model accounted for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- boat/prima/src/prima.ts | 5 ----- boat/prima/tests/prima.test.ts | 13 ++++++------- 2 files changed, 6 insertions(+), 12 deletions(-) diff --git a/boat/prima/src/prima.ts b/boat/prima/src/prima.ts index 451bbee7..a50391b5 100644 --- a/boat/prima/src/prima.ts +++ b/boat/prima/src/prima.ts @@ -146,7 +146,6 @@ export class Prima { conversation.addUserText(await this.instructionPrompt(instructions, await this.capturedResult(previousState))); const used: string[] = []; - const checks = new Map<string, boolean>(); const trace: Array<{ label: string; ok: boolean; proof: string }> = []; let failure: { code: string; message: string } | null = null; let aiError: unknown = null; @@ -201,7 +200,6 @@ export class Prima { const claim = execution.input?.assertion || 'verification'; let passed = execution.wasSuccessful; if (output.alreadyVerified) passed = output.verifications?.[claim] === true; - checks.set(claim, passed); trace.push({ label: `verify: ${claim}`, ok: passed, proof: output.code || '' }); continue; } @@ -253,9 +251,6 @@ export class Prima { envelope.changes = undefined; const unmet = unfinished.map((entry) => `${entry.status}: ${entry.text}${entry.proof ? ` — ${entry.proof}` : ''}`); - for (const [claim, passed] of checks) { - if (!passed) unmet.push(`unproven: ${claim}`); - } if (unmet.length) { envelope.ok = false; envelope.failure = { error: unmet.join('\n') }; diff --git a/boat/prima/tests/prima.test.ts b/boat/prima/tests/prima.test.ts index b1cd5cd6..59ea9943 100644 --- a/boat/prima/tests/prima.test.ts +++ b/boat/prima/tests/prima.test.ts @@ -723,7 +723,7 @@ describe('Prima.do', () => { expect(envelope.failure?.error).toContain('blocked: download the PDF — no PDF link exists on this page'); }); - test('a check that never passed fails the command even after later actions succeed', async () => { + test('a failed check is logged but does not overrule the instruction the model closed', async () => { const { prima } = fakePrima(); let calls = 0; (prima as any).bot.getProvider = () => @@ -731,17 +731,16 @@ describe('Prima.do', () => { calls++; if (calls === 1) { return { - toolExecutions: [{ toolName: 'verify', input: { assertion: 'unsaved indicator is visible' }, output: { success: false, action: 'verify', message: 'Verification failed' }, wasSuccessful: false }, toolExecution("I.click('Close')")], + toolExecutions: [{ toolName: 'verify', input: { assertion: 'debug panel is enabled' }, output: { success: false, action: 'verify', message: 'Verification failed' }, wasSuccessful: false }, toolExecution("I.click('Enable debug panel')")], }; } - return { toolExecutions: [completedExecution([1, 2], 'the editor is closed')] }; + return { toolExecutions: [completedExecution([1], 'the debug switch is on')] }; }); - const envelope = await prima.do(['confirm the unsaved indicator', 'close the editor']); + const envelope = await prima.do(['enable debug mode']); - expect(envelope.ok).toBe(false); - expect(envelope.failure?.error).toContain('unproven: unsaved indicator is visible'); - expect(envelope.steps?.[0]).toMatchObject({ label: 'verify: unsaved indicator is visible', ok: false }); + expect(envelope.ok).toBe(true); + expect(envelope.steps?.[0]).toMatchObject({ label: 'verify: debug panel is enabled', ok: false }); }); test('a check that passes on a retry does not fail the command', async () => { From 5aeea2d2bf765000778961c48bbbb44e46b6d352 Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Mon, 10 Aug 2026 23:14:29 +0300 Subject: [PATCH 31/39] Rename prima models to prima config, and show the three roles The command answers "what is prima running on", which is configuration, not a model list. It printed per-agent overrides too, which are noise next to the three roles that decide behaviour. prima config prints model, agenticModel and visionModel, and the file they came from. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- boat/prima/src/cli.ts | 4 ++-- boat/prima/src/prima.ts | 5 +---- 2 files changed, 3 insertions(+), 6 deletions(-) diff --git a/boat/prima/src/cli.ts b/boat/prima/src/cli.ts index c27d0c46..c9b367ef 100644 --- a/boat/prima/src/cli.ts +++ b/boat/prima/src/cli.ts @@ -178,10 +178,10 @@ export function createPrimaCommands(name = 'prima'): Command { await runPrima(options, `go ${target}`, (prima) => prima.go(target)); }); - addCommonOptions(cmd.command('models').description('Show which AI model answers for each role')).action(async (options) => { + addCommonOptions(cmd.command('config').description('Show the AI models prima runs on and the config file they come from')).action(async (options) => { setQuietMode(!options.verbose && !options.debug); const prima = primaFor(options); - console.log(await prima.models().catch((error: unknown) => browserErrorMessage(error))); + console.log(await prima.config().catch((error: unknown) => browserErrorMessage(error))); await prima.stop().catch(() => {}); process.exit(0); }); diff --git a/boat/prima/src/prima.ts b/boat/prima/src/prima.ts index a50391b5..e21d4af3 100644 --- a/boat/prima/src/prima.ts +++ b/boat/prima/src/prima.ts @@ -432,7 +432,7 @@ export class Prima { return stopped; } - async models(): Promise<string> { + async config(): Promise<string> { const [site] = listSites(); if (site && !this.configBaseUrl()) this.sessionUrl = site.url; const config = await this.loadConfig(); @@ -448,9 +448,6 @@ export class Prima { ['agenticModel', ai.agenticModel], ['visionModel', ai.visionModel], ]; - for (const [agent, agentConfig] of Object.entries(ai.agents || {})) { - if ((agentConfig as any)?.model) roles.push([agent, (agentConfig as any).model]); - } const lines = roles.filter(([, model]) => model).map(([role, model]) => `${role.padEnd(14)} ${named(model)}`); lines.push(`config ${ConfigParser.getInstance().getConfigPath() || 'built-in defaults'}`); From 1bb9f71db96a8a5e50ebba29c092a75b8798780c Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Mon, 10 Aug 2026 23:58:32 +0300 Subject: [PATCH 32/39] Record a prima session and report on it on demand Prima no longer runs explorbot's reporter: a check ended by writing the session HTML and Markdown reports, which belong to an explorbot run and not to one command. Instead every command appends itself to output/prima/sessions/<session>.jsonl in the format the Testomat.io reporter replays, and prima report turns that log into one HTML and Markdown report. The log is written as commands run, so the report needs no browser and outlives the session; the same file can be replayed to Testomat.io with the reporter CLI. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- CHANGELOG.md | 27 ++++++ boat/prima/src/cli.ts | 23 ++++- boat/prima/src/prima.ts | 50 ++++++++++- boat/prima/src/session-log.ts | 126 +++++++++++++++++++++++++++ boat/prima/tests/prima.test.ts | 8 ++ boat/prima/tests/session-log.test.ts | 92 +++++++++++++++++++ src/explorbot.ts | 5 +- src/reporter.ts | 8 +- 8 files changed, 333 insertions(+), 6 deletions(-) create mode 100644 boat/prima/src/session-log.ts create mode 100644 boat/prima/tests/session-log.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index a4895336..e80c3e4d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,32 @@ # Changelog +## 2026-08-10 + +### `prima report` collects a whole session into one report + +Every prima command is logged to `output/prima/sessions/` as it runs. `prima report` turns that +log into an HTML and a Markdown report — each command with its steps, expected outcomes and the +proof recorded for them. It needs no browser, so the report still comes out after the session is +closed. + +```bash +prima report # the session used most recently +prima report --pw-session my-app # a named playwright-cli session +``` + +The log is written in the format the Testomat.io reporter replays, so the same file can be sent +to Testomat.io as a run: + +```bash +TESTOMATIO=<apiKey> npx @testomatio/reporter replay <the path prima report prints> +``` + +### Changes + +- [Prima] Prima no longer generates a report of its own while it runs. A `check` used to end by + writing the session HTML and Markdown reports, which belong to an explorbot run rather than to + a single command. Reports are produced only when `prima report` asks for one. + ## 2026-08-09 ### `prima check` runs a whole scenario and reports what it proved diff --git a/boat/prima/src/cli.ts b/boat/prima/src/cli.ts index c9b367ef..6b5eb9f5 100644 --- a/boat/prima/src/cli.ts +++ b/boat/prima/src/cli.ts @@ -42,6 +42,11 @@ const verifyHelp = dedent` could not be expressed, which is not the same as false. `; +const reportHelp = dedent` + Commands are logged as they run, so the report needs no browser and outlives the session. + The most recent session is reported unless --pw-session names another. +`; + const sessionHelp = dedent` --endpoint <ep> attach to a browser server endpoint directly, skipping discovery --instance <name> which prima-owned browser you talk to; parallel work needs one each @@ -103,9 +108,10 @@ function primaFor(options: any): Prima { return new Prima(buildOptions(options)); } -async function runPrima(options: any, command: string, run: (prima: Prima) => Promise<EnvelopeData>): Promise<void> { +async function runPrima(options: any, command: string, run: (prima: Prima) => Promise<EnvelopeData>, record = true): Promise<void> { setQuietMode(!options.verbose && !options.debug); const prima = primaFor(options); + const startedAt = Date.now(); let envelope: EnvelopeData; try { @@ -115,6 +121,7 @@ async function runPrima(options: any, command: string, run: (prima: Prima) => Pr envelope = await prima.toolFailureEnvelope(command, error); } + if (record) prima.record(envelope, Date.now() - startedAt); console.log(renderEnvelope(envelope)); await prima.stop().catch(() => {}); process.exit(envelope.ok ? 0 : 1); @@ -187,9 +194,21 @@ export function createPrimaCommands(name = 'prima'): Command { }); addCommonOptions(cmd.command('status <hash>').description('Show the artifacts and page detail recorded for an earlier command')).action(async (hash, options) => { - await runPrima(options, `status ${hash}`, (prima) => prima.status(hash)); + await runPrima(options, `status ${hash}`, (prima) => prima.status(hash), false); }); + addCommonOptions(cmd.command('report').description('Turn every command of a session into one html and markdown report')) + .addHelpText('after', `\n${reportHelp}`) + .action(async (options) => { + setQuietMode(!options.verbose && !options.debug); + console.log( + await primaFor(options) + .report() + .catch((error: unknown) => browserErrorMessage(error)) + ); + process.exit(0); + }); + const browser = cmd.command('browser').description('Manage the browsers prima drives'); addCommonOptions(browser.command('start').description('Start a prima-owned browser and hold it open until Ctrl+C')) diff --git a/boat/prima/src/prima.ts b/boat/prima/src/prima.ts index e21d4af3..4ca50dc3 100644 --- a/boat/prima/src/prima.ts +++ b/boat/prima/src/prima.ts @@ -14,6 +14,8 @@ import { getAliveEndpoint, launchServer, listInstances, stopServer } from '../.. import { listSites } from '../../../src/global-config.ts'; import { ConfigMissingError, ConfigParser, type ExplorbotConfig, outputPath } from '../../../src/config.ts'; import { ExplorBot } from '../../../src/explorbot.ts'; +import { Reporter } from '../../../src/reporter.ts'; +import { Stats } from '../../../src/stats.ts'; import type { WebPageState } from '../../../src/state-manager.ts'; import { Task, Test, TestResult } from '../../../src/test-plan.ts'; import { getPreviousResearch } from '../../../src/ai/researcher/cache.ts'; @@ -24,6 +26,7 @@ import { pluralize } from '../../../src/utils/logger.ts'; import { safeFilename } from '../../../src/utils/strings.ts'; import { type EnvelopeData, type InstanceInfo, writeArtifacts } from './envelope.ts'; import { isFunctionExpression, takePwValue, toCodeceptWrapper } from './pw-parser.ts'; +import { type SessionRun, latestSessionFile, readSession, recordCommand, sessionFile, sessionsDir } from './session-log.ts'; import { type PwServerDescriptor, readDescriptors, selectDescriptor } from './pw-registry.ts'; const TESTER_ONLY_TOOLS = ['learnExperience', 'askUser']; @@ -69,6 +72,7 @@ export class Prima { private sessionUrl?: string; private server: { close: () => Promise<void> } | null = null; private attached: string | null = null; + private session: SessionRun | null = null; constructor(options: PrimaOptions = {}) { this.options = options; @@ -81,6 +85,7 @@ export class Prima { instance: options.instance, headless: true, optionalAi: true, + reporter: { enabled: false }, }); } @@ -455,6 +460,45 @@ export class Prima { return lines.join('\n'); } + record(envelope: EnvelopeData, durationMs: number): void { + if (!this.session) return; + recordCommand(sessionFile(this.session.key), this.session, envelope, durationMs); + } + + async report(): Promise<string> { + const [site] = listSites(); + if (site && !this.configBaseUrl()) this.sessionUrl = site.url; + await this.loadConfig(); + + let file = latestSessionFile(); + if (this.options.pwSession) file = sessionFile(this.options.pwSession); + if (!file || !existsSync(file)) return `No prima session was recorded under ${sessionsDir()}. Commands are recorded as they run.`; + + const session = readSession(file); + if (!session.tests.length) return `No commands are recorded in ${file}`; + + Stats.sessionName = path.basename(file, '.jsonl'); + process.env.TESTOMATIO_TITLE = session.title; + const reporter = new Reporter({ html: true, markdown: true }); + + // the report pipes narrate themselves on console.log; prima prints the paths itself + const speak = console.log; + console.log = () => {}; + try { + for (const test of session.tests) await reporter.reportTestData(test.status, test); + await reporter.finishRun(); + } finally { + console.log = speak; + } + + return [ + `${session.tests.length} ${pluralize(session.tests.length, 'command')} from ${file}`, + `html: ${outputPath('reports', `${Stats.sessionLabel()}.html`)}`, + `markdown: ${outputPath('reports', `${Stats.sessionLabel()}-tests.md`)}`, + `upload: TESTOMATIO=<apiKey> npx @testomatio/reporter replay ${file}`, + ].join('\n'); + } + async browserStatus(): Promise<string> { await this.loadConfig(); const info = await this.instanceInfo(); @@ -596,6 +640,7 @@ export class Prima { this.bot.attachBrowser(browser); this.attached = this.attachmentLabel(descriptor); + this.session = { key: descriptor.title || this.instanceName(), endpoint: descriptor.endpoint, title: `prima session "${descriptor.title || this.instanceName()}"` }; return true; } @@ -627,7 +672,10 @@ export class Prima { } private async connectOwnInstance(): Promise<boolean> { - return !!(await getAliveEndpoint(this.instanceName())); + const endpoint = await getAliveEndpoint(this.instanceName()); + if (!endpoint) return false; + this.session = { key: this.instanceName(), endpoint, title: `prima instance "${this.instanceName()}"` }; + return true; } private async launchOwnServer(opts: { browser?: string; show?: boolean }, instance: string): Promise<{ close: () => Promise<void> }> { diff --git a/boat/prima/src/session-log.ts b/boat/prima/src/session-log.ts new file mode 100644 index 00000000..cd234081 --- /dev/null +++ b/boat/prima/src/session-log.ts @@ -0,0 +1,126 @@ +import { appendFileSync, existsSync, mkdirSync, readFileSync, readdirSync, statSync, writeFileSync } from 'node:fs'; +import path from 'node:path'; +import { outputPath } from '../../../src/config.ts'; +import { safeFilename } from '../../../src/utils/strings.ts'; +import { type EnvelopeData, renderEnvelope } from './envelope.ts'; + +const EXPECTATION_STATUS = { passed: 'passed', failed: 'failed', unverified: 'skipped' }; + +export function sessionsDir(): string { + return outputPath('prima', 'sessions'); +} + +export function sessionFile(key: string): string { + return path.join(sessionsDir(), `${safeFilename(key)}.jsonl`); +} + +export function latestSessionFile(): string | null { + const dir = sessionsDir(); + if (!existsSync(dir)) return null; + + const files = readdirSync(dir) + .filter((name) => name.endsWith('.jsonl')) + .map((name) => path.join(dir, name)) + .sort((a, b) => statSync(b).mtimeMs - statSync(a).mtimeMs); + + return files[0] || null; +} + +export function recordCommand(file: string, session: SessionRun, envelope: EnvelopeData, durationMs: number): void { + mkdirSync(path.dirname(file), { recursive: true }); + if (recordedEndpoint(file) !== session.endpoint) writeFileSync(file, jsonLine({ action: 'createRun', endpoint: session.endpoint, params: { title: session.title } }), 'utf-8'); + appendFileSync(file, jsonLine({ action: 'addTest', testId: commandTest(envelope, durationMs) }), 'utf-8'); +} + +export function readSession(file: string): { title: string; tests: any[] } { + const entries = readEntries(file); + const run = entries.find((entry) => entry.action === 'createRun'); + const tests = entries.filter((entry) => entry.action === 'addTest').map((entry) => entry.testId); + return { title: run?.params?.title || 'prima session', tests }; +} + +function commandTest(envelope: EnvelopeData, durationMs: number): any { + let status = 'failed'; + if (envelope.ok) status = 'passed'; + + const test: any = { + rid: envelope.status, + title: envelope.command, + suite_title: 'Prima session', + status, + file: '<prima>', + description: [`page: ${envelope.page.url}`, envelope.stepFiles && `artifacts: ${envelope.stepFiles}`].filter(Boolean).join('\n'), + code: (envelope.used || []).join('\n'), + steps: commandSteps(envelope), + logs: renderEnvelope(envelope), + time: Math.round(durationMs), + }; + + if (envelope.failure) test.error = { message: envelope.failure.error }; + return test; +} + +function commandSteps(envelope: EnvelopeData): any[] { + const steps: any[] = []; + + for (const step of envelope.steps || []) { + const entry: any = { category: 'user', title: step.label, status: 'passed', duration: 0, log: step.proof }; + if (!step.ok) { + entry.status = 'failed'; + entry.error = step.proof; + } + steps.push(entry); + } + + for (const expectation of envelope.expectations || []) { + const entry: any = { category: 'user', title: `expected: ${expectation.text}`, status: EXPECTATION_STATUS[expectation.status], duration: 0 }; + if (expectation.status === 'failed') entry.error = 'the run did not reach this outcome'; + steps.push(entry); + } + + for (const assertion of envelope.assertions || []) { + const entry: any = { category: 'user', title: assertion.code, status: 'passed', duration: 0, log: assertion.proof.join('\n') }; + if (!assertion.passed) { + entry.status = 'failed'; + entry.error = 'the assertion did not hold on the page'; + } + steps.push(entry); + } + + return steps; +} + +function recordedEndpoint(file: string): string | null { + const [first] = readEntries(file); + return first?.endpoint || null; +} + +function readEntries(file: string): any[] { + if (!existsSync(file)) return []; + + const entries: any[] = []; + for (const line of readFileSync(file, 'utf-8').split('\n')) { + if (!line.trim()) continue; + const entry = parseLine(line); + if (entry) entries.push(entry); + } + return entries; +} + +function parseLine(line: string): any { + try { + return JSON.parse(line); + } catch { + return null; + } +} + +function jsonLine(entry: unknown): string { + return `${JSON.stringify(entry)}\n`; +} + +export interface SessionRun { + key: string; + endpoint: string; + title: string; +} diff --git a/boat/prima/tests/prima.test.ts b/boat/prima/tests/prima.test.ts index 59ea9943..8d19e152 100644 --- a/boat/prima/tests/prima.test.ts +++ b/boat/prima/tests/prima.test.ts @@ -228,6 +228,14 @@ describe('Prima.start', () => { await expect(prima.start()).rejects.toThrow(/playwright-cli open/); expect(started).toBe(false); }); + + test('leaves the explorbot reporter switched off', () => { + const prima = new Prima({ instance: 'default' }); + + expect((prima as any).bot.reporter().isEnabled()).toBe(false); + expect(process.env.TESTOMATIO_HTML_REPORT_SAVE).toBeUndefined(); + expect(process.env.TESTOMATIO_MARKDOWN_REPORT_SAVE).toBeUndefined(); + }); }); describe('Prima attach ladder', () => { diff --git a/boat/prima/tests/session-log.test.ts b/boat/prima/tests/session-log.test.ts new file mode 100644 index 00000000..c0180c14 --- /dev/null +++ b/boat/prima/tests/session-log.test.ts @@ -0,0 +1,92 @@ +import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test'; +import { existsSync, rmSync } from 'node:fs'; +import { ConfigParser } from '../../../src/config.ts'; +import type { EnvelopeData } from '../src/envelope.ts'; +import { latestSessionFile, readSession, recordCommand, sessionFile, sessionsDir } from '../src/session-log.ts'; + +const session = { key: 'default', endpoint: 'ws://127.0.0.1:5000/one', title: 'prima session "default"' }; +const restarted = { ...session, endpoint: 'ws://127.0.0.1:5000/two' }; + +beforeAll(() => { + ConfigParser.resetForTesting(); + ConfigParser.setupTestConfig(); +}); + +afterAll(() => { + ConfigParser.cleanupAllTestDirectories(); +}); + +beforeEach(() => { + rmSync(sessionsDir(), { recursive: true, force: true }); +}); + +function envelope(over: Partial<EnvelopeData> = {}): EnvelopeData { + return { + ok: true, + command: 'do "open the account menu"', + page: { url: 'https://app.example.com/settings', title: 'Settings', state: 'settings_h1_settings', visits: 1 }, + instance: { name: 'default', tabs: 1, others: [] }, + status: 'abc123', + ...over, + }; +} + +describe('prima session log', () => { + test('records every command of one session', () => { + const file = sessionFile(session.key); + recordCommand(file, session, envelope(), 1200); + recordCommand(file, session, envelope({ command: 'check "settings save"', ok: false, status: 'def456', failure: { error: 'the form never saved' } }), 3400); + + const recorded = readSession(file); + expect(recorded.title).toBe('prima session "default"'); + expect(recorded.tests.map((test: any) => test.title)).toEqual(['do "open the account menu"', 'check "settings save"']); + expect(recorded.tests.map((test: any) => test.status)).toEqual(['passed', 'failed']); + expect(recorded.tests[1].error.message).toBe('the form never saved'); + expect(recorded.tests[1].time).toBe(3400); + }); + + test('starts a new log when the session is a different browser', () => { + const file = sessionFile(session.key); + recordCommand(file, session, envelope(), 1000); + recordCommand(file, restarted, envelope({ command: 'pw "({ page }) => page.title()"' }), 1000); + + const recorded = readSession(file); + expect(recorded.tests.map((test: any) => test.title)).toEqual(['pw "({ page }) => page.title()"']); + }); + + test('carries steps, expectations and assertions as report steps', () => { + const file = sessionFile(session.key); + recordCommand( + file, + session, + envelope({ + steps: [{ label: 'done: open the account menu', ok: true, proof: 'the menu is expanded' }], + expectations: [ + { text: 'the theme is dark', status: 'passed' }, + { text: 'the choice survives a reload', status: 'unverified' }, + ], + assertions: [{ code: 'I.see("Dark")', passed: true, proof: ['locator("body").filter({ hasText: "Dark" })'] }], + stepFiles: '/tmp/prima/abc123', + }), + 900 + ); + + const [test] = readSession(file).tests; + expect(test.steps.map((step: any) => [step.title, step.status])).toEqual([ + ['done: open the account menu', 'passed'], + ['expected: the theme is dark', 'passed'], + ['expected: the choice survives a reload', 'skipped'], + ['I.see("Dark")', 'passed'], + ]); + expect(test.description).toContain('artifacts: /tmp/prima/abc123'); + expect(test.logs).toContain('### Result'); + }); + + test('reports the most recent session when none is named', () => { + expect(latestSessionFile()).toBe(null); + + recordCommand(sessionFile('checkout'), { ...session, key: 'checkout' }, envelope(), 500); + expect(latestSessionFile()).toBe(sessionFile('checkout')); + expect(existsSync(sessionFile('checkout'))).toBe(true); + }); +}); diff --git a/src/explorbot.ts b/src/explorbot.ts index 63b76910..e51fb745 100644 --- a/src/explorbot.ts +++ b/src/explorbot.ts @@ -20,7 +20,7 @@ import { createAgentTools } from './ai/tools.ts'; import { ApiClient } from './api/api-client.ts'; import { RequestStore } from './api/request-store.ts'; import { loadSpec } from './api/spec-reader.ts'; -import type { ExplorbotConfig } from './config.js'; +import type { ExplorbotConfig, ReporterConfig } from './config.js'; import { ConfigParser } from './config.ts'; import { ExperienceTracker } from './experience-tracker.ts'; import Explorer from './explorer.ts'; @@ -49,6 +49,7 @@ export interface ExplorBotOptions { session?: string | boolean; instance?: string; optionalAi?: boolean; + reporter?: ReporterConfig; attachedBrowser?: Browser; applicationSpec?: string; } @@ -175,7 +176,7 @@ export class ExplorBot { } reporter(): Reporter { - return (this._reporter ||= new Reporter(this.config.reporter, this.stateManager())); + return (this._reporter ||= new Reporter(this.options.reporter || this.config.reporter, this.stateManager())); } requestStore(): RequestStore { diff --git a/src/reporter.ts b/src/reporter.ts index 8b780f3b..4d0dd4c6 100644 --- a/src/reporter.ts +++ b/src/reporter.ts @@ -283,13 +283,19 @@ export class Reporter { debugLog(testData); - await this.client.addTestRun(status, testData); + await this.reportTestData(status, testData); debugLog(`Test reported: ${test.scenario} - ${status}`); } catch (error) { debugLog('Failed to report test:', error); } } + async reportTestData(status: string | null, testData: Record<string, unknown>): Promise<void> { + await this.startRun(); + if (!this.isRunStarted) return; + await this.client.addTestRun(status, testData); + } + async finishRun(): Promise<void> { if (!this.isRunStarted) { return; From 1dc1a74f243db1236ce65a7fc16ecb26be9282a2 Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Tue, 11 Aug 2026 01:35:40 +0300 Subject: [PATCH 33/39] Make prima --verbose print the log it promises The flag was only taken after the subcommand, so the natural prima --verbose <command> failed as an unknown option, and quiet mode was decided from the subcommand options alone rather than the merged set. Verbose also printed a fraction of what DEBUG='explorbot:*' printed in front of the same command: the debug library reads DEBUG when it loads, and the flag sets it later, so every namespace stayed off. Verbose mode now enables them directly, which fixes explorbot's own --verbose the same way. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- CHANGELOG.md | 19 +++++++++++++++++++ boat/prima/src/cli.ts | 8 +++++--- boat/prima/tests/cli.test.ts | 14 ++++++++++++++ src/utils/logger.ts | 2 ++ 4 files changed, 40 insertions(+), 3 deletions(-) create mode 100644 boat/prima/tests/cli.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index e80c3e4d..78566815 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,24 @@ # Changelog +## 2026-08-11 + +### `prima --verbose` prints the log of a command + +`-v` / `--verbose` is now taken before the command as well as after it, and prints everything the +run does: config and browser attachment, every step as it executes, and the debug stream of the +agents behind it. + +```bash +prima -v do "open the account menu" "switch the theme to dark" +prima --verbose check "a workflow can be created" +``` + +### Changes + +- `--verbose` now turns the debug log on for real, in prima and in explorbot alike. The flag set + the `DEBUG` variable after the logging library had already read it, so a verbose run printed a + fraction of what `DEBUG='explorbot:*'` printed in front of the same command. + ## 2026-08-10 ### `prima report` collects a whole session into one report diff --git a/boat/prima/src/cli.ts b/boat/prima/src/cli.ts index 6b5eb9f5..3fa8c318 100644 --- a/boat/prima/src/cli.ts +++ b/boat/prima/src/cli.ts @@ -109,7 +109,7 @@ function primaFor(options: any): Prima { } async function runPrima(options: any, command: string, run: (prima: Prima) => Promise<EnvelopeData>, record = true): Promise<void> { - setQuietMode(!options.verbose && !options.debug); + setQuietMode(!buildOptions(options).verbose); const prima = primaFor(options); const startedAt = Date.now(); @@ -142,6 +142,8 @@ async function runBrowser(options: any, run: (prima: Prima) => Promise<boolean>) export function createPrimaCommands(name = 'prima'): Command { const cmd = new Command(name); cmd.description('Tests and drives a web app through described behaviour instead of locators: one command carries a whole scenario, verifies it, and reports the proof'); + cmd.option('-v, --verbose', 'Print the logs of everything the command does'); + cmd.option('--debug', 'Enable debug logging (same as --verbose)'); cmd.option('--pw-session <title>', 'Title of the playwright-cli session to attach to'); cmd.option('--url <url>', 'Page to open when the session has no page yet'); cmd.addHelpText('after', `\n${helpContract}`); @@ -186,7 +188,7 @@ export function createPrimaCommands(name = 'prima'): Command { }); addCommonOptions(cmd.command('config').description('Show the AI models prima runs on and the config file they come from')).action(async (options) => { - setQuietMode(!options.verbose && !options.debug); + setQuietMode(!buildOptions(options).verbose); const prima = primaFor(options); console.log(await prima.config().catch((error: unknown) => browserErrorMessage(error))); await prima.stop().catch(() => {}); @@ -200,7 +202,7 @@ export function createPrimaCommands(name = 'prima'): Command { addCommonOptions(cmd.command('report').description('Turn every command of a session into one html and markdown report')) .addHelpText('after', `\n${reportHelp}`) .action(async (options) => { - setQuietMode(!options.verbose && !options.debug); + setQuietMode(!buildOptions(options).verbose); console.log( await primaFor(options) .report() diff --git a/boat/prima/tests/cli.test.ts b/boat/prima/tests/cli.test.ts new file mode 100644 index 00000000..70359612 --- /dev/null +++ b/boat/prima/tests/cli.test.ts @@ -0,0 +1,14 @@ +import { describe, expect, test } from 'bun:test'; +import { createPrimaCommands } from '../src/cli.ts'; + +describe('prima cli options', () => { + test('takes the logging flags before the command as well as after it', () => { + const cmd = createPrimaCommands(); + const root = cmd.options.map((option) => option.long); + const check = cmd.commands.find((command) => command.name() === 'check')!.options.map((option) => option.long); + + expect(root).toContain('--verbose'); + expect(root).toContain('--debug'); + expect(check).toContain('--verbose'); + }); +}); diff --git a/src/utils/logger.ts b/src/utils/logger.ts index ac3319e2..532ea665 100644 --- a/src/utils/logger.ts +++ b/src/utils/logger.ts @@ -142,6 +142,8 @@ class DebugDestination implements LogDestination { setVerboseMode(enabled: boolean): void { this.verboseMode = enabled; + // the debug package reads DEBUG when it loads, so a namespace turned on later needs enabling by hand + if (enabled) debug.enable(process.env.DEBUG || 'explorbot:*'); } write(namespace: string, ...args: any[]): void { From ce31dd76946d40763c2297eb1fb6bca295fd3286 Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Tue, 11 Aug 2026 02:03:02 +0300 Subject: [PATCH 34/39] Drop prima's logging flags for DEBUG DEBUG already selects what is logged, and with finer aim than a flag. Prima now leaves the whole console log on whenever DEBUG names any namespace, so DEBUG='explorbot:*' prints what --verbose used to and DEBUG='explorbot:tester' narrows it to one agent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- CHANGELOG.md | 18 +++++++++--------- boat/prima/src/cli.ts | 14 +++++--------- boat/prima/src/prima.ts | 2 -- boat/prima/tests/cli.test.ts | 14 -------------- 4 files changed, 14 insertions(+), 34 deletions(-) delete mode 100644 boat/prima/tests/cli.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 78566815..e40afcf3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,22 +2,22 @@ ## 2026-08-11 -### `prima --verbose` prints the log of a command +### Prima prints its log when `DEBUG` is set -`-v` / `--verbose` is now taken before the command as well as after it, and prints everything the -run does: config and browser attachment, every step as it executes, and the debug stream of the -agents behind it. +`DEBUG` in front of any prima command prints everything the run does: config and browser +attachment, every step as it executes, and the debug stream of the agents behind it. Prima carries +no logging flags of its own — `--verbose` and `--debug` are gone. ```bash -prima -v do "open the account menu" "switch the theme to dark" -prima --verbose check "a workflow can be created" +DEBUG='explorbot:*' prima do "open the account menu" "switch the theme to dark" +DEBUG='explorbot:tester' prima check "a workflow can be created" ``` ### Changes -- `--verbose` now turns the debug log on for real, in prima and in explorbot alike. The flag set - the `DEBUG` variable after the logging library had already read it, so a verbose run printed a - fraction of what `DEBUG='explorbot:*'` printed in front of the same command. +- Verbose mode turns the debug log on for real. It set the `DEBUG` variable after the logging + library had already read it, so `explorbot --verbose` printed a fraction of what + `DEBUG='explorbot:*'` printed in front of the same command. ## 2026-08-10 diff --git a/boat/prima/src/cli.ts b/boat/prima/src/cli.ts index 3fa8c318..c52a9c32 100644 --- a/boat/prima/src/cli.ts +++ b/boat/prima/src/cli.ts @@ -2,7 +2,7 @@ import { Command } from 'commander'; import dedent from 'dedent'; import { keepServerRunning } from '../../../src/browser-server.ts'; import { browserErrorMessage } from '../../../src/utils/browser-errors.ts'; -import { setQuietMode } from '../../../src/utils/logger.ts'; +import { isVerboseMode, setQuietMode } from '../../../src/utils/logger.ts'; import { type EnvelopeData, renderEnvelope } from './envelope.ts'; import { Prima, type PrimaOptions } from './prima.ts'; @@ -53,6 +53,7 @@ const sessionHelp = dedent` --session [file] cookies and storage persisted across processes; ignored while attached, since the attached session keeps its own --framework parsed but not active yet; reported code is CodeceptJS either way + DEBUG='explorbot:*' in front of a command prints the log of everything it does. When no AI model is usable pw still works; for everything else drive playwright-cli. `; @@ -61,7 +62,6 @@ let rootOptions: () => any = () => ({}); function buildOptions(subcommand: any): PrimaOptions { const options = { ...rootOptions(), ...stripEmpty(subcommand) }; return { - verbose: options.verbose || options.debug, config: options.config, path: options.path, instance: options.instance, @@ -89,8 +89,6 @@ function stripEmpty(options: any): any { function addCommonOptions(cmd: Command): Command { return cmd - .option('-v, --verbose', 'Enable verbose logging') - .option('--debug', 'Enable debug logging (same as --verbose)') .option('-c, --config <path>', 'Path to explorbot configuration file') .option('-p, --path <path>', 'Working directory path') .option('-i, --instance <name>', 'Browser instance to drive') @@ -109,7 +107,7 @@ function primaFor(options: any): Prima { } async function runPrima(options: any, command: string, run: (prima: Prima) => Promise<EnvelopeData>, record = true): Promise<void> { - setQuietMode(!buildOptions(options).verbose); + setQuietMode(!isVerboseMode()); const prima = primaFor(options); const startedAt = Date.now(); @@ -142,8 +140,6 @@ async function runBrowser(options: any, run: (prima: Prima) => Promise<boolean>) export function createPrimaCommands(name = 'prima'): Command { const cmd = new Command(name); cmd.description('Tests and drives a web app through described behaviour instead of locators: one command carries a whole scenario, verifies it, and reports the proof'); - cmd.option('-v, --verbose', 'Print the logs of everything the command does'); - cmd.option('--debug', 'Enable debug logging (same as --verbose)'); cmd.option('--pw-session <title>', 'Title of the playwright-cli session to attach to'); cmd.option('--url <url>', 'Page to open when the session has no page yet'); cmd.addHelpText('after', `\n${helpContract}`); @@ -188,7 +184,7 @@ export function createPrimaCommands(name = 'prima'): Command { }); addCommonOptions(cmd.command('config').description('Show the AI models prima runs on and the config file they come from')).action(async (options) => { - setQuietMode(!buildOptions(options).verbose); + setQuietMode(!isVerboseMode()); const prima = primaFor(options); console.log(await prima.config().catch((error: unknown) => browserErrorMessage(error))); await prima.stop().catch(() => {}); @@ -202,7 +198,7 @@ export function createPrimaCommands(name = 'prima'): Command { addCommonOptions(cmd.command('report').description('Turn every command of a session into one html and markdown report')) .addHelpText('after', `\n${reportHelp}`) .action(async (options) => { - setQuietMode(!buildOptions(options).verbose); + setQuietMode(!isVerboseMode()); console.log( await primaFor(options) .report() diff --git a/boat/prima/src/prima.ts b/boat/prima/src/prima.ts index 4ca50dc3..fd6b5de4 100644 --- a/boat/prima/src/prima.ts +++ b/boat/prima/src/prima.ts @@ -80,7 +80,6 @@ export class Prima { config: options.config, path: options.path, baseUrl: this.configBaseUrl(), - verbose: options.verbose, session: options.session, instance: options.instance, headless: true, @@ -1109,7 +1108,6 @@ interface LedgerEntry { } export interface PrimaOptions { - verbose?: boolean; config?: string; path?: string; instance?: string; diff --git a/boat/prima/tests/cli.test.ts b/boat/prima/tests/cli.test.ts deleted file mode 100644 index 70359612..00000000 --- a/boat/prima/tests/cli.test.ts +++ /dev/null @@ -1,14 +0,0 @@ -import { describe, expect, test } from 'bun:test'; -import { createPrimaCommands } from '../src/cli.ts'; - -describe('prima cli options', () => { - test('takes the logging flags before the command as well as after it', () => { - const cmd = createPrimaCommands(); - const root = cmd.options.map((option) => option.long); - const check = cmd.commands.find((command) => command.name() === 'check')!.options.map((option) => option.long); - - expect(root).toContain('--verbose'); - expect(root).toContain('--debug'); - expect(check).toContain('--verbose'); - }); -}); From 4199fe55f1330f85a9249df60a2e9e926867e3f7 Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Wed, 12 Aug 2026 21:59:00 +0300 Subject: [PATCH 35/39] Show prima's current activity on one rewritten line Prima ran silent until the envelope arrived. It now renders the tracked activity - the model being asked, the browser step underway - as a single stderr line that each new activity overwrites, cleared before the report. Only drawn on a terminal and only when DEBUG is not printing the log, so piped output stays byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- CHANGELOG.md | 12 ++++++++++++ boat/prima/src/activity-line.ts | 33 +++++++++++++++++++++++++++++++++ boat/prima/src/cli.ts | 3 +++ 3 files changed, 48 insertions(+) create mode 100644 boat/prima/src/activity-line.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index e40afcf3..9c0869f8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,17 @@ # Changelog +## 2026-08-12 + +### Prima shows what it is doing while it does it + +A prima command used to sit silent until the envelope arrived. It now writes the current activity — +the model it is asking, the browser step underway, the scenario it is testing — as a single line +that each new activity overwrites. The line is erased before the envelope is printed, so the report is the only thing left +on screen. + +It is drawn on stderr and only when that is a terminal: piped or captured output stays exactly as it +was, byte for byte. With `DEBUG` set the log takes over and the line stays out of its way. + ## 2026-08-11 ### Prima prints its log when `DEBUG` is set diff --git a/boat/prima/src/activity-line.ts b/boat/prima/src/activity-line.ts new file mode 100644 index 00000000..bcfc15af --- /dev/null +++ b/boat/prima/src/activity-line.ts @@ -0,0 +1,33 @@ +import chalk from 'chalk'; +import { type ActivityEntry, addActivityListener, removeActivityListener } from '../../../src/activity.ts'; +import { isVerboseMode } from '../../../src/utils/logger.ts'; + +const RESET_LINE = '\r\u001b[2K'; + +const stream = process.stderr; +let tracking = false; + +export function trackActivityLine(): void { + if (tracking) return; + if (!stream.isTTY) return; + if (isVerboseMode()) return; + + tracking = true; + addActivityListener(writeActivityLine); +} + +export function clearActivityLine(): void { + if (!tracking) return; + + tracking = false; + removeActivityListener(writeActivityLine); + stream.write(RESET_LINE); +} + +function writeActivityLine(activity: ActivityEntry | null): void { + if (!activity) return; + + const width = (stream.columns || 80) - 2; + const message = Array.from(activity.message.replace(/\s+/g, ' ').trim()).slice(0, width).join(''); + stream.write(`${RESET_LINE}${chalk.gray(message)}`); +} diff --git a/boat/prima/src/cli.ts b/boat/prima/src/cli.ts index c52a9c32..7414ab3b 100644 --- a/boat/prima/src/cli.ts +++ b/boat/prima/src/cli.ts @@ -3,6 +3,7 @@ import dedent from 'dedent'; import { keepServerRunning } from '../../../src/browser-server.ts'; import { browserErrorMessage } from '../../../src/utils/browser-errors.ts'; import { isVerboseMode, setQuietMode } from '../../../src/utils/logger.ts'; +import { clearActivityLine, trackActivityLine } from './activity-line.ts'; import { type EnvelopeData, renderEnvelope } from './envelope.ts'; import { Prima, type PrimaOptions } from './prima.ts'; @@ -108,6 +109,7 @@ function primaFor(options: any): Prima { async function runPrima(options: any, command: string, run: (prima: Prima) => Promise<EnvelopeData>, record = true): Promise<void> { setQuietMode(!isVerboseMode()); + trackActivityLine(); const prima = primaFor(options); const startedAt = Date.now(); @@ -120,6 +122,7 @@ async function runPrima(options: any, command: string, run: (prima: Prima) => Pr } if (record) prima.record(envelope, Date.now() - startedAt); + clearActivityLine(); console.log(renderEnvelope(envelope)); await prima.stop().catch(() => {}); process.exit(envelope.ok ? 0 : 1); From d4dc92b9d4156a1bd4dde73db00b0abc02d6c718 Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Sun, 16 Aug 2026 16:04:49 +0300 Subject: [PATCH 36/39] Document prima as it is, not as it was before this branch MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Prima boat reference still described the commands and the failure behaviour this branch replaced: `click` and `fill` as tiers to fall back to, `healed: true` and `--no-heal` as what happens when an action fails, `-v/--verbose` as the way to see the log, and a `### Verdict` block that no longer renders. `check`, `report`, `config` and `status` were missing entirely. The section now follows `boat/prima/src/cli.ts`: the `check` → `do` → `pw` ladder, per-command notes for the outcome and step reporting, the envelope sections as `renderEnvelope` emits them, `DEBUG` in place of the logging flags, and a session-report block. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- docs/reference/commands.md | 69 +++++++++++++++++++++++++++++--------- 1 file changed, 53 insertions(+), 16 deletions(-) diff --git a/docs/reference/commands.md b/docs/reference/commands.md index dc253a3c..258e0d18 100644 --- a/docs/reference/commands.md +++ b/docs/reference/commands.md @@ -614,27 +614,43 @@ Run it as `npx explorbot prima <command>` or through the standalone `prima` bin. | Command | Purpose | |---|---| +| `prima check <scenario>` | Run a scenario end to end as a test, verify it, and report the steps it took | +| `prima do <instructions...>` | Run high-level instructions tester-style, one argument per instruction | | `prima pw <fn>` | Run a Playwright function expression against the open page | -| `prima do <steps...>` | Run several described steps tester-style, one argument per step | -| `prima click <target>` | Click an element described in plain words | -| `prima fill <field> <value>` | Fill a field described in plain words | | `prima ask <question>` | Answer a question about the current page | | `prima verify <assertion>` | Assert a statement about the current page (alias: `assert`) | | `prima research` | Map the current page and return verified locators | | `prima go <target>` | Navigate to a url, a path, or a page described in plain words | +| `prima status <hash>` | Show the artifacts and page detail recorded for an earlier command | +| `prima report` | Turn every command of a session into one html and markdown report | +| `prima config` | Show the AI models prima runs on and the config file they come from | | `prima browser {start\|stop\|status\|list}` | Manage the browsers prima drives | ### Choosing a command -Pick by what you hold, not by how hard the step looks. +Start at the top and come down only when the tier above cannot hold the work. +- **`check`** takes an outcome rather than a click path, works out how to reach it, verifies it itself, and reports every step with its proof. +- **`do`** takes several described instructions and runs them tester-style in one process. - **`pw`** is precise: a function expression built from a locator you already verified. No AI on the happy path, so it also works when no model is configured. -- **`click` and `fill`** take one action described in words and let AI resolve it against the current page. -- **`do`** takes several described steps and runs them tester-style in one process. -Never pass a locator or a function expression to `click`, `fill`, or `do` — describe the target. Never pass a description to `pw` — it takes executable code only. +Never pass a locator or a function expression to `check` or `do` — describe the target. Never pass a description to `pw` — it takes executable code only. -The loop that works: `go` to the page, `research` it once for verified locators, drive it with `pw`, then `verify` the outcome. Fall back to `click`, `fill`, or `do` whenever research left you no locator to hold. +Pass `do` the whole remaining sequence rather than one instruction per call. Every command is a process of its own, so a sequence split across calls pays the startup and page-capture cost each time. `check` and `do` legitimately run for minutes. + +### `check`, `do`, and `verify` in detail + +`check` takes `--expected <outcome>`, repeatable for several; without it the scenario text is the single expected outcome. Each comes back under `### Expected outcomes` as `PASSED`, `FAILED` or `not verified` — "not verified" means the run never checked it, which is not the same as false. Page problems seen along the way appear under `### Answer` rather than as step failures. + +```bash +prima check "signup rejects a duplicate email" \ + --expected "an error names the email as taken" \ + --expected "no second account is created" +``` + +`do` numbers every instruction and accounts for it: `### Steps` reports each as `ok` or `FAIL` with what proved it, and one that could not be carried out fails the command. Nothing runs past the last instruction given. + +`verify` lists every assertion it could express as `PASSED` or `FAILED` with its Playwright form, and gives no overall verdict — read the lines and decide. `none ran` means the claim could not be expressed at all, which is not the same as false. ### The envelope @@ -657,20 +673,18 @@ ariaDiff: - button "Submit" ### Instance -instance: default (1 tab) | other instances: none -browser: attached (playwright-cli session "default", workspace /home/you/projects/shop) +default (1 tab) | attached to playwright-cli session "default" | details: prima status 7f3a91 ### Artifacts aria: /home/you/.explorbot/sites/app.example.com/output/prima/2026-08-04T10-04-22-285Z/aria.yml html: /home/you/.explorbot/sites/app.example.com/output/prima/2026-08-04T10-04-22-285Z/page.html -network: /home/you/.explorbot/sites/app.example.com/output/prima/2026-08-04T10-04-22-285Z/network.jsonl ``` -`used:` is code that already executed. For `click`, `fill`, `do`, and `go` those are CodeceptJS steps you can copy into a test as they are; for `pw` it is the Playwright expression you passed, which a CodeceptJS test needs wrapped in `I.usePlaywrightTo(...)`. Log lines can precede the envelope, so start parsing at the first `###` line. +`used:` is code that already executed. For `check`, `do`, and `go` those are CodeceptJS steps you can copy into a test as they are; for `pw` it is the Playwright expression you passed, which a CodeceptJS test needs wrapped in `I.usePlaywrightTo(...)`. Log lines can precede the envelope, so start parsing at the first `###` line. -`ask`, `research`, and `verify` replace the `### Changes` block with `### Answer`, `### Research`, or `### Verdict`. A failure adds `### Failure` with the error and the compact ARIA of the page, so you can retarget from the envelope itself instead of opening the artifact files. +`### Changes` renders on every action, saying `no change` when the page is identical — so a successful command proves what it did. `check` and `do` add `### Steps`, `check` adds `### Expected outcomes`, and `ask`, `research`, and `verify` add `### Answer`, `### Research`, or `### Assertions` beside the changes rather than instead of them. `pw` adds `### Value` when its expression returns one. `network:` appears under `### Artifacts` only when requests were captured, and everything else recorded for a command is behind `prima status <hash>`. -When an action fails, AI retries it along a different route; `healed: true` means the outcome was reached another way and `used:` holds the code that worked. `--no-heal` skips that and fails fast. +**A failed action is a failure.** Nothing is retried along a different route and no other element is substituted, so `ok: true` means the action you asked for is the one that landed. A failure adds `### Failure` with the error and the compact ARIA of the page, so you can retarget from the envelope itself instead of opening the artifact files. ### Browsers and sessions @@ -694,10 +708,9 @@ Every command takes these: | `-i, --instance <name>` | Which prima-owned browser to talk to; parallel work needs one each | | `--session [file]` | Cookies and storage persisted across processes; ignored while attached, since the attached session keeps its own | | `--url <url>` | Page to open when the session has no page yet | -| `--no-heal` | Fail immediately instead of letting AI retry a failed action | | `--ephemeral` | Keep no state between runs. Applies to config-free runs only — with a config file the output directory comes from the config | | `--framework <name>` | Parsed but not active yet; reported code is CodeceptJS whatever you pass | -| `-v, --verbose`, `--debug`, `-c, --config <path>`, `-p, --path <path>` | As on every other Explorbot command | +| `-c, --config <path>`, `-p, --path <path>` | As on every other Explorbot command | `--instance` and `--session` answer different questions: `--instance` picks *which browser process* prima drives, `--session` decides *whose cookies* it starts from. @@ -705,6 +718,7 @@ A few commands add their own: | Command | Option | Description | |---|---|---| +| `check` | `--expected <outcome>` | An outcome the run must reach; repeat the flag for several | | `ask` | `--no-vision` | Answer from page structure only, without a screenshot | | `research` | `--data` | Include data extraction in the map | | `research` | `--deep` | Expand hidden elements for a deeper map | @@ -712,6 +726,29 @@ A few commands add their own: | `browser start` | `-s, --show` / `--headless` | Launch the browser with or without a window | | `browser stop` | `--all` | Stop every running instance | +Prima carries no logging flags of its own. `DEBUG` in front of a command prints everything the run does — config and browser attachment, every step as it executes, and the debug stream of the agents behind it: + +```bash +DEBUG='explorbot:*' prima do "open the account menu" "switch the theme to dark" +``` + +Without it, a running command writes its current activity to stderr as a single line that each new activity overwrites, erased before the envelope is printed. Piped or captured output is unaffected. + +### Session reports + +Every command is logged to `output/prima/sessions/` as it runs, so `prima report` needs no browser and outlives the session: + +```bash +prima report # the session used most recently +prima report --pw-session my-app # a named playwright-cli session +``` + +It writes one html and one markdown report — each command with its steps, expected outcomes and the proof recorded for them. The log is written in the format the Testomat.io reporter replays, so the same file can be sent as a run: + +```bash +TESTOMATIO=<apiKey> npx @testomatio/reporter replay <the path prima report prints> +``` + ### Without a config file Prima follows the same [configuration ladder](#environment-variables) as every other command, so a provider name in the environment is enough: From c3953573c5d2f1bd441d275835815c1f99d6258e Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Sun, 16 Aug 2026 16:05:47 +0300 Subject: [PATCH 37/39] Count retried attempts, warn on a foreign origin, keep the session with its project MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three fixes that were sitting in the working tree, none of them prima's. Token accounting recorded usage from the response `withRetry` finally returned, so every attempt discarded on the way — an empty completion, a truncated one — was spent but never counted. Usage is now recorded per attempt, inside the retried block. `resolveTargetPath` warns when the target's origin differs from the configured base URL. Relative navigation still resolves against the base URL, so the run would otherwise drift back to the configured host with no explanation. `--session <file>` is resolved against `-p, --path`, so a run pointed at another project reads and writes its session file there rather than in the directory the command happened to start in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- CHANGELOG.md | 13 ++++++++ src/ai/provider.ts | 6 ++-- src/config.ts | 12 ++++++-- src/explorbot.ts | 1 + tests/unit/provider-usage-retry.test.ts | 40 +++++++++++++++++++++++++ 5 files changed, 66 insertions(+), 6 deletions(-) create mode 100644 tests/unit/provider-usage-retry.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index a9770c82..109a0ece 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,18 @@ # Changelog +## 2026-08-16 + +### Changes + +- Token counters include the attempts that were retried. Only the attempt that finally succeeded + was counted, so a run that retried its way through a flaky model reported a fraction of the + tokens it had actually spent. +- Starting on a host other than the configured one now says so. Relative navigation still resolves + against the base URL, so the warning names the `web.url` to set to make the two agree. +- `--session <file>` resolves against the working directory given by `-p, --path` rather than the + directory the command was started from, so a run pointed at another project keeps its session + file with that project. + ## 2026-08-12 ### Prima shows what it is doing while it does it diff --git a/src/ai/provider.ts b/src/ai/provider.ts index 280037b8..678659f6 100644 --- a/src/ai/provider.ts +++ b/src/ai/provider.ts @@ -326,6 +326,7 @@ export class Provider { try { const response = await withRetry(async () => { const result = await generateText({ messages, ...config }); + this.recordUsage(options.agentName || 'unknown', modelName, result.usage); if (!result.text) { debugLog(result); if (result.finishReason === 'length') { @@ -342,8 +343,6 @@ export class Provider { clearActivity(); responseLog(response.text); - this.recordUsage(options.agentName || 'unknown', modelName, response.usage); - return response; } catch (error: any) { clearActivity(); @@ -378,6 +377,7 @@ export class Provider { try { const response = await withRetry(async () => { const result = (await this.raceWithIdleTimeout((signal) => generateText({ messages, ...config, abortSignal: signal }), config.timeout || 30000)) as any; + this.recordUsage(options.agentName || 'unknown', modelName, result.usage); const hasToolCall = (result.toolCalls?.length || 0) > 0; if (!result.text && !hasToolCall && result.finishReason === 'length') { throw new ContextLengthError('AI response empty: output truncated at maxTokens. Increase maxOutputTokens in config or use a model with higher output capacity.'); @@ -397,8 +397,6 @@ export class Provider { responseLog(response.text); - this.recordUsage(options.agentName || 'unknown', modelName, response.usage); - return response; } catch (error: any) { clearActivity(); diff --git a/src/config.ts b/src/config.ts index 90dfa8b5..2e892069 100644 --- a/src/config.ts +++ b/src/config.ts @@ -6,7 +6,7 @@ import dedent from 'dedent'; import matter from 'gray-matter'; import { type SiteRecord, findGlobalConfig, globalEnvPath, isGlobalConfigPath, registerSite, resolveSiteTarget } from './global-config.js'; import { getCliName } from './utils/cli-name.js'; -import { log } from './utils/logger.js'; +import { log, tag } from './utils/logger.js'; export const PROVIDERS: Record<string, ProviderInfo> = { openai: { envKey: 'OPENAI_API_KEY', load: async () => (await import('@ai-sdk/openai')).createOpenAI() }, @@ -455,7 +455,15 @@ export class ConfigParser { } public resolveTargetPath(target?: string): string { - if (!this.site) return target || '/'; + if (!this.site) { + const configured = this.config?.playwright?.url || this.config?.web?.url; + const targetOrigin = target ? URL.parse(target)?.origin : null; + const baseOrigin = configured ? URL.parse(configured)?.origin : null; + if (targetOrigin && baseOrigin && targetOrigin !== baseOrigin) { + tag('warning').log(`Exploring ${targetOrigin} but base URL is ${baseOrigin}. Relative navigation resolves against the base URL — set web.url to ${targetOrigin} to avoid it.`); + } + return target || '/'; + } if (!target) return this.siteStartPath; const resolved = resolveSiteTarget(target, this.site.url); diff --git a/src/explorbot.ts b/src/explorbot.ts index e51fb745..7278718e 100644 --- a/src/explorbot.ts +++ b/src/explorbot.ts @@ -129,6 +129,7 @@ export class ExplorBot { if (this.provider) return; this.config = await this.configParser.loadConfig(this.options); if (this.options.session === true) this.options.session = path.join(this.configParser.getOutputDir(), 'session.json'); + if (typeof this.options.session === 'string') this.options.session = path.resolve(this.options.path || '.', this.options.session); if (this.options.optionalAi) return this.bootstrapOptionalProvider(); this.provider = new AIProvider(this.config.ai); await this.provider.validateConnection(); diff --git a/tests/unit/provider-usage-retry.test.ts b/tests/unit/provider-usage-retry.test.ts new file mode 100644 index 00000000..9646ffae --- /dev/null +++ b/tests/unit/provider-usage-retry.test.ts @@ -0,0 +1,40 @@ +import { beforeEach, describe, expect, it } from 'bun:test'; +import { MockLanguageModelV3 } from 'ai/test'; +import { Provider } from '../../src/ai/provider.js'; +import { Stats } from '../../src/stats.js'; + +const USAGE = { + inputTokens: { total: 1000, noCache: 1000, cacheRead: 0, cacheWrite: 0 }, + outputTokens: { total: 100, text: 100, reasoning: 0 }, +}; + +function buildFlakyModel(): MockLanguageModelV3 { + let call = 0; + return new MockLanguageModelV3({ + provider: 'test', + modelId: 'flaky-model', + doGenerate: async () => { + call++; + if (call === 1) { + return { content: [], finishReason: 'stop' as const, usage: USAGE, warnings: [] }; + } + return { content: [{ type: 'text' as const, text: 'recovered' }], finishReason: 'stop' as const, usage: USAGE, warnings: [] }; + }, + }); +} + +describe('token accounting across retries', () => { + beforeEach(() => { + Stats.models = {}; + }); + + it('counts tokens burned by an attempt that is later retried', async () => { + const model = buildFlakyModel(); + const provider = new Provider({ model, apiKey: 'test-key', config: {}, vision: false } as any); + + const response = await provider.chat([{ role: 'user', content: 'hi' }], model, { maxRetries: 3, retryDelay: 0 }); + + expect(response.text).toBe('recovered'); + expect(Stats.models['flaky-model']?.total).toBe(2200); + }); +}); From d2489f2c71286b27956cc097e95d6cdf1e998471 Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Sun, 16 Aug 2026 16:05:53 +0300 Subject: [PATCH 38/39] Ignore the session and browser state a run leaves in the repo MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Running the CLI from the checkout drops `session.json` and `.playwright-cli/` next to the source. The session file holds the cookies and storage of whatever app was being driven, so it is not something to offer up for staging. `testomatio.debug.json` went in with it — a symlink to a file under `/tmp` that this branch committed by accident and that has been broken ever since. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- .gitignore | 7 +++++++ testomatio.debug.json | 1 - 2 files changed, 7 insertions(+), 1 deletion(-) delete mode 120000 testomatio.debug.json diff --git a/.gitignore b/.gitignore index e236cd27..e037b9d6 100644 --- a/.gitignore +++ b/.gitignore @@ -9,6 +9,10 @@ bun.lockb # Regression harness run artifacts tests/regression/.runs/ +# Session state and browser sessions written by running the CLI in this repo +session.json +.playwright-cli/ + # Environment variables .env .env.local @@ -148,3 +152,6 @@ Thumbs.db # Keep the temporarily vendored OpenRouter AI-SDK-7 provider (PR #511) !vendor/*.tgz + +# Reporter debug dump +testomatio.debug.json diff --git a/testomatio.debug.json b/testomatio.debug.json deleted file mode 120000 index 3c3f2de6..00000000 --- a/testomatio.debug.json +++ /dev/null @@ -1 +0,0 @@ -/tmp/testomatio.debug.2026-08-09T14-20-32.json \ No newline at end of file From 85887f020dc363e85c2831ea77002c99b20cf17c Mon Sep 17 00:00:00 2001 From: DavertMik <davert@testomat.io> Date: Sun, 16 Aug 2026 16:10:47 +0300 Subject: [PATCH 39/39] Say which commands actually carry used: and Changes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `do` clears both before returning — its code and its diffs belong to the step log — and `check` reports through `### Steps` and `### Expected outcomes` without ever setting them. The section claimed both blocks for all four commands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --- docs/reference/commands.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/commands.md b/docs/reference/commands.md index 258e0d18..fc99a734 100644 --- a/docs/reference/commands.md +++ b/docs/reference/commands.md @@ -680,9 +680,9 @@ aria: /home/you/.explorbot/sites/app.example.com/output/prima/2026-08-04T10-04-2 html: /home/you/.explorbot/sites/app.example.com/output/prima/2026-08-04T10-04-22-285Z/page.html ``` -`used:` is code that already executed. For `check`, `do`, and `go` those are CodeceptJS steps you can copy into a test as they are; for `pw` it is the Playwright expression you passed, which a CodeceptJS test needs wrapped in `I.usePlaywrightTo(...)`. Log lines can precede the envelope, so start parsing at the first `###` line. +`used:` is code that already executed: for `go` the CodeceptJS step it ran, for `pw` the Playwright expression you passed, which a CodeceptJS test needs wrapped in `I.usePlaywrightTo(...)`. Log lines can precede the envelope, so start parsing at the first `###` line. -`### Changes` renders on every action, saying `no change` when the page is identical — so a successful command proves what it did. `check` and `do` add `### Steps`, `check` adds `### Expected outcomes`, and `ask`, `research`, and `verify` add `### Answer`, `### Research`, or `### Assertions` beside the changes rather than instead of them. `pw` adds `### Value` when its expression returns one. `network:` appears under `### Artifacts` only when requests were captured, and everything else recorded for a command is behind `prima status <hash>`. +`### Changes` renders on every action envelope, saying `no change` when the tree is identical — so a successful command proves what it did instead of leaving you to check. `check` and `do` report per step rather than in aggregate: `### Steps` names each step with the code it ran and what proved it, and `page after each step:` points at the captures. `check` adds `### Expected outcomes`; `ask`, `research`, and `verify` add `### Answer`, `### Research`, or `### Assertions`; `pw` adds `### Value` when its expression returns one. `network:` appears under `### Artifacts` only when requests were captured, and everything else recorded for a command is behind `prima status <hash>`. **A failed action is a failure.** Nothing is retried along a different route and no other element is substituted, so `ok: true` means the action you asked for is the one that landed. A failure adds `### Failure` with the error and the compact ARIA of the page, so you can retarget from the envelope itself instead of opening the artifact files.