diff --git a/.gitignore b/.gitignore index e236cd27..e037b9d6 100644 --- a/.gitignore +++ b/.gitignore @@ -9,6 +9,10 @@ bun.lockb # Regression harness run artifacts tests/regression/.runs/ +# Session state and browser sessions written by running the CLI in this repo +session.json +.playwright-cli/ + # Environment variables .env .env.local @@ -148,3 +152,6 @@ Thumbs.db # Keep the temporarily vendored OpenRouter AI-SDK-7 provider (PR #511) !vendor/*.tgz + +# Reporter debug dump +testomatio.debug.json diff --git a/CHANGELOG.md b/CHANGELOG.md index 3201e4c1..109a0ece 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,13 +1,187 @@ # Changelog +## 2026-08-16 + +### Changes + +- Token counters include the attempts that were retried. Only the attempt that finally succeeded + was counted, so a run that retried its way through a flaky model reported a fraction of the + tokens it had actually spent. +- Starting on a host other than the configured one now says so. Relative navigation still resolves + against the base URL, so the warning names the `web.url` to set to make the two agree. +- `--session ` resolves against the working directory given by `-p, --path` rather than the + directory the command was started from, so a run pointed at another project keeps its session + file with that project. + +## 2026-08-12 + +### Prima shows what it is doing while it does it + +A prima command used to sit silent until the envelope arrived. It now writes the current activity — +the model it is asking, the browser step underway, the scenario it is testing — as a single line +that each new activity overwrites. The line is erased before the envelope is printed, so the report is the only thing left +on screen. + +It is drawn on stderr and only when that is a terminal: piped or captured output stays exactly as it +was, byte for byte. With `DEBUG` set the log takes over and the line stays out of its way. + ## 2026-08-11 +### Prima prints its log when `DEBUG` is set + +`DEBUG` in front of any prima command prints everything the run does: config and browser +attachment, every step as it executes, and the debug stream of the agents behind it. Prima carries +no logging flags of its own — `--verbose` and `--debug` are gone. + +```bash +DEBUG='explorbot:*' prima do "open the account menu" "switch the theme to dark" +DEBUG='explorbot:tester' prima check "a workflow can be created" +``` + ### Changes +- Verbose mode turns the debug log on for real. It set the `DEBUG` variable after the logging + library had already read it, so `explorbot --verbose` printed a fraction of what + `DEBUG='explorbot:*'` printed in front of the same command. - Page snapshots no longer carry variant-prefixed classes such as `dark:`, `hover:` or `md:`. Those classes describe how an element looks in some other state, not what it is, so the agents now read a smaller and less noisy page. +## 2026-08-10 + +### `prima report` collects a whole session into one report + +Every prima command is logged to `output/prima/sessions/` as it runs. `prima report` turns that +log into an HTML and a Markdown report — each command with its steps, expected outcomes and the +proof recorded for them. It needs no browser, so the report still comes out after the session is +closed. + +```bash +prima report # the session used most recently +prima report --pw-session my-app # a named playwright-cli session +``` + +The log is written in the format the Testomat.io reporter replays, so the same file can be sent +to Testomat.io as a run: + +```bash +TESTOMATIO= npx @testomatio/reporter replay +``` + +### Changes + +- [Prima] Prima no longer generates a report of its own while it runs. A `check` used to end by + writing the session HTML and Markdown reports, which belong to an explorbot run rather than to + a single command. Reports are produced only when `prima report` asks for one. + +## 2026-08-09 + +### `prima check` runs a whole scenario and reports what it proved + +`prima check` takes a behaviour, not a click path, and runs it the way a tester would: it drives +the page, verifies the outcome itself, and reports every step with the proof for each. Use it when +you want a verdict; use `prima do` when you already know the steps. + +```bash +prima check "a workflow can be created and appears in the list" --url http://app.test +prima check "signup rejects a duplicate email" \ + --expected "an error names the email as taken" \ + --expected "no second account is created" +``` + +- **`--expected `** — an outcome the run must reach; repeat the flag for several. Without + it the scenario text is the single expected outcome. Each one comes back under + `### Expected outcomes` as `PASSED`, `FAILED` or `not verified` — "not verified" means the run + never checked it, which is not the same as false. +- **`--url `** — open that page before starting, when the browser is not already on it. + +### `prima status ` reopens an earlier command + +Every envelope prints a hash on its `### Instance` line. `prima status ` returns the page +detail and artifact paths recorded for that command, so envelopes stay short and the full ARIA +tree, HTML and network log are one command away instead of inline. + +```bash +prima status 66800d6d2c8c553 +``` + +### Changes + +- [Prima] `do` now ends at the last step you gave it. It used to keep acting after the sequence + was finished, and could report success while a step it was asked to check never held. A step it + could not carry out is named under `### Failure` and fails the command. +- [Prima] `do` carries the same tools a test run does, so a single call can act, look and assert + across a long sequence rather than being split into one command per step. +- [Prima] `pw` returns the value of the expression under `### Value`. A `page.title()` or + `locator.count()` used to run and have its answer thrown away. +- [Prima] `verify` lists every assertion it ran with its own `PASSED` or `FAILED`, plus the + Playwright form of the ones that held, and gives no overall verdict — read the lines and decide. +- [Prima] Research output drops CSS selectors, XPaths and coordinates, which were the bulk of the + map and are not what you act on. +- [Prima] Prima commands print the envelope and nothing else. The banner, config line, browser + startup and disconnect chatter are hidden unless `--verbose` or `--debug` is passed. +- [Prima] `click` and `fill` are removed — describe the whole sequence to `do` instead, which + attaches once and carries all of it. +- [Tester] A test that stops making progress is handed to final review instead of being marked + failed on the spot. A run that had already done its work and gone quiet was reported as a + failure; the verdict now comes from reviewing the result. +- [Tester] Console and network errors seen during a run are reported as page problems rather than + as failed steps, so they no longer sink a test that otherwise passed. +- [Tester] An expected outcome counts as settled only when it is recorded back word for word, + and the prompt now says so — outcomes phrased differently were silently left unaccounted for. +- Page changes report values typed into fields, under a `typed:` section, alongside what was added, + removed and toggled. Filling a form previously showed as no change at all. +- Long field values in page snapshots are cut to an excerpt with a pointer to the full text, which + keeps a page holding a large document readable. +- Pages are considered ready as soon as the DOM goes quiet, instead of waiting on network idle. + Applications with a live websocket never reached network idle, so every snapshot paid the full + timeout. +- Generated tests assert element visibility, hidden state and field values through real Playwright + locators instead of leaving a TODO comment. + +## 2026-08-07 + +### Prima never substitutes your target + +A failed action now fails. Previously a `pw` call on a selector that did not exist could be +"healed" into clicking a different element and still report `ok: true` — so `ok: true` did not +mean your own action landed. Automatic retry along a different route is gone entirely, together +with the `--no-heal` flag that used to switch it off, and the `healed:` line and +`### Healing attempts` block in the envelope. + +### Configuration + +- **`ai.agents.prima.researchAfterVisits`** — how many visits to a page before `prima do` works + from that page's stored research map instead of its accessibility tree. Default: `3`. + +### Changes + +- [Prima] `### Changes` now appears on every action, showing what the accessibility tree gained, + lost or toggled — or saying `no change` outright. It previously went missing whenever the + command also produced an answer, a research map or a verdict, so a successful click proved + nothing. +- [Prima] Attaching to a running `playwright-cli` session works again. Prima was matching on a + field that no release of `playwright-cli` writes, so it reported no browser while one was open + and told you to open the session you already had. +- [Prima] Attached sessions are driven through the browser's own Playwright build, which is what + makes reading the page work across versions. +- [Prima] New `context()` step for `do`: when the element an instruction needs is missing from the + context it holds, it returns the page again, and drops to raw markup if asked a second time on + the same page. +- [Prima] `network.jsonl` is listed only when requests were actually recorded, instead of always + pointing at an empty file. +- [Navigator] `verify` now separates "this claim is false" from "no assertion can express this + claim". A claim it cannot phrase is no longer reported as a failing check, and is no longer + remembered as one. +- [Navigator] Verification can assert whether a control is enabled, disabled, checked, selected or + expanded. Only presence and text could be asserted before, so state claims always failed. +- Navigation: a redirect that only adds query parameters — a session id, a workspace flag — counts + as arriving. Reaching such a page previously burned minutes of retries and could still end in a + failure while the page sat correctly loaded. +- Page snapshots keep every attribute on an element. A control marked `[pressed]` or `[disabled]` + used to lose its ref, which hid exactly the controls worth acting on. +- Playwright updated to 1.62. + ## 2026-08-06 ### Global Installation diff --git a/bin/explorbot-cli.ts b/bin/explorbot-cli.ts index 02bb38e4..fdd06b66 100755 --- a/bin/explorbot-cli.ts +++ b/bin/explorbot-cli.ts @@ -27,7 +27,7 @@ const pkgVersion = JSON.parse(fs.readFileSync(pkgPath, 'utf-8')).version as stri program.name(cli).description('AI-powered web exploration tool').version(pkgVersion, '-V, --version'); -if (!process.env.EXPLORBOT_NO_BANNER) { +if (!process.env.EXPLORBOT_NO_BANNER && !process.argv.includes('prima')) { console.log(`⛵ ${chalk.yellow.bold(`Explorbot v${pkgVersion}`)} ${chalk.dim('Autonomous Testing Agent')}`); } diff --git a/boat/prima/bin/prima-cli.ts b/boat/prima/bin/prima-cli.ts old mode 100644 new mode 100755 diff --git a/boat/prima/src/activity-line.ts b/boat/prima/src/activity-line.ts new file mode 100644 index 00000000..bcfc15af --- /dev/null +++ b/boat/prima/src/activity-line.ts @@ -0,0 +1,33 @@ +import chalk from 'chalk'; +import { type ActivityEntry, addActivityListener, removeActivityListener } from '../../../src/activity.ts'; +import { isVerboseMode } from '../../../src/utils/logger.ts'; + +const RESET_LINE = '\r\u001b[2K'; + +const stream = process.stderr; +let tracking = false; + +export function trackActivityLine(): void { + if (tracking) return; + if (!stream.isTTY) return; + if (isVerboseMode()) return; + + tracking = true; + addActivityListener(writeActivityLine); +} + +export function clearActivityLine(): void { + if (!tracking) return; + + tracking = false; + removeActivityListener(writeActivityLine); + stream.write(RESET_LINE); +} + +function writeActivityLine(activity: ActivityEntry | null): void { + if (!activity) return; + + const width = (stream.columns || 80) - 2; + const message = Array.from(activity.message.replace(/\s+/g, ' ').trim()).slice(0, width).join(''); + stream.write(`${RESET_LINE}${chalk.gray(message)}`); +} diff --git a/boat/prima/src/cli.ts b/boat/prima/src/cli.ts index 25612859..7414ab3b 100644 --- a/boat/prima/src/cli.ts +++ b/boat/prima/src/cli.ts @@ -2,83 +2,71 @@ import { Command } from 'commander'; import dedent from 'dedent'; import { keepServerRunning } from '../../../src/browser-server.ts'; import { browserErrorMessage } from '../../../src/utils/browser-errors.ts'; -import { setPreserveConsoleLogs } from '../../../src/utils/logger.ts'; +import { isVerboseMode, setQuietMode } from '../../../src/utils/logger.ts'; +import { clearActivityLine, trackActivityLine } from './activity-line.ts'; import { type EnvelopeData, renderEnvelope } from './envelope.ts'; import { Prima, type PrimaOptions } from './prima.ts'; const helpContract = dedent` - Prima drives a browser that is already open. One command per process; every command - prints a plain-text envelope on stdout and exits 0 when ok, 1 when not. - - TIERS - choose by what you hold, not by how hard the step looks - pw Precise. A Playwright function expression built from a locator you - already verified. No AI on the happy path. - prima pw "({ page }) => page.click('[data-test=submit]')" - click / fill One action described in words; AI resolves it on the current page. - prima click "the primary action button in the header" - prima fill "the search box" "a search term" - do Several described steps, run tester-style in one process. - prima do "open the account menu" "choose the settings entry" - Never pass a locator or a function expression to click/fill/do - describe the target. - Never pass a description to pw - it takes executable code only. - - LOOP - prima go reach the page you want to work on - prima research once per new page; returns verified locators - prima pw "..." drive the page with those locators - prima verify "..." assert the outcome (prima ask "..." to inspect instead) - Fall back to click/fill/do whenever research left you no locator to hold. - - ENVELOPE - ### Result ok, command, healed, used - ### Page url, title, state hash, visit count - ### Changes what the accessibility tree gained or lost - ### Answer | ### Research | ### Verdict output of ask, research, verify - ### Failure error, reasoning, healing attempts, compact ARIA of the page - ### Instance the browser you are on and the other instances running - ### Artifacts paths to the full aria.yml, page.html and network.jsonl - used: is code that already executed - CodeceptJS steps to copy as they are, except - for pw, whose Playwright expression a test needs inside I.usePlaywrightTo(...). - Log lines can precede the envelope; start parsing at the first ### line. - - HEALING AND FAILURE - A failed action is retried by AI along a different route; healed: true means the - outcome was reached another way and used: holds the code that worked. - --no-heal skips that and fails fast. - Failures print compact ARIA inline, so retarget from the envelope itself and open - the artifact files only when the inline snapshot is not enough. - - SESSIONS - By default prima attaches to the playwright-cli browser of this workspace and works - on the tabs it already has open; driving the same session from both tools is the - intended usage. - playwright-cli open the session prima attaches to - --pw-session which playwright-cli session, when several are open - --endpoint <ep> attach to a browser server endpoint directly - prima browser start a prima-owned browser instead, when no session is open - --instance <name> which prima-owned browser you talk to; parallel work - needs one each - --session [file] cookies and storage persisted across processes; ignored - while attached, the attached session keeps its own - Prima never launches a browser implicitly and never closes an attached one - it - disconnects. browser list shows both kinds; ### Instance names the one you are on. - Every browser is reached over a Playwright browser-server endpoint, which needs the - Node build - run prima as "npx explorbot prima ..." or through the published prima - bin; from source under Bun the connection does not open. - When no AI model is usable pw still works; for everything else drive - playwright-cli directly. - Parsed but not active yet: --framework, so reported code is CodeceptJS whatever - you pass. + Prima is a high-level AI extension to playwright-cli, driving the browser it has open. + + playwright-cli open <url> starts the session + prima <command> ... drives it + playwright-cli close ends it + + One call takes a whole job: + + prima check "a workflow can be created and appears in the list" --expected "the new workflow is listed" + prima do "open the account menu" "choose the settings entry" "switch the theme to dark" "check it took effect" + prima pw "({ page }) => page.click('[data-test=submit]')" +`; + +const checkHelp = dedent` + check takes an outcome rather than a click path, and works out how to reach it. + --expected one outcome the run must reach, repeatable for several. Without it the + scenario text is the single expected outcome. Each comes back under + ### Expected outcomes as PASSED, FAILED or not verified - "not verified" + means the run never checked it, which is not the same as false. + Page problems seen on the way appear under ### Answer, not as step failures. +`; + +const doHelp = dedent` + Each instruction is numbered and accounted for: ### Steps reports each as ok or FAIL + with what proved it. One that could not be carried out fails the command and says why. + Nothing runs past the last instruction given. A whole remaining sequence in one call is + what makes this tier cheap. +`; + +const verifyHelp = dedent` + Reports each assertion it could express as PASSED or FAILED with its playwright form, + and gives no overall verdict - read the lines and decide. "none ran" means the claim + could not be expressed, which is not the same as false. +`; + +const reportHelp = dedent` + Commands are logged as they run, so the report needs no browser and outlives the session. + The most recent session is reported unless --pw-session names another. +`; + +const sessionHelp = dedent` + --endpoint <ep> attach to a browser server endpoint directly, skipping discovery + --instance <name> which prima-owned browser you talk to; parallel work needs one each + --session [file] cookies and storage persisted across processes; ignored while + attached, since the attached session keeps its own + --framework parsed but not active yet; reported code is CodeceptJS either way + DEBUG='explorbot:*' in front of a command prints the log of everything it does. + When no AI model is usable pw still works; for everything else drive playwright-cli. `; -function buildOptions(options: any): PrimaOptions { +let rootOptions: () => any = () => ({}); + +function buildOptions(subcommand: any): PrimaOptions { + const options = { ...rootOptions(), ...stripEmpty(subcommand) }; return { - verbose: options.verbose || options.debug, config: options.config, path: options.path, instance: options.instance, session: options.session, - heal: options.heal, ephemeral: options.ephemeral, framework: options.framework, noVision: options.vision === false, @@ -91,30 +79,39 @@ function buildOptions(options: any): PrimaOptions { }; } +function stripEmpty(options: any): any { + const present: any = {}; + for (const [key, value] of Object.entries(options || {})) { + if (value === undefined) continue; + present[key] = value; + } + return present; +} + function addCommonOptions(cmd: Command): Command { return cmd - .option('-v, --verbose', 'Enable verbose logging') - .option('--debug', 'Enable debug logging (same as --verbose)') .option('-c, --config <path>', 'Path to explorbot configuration file') .option('-p, --path <path>', 'Working directory path') .option('-i, --instance <name>', 'Browser instance to drive') .option('--session [file]', 'Persist cookies and storage to a session file') - .option('--no-heal', 'Fail immediately instead of letting AI retry a failed action') .option('--ephemeral', 'Keep no state between runs; applies to config-free runs, where output goes to a temp directory') .option('--framework <name>', 'Not active yet: framework the reported code targets, codeceptjs or playwright') .option('--url <url>', 'Page to open when the session has no page yet') .option('--endpoint <ep>', 'Websocket endpoint of a browser server to attach to, skipping discovery') - .option('--pw-session <title>', 'Title of the playwright-cli session to attach to'); + .option('--pw-session <title>', 'Title of the playwright-cli session to attach to') + .addHelpText('after', `\n${sessionHelp}`); } function primaFor(options: any): Prima { - setPreserveConsoleLogs(true); if (options.ephemeral) process.env.EXPLORBOT_EPHEMERAL = '1'; return new Prima(buildOptions(options)); } -async function runPrima(options: any, command: string, run: (prima: Prima) => Promise<EnvelopeData>): Promise<void> { +async function runPrima(options: any, command: string, run: (prima: Prima) => Promise<EnvelopeData>, record = true): Promise<void> { + setQuietMode(!isVerboseMode()); + trackActivityLine(); const prima = primaFor(options); + const startedAt = Date.now(); let envelope: EnvelopeData; try { @@ -124,6 +121,8 @@ async function runPrima(options: any, command: string, run: (prima: Prima) => Pr envelope = await prima.toolFailureEnvelope(command, error); } + if (record) prima.record(envelope, Date.now() - startedAt); + clearActivityLine(); console.log(renderEnvelope(envelope)); await prima.stop().catch(() => {}); process.exit(envelope.ok ? 0 : 1); @@ -143,32 +142,38 @@ async function runBrowser(options: any, run: (prima: Prima) => Promise<boolean>) export function createPrimaCommands(name = 'prima'): Command { const cmd = new Command(name); - cmd.description('Drive an already-open browser one command at a time and report back in a plain-text envelope'); + cmd.description('Tests and drives a web app through described behaviour instead of locators: one command carries a whole scenario, verifies it, and reports the proof'); + cmd.option('--pw-session <title>', 'Title of the playwright-cli session to attach to'); + cmd.option('--url <url>', 'Page to open when the session has no page yet'); cmd.addHelpText('after', `\n${helpContract}`); + rootOptions = () => cmd.opts(); addCommonOptions(cmd.command('pw <fn>').description('Run a Playwright function expression against the open page')).action(async (fn, options) => { await runPrima(options, `pw ${fn}`, (prima) => prima.pw(fn)); }); - addCommonOptions(cmd.command('do <instructions...>').description('Run high-level instructions tester-style, one argument per instruction')).action(async (instructions, options) => { - await runPrima(options, `do ${instructions.join(' ')}`, (prima) => prima.do(instructions)); - }); - - addCommonOptions(cmd.command('click <target>').description('Click an element described in plain words')).action(async (target, options) => { - await runPrima(options, `click ${target}`, (prima) => prima.click(target)); - }); + addCommonOptions(cmd.command('do <instructions...>').description('Run high-level instructions tester-style, one argument per instruction')) + .addHelpText('after', `\n${doHelp}`) + .action(async (instructions, options) => { + await runPrima(options, `do ${instructions.join(' ')}`, (prima) => prima.do(instructions)); + }); - addCommonOptions(cmd.command('fill <field> <value>').description('Fill a field described in plain words')).action(async (field, value, options) => { - await runPrima(options, `fill ${field} ${value}`, (prima) => prima.fill(field, value)); - }); + addCommonOptions(cmd.command('check <scenario>').description('Run a scenario end to end as a test, with its own verification, and report the steps it took')) + .option('--expected <outcome>', 'An outcome the run must reach; repeat the flag for several', (value: string, all: string[]) => [...all, value], []) + .addHelpText('after', `\n${checkHelp}`) + .action(async (scenario, options) => { + await runPrima(options, `check ${scenario}`, (prima) => prima.check(scenario, options.expected)); + }); addCommonOptions(cmd.command('ask <question>').description('Answer a question about the current page').option('--no-vision', 'Answer from page structure only, without a screenshot')).action(async (question, options) => { await runPrima(options, `ask ${question}`, (prima) => prima.ask(question)); }); - addCommonOptions(cmd.command('verify <assertion>').alias('assert').description('Assert a statement about the current page')).action(async (assertion, options) => { - await runPrima(options, `verify ${assertion}`, (prima) => prima.verify(assertion)); - }); + addCommonOptions(cmd.command('verify <assertion>').alias('assert').description('Assert a statement about the current page')) + .addHelpText('after', `\n${verifyHelp}`) + .action(async (assertion, options) => { + await runPrima(options, `verify ${assertion}`, (prima) => prima.verify(assertion)); + }); addCommonOptions( cmd.command('research').description('Map the current page and return verified locators').option('--data', 'Include data extraction in the map').option('--deep', 'Expand hidden elements for a deeper map').option('--fresh', 'Ignore the cached map and research the page again') @@ -181,6 +186,30 @@ export function createPrimaCommands(name = 'prima'): Command { await runPrima(options, `go ${target}`, (prima) => prima.go(target)); }); + addCommonOptions(cmd.command('config').description('Show the AI models prima runs on and the config file they come from')).action(async (options) => { + setQuietMode(!isVerboseMode()); + const prima = primaFor(options); + console.log(await prima.config().catch((error: unknown) => browserErrorMessage(error))); + await prima.stop().catch(() => {}); + process.exit(0); + }); + + addCommonOptions(cmd.command('status <hash>').description('Show the artifacts and page detail recorded for an earlier command')).action(async (hash, options) => { + await runPrima(options, `status ${hash}`, (prima) => prima.status(hash), false); + }); + + addCommonOptions(cmd.command('report').description('Turn every command of a session into one html and markdown report')) + .addHelpText('after', `\n${reportHelp}`) + .action(async (options) => { + setQuietMode(!isVerboseMode()); + console.log( + await primaFor(options) + .report() + .catch((error: unknown) => browserErrorMessage(error)) + ); + process.exit(0); + }); + const browser = cmd.command('browser').description('Manage the browsers prima drives'); addCommonOptions(browser.command('start').description('Start a prima-owned browser and hold it open until Ctrl+C')) diff --git a/boat/prima/src/envelope.ts b/boat/prima/src/envelope.ts index 46e68ffc..78411c5d 100644 --- a/boat/prima/src/envelope.ts +++ b/boat/prima/src/envelope.ts @@ -1,6 +1,12 @@ import { mkdirSync, writeFileSync } from 'node:fs'; import path from 'node:path'; +const EXPECTATION_LABELS = { + passed: 'PASSED ', + failed: 'FAILED ', + unverified: 'not verified', +}; + export interface InstanceInfo { name: string; tabs: number; @@ -9,59 +15,52 @@ export interface InstanceInfo { others: Array<{ name: string; tabs: number }>; } -export interface HealAttempt { - code: string; - outcome: string; -} - export interface EnvelopeData { ok: boolean; command: string; - healed?: boolean; - healNote?: string; used?: string[]; page: { url: string; previousUrl?: string; title: string; state: string; visits: number }; changes?: string | null; + steps?: Array<{ label: string; ok: boolean; proof: string }>; + expectations?: Array<{ text: string; status: 'passed' | 'failed' | 'unverified' }>; + stepFiles?: string; + value?: string; answer?: string; research?: string; - verdict?: { passed: boolean; evidence: string; code: string }; - failure?: { error: string; attempts: HealAttempt[]; reasoning?: string; compactAria?: string }; + assertions?: Array<{ code: string; passed: boolean; proof: string[] }>; + failure?: { error: string; compactAria?: string }; instance: InstanceInfo; - artifacts?: { aria: string; html: string; network: string }; + status?: string; + artifacts?: { aria: string; html: string; network?: string }; } export function renderEnvelope(data: EnvelopeData): string { - const sections = [renderResult(data), renderPage(data), renderOutcome(data), ...renderFailure(data), renderInstance(data.instance), renderArtifacts(data)]; + const sections = [renderResult(data), renderPage(data), renderValue(data), renderChanges(data), renderSteps(data), renderExpectations(data), renderOutcome(data), ...renderFailure(data), renderInstance(data), renderArtifacts(data)]; return sections.filter((section) => section).join('\n\n'); } -export function writeArtifacts(dir: string, snapshot: { aria: string | null; html: string | null; requests: unknown[] }): { aria: string; html: string; network: string } { +export function writeArtifacts(dir: string, snapshot: { aria: string | null; html: string | null; requests: unknown[] }): { aria: string; html: string; network?: string } { mkdirSync(dir, { recursive: true }); - const paths = { + const paths: { aria: string; html: string; network?: string } = { aria: path.resolve(dir, 'aria.yml'), html: path.resolve(dir, 'page.html'), - network: path.resolve(dir, 'network.jsonl'), }; writeFileSync(paths.aria, snapshot.aria ?? '', 'utf-8'); writeFileSync(paths.html, snapshot.html ?? '', 'utf-8'); + + if (!snapshot.requests.length) return paths; + + paths.network = path.resolve(dir, 'network.jsonl'); writeFileSync(paths.network, snapshot.requests.map((request) => `${JSON.stringify(request)}\n`).join(''), 'utf-8'); return paths; } function renderResult(data: EnvelopeData): string { const lines = [`ok: ${data.ok}`, `command: ${data.command}`]; - const healed = renderHealed(data); - if (healed) lines.push(healed); if (data.used?.length) lines.push(`used: ${data.used.join('; ')}`); return section('Result', lines.join('\n')); } -function renderHealed(data: EnvelopeData): string | null { - if (data.healed === undefined) return null; - if (data.healNote) return `healed: ${data.healed} (${data.healNote})`; - return `healed: ${data.healed}`; -} - function renderPage(data: EnvelopeData): string { const { url, previousUrl, title, state, visits } = data.page; const urlLabel = `url: ${url}`; @@ -73,28 +72,59 @@ function renderPage(data: EnvelopeData): string { return section('Page', lines.join('\n')); } +function renderValue(data: EnvelopeData): string | null { + if (data.value === undefined) return null; + return section('Value', data.value); +} + +function renderChanges(data: EnvelopeData): string | null { + if (data.changes === undefined || data.changes === null) return null; + return section('Changes', data.changes); +} + +function renderSteps(data: EnvelopeData): string | null { + if (!data.steps?.length) return null; + + const lines: string[] = []; + data.steps.forEach((step, index) => { + lines.push(`${index + 1}. ${step.ok ? 'ok ' : 'FAIL'} ${step.label}`); + for (const line of (step.proof || '').split('\n').filter(Boolean)) lines.push(` ${line}`); + }); + if (data.stepFiles) lines.push('', `page after each step: ${data.stepFiles}`); + return section('Steps', lines.join('\n')); +} + +function renderExpectations(data: EnvelopeData): string | null { + if (!data.expectations?.length) return null; + const lines = data.expectations.map((expectation, index) => `${index + 1}. ${EXPECTATION_LABELS[expectation.status]} ${expectation.text}`); + return section('Expected outcomes', lines.join('\n')); +} + function renderOutcome(data: EnvelopeData): string | null { - if (data.changes) return section('Changes', data.changes); if (data.answer) return section('Answer', data.answer); if (data.research) return section('Research', data.research); - if (!data.verdict) return null; - const lines = [`passed: ${data.verdict.passed}`, `evidence: ${data.verdict.evidence}`, `code: ${data.verdict.code}`]; - return section('Verdict', lines.join('\n')); + if (!data.assertions) return null; + + if (!data.assertions.length) return section('Assertions', 'none ran — no assertion could express this claim, so nothing was checked against the page'); + + const lines = data.assertions.map((assertion) => { + const code = assertion.code + .split('\n') + .map((line) => line.trim()) + .filter((line) => line && !line.startsWith('//')) + .join(' '); + return `${code} => ${assertion.passed ? 'PASSED' : 'FAILED'}`; + }); + + const proof = data.assertions.flatMap((assertion) => assertion.proof); + if (proof.length) lines.push('', 'playwright:', ...proof); + + return section('Assertions', lines.join('\n')); } function renderFailure(data: EnvelopeData): Array<string | null> { if (!data.failure) return []; - const lines = [`error: ${data.failure.error}`]; - if (data.failure.reasoning) lines.push(`reasoning: ${data.failure.reasoning}`); - return [section('Failure', lines.join('\n')), renderAttempts(data.failure.attempts), renderCompactAria(data.failure.compactAria)]; -} - -function renderAttempts(attempts: HealAttempt[]): string | null { - if (!attempts?.length) return null; - const labels = attempts.map((attempt, index) => `${index + 1}. ${attempt.code}`); - const width = Math.max(...labels.map((label) => label.length)) + 3; - const lines = labels.map((label, index) => align(label, `→ ${attempts[index].outcome}`, width)); - return section(`Healing attempts (${attempts.length})`, lines.join('\n')); + return [section('Failure', `error: ${data.failure.error}`), renderCompactAria(data.failure.compactAria)]; } function renderCompactAria(compactAria?: string): string | null { @@ -102,22 +132,19 @@ function renderCompactAria(compactAria?: string): string | null { return section('Current page (compact ARIA)', compactAria); } -function renderInstance(instance: InstanceInfo): string { - const others = instance.others.map((other) => `${other.name} (${tabsLabel(other.tabs)})`); - const lines = [`instance: ${instance.name} (${tabsLabel(instance.tabs)}) | other instances: ${otherInstances(others)}`, browserLine(instance)]; - return section('Instance', lines.join('\n')); -} - -function otherInstances(others: string[]): string { - if (!others.length) return 'none'; - return others.join(', '); +function renderInstance(data: EnvelopeData): string | null { + const instance = data.instance; + const parts = [`${instance.name} (${tabsLabel(instance.tabs)})`, browserLine(instance)]; + if (instance.others.length) parts.push(`other instances: ${instance.others.map((other) => `${other.name} (${tabsLabel(other.tabs)})`).join(', ')}`); + if (data.status) parts.push(`details: prima status ${data.status}`); + return section('Instance', parts.join(' | ')); } function browserLine(instance: InstanceInfo): string { - if (instance.attached) return `browser: attached (${instance.attached})`; - if (instance.startedAgo) return `browser: running, started ${instance.startedAgo} ago`; - if (instance.tabs > 0) return 'browser: running'; - return 'browser: not running'; + if (instance.attached) return `attached to ${instance.attached}`; + if (instance.startedAgo) return `running, started ${instance.startedAgo} ago`; + if (instance.tabs > 0) return 'running'; + return 'not running'; } function tabsLabel(tabs: number): string { @@ -125,9 +152,10 @@ function tabsLabel(tabs: number): string { return `${tabs} tabs`; } -function renderArtifacts(data: EnvelopeData): string | null { +export function renderArtifacts(data: EnvelopeData): string | null { if (!data.artifacts) return null; - const lines = [`aria: ${data.artifacts.aria}`, `html: ${data.artifacts.html}`, `network: ${data.artifacts.network}`]; + const lines = [`aria: ${data.artifacts.aria}`, `html: ${data.artifacts.html}`]; + if (data.artifacts.network) lines.push(`network: ${data.artifacts.network}`); return section('Artifacts', lines.join('\n')); } diff --git a/boat/prima/src/prima.ts b/boat/prima/src/prima.ts index 94338944..fd6b5de4 100644 --- a/boat/prima/src/prima.ts +++ b/boat/prima/src/prima.ts @@ -1,36 +1,78 @@ +import { createHash } from 'node:crypto'; +import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'; import { createRequire } from 'node:module'; import path from 'node:path'; +import { tool } from 'ai'; import dedent from 'dedent'; +import { z } from 'zod'; import * as playwright from 'playwright'; import type { Browser } from 'playwright'; import { ActionResult } from '../../../src/action-result.ts'; -import type { Navigator } from '../../../src/ai/navigator.ts'; import { actionRule, locatorRule } from '../../../src/ai/rules.ts'; -import { createCodeceptJSTools } from '../../../src/ai/tools.ts'; +import { createAgentTools, createCodeceptJSTools } from '../../../src/ai/tools.ts'; import { getAliveEndpoint, launchServer, listInstances, stopServer } from '../../../src/browser-server.ts'; +import { listSites } from '../../../src/global-config.ts'; import { ConfigMissingError, ConfigParser, type ExplorbotConfig, outputPath } from '../../../src/config.ts'; import { ExplorBot } from '../../../src/explorbot.ts'; +import { Reporter } from '../../../src/reporter.ts'; +import { Stats } from '../../../src/stats.ts'; import type { WebPageState } from '../../../src/state-manager.ts'; -import { Task } from '../../../src/test-plan.ts'; +import { Task, Test, TestResult } from '../../../src/test-plan.ts'; +import { getPreviousResearch } from '../../../src/ai/researcher/cache.ts'; import { compactAriaSnapshot } from '../../../src/utils/aria.ts'; +import { mdq } from '../../../src/utils/markdown-query.ts'; import { browserErrorMessage } from '../../../src/utils/browser-errors.ts'; import { pluralize } from '../../../src/utils/logger.ts'; -import { type EnvelopeData, type HealAttempt, type InstanceInfo, writeArtifacts } from './envelope.ts'; -import { isFunctionExpression, toCodeceptWrapper } from './pw-parser.ts'; +import { safeFilename } from '../../../src/utils/strings.ts'; +import { type EnvelopeData, type InstanceInfo, writeArtifacts } from './envelope.ts'; +import { isFunctionExpression, takePwValue, toCodeceptWrapper } from './pw-parser.ts'; +import { type SessionRun, latestSessionFile, readSession, recordCommand, sessionFile, sessionsDir } from './session-log.ts'; import { type PwServerDescriptor, readDescriptors, selectDescriptor } from './pw-registry.ts'; -const MAX_INSTRUCTION_ITERATIONS = 6; +const TESTER_ONLY_TOOLS = ['learnExperience', 'askUser']; +const ITERATIONS_PER_INSTRUCTION = 2; +const MAX_INSTRUCTION_ITERATIONS = 24; +const DEFAULT_RESEARCH_AFTER_VISITS = 3; +const CONTEXT_HTML_CAP = 6000; const MAX_TOOL_ROUNDTRIPS = 5; const AI_AGENT_NAME = 'prima'; const CONNECT_TIMEOUT = 3000; const requireLib = createRequire(import.meta.url); +const VOLATILE_COLUMNS = ['CSS', 'XPath', 'Coordinates', 'eidx']; +const UNACCOUNTED: Record<string, string> = { open: 'never reported — the run ended with this instruction still open' }; + +function dropVolatileColumns(markdown: string): string { + return mdq(markdown) + .query('table') + .replaceEach((table) => { + const rows = table.toJson(); + if (!rows.length) return table.text(); + + const columns = Object.keys(rows[0]).filter((name) => !VOLATILE_COLUMNS.includes(name)); + if (!columns.length) return table.text(); + + const header = `| ${columns.join(' | ')} |`; + const divider = `|${columns.map(() => '------').join('|')}|`; + const body = rows.map((row) => `| ${columns.map((name) => row[name] || '-').join(' | ')} |`); + return [header, divider, ...body, ''].join('\n'); + }); +} + +function cap(text: string, max: number): string { + if (text.length <= max) return text; + return `${text.slice(0, max)}\n[...truncated; ${text.length - max} chars omitted...]`; +} + export class Prima { private options: PrimaOptions; private bot: ExplorBot; private artifactsDir?: string; + private hash?: string; + private sessionUrl?: string; private server: { close: () => Promise<void> } | null = null; private attached: string | null = null; + private session: SessionRun | null = null; constructor(options: PrimaOptions = {}) { this.options = options; @@ -38,17 +80,23 @@ export class Prima { config: options.config, path: options.path, baseUrl: this.configBaseUrl(), - verbose: options.verbose, session: options.session, instance: options.instance, headless: true, optionalAi: true, + reporter: { enabled: false }, }); } async start(): Promise<void> { + let discovery: Discovery | undefined; + if (!this.options.endpoint) { + discovery = await this.discover(); + this.adoptSessionUrl(discovery); + } + const config = await this.loadConfig(); - await this.resolveBrowser(config); + await this.resolveBrowser(config, discovery); await this.bot.start(); if (!this.options.url) return; @@ -65,46 +113,65 @@ export class Prima { const validation = isFunctionExpression(expression); if (!validation.valid) return this.toolFailureEnvelope(command, validation.error!); - const previousState = this.bot.stateManager().getCurrentState(); + const previousState = await this.baselineState(); let result: ActionResult | null = null; + let returnedValue: unknown; let executionError: unknown = null; try { const executed = await this.bot.getExplorer().action().execute(toCodeceptWrapper(expression), { verbatim: true }); result = executed.actionResult; + returnedValue = executed.lastValue; } catch (error) { executionError = error; } - if (executionError) return this.heal(command, expression, executionError, previousState); + if (executionError) return this.failureEnvelope(command, executionError, previousState); result ||= await this.capturedResult(previousState); - return this.successEnvelope(command, [expression], result, previousState); + const envelope = await this.successEnvelope(command, [expression], result, previousState); + envelope.value = takePwValue(returnedValue); + return envelope; } - async do(instructions: string[]): Promise<EnvelopeData> { - const command = `do ${instructions.map((instruction) => `"${instruction}"`).join(' ')}`; + async do(instructions: string[], label?: string): Promise<EnvelopeData> { + const command = label || `do ${instructions.map((instruction) => `"${instruction}"`).join(' ')}`; const guard = await this.aiGuard(command); if (guard) return guard; const provider = this.bot.getProvider(); - const previousState = this.bot.stateManager().getCurrentState(); + const previousState = await this.baselineState(); const conversation = provider.startConversation(this.instructionSystemPrompt(), AI_AGENT_NAME); const task = new Task(instructions.join('; '), previousState?.url || ''); - const tools = createCodeceptJSTools({ explorer: this.bot.getExplorer(), stateManager: this.bot.stateManager(), ai: provider }, task); - conversation.addUserText(this.instructionPrompt(instructions, await this.capturedResult(previousState))); + const deps = { explorer: this.bot.getExplorer(), stateManager: this.bot.stateManager(), ai: provider }; + const ledger: LedgerEntry[] = instructions.map((text) => ({ text, status: 'open', proof: '' })); + const descent = { markup: false }; + const tools = { ...createCodeceptJSTools(deps, task), ...this.testerTools(deps), context: this.contextTool(descent), completed: this.completedTool(), blocked: this.blockedTool() }; + conversation.addUserText(await this.instructionPrompt(instructions, await this.capturedResult(previousState))); const used: string[] = []; + const trace: Array<{ label: string; ok: boolean; proof: string }> = []; let failure: { code: string; message: string } | null = null; let aiError: unknown = null; let narration = ''; + let nudged = false; let contextHash = this.bot.stateManager().getCurrentState()?.hash; - for (let iteration = 1; iteration <= Math.min(instructions.length + 2, MAX_INSTRUCTION_ITERATIONS); iteration++) { + for (let iteration = 1; iteration <= Math.min(instructions.length * ITERATIONS_PER_INSTRUCTION + 2, MAX_INSTRUCTION_ITERATIONS); iteration++) { const state = this.bot.stateManager().getCurrentState(); if (iteration > 1 && state && state.hash !== contextHash) { contextHash = state.hash; - conversation.addUserText(this.pageContext(ActionResult.fromState(state))); + conversation.addUserText(await this.pageContext(ActionResult.fromState(state))); + } + + if (iteration > 1) { + conversation.addUserText(dedent` + <progress> + ${this.ledgerProgress(ledger)} + </progress> + + Call completed() now for every open instruction the page already shows is satisfied, before you act again. + `); } const invoked = await provider.invokeConversation(conversation, tools, { maxToolRoundtrips: MAX_TOOL_ROUNDTRIPS, agentName: AI_AGENT_NAME }).catch((error: unknown) => { @@ -116,42 +183,176 @@ export class Prima { const executions = invoked.toolExecutions || []; if (!executions.length) { narration = invoked.response?.text?.trim() || ''; - break; + const unreported = this.openInstructions(ledger); + if (!unreported || nudged) break; + nudged = true; + conversation.addUserText(dedent` + These instructions are still unreported: + ${unreported} + + Report each one with completed() or blocked(). Do not act again on anything you have already carried out. + `); + continue; } for (const execution of executions) { + const output = execution.output || {}; + + if (this.applyLedgerReport(execution, ledger, trace)) continue; + + if (output.action === 'verify' && !output.inexpressible) { + const claim = execution.input?.assertion || 'verification'; + let passed = execution.wasSuccessful; + if (output.alreadyVerified) passed = output.verifications?.[claim] === true; + trace.push({ label: `verify: ${claim}`, ok: passed, proof: output.code || '' }); + continue; + } + if (!execution.wasSuccessful) { - failure = { code: execution.output?.code || '', message: execution.output?.message || 'action failed' }; + failure = { code: output.code || '', message: output.message || 'action failed' }; + trace.push({ label: output.code || execution.toolName || 'action', ok: false, proof: output.message || '' }); + await this.writeStepFiles(trace.length, output.code || execution.toolName || 'action', ''); continue; } - used.push(...this.executedCodes(execution.output?.code)); + + const codes = this.executedCodes(output.code); + used.push(...codes); + trace.push({ label: codes.join('; ') || execution.toolName || 'action', ok: true, proof: '' }); + await this.writeStepFiles(trace.length, codes.join(' ') || execution.toolName || 'action', output.pageDiff?.ariaChanges || ''); failure = null; } + + if (ledger.every((entry) => entry.status !== 'open')) break; } if (aiError) return this.failureEnvelope(command, aiError, previousState); - if (failure) { - const envelope = await this.heal(command, failure.code || instructions.join('; '), failure.message, previousState); - envelope.used = [...used, ...(envelope.used || [])]; + if (used.length && ledger.some((entry) => entry.status === 'open')) { + await this.settleLedger(conversation, provider, ledger, trace); + } + + const unfinished = ledger.filter((entry) => entry.status !== 'done'); + const steps = [...trace, ...ledger.filter((entry) => entry.status === 'open').map((entry) => ({ label: `unreported: ${entry.text}`, ok: false, proof: UNACCOUNTED.open }))]; + + if (failure && unfinished.length) { + const envelope = await this.failureEnvelope(command, failure.message, previousState); + envelope.steps = steps; + envelope.stepFiles = this.statusDir(); return envelope; } - if (!used.length) { + if (!used.length && unfinished.length === ledger.length) { const reason = ['No action was performed for these instructions on the current page.', narration].filter(Boolean).join(' '); return this.failureEnvelope(command, reason, previousState); } const result = await this.capturedResult(this.bot.stateManager().getCurrentState()); - return this.successEnvelope(command, used, result, previousState); + const envelope = await this.successEnvelope(command, used, result, previousState); + envelope.steps = steps; + envelope.stepFiles = this.statusDir(); + // the step log already reports every action and what it changed + envelope.used = undefined; + envelope.changes = undefined; + + const unmet = unfinished.map((entry) => `${entry.status}: ${entry.text}${entry.proof ? ` — ${entry.proof}` : ''}`); + if (unmet.length) { + envelope.ok = false; + envelope.failure = { error: unmet.join('\n') }; + } + return envelope; } - async click(target: string): Promise<EnvelopeData> { - return this.do([`click ${target}`]); + private openInstructions(ledger: LedgerEntry[]): string { + return ledger + .map((entry, index) => ({ entry, number: index + 1 })) + .filter(({ entry }) => entry.status === 'open') + .map(({ entry, number }) => `${number}. ${entry.text}`) + .join('\n'); } - async fill(field: string, value: string): Promise<EnvelopeData> { - return this.do([`fill ${field} with value: ${value}`]); + private applyLedgerReport(execution: any, ledger: LedgerEntry[], trace: Array<{ label: string; ok: boolean; proof: string }>): boolean { + const action = execution.output?.action; + + if (action === 'completed') { + const closed: string[] = []; + for (const number of execution.input?.numbers || []) { + const entry = ledger[number - 1]; + if (entry?.status !== 'open') continue; + entry.status = 'done'; + entry.proof = execution.input?.proof || ''; + closed.push(entry.text); + } + // one report carries one proof, however many instructions it closed + if (closed.length) trace.push({ label: `done: ${closed.join('; ')}`, ok: true, proof: execution.input?.proof || '' }); + return true; + } + + if (action !== 'blocked') return false; + + const entry = ledger[(execution.input?.instruction || 0) - 1]; + if (entry?.status === 'open') { + entry.status = 'blocked'; + entry.proof = execution.input?.reason || ''; + trace.push({ label: `blocked: ${entry.text}`, ok: false, proof: entry.proof }); + } + return true; + } + + private async settleLedger(conversation: any, provider: any, ledger: LedgerEntry[], trace: Array<{ label: string; ok: boolean; proof: string }>): Promise<void> { + conversation.addUserText(dedent` + The run is over and these instructions were never reported: + + ${this.openInstructions(ledger)} + + Judge each one against what you saw at the time it was due, not against the page as it stands now — later + instructions have moved it on, and something you confirmed earlier stays confirmed even if it is gone. + completed() for those, blocked() for the ones the page could not do. Report every one — nothing else runs after this. + `); + + const invoked = await provider.invokeConversation(conversation, { completed: this.completedTool(), blocked: this.blockedTool() }, { maxToolRoundtrips: 2, toolChoice: 'required', agentName: AI_AGENT_NAME }).catch(() => null); + + for (const execution of invoked?.toolExecutions || []) { + this.applyLedgerReport(execution, ledger, trace); + } + } + + private ledgerProgress(ledger: LedgerEntry[]): string { + return ledger + .map((entry, index) => { + const head = `${index + 1}. ${entry.status} — ${entry.text}`; + if (entry.status === 'open') return head; + return `${head} (${entry.proof})`; + }) + .join('\n'); + } + + async check(scenario: string, expected: string[] = []): Promise<EnvelopeData> { + const command = `check ${scenario}`; + const guard = await this.aiGuard(command); + if (guard) return guard; + + const previousState = await this.baselineState(); + const outcomes = expected.length ? expected : [scenario]; + const test = new Test(scenario, 'normal', outcomes, previousState?.url || this.options.url || ''); + const tester = this.bot.agentTester(); + + const outcome = await tester.test(test); + + const notes = Object.values(test.notes || {}) as Array<{ message: string; status?: string; log?: string; observation?: boolean }>; + const result = await this.capturedResult(this.bot.stateManager().getCurrentState()); + const envelope = await this.reportEnvelope(command, result, previousState, { ok: outcome.success }); + const recorded = notes.filter((note) => !note.observation && !outcomes.includes(note.message)); + const failed = recorded.filter((note) => note.status === TestResult.FAILED); + envelope.steps = failed.map((note) => ({ label: note.message, ok: false, proof: note.log || '' })); + + const routine = recorded.length - failed.length; + if (routine) envelope.steps.push({ label: `${routine} further ${pluralize(routine, 'step')} ran without failing — prima status ${envelope.status} for the full log`, ok: true, proof: '' }); + + envelope.expectations = await this.bot.agentPilot().settleExpectations(test); + + const observations = notes.filter((note) => note.observation).map((note) => note.message); + if (observations.length) envelope.answer = ['Page problems noticed while running, not step failures:', ...observations.map((line) => `- ${line}`)].join('\n'); + return envelope; } async ask(question: string): Promise<EnvelopeData> { @@ -172,9 +373,7 @@ export class Prima { const previousState = this.bot.stateManager().getCurrentState(); const result = await this.capturedResult(previousState); const verification = await this.bot.agentNavigator().verifyState(assertion, result); - const codes = verification.successfulCodes || []; - const verdict = { passed: verification.verified, evidence: this.verdictEvidence(verification.verified, codes), code: codes.join('\n') }; - return this.reportEnvelope(command, result, previousState, { ok: verification.verified, verdict }); + return this.reportEnvelope(command, result, previousState, { assertions: verification.results || [] }); } async research(opts: { data?: boolean; deep?: boolean; fresh?: boolean } = {}): Promise<EnvelopeData> { @@ -186,7 +385,7 @@ export class Prima { const previousState = this.bot.stateManager().getCurrentState(); const result = await this.capturedResult(previousState); const uiMap = await this.bot.agentResearcher().research(result, { screenshot: true, data: opts.data, deep: opts.deep, force: opts.fresh }); - return this.reportEnvelope(command, result, previousState, { research: uiMap }); + return this.reportEnvelope(command, result, previousState, { research: dropVolatileColumns(uiMap) }); } async go(target: string): Promise<EnvelopeData> { @@ -208,7 +407,7 @@ export class Prima { navigationError = error; } - if (navigationError) return this.heal(command, code, navigationError, previousState); + if (navigationError) return this.failureEnvelope(command, navigationError, previousState); const used: string[] = []; if (isUrl) used.push(code); @@ -237,6 +436,68 @@ export class Prima { return stopped; } + async config(): Promise<string> { + const [site] = listSites(); + if (site && !this.configBaseUrl()) this.sessionUrl = site.url; + const config = await this.loadConfig(); + + const named = (model: unknown): string => { + if (typeof model === 'string') return model; + return (model as any)?.modelId || (model as any)?.model || 'unknown'; + }; + + const ai = config.ai || ({} as any); + const roles: Array<[string, unknown]> = [ + ['model', ai.model], + ['agenticModel', ai.agenticModel], + ['visionModel', ai.visionModel], + ]; + + const lines = roles.filter(([, model]) => model).map(([role, model]) => `${role.padEnd(14)} ${named(model)}`); + lines.push(`config ${ConfigParser.getInstance().getConfigPath() || 'built-in defaults'}`); + if (ai.langfuse?.enabled) lines.push('telemetry langfuse'); + return lines.join('\n'); + } + + record(envelope: EnvelopeData, durationMs: number): void { + if (!this.session) return; + recordCommand(sessionFile(this.session.key), this.session, envelope, durationMs); + } + + async report(): Promise<string> { + const [site] = listSites(); + if (site && !this.configBaseUrl()) this.sessionUrl = site.url; + await this.loadConfig(); + + let file = latestSessionFile(); + if (this.options.pwSession) file = sessionFile(this.options.pwSession); + if (!file || !existsSync(file)) return `No prima session was recorded under ${sessionsDir()}. Commands are recorded as they run.`; + + const session = readSession(file); + if (!session.tests.length) return `No commands are recorded in ${file}`; + + Stats.sessionName = path.basename(file, '.jsonl'); + process.env.TESTOMATIO_TITLE = session.title; + const reporter = new Reporter({ html: true, markdown: true }); + + // the report pipes narrate themselves on console.log; prima prints the paths itself + const speak = console.log; + console.log = () => {}; + try { + for (const test of session.tests) await reporter.reportTestData(test.status, test); + await reporter.finishRun(); + } finally { + console.log = speak; + } + + return [ + `${session.tests.length} ${pluralize(session.tests.length, 'command')} from ${file}`, + `html: ${outputPath('reports', `${Stats.sessionLabel()}.html`)}`, + `markdown: ${outputPath('reports', `${Stats.sessionLabel()}-tests.md`)}`, + `upload: TESTOMATIO=<apiKey> npx @testomatio/reporter replay ${file}`, + ].join('\n'); + } + async browserStatus(): Promise<string> { await this.loadConfig(); const info = await this.instanceInfo(); @@ -282,7 +543,7 @@ export class Prima { async toolFailureEnvelope(command: string, error: unknown): Promise<EnvelopeData> { const state = this.bot.getCurrentState(); const instance = await this.instanceInfo().catch(() => ({ name: this.instanceName(), tabs: 0, others: [] })); - const failure: EnvelopeData['failure'] = { error: `tool: ${browserErrorMessage(error)}`, attempts: [] }; + const failure: EnvelopeData['failure'] = { error: `tool: ${browserErrorMessage(error)}` }; if (error instanceof ConfigMissingError) failure.error = browserErrorMessage(error); if (state?.ariaSnapshot) failure.compactAria = compactAriaSnapshot(state.ariaSnapshot, true); @@ -300,31 +561,41 @@ export class Prima { } private configBaseUrl(): string | undefined { - const url = this.options.baseUrl || this.options.url; + const url = this.options.baseUrl || this.options.url || this.sessionUrl; if (!url) return undefined; if (!URL.canParse(url)) return undefined; return url; } - private async resolveBrowser(config: ExplorbotConfig): Promise<void> { + private adoptSessionUrl(discovery: Discovery): void { + if (this.options.baseUrl || this.options.url) return; + + const url = discovery.browser?.contexts()[0]?.pages()[0]?.url(); + if (!url?.startsWith('http')) return; + this.sessionUrl = new URL(url).origin; + this.bot.getOptions().baseUrl = this.sessionUrl; + } + + private async resolveBrowser(config: ExplorbotConfig, discovered?: Discovery): Promise<void> { if (this.options.endpoint) { const endpoint = this.options.endpoint; const browserName = config.playwright.browser || 'chromium'; - if (await this.attachToEndpoint({ file: '', title: '', endpoint, workspaceDir: '', browserName, playwrightLib: '' })) return; + const known = readDescriptors().find((descriptor) => descriptor.endpoint === endpoint); + if (await this.attachToEndpoint({ file: '', title: '', endpoint, workspaceDir: '', browserName, playwrightLib: known?.playwrightLib || '' })) return; throw new Error(dedent` No browser answered at ${endpoint}. - Check the endpoint of the running session, or drop --endpoint to attach to the - playwright-cli browser of this workspace. + Check the endpoint of the running session, or drop --endpoint to let prima pick + the playwright-cli session itself. `); } - const { match, candidates, browser } = await this.discover(); + const { match, candidates, browser } = discovered || (await this.discover()); if (match && (await this.attachToEndpoint(match, browser))) return; if (!match && candidates.length) { const titles = candidates.map((candidate) => candidate.title).join(', '); throw new Error(dedent` - Several playwright-cli sessions are open for this workspace: ${titles} + Several playwright-cli sessions are open: ${titles} Pick one with --pw-session <title>. `); } @@ -342,7 +613,7 @@ export class Prima { private async discover(descriptors = readDescriptors()): Promise<Discovery> { const title = this.options.pwSession ?? process.env.PLAYWRIGHT_CLI_SESSION; - const opts = { workspaceDir: this.workspaceDir(), title }; + const opts = { title }; const alive = new Map<PwServerDescriptor, Browser>(); for (const candidate of selectDescriptor(descriptors, opts).candidates) { @@ -368,13 +639,14 @@ export class Prima { this.bot.attachBrowser(browser); this.attached = this.attachmentLabel(descriptor); + this.session = { key: descriptor.title || this.instanceName(), endpoint: descriptor.endpoint, title: `prima session "${descriptor.title || this.instanceName()}"` }; return true; } private async connectDescriptor(descriptor: PwServerDescriptor): Promise<Browser | null> { - const connected = await this.connectWith(playwright, descriptor); + const connected = await this.connectWith(this.descriptorLib(descriptor), descriptor); if (connected) return connected; - return this.connectWith(this.descriptorLib(descriptor), descriptor); + return this.connectWith(playwright, descriptor); } private async connectWith(lib: any, descriptor: PwServerDescriptor): Promise<Browser | null> { @@ -394,15 +666,15 @@ export class Prima { private attachmentLabel(descriptor: PwServerDescriptor): string { if (!descriptor.title) return `endpoint ${descriptor.endpoint}`; + if (!descriptor.workspaceDir) return `playwright-cli session "${descriptor.title}"`; return `playwright-cli session "${descriptor.title}", workspace ${descriptor.workspaceDir}`; } - private workspaceDir(): string { - return path.resolve(this.options.path || process.cwd()); - } - private async connectOwnInstance(): Promise<boolean> { - return !!(await getAliveEndpoint(this.instanceName())); + const endpoint = await getAliveEndpoint(this.instanceName()); + if (!endpoint) return false; + this.session = { key: this.instanceName(), endpoint, title: `prima instance "${this.instanceName()}"` }; + return true; } private async launchOwnServer(opts: { browser?: string; show?: boolean }, instance: string): Promise<{ close: () => Promise<void> }> { @@ -425,49 +697,6 @@ export class Prima { return URL.canParse(value); } - private async heal(command: string, expression: string, error: unknown, previousState: WebPageState | null): Promise<EnvelopeData> { - if (this.options.heal === false) return this.failureEnvelope(command, error, previousState); - - const navigator = this.healNavigator(); - if (!navigator) { - const envelope = await this.failureEnvelope(command, error, previousState); - envelope.healed = false; - envelope.healNote = this.aiUnavailableNote(); - return envelope; - } - - const message = dedent` - I tried to run this command on the page: ${expression} - But it failed with: ${browserErrorMessage(error)} - Reach the same outcome on the current page in a different way. - `; - - const attempts: Array<{ code: string; error?: string }> = []; - const failedResult = await this.capturedResult(previousState); - const resolved = await navigator.resolveState(message, failedResult, { onAttempt: (attempt) => attempts.push(attempt) }).catch(() => false); - - if (!resolved) { - const healAttempts = attempts.map((attempt) => ({ code: attempt.code, outcome: attempt.error || 'ok' })); - return this.failureEnvelope(command, error, previousState, healAttempts); - } - - const used = attempts.filter((attempt) => !attempt.error).map((attempt) => attempt.code); - const result = await this.capturedResult(this.bot.stateManager().getCurrentState()); - const envelope = await this.successEnvelope(command, used, result, previousState); - envelope.healed = true; - envelope.healNote = `recovered after ${attempts.length} ${pluralize(attempts.length, 'attempt')}`; - return envelope; - } - - private healNavigator(): Navigator | null { - if (this.aiUnavailable()) return null; - try { - return this.bot.agentNavigator?.() ?? null; - } catch { - return null; - } - } - private aiUnavailable(): string | null { try { if (this.bot.getProvider?.()) return null; @@ -477,12 +706,6 @@ export class Prima { return this.bot.aiFailureReason?.() || 'no AI model is configured'; } - private aiUnavailableNote(): string { - const reason = this.aiUnavailable(); - if (!reason) return 'ai unavailable'; - return `ai unavailable: ${reason}`; - } - private async aiGuard(command: string): Promise<EnvelopeData | null> { const reason = this.aiUnavailable(); if (!reason) return null; @@ -497,47 +720,191 @@ export class Prima { </role> <approach> - 1. Read the page context and perform the instructions in the order they are listed. + 1. Read the page context and carry out the instructions in the order they are listed. 2. Interact with the page only through the provided tools. 3. Pick the smallest interaction that fulfills an instruction, then move to the next one. 4. After the page changes, work from the updated context you are given, not from the earlier one. - 5. Stop calling tools when every instruction is done, or when an instruction cannot be performed on this page — say what is missing instead. + 5. Account for every instruction: completed() as soon as one is satisfied, blocked() when the page cannot do what it asks. </approach> + <ledger> + Instructions are numbered and those numbers never change. Report by number. + Saying in your reply that something is done does not report it — only completed() does. Nothing you write is read as a report. + Report an instruction the moment the page shows it is satisfied, before moving on. Waiting until later is how work gets repeated. + An instruction you have reported is finished. Never act on it again, and never report it twice. + After each turn you are shown every instruction with its state. Act only on the ones still open — repeating an action that already + landed can undo it, since a control that opened something will close it again. + Reaching for blocked() after a couple of honest attempts costs less than a third attempt that fails the same way. + </ledger> + + <scope> + Do only what the instructions ask. An action that looks helpful but was not asked for is out of scope — report it as something you noticed, never perform it. + Continuing past the last instruction is a failure, even when the next step seems obvious. + An instruction worded as a condition — do X if Y appears — is satisfied the moment you can see Y is absent. Say so and move on. Never search for something the page does not show. + </scope> + + <pace> + Work in as few turns as you can. When the next actions are already determined by what you can see, ask for them together in one turn rather than one at a time — each turn costs a full round trip. + Only stop to look again when what you find changes what you would do next. + A batch may not run past an instruction that inspects the page — settle that one first, because the actions after it destroy the state it would have read. + </pace> + + <proof> + An instruction is done only when a change on the page shows it. After each action read the reported change and decide which part of it proves the instruction. + That part is what completed() takes as its proof. Do not restate the action as if it were the outcome. + An instruction that only inspects the page is satisfied by what you can see, including seeing that something is absent — those need no action at all. + </proof> + + <targets> + The page context lists every element with a ref, like [ref=e14]. To click one, pass that ref to clickRef — a ref names one + exact element, so it cannot match several by mistake and costs nothing to resolve. This is the cheapest way to act. + Use click() with a role and name for anything clickRef cannot take, and narrow with the container it sits in when a name + appears more than once, rather than guessing at an id or a class. + Refs belong to the context you were given. Use the ones in your newest context, never one you invented or remembered from + an older page. When the element an instruction needs is missing from that context, call context() and act on what it returns. + </targets> + ${locatorRule} ${actionRule} + + <targets_first> + Everything above about composing locators applies to click() and the other locator tools. It does not apply when the + element carries a ref: pass that ref to clickRef instead and compose nothing. Reach for a locator only for elements + that have no ref, or when a ref has stopped resolving. + </targets_first> `; } - private instructionPrompt(instructions: string[], result: ActionResult): string { + private async instructionPrompt(instructions: string[], result: ActionResult): Promise<string> { const list = instructions.map((instruction, index) => `${index + 1}. ${instruction}`).join('\n'); return dedent` <instructions> ${list} </instructions> - ${this.pageContext(result)} + ${await this.pageContext(result)} `; } - private pageContext(result: ActionResult): string { + private testerTools(deps: any): any { + const researcher = this.bot.agentResearcher?.(); + const navigator = this.bot.agentNavigator?.(); + if (!researcher || !navigator) return {}; + + const tools = createAgentTools({ ...deps, researcher, navigator, withExperience: false }); + for (const name of TESTER_ONLY_TOOLS) delete tools[name]; + return tools; + } + + private completedTool(): any { + return tool({ + description: dedent` + Report the instructions you have just satisfied, by their number. Report several together when one turn satisfied several. + A reported instruction is finished — you will not be asked for it again and must not act on it again. + `, + inputSchema: z.object({ + numbers: z.array(z.number()).describe('Numbers of the instructions now satisfied, as they are numbered in the instruction list'), + proof: z.string().describe('What on the page shows they are satisfied'), + }), + execute: async () => ({ success: true, action: 'completed' }), + }); + } + + private blockedTool(): any { + return tool({ + description: dedent` + Report one instruction that cannot be carried out on this page, by its number. Reach for this instead of trying the same thing again. + The rest of the sequence continues without it. + `, + inputSchema: z.object({ + instruction: z.number().describe('Number of the instruction that cannot be carried out'), + reason: z.string().describe('What stopped it — what you looked for and what the page showed instead'), + }), + execute: async () => ({ success: true, action: 'blocked' }), + }); + } + + private contextTool(descent: { markup: boolean }): any { + let refreshed = false; + return tool({ + description: dedent` + Look at the page again when the refs you hold no longer resolve, or when the element an instruction needs is not in the context you were given. + The first call returns the page as it is now, with fresh refs that replace every ref you were holding. + A later call on the same page drops to the raw markup, for elements the accessibility tree does not describe. + Do not call it to confirm an action worked — the change is already reported back to you. + `, + inputSchema: z.object({ + reason: z.string().describe('Which element you cannot reach and what you already tried'), + }), + execute: async () => { + const result = await this.capturedResult(this.bot.stateManager().getCurrentState()); + if (!refreshed) { + refreshed = true; + return { success: true, context: await this.pageContext(result) }; + } + descent.markup = true; + return { success: true, context: cap(await result.simplifiedHtml(), CONTEXT_HTML_CAP) }; + }, + }); + } + + private async pageContext(result: ActionResult): Promise<string> { const experience = this.bot.experienceTracker?.()?.renderExperienceTocFor?.(result) || ''; + const map = this.researchMap(result); + if (map) { + return dedent` + <page_ui_map url="${result.url}" title="${result.title}"> + ${map} + </page_ui_map> + + ${experience} + `; + } + return dedent` <page url="${result.url}" title="${result.title}"> - ${compactAriaSnapshot(result.ariaSnapshot, true)} + ${compactAriaSnapshot(await this.refAriaSnapshot(result), true, (value) => this.offloadValue(value))} </page> ${experience} `; } + private researchMap(result: ActionResult): string { + if (this.bot.stateManager().getVisitCount(result.url) < this.researchAfterVisits()) return ''; + return getPreviousResearch(result.getStateHash()); + } + + private researchAfterVisits(): number { + const configured = this.bot.getConfig?.()?.ai?.agents?.prima?.researchAfterVisits; + if (typeof configured === 'number') return configured; + return DEFAULT_RESEARCH_AFTER_VISITS; + } + + private offloadValue(value: string): string | undefined { + const dir = this.statusDir(); + const name = `value-${createHash('sha1').update(value).digest('hex').slice(0, 8)}.txt`; + try { + mkdirSync(dir, { recursive: true }); + writeFileSync(path.join(dir, name), value, 'utf-8'); + } catch { + return undefined; + } + return path.join(path.basename(dir), name); + } + + private async refAriaSnapshot(result: ActionResult): Promise<string | null> { + const snapshot = await Promise.resolve(this.bot.getExplorer()?.withPage?.((page: any) => page.locator('body').ariaSnapshot({ mode: 'ai' }))).catch(() => null); + return snapshot || result.ariaSnapshot; + } + private executedCodes(code: unknown): string[] { if (typeof code !== 'string') return []; return code .split('\n') .map((line) => line.trim()) - .filter((line) => line); + .filter((line) => line && !line.startsWith('//')); } private visionEnabled(): boolean { @@ -585,12 +952,6 @@ export class Prima { return response?.text || ''; } - private verdictEvidence(verified: boolean, codes: string[]): string { - if (!verified) return 'no assertion held on the current page'; - if (!codes.length) return 'already verified on this page'; - return `${codes[0]} passed`; - } - private async successEnvelope(command: string, used: string[], result: ActionResult, previousState: WebPageState | null): Promise<EnvelopeData> { return { ok: true, @@ -599,15 +960,14 @@ export class Prima { page: this.pageBlock(result, previousState), changes: await this.pageChanges(result, previousState, used[0]), instance: await this.instanceInfo(), - artifacts: await this.writeSnapshot(result), + status: await this.saveStatus(result), }; } - private async failureEnvelope(command: string, error: unknown, previousState: WebPageState | null, attempts: HealAttempt[] = []): Promise<EnvelopeData> { + private async failureEnvelope(command: string, error: unknown, previousState: WebPageState | null): Promise<EnvelopeData> { const result = await this.capturedResult(previousState); - const failure: EnvelopeData['failure'] = { error: browserErrorMessage(error), attempts }; + const failure: EnvelopeData['failure'] = { error: browserErrorMessage(error) }; if (result.ariaSnapshot) failure.compactAria = compactAriaSnapshot(result.ariaSnapshot, true); - if (attempts.length) failure.reasoning = [...new Set(attempts.map((attempt) => attempt.outcome))].join('; '); return { ok: false, @@ -615,7 +975,7 @@ export class Prima { page: this.pageBlock(result, previousState), failure, instance: await this.instanceInfo(), - artifacts: await this.writeSnapshot(result), + status: await this.saveStatus(result), }; } @@ -626,7 +986,7 @@ export class Prima { page: this.pageBlock(result, previousState), ...outcome, instance: await this.instanceInfo(), - artifacts: await this.writeSnapshot(result), + status: await this.saveStatus(result), }; } @@ -650,23 +1010,78 @@ export class Prima { }; } - private async pageChanges(result: ActionResult, previousState: WebPageState | null, code: string): Promise<string | null> { - if (!previousState) return null; + private async baselineState(): Promise<WebPageState | null> { + const existing = this.bot.stateManager?.()?.getCurrentState(); + if (existing) return existing; + + const result = await Promise.resolve(this.bot.getExplorer?.()?.capture?.()).catch(() => null); + if (!result) return null; + return this.bot.stateManager?.()?.updateState(result) ?? null; + } + + private async pageChanges(result: ActionResult, previousState: WebPageState | null, code: string): Promise<string> { + if (!previousState) return 'no snapshot was captured before this command, so nothing could be compared'; const toolResult = await result.toToolResult(ActionResult.fromState(previousState), code); - return toolResult.pageDiff?.ariaChanges ?? null; + return toolResult.pageDiff?.ariaChanges || 'no change'; + } + + async status(hash: string): Promise<EnvelopeData> { + const dir = this.statusDir(hash); + const statusFile = path.join(dir, 'status.json'); + if (!existsSync(statusFile)) return this.toolFailureEnvelope(`status ${hash}`, `No command was recorded under ${hash}. Every envelope prints its own hash on the Instance line.`); + + const saved = JSON.parse(readFileSync(statusFile, 'utf-8')); + return { + ok: true, + command: `status ${hash}`, + page: saved.page, + changes: saved.changes, + instance: await this.instanceInfo(), + artifacts: { aria: path.join(dir, 'aria.yml'), html: path.join(dir, 'page.html') }, + }; } - private async writeSnapshot(result: ActionResult): Promise<EnvelopeData['artifacts']> { - return writeArtifacts(this.nextArtifactDir(), { + private async saveStatus(result: ActionResult): Promise<string> { + const hash = this.statusHash(); + await this.writeSnapshot(result); + writeFileSync(path.join(this.statusDir(hash), 'status.json'), JSON.stringify({ page: this.pageBlock(result, null), changes: compactAriaSnapshot(result.ariaSnapshot, true) }), 'utf-8'); + return hash; + } + + private async writeStepFiles(index: number, label: string, diff: string): Promise<void> { + const state = this.bot.stateManager().getCurrentState(); + if (!state) return; + + const dir = this.statusDir(); + mkdirSync(dir, { recursive: true }); + const stem = path.join(dir, `${index}-${safeFilename(label.slice(0, 60))}`); + const result = ActionResult.fromState(state); + + writeFileSync(`${stem}.aria.yaml`, result.ariaSnapshot ?? '', 'utf-8'); + writeFileSync(`${stem}.html`, await result.combinedHtml(), 'utf-8'); + if (diff) writeFileSync(`${stem}.diff.yaml`, diff, 'utf-8'); + } + + private async writeSnapshot(result: ActionResult): Promise<undefined> { + writeArtifacts(this.statusDir(), { aria: result.ariaSnapshot, html: await result.combinedHtml(), requests: this.bot.requestStore().getRequests(), }); + return undefined; } - private nextArtifactDir(): string { + private statusHash(): string { + this.hash ||= createHash('sha1') + .update(`${this.options.path || process.cwd()}-${Date.now()}`) + .digest('hex') + .slice(0, 15); + return this.hash; + } + + private statusDir(hash = this.statusHash()): string { this.artifactsDir ||= outputPath('prima'); - return path.join(this.artifactsDir, new Date().toISOString().replace(/[:.]/g, '-')); + return path.join(this.artifactsDir, hash); } private tabCount(): number { @@ -686,13 +1101,17 @@ interface Discovery { browser?: Browser; } +interface LedgerEntry { + text: string; + status: 'open' | 'done' | 'blocked'; + proof: string; +} + export interface PrimaOptions { - verbose?: boolean; config?: string; path?: string; instance?: string; session?: string | boolean; - heal?: boolean; ephemeral?: boolean; framework?: 'codeceptjs' | 'playwright'; noVision?: boolean; diff --git a/boat/prima/src/pw-parser.ts b/boat/prima/src/pw-parser.ts index 50f942ed..7fa9ccf3 100644 --- a/boat/prima/src/pw-parser.ts +++ b/boat/prima/src/pw-parser.ts @@ -13,5 +13,15 @@ export function isFunctionExpression(expr: string): { valid: boolean; error?: st } export function toCodeceptWrapper(expr: string): string { - return `I.usePlaywrightTo('pw', async (playwright) => (${expr.trim()})(playwright))`; + return `return I.usePlaywrightTo('pw', async (playwright) => (${expr.trim()})(playwright))`; +} + +export function takePwValue(held: unknown): string | undefined { + if (held === undefined || held === null) return undefined; + if (typeof held === 'string') return held; + try { + return JSON.stringify(held, null, 2); + } catch { + return String(held); + } } diff --git a/boat/prima/src/pw-registry.ts b/boat/prima/src/pw-registry.ts index 43168717..b689d796 100644 --- a/boat/prima/src/pw-registry.ts +++ b/boat/prima/src/pw-registry.ts @@ -20,9 +20,8 @@ export function readDescriptors(dir = registryDir()): PwServerDescriptor[] { return descriptors; } -export function selectDescriptor(descriptors: PwServerDescriptor[], opts: { workspaceDir: string; title?: string }): { match?: PwServerDescriptor; candidates: PwServerDescriptor[] } { - const workspaceDir = path.resolve(opts.workspaceDir); - const candidates = descriptors.filter((descriptor) => path.resolve(descriptor.workspaceDir) === workspaceDir); +export function selectDescriptor(descriptors: PwServerDescriptor[], opts: { title?: string } = {}): { match?: PwServerDescriptor; candidates: PwServerDescriptor[] } { + const candidates = descriptors; if (!candidates.length) return { candidates }; if (opts.title) { @@ -53,13 +52,13 @@ function parseDescriptor(file: string): PwServerDescriptor | null { return null; } - if (!data?.endpoint || !data?.title || !data?.workspaceDir) return null; + if (!data?.endpoint || !data?.title) return null; return { file, title: data.title, endpoint: data.endpoint, - workspaceDir: data.workspaceDir, + workspaceDir: data.workspaceDir || '', browserName: data.browser?.browserName || DEFAULT_BROWSER, playwrightLib: data.playwrightLib || '', }; diff --git a/boat/prima/src/session-log.ts b/boat/prima/src/session-log.ts new file mode 100644 index 00000000..cd234081 --- /dev/null +++ b/boat/prima/src/session-log.ts @@ -0,0 +1,126 @@ +import { appendFileSync, existsSync, mkdirSync, readFileSync, readdirSync, statSync, writeFileSync } from 'node:fs'; +import path from 'node:path'; +import { outputPath } from '../../../src/config.ts'; +import { safeFilename } from '../../../src/utils/strings.ts'; +import { type EnvelopeData, renderEnvelope } from './envelope.ts'; + +const EXPECTATION_STATUS = { passed: 'passed', failed: 'failed', unverified: 'skipped' }; + +export function sessionsDir(): string { + return outputPath('prima', 'sessions'); +} + +export function sessionFile(key: string): string { + return path.join(sessionsDir(), `${safeFilename(key)}.jsonl`); +} + +export function latestSessionFile(): string | null { + const dir = sessionsDir(); + if (!existsSync(dir)) return null; + + const files = readdirSync(dir) + .filter((name) => name.endsWith('.jsonl')) + .map((name) => path.join(dir, name)) + .sort((a, b) => statSync(b).mtimeMs - statSync(a).mtimeMs); + + return files[0] || null; +} + +export function recordCommand(file: string, session: SessionRun, envelope: EnvelopeData, durationMs: number): void { + mkdirSync(path.dirname(file), { recursive: true }); + if (recordedEndpoint(file) !== session.endpoint) writeFileSync(file, jsonLine({ action: 'createRun', endpoint: session.endpoint, params: { title: session.title } }), 'utf-8'); + appendFileSync(file, jsonLine({ action: 'addTest', testId: commandTest(envelope, durationMs) }), 'utf-8'); +} + +export function readSession(file: string): { title: string; tests: any[] } { + const entries = readEntries(file); + const run = entries.find((entry) => entry.action === 'createRun'); + const tests = entries.filter((entry) => entry.action === 'addTest').map((entry) => entry.testId); + return { title: run?.params?.title || 'prima session', tests }; +} + +function commandTest(envelope: EnvelopeData, durationMs: number): any { + let status = 'failed'; + if (envelope.ok) status = 'passed'; + + const test: any = { + rid: envelope.status, + title: envelope.command, + suite_title: 'Prima session', + status, + file: '<prima>', + description: [`page: ${envelope.page.url}`, envelope.stepFiles && `artifacts: ${envelope.stepFiles}`].filter(Boolean).join('\n'), + code: (envelope.used || []).join('\n'), + steps: commandSteps(envelope), + logs: renderEnvelope(envelope), + time: Math.round(durationMs), + }; + + if (envelope.failure) test.error = { message: envelope.failure.error }; + return test; +} + +function commandSteps(envelope: EnvelopeData): any[] { + const steps: any[] = []; + + for (const step of envelope.steps || []) { + const entry: any = { category: 'user', title: step.label, status: 'passed', duration: 0, log: step.proof }; + if (!step.ok) { + entry.status = 'failed'; + entry.error = step.proof; + } + steps.push(entry); + } + + for (const expectation of envelope.expectations || []) { + const entry: any = { category: 'user', title: `expected: ${expectation.text}`, status: EXPECTATION_STATUS[expectation.status], duration: 0 }; + if (expectation.status === 'failed') entry.error = 'the run did not reach this outcome'; + steps.push(entry); + } + + for (const assertion of envelope.assertions || []) { + const entry: any = { category: 'user', title: assertion.code, status: 'passed', duration: 0, log: assertion.proof.join('\n') }; + if (!assertion.passed) { + entry.status = 'failed'; + entry.error = 'the assertion did not hold on the page'; + } + steps.push(entry); + } + + return steps; +} + +function recordedEndpoint(file: string): string | null { + const [first] = readEntries(file); + return first?.endpoint || null; +} + +function readEntries(file: string): any[] { + if (!existsSync(file)) return []; + + const entries: any[] = []; + for (const line of readFileSync(file, 'utf-8').split('\n')) { + if (!line.trim()) continue; + const entry = parseLine(line); + if (entry) entries.push(entry); + } + return entries; +} + +function parseLine(line: string): any { + try { + return JSON.parse(line); + } catch { + return null; + } +} + +function jsonLine(entry: unknown): string { + return `${JSON.stringify(entry)}\n`; +} + +export interface SessionRun { + key: string; + endpoint: string; + title: string; +} diff --git a/boat/prima/tests/envelope.test.ts b/boat/prima/tests/envelope.test.ts index 0071cef6..4f475195 100644 --- a/boat/prima/tests/envelope.test.ts +++ b/boat/prima/tests/envelope.test.ts @@ -17,14 +17,15 @@ const base: EnvelopeData = { describe('renderEnvelope', () => { test('success envelope contains all sections in order', () => { const out = renderEnvelope(base); - const sections = ['### Result', '### Page', '### Changes', '### Instance', '### Artifacts']; + const sections = ['### Result', '### Page', '### Changes', '### Instance']; const positions = sections.map((s) => out.indexOf(s)); expect(positions.every((p) => p >= 0)).toBe(true); expect([...positions].sort((a, b) => a - b)).toEqual(positions); expect(out).toContain('ok: true'); expect(out).toContain("used: I.click('Login')"); expect(out).toContain('(changed: https://app.example.com/login → https://app.example.com/dashboard)'); - expect(out).toContain('instance: default (3 tabs) | other instances: auth-test (1 tab)'); + expect(out).toContain('default (3 tabs)'); + expect(out).toContain('other instances: auth-test (1 tab)'); }); test('unchanged url renders without changed marker', () => { @@ -32,26 +33,21 @@ describe('renderEnvelope', () => { expect(out).not.toContain('(changed:'); }); - test('failure envelope renders attempts, reasoning and compact aria', () => { + test('failure envelope renders the error and compact aria, and nothing about retries', () => { const out = renderEnvelope({ ...base, ok: false, failure: { error: "locator 'text=Login' not found", - attempts: [ - { code: "I.click('Login')", outcome: 'not visible' }, - { code: 'scroll + retry', outcome: 'covered by cookie banner' }, - ], - reasoning: 'element hidden behind consent overlay', compactAria: '- button "Accept all"', }, }); expect(out).toContain('### Failure'); - expect(out).toContain('### Healing attempts (2)'); - expect(out).toContain("1. I.click('Login')"); - expect(out).toContain('→ not visible'); + expect(out).toContain("locator 'text=Login' not found"); expect(out).toContain('### Current page (compact ARIA)'); expect(out).toContain('- button "Accept all"'); + expect(out).not.toContain('Healing'); + expect(out).not.toContain('healed'); }); test('answer replaces changes for ask', () => { @@ -68,32 +64,106 @@ describe('renderEnvelope', () => { expect(out).not.toContain('### Answer'); }); - test('verdict replaces changes for verify', () => { - const out = renderEnvelope({ ...base, changes: undefined, verdict: { passed: true, evidence: 'heading "Dashboard" present', code: "I.see('Dashboard')" } }); - expect(out).toContain('### Verdict'); - expect(out).toContain('passed: true'); - expect(out).toContain("I.see('Dashboard')"); + test('assertions render one line per check with its own result, and no overall verdict', () => { + const out = renderEnvelope({ + ...base, + changes: undefined, + assertions: [ + { code: "I.see('Dashboard')", passed: true, proof: ['await expect(page).toContainText("Dashboard");'] }, + { code: "I.seeElement('.chart')", passed: false, proof: [] }, + ], + }); + + expect(out).toContain('### Assertions'); + expect(out).toContain("I.see('Dashboard') => PASSED"); + expect(out).toContain("I.seeElement('.chart') => FAILED"); + expect(out).toContain('playwright:'); + expect(out).toContain('await expect(page).toContainText("Dashboard");'); + expect(out).not.toContain('passed: '); + }); + + test('a multi-line assertion gets one result, not one per line', () => { + const out = renderEnvelope({ + ...base, + changes: undefined, + assertions: [{ code: '// check the box\nI.seeElement({\n role: "button",\n text: "Submit"\n});', passed: false, proof: [] }], + }); + + expect(out).toContain('I.seeElement({ role: "button", text: "Submit" }); => FAILED'); + expect(out.match(/=> FAILED/g)).toHaveLength(1); + expect(out).not.toContain('// check the box'); + }); + + test('no expressible assertion is stated as such, not as a failure', () => { + const out = renderEnvelope({ ...base, changes: undefined, assertions: [] }); + expect(out).toContain('none ran'); + expect(out).not.toContain('FAILED'); + }); + + test('changes render on every action envelope, including when nothing moved', () => { + expect(renderEnvelope({ ...base, changes: 'no change' })).toContain('### Changes\nno change'); }); - test('healed success carries note', () => { - const out = renderEnvelope({ ...base, healed: true, healNote: 'dismissed overlay first' }); - expect(out).toContain('healed: true (dismissed overlay first)'); + test('changes render alongside assertions rather than replacing them', () => { + const out = renderEnvelope({ ...base, changes: 'no change', assertions: [{ code: 'I.seeElement()', passed: true, proof: [] }] }); + expect(out).toContain('### Changes'); + expect(out).toContain('### Assertions'); + }); + + test('steps report every action and check in order, with its proof', () => { + const out = renderEnvelope({ + ...base, + steps: [ + { label: "I.click('Add workflow')", ok: true, proof: 'ariaDiff:\n added:\n - textbox "Title"' }, + { label: 'I.seeAttributesOnElements({"role":"button"}, { disabled: true })', ok: true, proof: 'await expect(page.getByRole("button")).toBeDisabled();' }, + { label: "I.fillField('Title', 'x')", ok: false, proof: 'element not found' }, + ], + }); + + expect(out).toContain("1. ok I.click('Add workflow')"); + expect(out).toContain('2. ok I.seeAttributesOnElements'); + expect(out).toContain("3. FAIL I.fillField('Title', 'x')"); + expect(out).toContain(' element not found'); + expect(out).toContain(' await expect(page.getByRole("button")).toBeDisabled();'); + }); + + test('every expected outcome is echoed with its own result, including the ones nothing checked', () => { + const out = renderEnvelope({ + ...base, + changes: undefined, + expectations: [ + { text: 'the editor opens', status: 'passed' }, + { text: 'the draft is saved', status: 'failed' }, + { text: 'the list refreshes', status: 'unverified' }, + ], + }); + + expect(out).toContain('### Expected outcomes'); + expect(out).toContain('1. PASSED the editor opens'); + expect(out).toContain('2. FAILED the draft is saved'); + expect(out).toContain('3. not verified the list refreshes'); }); test('attached instance renders attached browser line', () => { const out = renderEnvelope({ ...base, instance: { ...base.instance, attached: 'playwright-cli session "default", workspace /w' } }); - expect(out).toContain('browser: attached (playwright-cli session "default"'); + expect(out).toContain('attached to playwright-cli session "default"'); expect(out).not.toContain('started 12m ago'); }); test('instance without evidence of a live browser reports it as not running', () => { const out = renderEnvelope({ ...base, instance: { name: 'default', tabs: 0, others: [] } }); - expect(out).toContain('browser: not running'); + expect(out).toContain('not running'); + }); + + test('the status hash is offered as the way to reach details', () => { + const out = renderEnvelope({ ...base, status: 'abc123def456789', artifacts: undefined }); + expect(out).toContain('details: prima status abc123def456789'); + expect(out).not.toContain('### Artifacts'); }); test('open tabs alone are evidence enough for a running browser', () => { const out = renderEnvelope({ ...base, instance: { name: 'default', tabs: 2, others: [] } }); - expect(out).toContain('browser: running'); + expect(out).toContain('| running'); expect(out).not.toContain('browser: not running'); }); }); diff --git a/boat/prima/tests/prima.test.ts b/boat/prima/tests/prima.test.ts index 2552f96b..8d19e152 100644 --- a/boat/prima/tests/prima.test.ts +++ b/boat/prima/tests/prima.test.ts @@ -1,11 +1,12 @@ import { afterAll, afterEach, beforeAll, describe, expect, spyOn, test } from 'bun:test'; -import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; import { tmpdir } from 'node:os'; import path from 'node:path'; import { Navigator } from '../../../src/ai/navigator.ts'; import { getEndpointFilePath, listInstances } from '../../../src/browser-server.ts'; import { ConfigParser } from '../../../src/config.ts'; import { ExplorBot } from '../../../src/explorbot.ts'; +import { TestResult } from '../../../src/test-plan.ts'; import { Prima } from '../src/prima.ts'; let artifactsRoot: string; @@ -29,6 +30,7 @@ function fakeState(over: Record<string, unknown> = {}) { getStateHash: () => 'login_h1_login', ariaSnapshot: '- textbox "Email"\n- button "Sign in"', combinedHtml: () => '<form></form>', + simplifiedHtml: async () => '<form><button>Sign in</button></form>', toToolResult: async () => ({ pageDiff: { urlChanged: false, ariaChanges: null } }), ...over, }; @@ -49,6 +51,14 @@ function toolExecution(code: string, success = true, message?: string) { return { toolName: 'click', input: { commands: [code] }, output: { success, code, message }, wasSuccessful: success }; } +function completedExecution(numbers: number[], proof: string) { + return { toolName: 'completed', input: { numbers, proof }, output: { success: true, action: 'completed' }, wasSuccessful: true }; +} + +function blockedExecution(instruction: number, reason: string) { + return { toolName: 'blocked', input: { instruction, reason }, output: { success: true, action: 'blocked' }, wasSuccessful: true }; +} + function fakeProvider(invokeConversation: (...args: any[]) => Promise<any>, prompts: string[] = []) { return { startConversation: () => ({ addUserText: (text: string) => prompts.push(text) }), @@ -77,12 +87,14 @@ function fakePrima(options: Record<string, unknown> = {}) { }, }), capture: async () => after, + withPage: async (fn: (page: any) => any) => fn({ locator: () => ({ ariaSnapshot: async () => `${(prima as any).bot.stateManager().getCurrentState().ariaSnapshot}\n- button "Refreshed" [ref=e7]` }) }), }), stateManager: () => ({ getCurrentState: () => fakeState(), getVisitCount: () => 1, }), getCurrentState: () => fakeState(), + getConfig: () => ({}), requestStore: () => ({ getRequests: () => [] }), getProvider: () => ({ chat: async () => '' }), }; @@ -119,9 +131,10 @@ describe('Prima.pw', () => { const { prima } = fakePrima(); const envelope = await prima.pw("({ page }) => page.click('text=Login')"); expect(envelope.changes).toContain('heading "Dashboard"'); - expect(existsSync(envelope.artifacts!.aria)).toBe(true); - expect(existsSync(envelope.artifacts!.html)).toBe(true); - expect(existsSync(envelope.artifacts!.network)).toBe(true); + expect(envelope.status).toMatch(/^[0-9a-f]{15}$/); + expect(existsSync(path.join(artifactsRoot, envelope.status!, 'aria.yml'))).toBe(true); + expect(existsSync(path.join(artifactsRoot, envelope.status!, 'page.html'))).toBe(true); + expect(existsSync(path.join(artifactsRoot, envelope.status!, 'network.jsonl'))).toBe(false); expect(envelope.instance.name).toBe('default'); }); @@ -138,9 +151,8 @@ describe('Prima.pw', () => { const envelope = await prima.pw("({ page }) => page.click('text=Login')"); expect(envelope.ok).toBe(false); expect(envelope.failure?.error).toContain("locator 'text=Login' not found"); - expect(envelope.failure?.attempts).toEqual([]); expect(envelope.page.url).toBe('https://app.example.com/login'); - expect(envelope.artifacts).toBeTruthy(); + expect(envelope.status).toBeTruthy(); }); test('unrecoverable browser still returns a failure envelope when capture fails too', async () => { @@ -159,7 +171,7 @@ describe('Prima.pw', () => { expect(envelope.ok).toBe(false); expect(envelope.failure?.error).toContain('Browser page is unavailable'); expect(envelope.page.url).toContain('/login'); - expect(envelope.artifacts).toBeTruthy(); + expect(envelope.status).toBeTruthy(); }); test('missing explorer returns a failure envelope instead of rejecting', async () => { @@ -172,76 +184,34 @@ describe('Prima.pw', () => { }); }); -describe('Prima heal', () => { - test('failed pw heals via navigator and reports healed envelope', async () => { - const { prima } = fakePrima(); - (prima as any).bot.getExplorer = failingExplorer; - (prima as any).bot.agentNavigator = () => ({ - resolveState: async (_msg: string, _result: unknown, opts: any) => { - opts?.onAttempt?.({ code: "I.click('Login')", error: 'not visible' }); - opts?.onAttempt?.({ code: "I.click('#login-btn')" }); - return true; - }, - }); - const envelope = await prima.pw("({ page }) => page.click('text=Login')"); - expect(envelope.ok).toBe(true); - expect(envelope.healed).toBe(true); - expect(envelope.healNote).toBeTruthy(); - expect(envelope.used).toEqual(["I.click('#login-btn')"]); - }); - - test('exhausted heal returns failure envelope with attempts and compact aria', async () => { +describe('Prima failure never substitutes a target', () => { + test('a failed pw fails without consulting the navigator', async () => { const { prima } = fakePrima(); - (prima as any).bot.getExplorer = failingExplorer; - (prima as any).bot.agentNavigator = () => ({ - resolveState: async (_msg: string, _result: unknown, opts: any) => { - opts?.onAttempt?.({ code: "I.click('Login')", error: 'not visible' }); - return false; - }, - }); - const envelope = await prima.pw("({ page }) => page.click('text=Login')"); - expect(envelope.ok).toBe(false); - expect(envelope.failure?.error).toContain("locator 'text=Login' not found"); - expect(envelope.failure?.attempts).toEqual([{ code: "I.click('Login')", outcome: 'not visible' }]); - expect(envelope.failure?.reasoning).toBe('not visible'); - expect(envelope.failure?.compactAria).toContain('button'); - expect(envelope.artifacts).toBeTruthy(); - }); - - test('heal disabled skips navigator entirely', async () => { - const { prima } = fakePrima({ heal: false }); let called = false; (prima as any).bot.getExplorer = failingExplorer; (prima as any).bot.agentNavigator = () => { called = true; return { resolveState: async () => true }; }; + const envelope = await prima.pw("({ page }) => page.click('text=Login')"); + expect(called).toBe(false); expect(envelope.ok).toBe(false); - expect(envelope.healed).toBeUndefined(); - expect(envelope.failure?.attempts).toEqual([]); + expect(envelope.used).toBeUndefined(); + expect(envelope.failure?.error).toContain("locator 'text=Login' not found"); expect(envelope.failure?.compactAria).toContain('button'); + expect(envelope.status).toBeTruthy(); }); - test('unusable ai provider skips healing and notes it in the envelope', async () => { + test('a failed pw carries no healed marker', async () => { const { prima } = fakePrima(); - let called = false; (prima as any).bot.getExplorer = failingExplorer; - (prima as any).bot.getProvider = () => { - throw new Error('AI provider is not configured'); - }; - (prima as any).bot.agentNavigator = () => { - called = true; - return { resolveState: async () => true }; - }; + const envelope = await prima.pw("({ page }) => page.click('text=Login')"); - expect(called).toBe(false); - expect(envelope.ok).toBe(false); - expect(envelope.healed).toBe(false); - expect(envelope.healNote).toContain('ai unavailable'); - expect(envelope.healNote).toContain('AI provider is not configured'); - expect(envelope.failure?.error).toContain("locator 'text=Login' not found"); + + expect(envelope).not.toHaveProperty('healed'); + expect(envelope).not.toHaveProperty('healNote'); }); }); @@ -258,6 +228,14 @@ describe('Prima.start', () => { await expect(prima.start()).rejects.toThrow(/playwright-cli open/); expect(started).toBe(false); }); + + test('leaves the explorbot reporter switched off', () => { + const prima = new Prima({ instance: 'default' }); + + expect((prima as any).bot.reporter().isEnabled()).toBe(false); + expect(process.env.TESTOMATIO_HTML_REPORT_SAVE).toBeUndefined(); + expect(process.env.TESTOMATIO_MARKDOWN_REPORT_SAVE).toBeUndefined(); + }); }); describe('Prima attach ladder', () => { @@ -461,7 +439,7 @@ describe('Prima.do', () => { fakeProvider(async () => { calls++; if (calls > 1) return { toolExecutions: [] }; - return { toolExecutions: [toolExecution("I.click('Login')")] }; + return { toolExecutions: [toolExecution("I.click('Login')"), completedExecution([1, 2], 'the invoice PDF is open')] }; }, prompts); const envelope = await prima.do(['open the first invoice', 'download its PDF']); @@ -469,9 +447,171 @@ describe('Prima.do', () => { expect(prompts.join('\n')).toContain('open the first invoice'); expect(prompts.join('\n')).toContain('download its PDF'); expect(prompts.join('\n')).toContain('button "Sign in"'); - expect(envelope.used).toEqual(["I.click('Login')"]); + expect(envelope.steps?.[0]).toMatchObject({ label: "I.click('Login')", ok: true }); + expect(envelope.ok).toBe(true); + expect(calls).toBe(1); + }); + + test('every instruction is reported in order with the actions that proved it', async () => { + const { prima } = fakePrima(); + let calls = 0; + (prima as any).bot.getProvider = () => + fakeProvider(async () => { + calls++; + if (calls === 1) return { toolExecutions: [toolExecution("I.click('Invoices')"), completedExecution([1], 'the invoice list is open')] }; + return { toolExecutions: [toolExecution("I.click('Download')"), completedExecution([2], 'the PDF opened in a new tab')] }; + }); + + const envelope = await prima.do(['open the invoices page', 'download the PDF']); + + expect(envelope.ok).toBe(true); + expect(envelope.steps).toEqual([ + { label: "I.click('Invoices')", ok: true, proof: '' }, + { label: 'done: open the invoices page', ok: true, proof: 'the invoice list is open' }, + { label: "I.click('Download')", ok: true, proof: '' }, + { label: 'done: download the PDF', ok: true, proof: 'the PDF opened in a new tab' }, + ]); + }); + + test('one report closing several instructions is logged once, not once per instruction', async () => { + const { prima } = fakePrima(); + (prima as any).bot.getProvider = () => fakeProvider(async () => ({ toolExecutions: [toolExecution("I.click('Overview')"), completedExecution([1, 2], 'the workflow list is shown')] })); + + const envelope = await prima.do(['click the overview button', 'confirm the workflow list appears']); + + expect(envelope.steps?.map((step) => step.label)).toEqual(["I.click('Overview')", 'done: click the overview button; confirm the workflow list appears']); + expect(envelope.steps?.filter((step) => step.proof === 'the workflow list is shown')).toHaveLength(1); + }); + + test('the page diff is reported once, under changes, not again under every step', async () => { + const { prima } = fakePrima(); + (prima as any).bot.getProvider = () => fakeProvider(async () => ({ toolExecutions: [toolExecution("I.click('Overview')"), completedExecution([1], 'the list is shown')] })); + + const envelope = await prima.do(['click the overview button']); + + expect(envelope.changes).toBeUndefined(); + expect(envelope.used).toBeUndefined(); + expect(envelope.steps?.some((step) => step.label === "I.click('Overview')")).toBe(true); + }); + + test('an instruction reported without any action behind it says so rather than being rejected', async () => { + const { prima } = fakePrima(); + (prima as any).bot.getProvider = () => fakeProvider(async () => ({ toolExecutions: [completedExecution([1], 'no cookie banner is present on this page')] })); + + const envelope = await prima.do(['dismiss the cookie banner if one appeared']); + + expect(envelope.ok).toBe(true); + expect(envelope.steps).toEqual([{ label: 'done: dismiss the cookie banner if one appeared', ok: true, proof: 'no cookie banner is present on this page' }]); + }); + + test('the remaining instructions are re-stated as the ledger closes, and finished ones are not', async () => { + const { prima } = fakePrima(); + const prompts: string[] = []; + let calls = 0; + (prima as any).bot.getProvider = () => + fakeProvider(async () => { + calls++; + if (calls === 1) return { toolExecutions: [toolExecution("I.click('Invoices')"), completedExecution([1], 'list is open')] }; + return { toolExecutions: [toolExecution("I.click('Download')"), completedExecution([2], 'pdf is open')] }; + }, prompts); + + await prima.do(['open the invoices page', 'download the PDF']); + + const progress = prompts.find((prompt) => prompt.startsWith('<progress>')); + expect(progress).toContain('1. done — open the invoices page (list is open)'); + expect(progress).toContain('2. open — download the PDF'); + }); + + test('a model that narrates instead of reporting is asked once for the ledger', async () => { + const { prima } = fakePrima(); + const prompts: string[] = []; + let calls = 0; + (prima as any).bot.getProvider = () => + fakeProvider(async () => { + calls++; + if (calls === 1) return { toolExecutions: [toolExecution("I.click('Invoices')")] }; + if (calls === 2) return { toolExecutions: [], response: { text: 'I opened the invoices page.' } }; + return { toolExecutions: [completedExecution([1], 'the invoice list is open')] }; + }, prompts); + + const envelope = await prima.do(['open the invoices page']); + + expect(prompts.join('\n')).toContain('still unreported'); + expect(envelope.ok).toBe(true); + expect(calls).toBe(3); + }); + + test('a model that will not report even when asked stops rather than looping', async () => { + const { prima } = fakePrima(); + let calls = 0; + (prima as any).bot.getProvider = () => + fakeProvider(async () => { + calls++; + if (calls === 1) return { toolExecutions: [toolExecution("I.click('Invoices')")] }; + return { toolExecutions: [], response: { text: 'I opened the invoices page.' } }; + }); + + const envelope = await prima.do(['open the invoices page']); + + expect(envelope.ok).toBe(false); + expect(calls).toBe(4); + }); + + test('instructions left unreported are settled from what the run actually did', async () => { + const { prima } = fakePrima(); + const prompts: string[] = []; + let calls = 0; + (prima as any).bot.getProvider = () => + fakeProvider(async () => { + calls++; + if (calls === 1) return { toolExecutions: [toolExecution("I.click('Invoices')"), toolExecution("I.click('Download')")] }; + if (calls === 2) return { toolExecutions: [], response: { text: 'Both done.' } }; + if (calls === 3) return { toolExecutions: [], response: { text: 'Still both done.' } }; + return { toolExecutions: [completedExecution([1, 2], 'the invoice list opened and the PDF downloaded')] }; + }, prompts); + + const envelope = await prima.do(['open the invoices page', 'download the PDF']); + + expect(envelope.ok).toBe(true); + expect(prompts.join('\n')).toContain('The run is over and these instructions were never reported'); + expect(envelope.steps?.every((step) => step.ok)).toBe(true); + }); + + test('each action leaves the page it produced beside the log', async () => { + const { prima } = fakePrima(); + const clicked = { ...toolExecution("I.click('Invoices')"), output: { success: true, code: "I.click('Invoices')", pageDiff: { ariaChanges: 'added:\n - heading "Dashboard"' } } }; + (prima as any).bot.getProvider = () => fakeProvider(async () => ({ toolExecutions: [clicked, completedExecution([1], 'the list is open')] })); + + const envelope = await prima.do(['open the invoices page']); + const dir = envelope.stepFiles!; + + expect(existsSync(path.join(dir, '1-i_click__invoices__.aria.yaml'))).toBe(true); + expect(existsSync(path.join(dir, '1-i_click__invoices__.html'))).toBe(true); + expect(readFileSync(path.join(dir, '1-i_click__invoices__.diff.yaml'), 'utf-8')).toContain('heading "Dashboard"'); + }); + + test('an instruction the model never reported is named as unaccounted for', async () => { + const { prima } = fakePrima(); + (prima as any).bot.getProvider = () => fakeProvider(async () => ({ toolExecutions: [toolExecution("I.click('Invoices')"), completedExecution([1], 'list is open')] })); + + const envelope = await prima.do(['open the invoices page', 'download the PDF']); + + expect(envelope.ok).toBe(false); + expect(envelope.failure?.error).toContain('open: download the PDF'); + expect(envelope.steps?.at(-1)).toMatchObject({ label: 'unreported: download the PDF', ok: false }); + expect(envelope.steps?.at(-1)?.proof).toContain('never reported'); + }); + + test('a stray failed action does not fail a sequence whose instructions all closed', async () => { + const { prima } = fakePrima(); + (prima as any).bot.getProvider = () => + fakeProvider(async () => ({ + toolExecutions: [toolExecution("I.click('Invoices')"), completedExecution([1], 'list is open'), toolExecution("I.pressKey('End')", false, 'No element is focused')], + })); + + const envelope = await prima.do(['open the invoices page']); + expect(envelope.ok).toBe(true); - expect(calls).toBe(2); }); test('caps iterations at the instruction budget', async () => { @@ -483,28 +623,35 @@ describe('Prima.do', () => { return { toolExecutions: [toolExecution("I.click('Next')")] }; }); + // one extra call past each cap is the closing pass that asks for the unreported instructions await prima.do(['first step', 'second step']); - expect(calls).toBe(4); + expect(calls).toBe(6 + 1); calls = 0; await prima.do(['a', 'b', 'c', 'd', 'e', 'f', 'g']); - expect(calls).toBe(6); + expect(calls).toBe(16 + 1); + + calls = 0; + await prima.do(Array.from({ length: 20 }, (_, i) => `step ${i}`)); + expect(calls).toBe(24 + 1); }); - test('routes a failed tool execution through heal', async () => { + test('a failed tool execution fails the command instead of reaching for another element', async () => { const { prima } = fakePrima(); + let resolved = false; (prima as any).bot.getProvider = () => fakeProvider(async () => ({ toolExecutions: [toolExecution("I.click('Login')", false, 'element not found')] })); (prima as any).bot.agentNavigator = () => ({ - resolveState: async (_msg: string, _result: unknown, opts: any) => { - opts?.onAttempt?.({ code: "I.click('#login-btn')" }); + resolveState: async () => { + resolved = true; return true; }, }); const envelope = await prima.do(['click the login link']); - expect(envelope.ok).toBe(true); - expect(envelope.healed).toBe(true); - expect(envelope.used).toEqual(["I.click('#login-btn')"]); + + expect(resolved).toBe(false); + expect(envelope.ok).toBe(false); + expect(envelope.failure?.error).toContain('element not found'); }); test('reports a failure when the model performs no action and keeps its explanation', async () => { @@ -534,9 +681,10 @@ describe('Prima.do', () => { await prima.do(['open the invoices page', 'read the first invoice']); - expect(prompts.length).toBe(2); - expect(prompts[0]).toContain('button "Sign in"'); - expect(prompts[1]).toContain('heading "Dashboard"'); + const contexts = prompts.filter((prompt) => prompt.includes('<page url=')); + expect(contexts.length).toBe(2); + expect(contexts[0]).toContain('button "Sign in"'); + expect(contexts[1]).toContain('heading "Dashboard"'); }); test('keeps the context when the state did not change between iterations', async () => { @@ -551,34 +699,156 @@ describe('Prima.do', () => { }, prompts); await prima.do(['open the invoices page', 'read the first invoice']); - expect(prompts.length).toBe(1); + expect(prompts.filter((prompt) => prompt.includes('<page url=')).length).toBe(1); }); - test('click is a single-instruction alias over do', async () => { + test('a closed ledger ends the sequence instead of burning the remaining iteration budget', async () => { const { prima } = fakePrima(); - const received: string[][] = []; - (prima as any).do = async (instructions: string[]) => { - received.push(instructions); - return { ok: true }; - }; + let calls = 0; + (prima as any).bot.getProvider = () => + fakeProvider(async () => { + calls++; + if (calls === 1) return { toolExecutions: [toolExecution("I.click('Invoices')"), completedExecution([1], 'the invoice list is open')] }; + return { toolExecutions: [completedExecution([2], 'the first invoice is on screen')] }; + }); + + const envelope = await prima.do(['open the invoices page', 'read the first invoice']); - await prima.click('the login link'); - expect(received[0].length).toBe(1); - expect(received[0][0]).toContain('the login link'); + expect(calls).toBe(2); + expect(envelope.ok).toBe(true); }); - test('fill is a single-instruction alias carrying field and value', async () => { + test('an instruction the model reports as blocked fails the command and keeps its reason', async () => { const { prima } = fakePrima(); - const received: string[][] = []; - (prima as any).do = async (instructions: string[]) => { - received.push(instructions); - return { ok: true }; - }; + (prima as any).bot.getProvider = () => + fakeProvider(async () => ({ + toolExecutions: [toolExecution("I.click('Invoices')"), completedExecution([1], 'the list is open'), blockedExecution(2, 'no PDF link exists on this page')], + })); + + const envelope = await prima.do(['open the invoices page', 'download the PDF']); - await prima.fill('the email field', 'user@example.com'); - expect(received[0].length).toBe(1); - expect(received[0][0]).toContain('the email field'); - expect(received[0][0]).toContain('user@example.com'); + expect(envelope.ok).toBe(false); + expect(envelope.failure?.error).toContain('blocked: download the PDF — no PDF link exists on this page'); + }); + + test('a failed check is logged but does not overrule the instruction the model closed', async () => { + const { prima } = fakePrima(); + let calls = 0; + (prima as any).bot.getProvider = () => + fakeProvider(async () => { + calls++; + if (calls === 1) { + return { + toolExecutions: [{ toolName: 'verify', input: { assertion: 'debug panel is enabled' }, output: { success: false, action: 'verify', message: 'Verification failed' }, wasSuccessful: false }, toolExecution("I.click('Enable debug panel')")], + }; + } + return { toolExecutions: [completedExecution([1], 'the debug switch is on')] }; + }); + + const envelope = await prima.do(['enable debug mode']); + + expect(envelope.ok).toBe(true); + expect(envelope.steps?.[0]).toMatchObject({ label: 'verify: debug panel is enabled', ok: false }); + }); + + test('a check that passes on a retry does not fail the command', async () => { + const { prima } = fakePrima(); + let calls = 0; + (prima as any).bot.getProvider = () => + fakeProvider(async () => { + calls++; + if (calls === 1) return { toolExecutions: [toolExecution("I.click('Delete')"), { toolName: 'verify', input: { assertion: 'the row is gone' }, output: { success: false, action: 'verify', message: 'Verification failed' }, wasSuccessful: false }] }; + return { toolExecutions: [{ toolName: 'verify', input: { assertion: 'the row is gone' }, output: { success: true, action: 'verify', code: "I.dontSee('Item')" }, wasSuccessful: true }, completedExecution([1, 2], 'the row is gone')] }; + }); + + const envelope = await prima.do(['delete the row', 'confirm it is gone']); + + expect(envelope.ok).toBe(true); + }); + test('context() hands back the page tree first and drops to markup only when asked again', async () => { + const { prima } = fakePrima(); + let captured: any; + (prima as any).bot.getProvider = () => + fakeProvider(async (_conversation: unknown, tools: any) => { + captured = tools.context; + return { toolExecutions: [] }; + }); + + await prima.do(['open the invoices page']); + + const first = await captured.execute({ reason: 'the control is not in my context' }); + expect(first.context).toContain('button "Refreshed"'); + expect(first.context).toContain('ref=e7'); + + const second = await captured.execute({ reason: 'still cannot reach it' }); + expect(second.context).toContain('<form>'); + }); +}); + +describe('Prima.check', () => { + test('every --expected outcome is reported with its own result', async () => { + const { prima } = fakePrima(); + (prima as any).bot.agentTester = () => ({ + test: async (test: any) => { + test.addNote('the editor opens', TestResult.PASSED); + test.addNote('the draft is saved', TestResult.FAILED); + return { success: false }; + }, + }); + (prima as any).bot.agentPilot = () => ({ + settleExpectations: async () => [ + { text: 'the editor opens', status: 'passed' }, + { text: 'the draft is saved', status: 'failed' }, + { text: 'the list refreshes', status: 'unverified' }, + ], + }); + + const envelope = await prima.check('edit and save a skill', ['the editor opens', 'the draft is saved', 'the list refreshes']); + + expect(envelope.expectations).toEqual([ + { text: 'the editor opens', status: 'passed' }, + { text: 'the draft is saved', status: 'failed' }, + { text: 'the list refreshes', status: 'unverified' }, + ]); + }); + + test('steps report the failures and count the rest, leaving verdicts to the outcomes section', async () => { + const { prima } = fakePrima(); + (prima as any).bot.agentTester = () => ({ + test: async (test: any) => { + test.addNote('opened the panel', TestResult.PASSED); + test.addNote('clicked the row', TestResult.PASSED); + test.addNote('Failed to open the editor (click)', TestResult.FAILED); + test.addNote('the editor opens', TestResult.PASSED); + return { success: false }; + }, + }); + (prima as any).bot.agentPilot = () => ({ settleExpectations: async () => [{ text: 'the editor opens', status: 'passed' }] }); + + const envelope = await prima.check('edit a skill', ['the editor opens']); + const labels = envelope.steps?.map((step) => step.label) || []; + + expect(labels).toContain('Failed to open the editor (click)'); + expect(labels).not.toContain('the editor opens'); + expect(labels).not.toContain('opened the panel'); + expect(labels.at(-1)).toContain('2 further steps ran without failing'); + expect(labels.at(-1)).toContain('prima status'); + }); + + test('the scenario stands in as the only outcome when none was given', async () => { + const { prima } = fakePrima(); + (prima as any).bot.agentTester = () => ({ + test: async (test: any) => { + test.addNote('edit and save a skill', TestResult.PASSED); + return { success: true }; + }, + }); + (prima as any).bot.agentPilot = () => ({ settleExpectations: async () => [{ text: 'edit and save a skill', status: 'passed' }] }); + + const envelope = await prima.check('edit and save a skill'); + + expect(envelope.ok).toBe(true); + expect(envelope.expectations).toEqual([{ text: 'edit and save a skill', status: 'passed' }]); }); }); @@ -591,7 +861,7 @@ describe('Prima.ask, verify, research', () => { const envelope = await prima.ask('what do I see?'); expect(envelope.answer).toContain('login'); expect(envelope.changes).toBeUndefined(); - expect(envelope.artifacts).toBeTruthy(); + expect(envelope.status).toBeTruthy(); }); test('ask with noVision answers from researcher summary', async () => { @@ -625,28 +895,41 @@ describe('Prima.ask, verify, research', () => { expect(envelope.answer).toContain('vision'); }); - test('verify returns verdict with assertion code', async () => { + test('verify reports each assertion with its own result and gives no verdict', async () => { const { prima } = fakePrima(); (prima as any).bot.agentNavigator = () => ({ - verifyState: async () => ({ verified: true, successfulCodes: ["I.see('Dashboard')"], assertionSteps: [], totalAttempted: 1 }), + verifyState: async () => ({ + verified: false, + results: [ + { code: "I.see('Dashboard')", passed: true, proof: ['await expect(page).toContainText("Dashboard");'] }, + { code: "I.seeElement('.chart')", passed: false, proof: [] }, + ], + successfulCodes: ["I.see('Dashboard')"], + assertionSteps: [], + totalAttempted: 2, + }), }); const envelope = await prima.verify('user sees the dashboard'); - expect(envelope.verdict?.passed).toBe(true); - expect(envelope.verdict?.code).toBe("I.see('Dashboard')"); + + expect(envelope).not.toHaveProperty('verdict'); + expect(envelope.assertions).toEqual([ + { code: "I.see('Dashboard')", passed: true, proof: ['await expect(page).toContainText("Dashboard");'] }, + { code: "I.seeElement('.chart')", passed: false, proof: [] }, + ]); expect(envelope.ok).toBe(true); }); - test('failed verification is reported as a failed verdict', async () => { + test('verify with nothing expressible reports no assertions rather than a failure', async () => { const { prima } = fakePrima(); (prima as any).bot.agentNavigator = () => ({ - verifyState: async () => ({ verified: false, successfulCodes: [], assertionSteps: [], totalAttempted: 3 }), + verifyState: async () => ({ verified: false, inexpressible: true, results: [], successfulCodes: [], assertionSteps: [], totalAttempted: 0 }), }); - const envelope = await prima.verify('user sees the dashboard'); - expect(envelope.verdict?.passed).toBe(false); - expect(envelope.verdict?.evidence).toBeTruthy(); - expect(envelope.ok).toBe(false); + const envelope = await prima.verify('the save button is disabled'); + + expect(envelope.assertions).toEqual([]); + expect(envelope.ok).toBe(true); }); test('research returns UI map in envelope', async () => { @@ -795,27 +1078,28 @@ describe('Prima.go', () => { const envelope = await prima.go('/billing'); expect(envelope.ok).toBe(false); - expect(envelope.healed).toBe(false); - expect(envelope.healNote).toContain('ai unavailable'); + expect(envelope).not.toHaveProperty('healed'); expect(envelope.failure?.error).toContain('AI-assisted recovery is unavailable'); }); - test('navigation error is routed through heal', async () => { + test('navigation error fails instead of reaching the target another way', async () => { const { prima } = fakePrima(); + let resolved = false; (prima as any).bot.agentNavigator = () => ({ visit: async () => { throw new Error('Navigation to /billing failed'); }, - resolveState: async (_message: string, _result: unknown, opts: any) => { - opts?.onAttempt?.({ code: "I.click('Billing')" }); + resolveState: async () => { + resolved = true; return true; }, }); const envelope = await prima.go('/billing'); - expect(envelope.ok).toBe(true); - expect(envelope.healed).toBe(true); - expect(envelope.used).toEqual(["I.click('Billing')"]); + + expect(resolved).toBe(false); + expect(envelope.ok).toBe(false); + expect(envelope.failure?.error).toContain('Navigation to /billing failed'); }); test('go cannot run before a session exists and never launches a browser', async () => { diff --git a/boat/prima/tests/pw-parser.test.ts b/boat/prima/tests/pw-parser.test.ts index db6f9608..f29f0320 100644 --- a/boat/prima/tests/pw-parser.test.ts +++ b/boat/prima/tests/pw-parser.test.ts @@ -22,12 +22,16 @@ describe('isFunctionExpression', () => { describe('toCodeceptWrapper', () => { test('calls the function from an async wrapper the native helper API accepts', () => { const code = toCodeceptWrapper("({ page }) => page.click('text=Login')"); - expect(code).toBe("I.usePlaywrightTo('pw', async (playwright) => (({ page }) => page.click('text=Login'))(playwright))"); + expect(code).toBe("return I.usePlaywrightTo('pw', async (playwright) => (({ page }) => page.click('text=Login'))(playwright))"); + }); + + test('returns the expression result so the caller can read it', () => { + expect(toCodeceptWrapper('({ page }) => page.title()')).toStartWith('return '); }); test('keeps an already async function callable', () => { const code = toCodeceptWrapper("async ({ page }) => { await page.click('text=Login') }"); expect(code).toContain("async ({ page }) => { await page.click('text=Login') }"); - expect(code).toStartWith("I.usePlaywrightTo('pw', async (playwright) =>"); + expect(code).toStartWith("return I.usePlaywrightTo('pw', async (playwright) =>"); }); }); diff --git a/boat/prima/tests/pw-registry.test.ts b/boat/prima/tests/pw-registry.test.ts index 09ed9dce..a217e7ab 100644 --- a/boat/prima/tests/pw-registry.test.ts +++ b/boat/prima/tests/pw-registry.test.ts @@ -26,14 +26,21 @@ describe('readDescriptors', () => { expect(list[0].playwrightLib).toBe('/lib/pw'); }); - test('skips descriptors without an endpoint, a title or a workspace', () => { + test('skips descriptors without an endpoint or a title', () => { const dir = mkdtempSync(path.join(tmpdir(), 'pwb-')); - writeDescriptor(dir, 'a', { title: 'default' }); writeDescriptor(dir, 'b', { workspaceDir: '/work/app' }); writeDescriptor(dir, 'c', { title: 'default', workspaceDir: '/work/app', endpoint: undefined }); expect(readDescriptors(dir)).toEqual([]); }); + test('keeps descriptors that carry no workspaceDir, as playwright-cli writes them', () => { + const dir = mkdtempSync(path.join(tmpdir(), 'pwb-')); + writeDescriptor(dir, 'a', { title: 'default' }); + const list = readDescriptors(dir); + expect(list.length).toBe(1); + expect(list[0].workspaceDir).toBe(''); + }); + test('returns nothing when the registry directory is missing', () => { expect(readDescriptors(path.join(tmpdir(), 'pwb-missing-registry'))).toEqual([]); }); @@ -41,28 +48,25 @@ describe('readDescriptors', () => { describe('selectDescriptor', () => { const dir = mkdtempSync(path.join(tmpdir(), 'pwb-')); - writeDescriptor(dir, 'a', { title: 'default', workspaceDir: '/work/app' }); - writeDescriptor(dir, 'b', { title: 'auth', workspaceDir: '/work/app' }); - writeDescriptor(dir, 'c', { title: 'default', workspaceDir: '/work/other' }); + writeDescriptor(dir, 'a', { title: 'default' }); + writeDescriptor(dir, 'b', { title: 'auth' }); const all = readDescriptors(dir); - test('explicit title wins within workspace', () => { - const { match } = selectDescriptor(all, { workspaceDir: '/work/app', title: 'auth' }); - expect(match?.title).toBe('auth'); + test('explicit title wins', () => { + expect(selectDescriptor(all, { title: 'auth' }).match?.title).toBe('auth'); }); - test('default title picked for workspace when present', () => { - const { match } = selectDescriptor(all, { workspaceDir: '/work/app' }); - expect(match?.title).toBe('default'); + test('default title picked when no title is asked for', () => { + expect(selectDescriptor(all).match?.title).toBe('default'); }); - test('single survivor for workspace picked without title', () => { - const { match } = selectDescriptor(all, { workspaceDir: '/work/other' }); - expect(match?.workspaceDir).toBe('/work/other'); + test('the single live session is picked without a title', () => { + const single = all.filter((descriptor) => descriptor.title === 'auth'); + expect(selectDescriptor(single).match?.title).toBe('auth'); }); - test('no workspace match returns candidates empty', () => { - const { match, candidates } = selectDescriptor(all, { workspaceDir: '/elsewhere' }); + test('no descriptors returns no match and no candidates', () => { + const { match, candidates } = selectDescriptor([]); expect(match).toBeUndefined(); expect(candidates.length).toBe(0); }); @@ -70,19 +74,14 @@ describe('selectDescriptor', () => { test('ambiguity returns no match with candidates listed', () => { const noDefault = all.filter((d) => d.title !== 'default'); const extra = [...noDefault, { ...noDefault[0], title: 'second' }]; - const { match, candidates } = selectDescriptor(extra, { workspaceDir: '/work/app' }); + const { match, candidates } = selectDescriptor(extra); expect(match).toBeUndefined(); expect(candidates.length).toBe(2); }); - test('unknown title in a populated workspace returns the workspace candidates', () => { - const { match, candidates } = selectDescriptor(all, { workspaceDir: '/work/app', title: 'missing' }); + test('unknown title returns every candidate', () => { + const { match, candidates } = selectDescriptor(all, { title: 'missing' }); expect(match).toBeUndefined(); expect(candidates.map((candidate) => candidate.title).sort()).toEqual(['auth', 'default']); }); - - test('workspace paths are compared resolved', () => { - const { match } = selectDescriptor(all, { workspaceDir: '/work/app/../app' }); - expect(match?.title).toBe('default'); - }); }); diff --git a/boat/prima/tests/session-log.test.ts b/boat/prima/tests/session-log.test.ts new file mode 100644 index 00000000..c0180c14 --- /dev/null +++ b/boat/prima/tests/session-log.test.ts @@ -0,0 +1,92 @@ +import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test'; +import { existsSync, rmSync } from 'node:fs'; +import { ConfigParser } from '../../../src/config.ts'; +import type { EnvelopeData } from '../src/envelope.ts'; +import { latestSessionFile, readSession, recordCommand, sessionFile, sessionsDir } from '../src/session-log.ts'; + +const session = { key: 'default', endpoint: 'ws://127.0.0.1:5000/one', title: 'prima session "default"' }; +const restarted = { ...session, endpoint: 'ws://127.0.0.1:5000/two' }; + +beforeAll(() => { + ConfigParser.resetForTesting(); + ConfigParser.setupTestConfig(); +}); + +afterAll(() => { + ConfigParser.cleanupAllTestDirectories(); +}); + +beforeEach(() => { + rmSync(sessionsDir(), { recursive: true, force: true }); +}); + +function envelope(over: Partial<EnvelopeData> = {}): EnvelopeData { + return { + ok: true, + command: 'do "open the account menu"', + page: { url: 'https://app.example.com/settings', title: 'Settings', state: 'settings_h1_settings', visits: 1 }, + instance: { name: 'default', tabs: 1, others: [] }, + status: 'abc123', + ...over, + }; +} + +describe('prima session log', () => { + test('records every command of one session', () => { + const file = sessionFile(session.key); + recordCommand(file, session, envelope(), 1200); + recordCommand(file, session, envelope({ command: 'check "settings save"', ok: false, status: 'def456', failure: { error: 'the form never saved' } }), 3400); + + const recorded = readSession(file); + expect(recorded.title).toBe('prima session "default"'); + expect(recorded.tests.map((test: any) => test.title)).toEqual(['do "open the account menu"', 'check "settings save"']); + expect(recorded.tests.map((test: any) => test.status)).toEqual(['passed', 'failed']); + expect(recorded.tests[1].error.message).toBe('the form never saved'); + expect(recorded.tests[1].time).toBe(3400); + }); + + test('starts a new log when the session is a different browser', () => { + const file = sessionFile(session.key); + recordCommand(file, session, envelope(), 1000); + recordCommand(file, restarted, envelope({ command: 'pw "({ page }) => page.title()"' }), 1000); + + const recorded = readSession(file); + expect(recorded.tests.map((test: any) => test.title)).toEqual(['pw "({ page }) => page.title()"']); + }); + + test('carries steps, expectations and assertions as report steps', () => { + const file = sessionFile(session.key); + recordCommand( + file, + session, + envelope({ + steps: [{ label: 'done: open the account menu', ok: true, proof: 'the menu is expanded' }], + expectations: [ + { text: 'the theme is dark', status: 'passed' }, + { text: 'the choice survives a reload', status: 'unverified' }, + ], + assertions: [{ code: 'I.see("Dark")', passed: true, proof: ['locator("body").filter({ hasText: "Dark" })'] }], + stepFiles: '/tmp/prima/abc123', + }), + 900 + ); + + const [test] = readSession(file).tests; + expect(test.steps.map((step: any) => [step.title, step.status])).toEqual([ + ['done: open the account menu', 'passed'], + ['expected: the theme is dark', 'passed'], + ['expected: the choice survives a reload', 'skipped'], + ['I.see("Dark")', 'passed'], + ]); + expect(test.description).toContain('artifacts: /tmp/prima/abc123'); + expect(test.logs).toContain('### Result'); + }); + + test('reports the most recent session when none is named', () => { + expect(latestSessionFile()).toBe(null); + + recordCommand(sessionFile('checkout'), { ...session, key: 'checkout' }, envelope(), 500); + expect(latestSessionFile()).toBe(sessionFile('checkout')); + expect(existsSync(sessionFile('checkout'))).toBe(true); + }); +}); diff --git a/bun.lock b/bun.lock index 2a159374..7bb52d82 100644 --- a/bun.lock +++ b/bun.lock @@ -52,7 +52,7 @@ "ora-classic": "^5.4.2", "parse5": "^8.0.0", "pixelmatch": "^7.2.0", - "playwright": "^1.60", + "playwright": "^1.62", "pngjs": "^7.0.0", "react": "^19.1.1", "sambanova-ai-provider": "^1.2.2", @@ -2153,7 +2153,7 @@ "pkce-challenge": ["pkce-challenge@5.0.1", "", {}, "sha512-wQ0b/W4Fr01qtpHlqSqspcj3EhBvimsdh0KlHhH8HRZnMsEa0ea2fTULOXOS9ccQr3om+GcGRk4e+isrZWV8qQ=="], - "playwright": ["playwright@1.60.0", "", { "dependencies": { "playwright-core": "1.60.0" }, "optionalDependencies": { "fsevents": "2.3.2" }, "bin": { "playwright": "cli.js" } }, "sha512-hheHdokM8cdqCb0lcE3s+zT4t4W+vvjpGxsZlDnikarzx8tSzMebh3UiFtgqwFwnTnjYQcsyMF8ei2mCO/tpeA=="], + "playwright": ["playwright@1.62.1", "", { "dependencies": { "playwright-core": "1.62.1" }, "optionalDependencies": { "fsevents": "2.3.2" }, "bin": { "playwright": "cli.js" } }, "sha512-0M+L3LAD8/nm554LOla9Ayx0j0tmFZ0FBcoQ7F1VuVHpM/XpiC8RcDzBQB8W5+hA8L22THxELzeF+2WcUzvcLg=="], "playwright-core": ["playwright-core@1.54.1", "", { "bin": { "playwright-core": "cli.js" } }, "sha512-Nbjs2zjj0htNhzgiy5wu+3w09YetDx5pkrpI/kZotDlDUaYk0HVA5xrBVPdow4SAUIlhgKcJeJg4GRKW6xHusA=="], @@ -3209,7 +3209,7 @@ "path-scurry/lru-cache": ["lru-cache@10.4.3", "", {}, "sha512-JNAzZcXrCt42VGLuYz0zfAzDfAvJWW6AfYlDBQyDV5DClI2m5sAmK+OIO7s59XfsRsWHp02jAJrRadPRGTt6SQ=="], - "playwright/playwright-core": ["playwright-core@1.60.0", "", { "bin": { "playwright-core": "cli.js" } }, "sha512-9bW6zvX/m0lEbgTKJ6YppOKx8H3VOPBMOCFh2irXFOT4BbHgrx5hPjwJYLT40Lu+4qtD36qKc/Hn56StUW57IA=="], + "playwright/playwright-core": ["playwright-core@1.62.1", "", { "bin": { "playwright-core": "cli.js" } }, "sha512-wPYSwEBJY9GHraISXqyqtx0na0LpO3XEX7jNDhntbex7tzUS7kLnZsOlFruFJB4Hi/rhDMjXGqHewDZ68nYZVw=="], "pretty-format/ansi-regex": ["ansi-regex@5.0.1", "", {}, "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ=="], diff --git a/docs/reference/commands.md b/docs/reference/commands.md index dc253a3c..fc99a734 100644 --- a/docs/reference/commands.md +++ b/docs/reference/commands.md @@ -614,27 +614,43 @@ Run it as `npx explorbot prima <command>` or through the standalone `prima` bin. | Command | Purpose | |---|---| +| `prima check <scenario>` | Run a scenario end to end as a test, verify it, and report the steps it took | +| `prima do <instructions...>` | Run high-level instructions tester-style, one argument per instruction | | `prima pw <fn>` | Run a Playwright function expression against the open page | -| `prima do <steps...>` | Run several described steps tester-style, one argument per step | -| `prima click <target>` | Click an element described in plain words | -| `prima fill <field> <value>` | Fill a field described in plain words | | `prima ask <question>` | Answer a question about the current page | | `prima verify <assertion>` | Assert a statement about the current page (alias: `assert`) | | `prima research` | Map the current page and return verified locators | | `prima go <target>` | Navigate to a url, a path, or a page described in plain words | +| `prima status <hash>` | Show the artifacts and page detail recorded for an earlier command | +| `prima report` | Turn every command of a session into one html and markdown report | +| `prima config` | Show the AI models prima runs on and the config file they come from | | `prima browser {start\|stop\|status\|list}` | Manage the browsers prima drives | ### Choosing a command -Pick by what you hold, not by how hard the step looks. +Start at the top and come down only when the tier above cannot hold the work. +- **`check`** takes an outcome rather than a click path, works out how to reach it, verifies it itself, and reports every step with its proof. +- **`do`** takes several described instructions and runs them tester-style in one process. - **`pw`** is precise: a function expression built from a locator you already verified. No AI on the happy path, so it also works when no model is configured. -- **`click` and `fill`** take one action described in words and let AI resolve it against the current page. -- **`do`** takes several described steps and runs them tester-style in one process. -Never pass a locator or a function expression to `click`, `fill`, or `do` — describe the target. Never pass a description to `pw` — it takes executable code only. +Never pass a locator or a function expression to `check` or `do` — describe the target. Never pass a description to `pw` — it takes executable code only. -The loop that works: `go` to the page, `research` it once for verified locators, drive it with `pw`, then `verify` the outcome. Fall back to `click`, `fill`, or `do` whenever research left you no locator to hold. +Pass `do` the whole remaining sequence rather than one instruction per call. Every command is a process of its own, so a sequence split across calls pays the startup and page-capture cost each time. `check` and `do` legitimately run for minutes. + +### `check`, `do`, and `verify` in detail + +`check` takes `--expected <outcome>`, repeatable for several; without it the scenario text is the single expected outcome. Each comes back under `### Expected outcomes` as `PASSED`, `FAILED` or `not verified` — "not verified" means the run never checked it, which is not the same as false. Page problems seen along the way appear under `### Answer` rather than as step failures. + +```bash +prima check "signup rejects a duplicate email" \ + --expected "an error names the email as taken" \ + --expected "no second account is created" +``` + +`do` numbers every instruction and accounts for it: `### Steps` reports each as `ok` or `FAIL` with what proved it, and one that could not be carried out fails the command. Nothing runs past the last instruction given. + +`verify` lists every assertion it could express as `PASSED` or `FAILED` with its Playwright form, and gives no overall verdict — read the lines and decide. `none ran` means the claim could not be expressed at all, which is not the same as false. ### The envelope @@ -657,20 +673,18 @@ ariaDiff: - button "Submit" ### Instance -instance: default (1 tab) | other instances: none -browser: attached (playwright-cli session "default", workspace /home/you/projects/shop) +default (1 tab) | attached to playwright-cli session "default" | details: prima status 7f3a91 ### Artifacts aria: /home/you/.explorbot/sites/app.example.com/output/prima/2026-08-04T10-04-22-285Z/aria.yml html: /home/you/.explorbot/sites/app.example.com/output/prima/2026-08-04T10-04-22-285Z/page.html -network: /home/you/.explorbot/sites/app.example.com/output/prima/2026-08-04T10-04-22-285Z/network.jsonl ``` -`used:` is code that already executed. For `click`, `fill`, `do`, and `go` those are CodeceptJS steps you can copy into a test as they are; for `pw` it is the Playwright expression you passed, which a CodeceptJS test needs wrapped in `I.usePlaywrightTo(...)`. Log lines can precede the envelope, so start parsing at the first `###` line. +`used:` is code that already executed: for `go` the CodeceptJS step it ran, for `pw` the Playwright expression you passed, which a CodeceptJS test needs wrapped in `I.usePlaywrightTo(...)`. Log lines can precede the envelope, so start parsing at the first `###` line. -`ask`, `research`, and `verify` replace the `### Changes` block with `### Answer`, `### Research`, or `### Verdict`. A failure adds `### Failure` with the error and the compact ARIA of the page, so you can retarget from the envelope itself instead of opening the artifact files. +`### Changes` renders on every action envelope, saying `no change` when the tree is identical — so a successful command proves what it did instead of leaving you to check. `check` and `do` report per step rather than in aggregate: `### Steps` names each step with the code it ran and what proved it, and `page after each step:` points at the captures. `check` adds `### Expected outcomes`; `ask`, `research`, and `verify` add `### Answer`, `### Research`, or `### Assertions`; `pw` adds `### Value` when its expression returns one. `network:` appears under `### Artifacts` only when requests were captured, and everything else recorded for a command is behind `prima status <hash>`. -When an action fails, AI retries it along a different route; `healed: true` means the outcome was reached another way and `used:` holds the code that worked. `--no-heal` skips that and fails fast. +**A failed action is a failure.** Nothing is retried along a different route and no other element is substituted, so `ok: true` means the action you asked for is the one that landed. A failure adds `### Failure` with the error and the compact ARIA of the page, so you can retarget from the envelope itself instead of opening the artifact files. ### Browsers and sessions @@ -694,10 +708,9 @@ Every command takes these: | `-i, --instance <name>` | Which prima-owned browser to talk to; parallel work needs one each | | `--session [file]` | Cookies and storage persisted across processes; ignored while attached, since the attached session keeps its own | | `--url <url>` | Page to open when the session has no page yet | -| `--no-heal` | Fail immediately instead of letting AI retry a failed action | | `--ephemeral` | Keep no state between runs. Applies to config-free runs only — with a config file the output directory comes from the config | | `--framework <name>` | Parsed but not active yet; reported code is CodeceptJS whatever you pass | -| `-v, --verbose`, `--debug`, `-c, --config <path>`, `-p, --path <path>` | As on every other Explorbot command | +| `-c, --config <path>`, `-p, --path <path>` | As on every other Explorbot command | `--instance` and `--session` answer different questions: `--instance` picks *which browser process* prima drives, `--session` decides *whose cookies* it starts from. @@ -705,6 +718,7 @@ A few commands add their own: | Command | Option | Description | |---|---|---| +| `check` | `--expected <outcome>` | An outcome the run must reach; repeat the flag for several | | `ask` | `--no-vision` | Answer from page structure only, without a screenshot | | `research` | `--data` | Include data extraction in the map | | `research` | `--deep` | Expand hidden elements for a deeper map | @@ -712,6 +726,29 @@ A few commands add their own: | `browser start` | `-s, --show` / `--headless` | Launch the browser with or without a window | | `browser stop` | `--all` | Stop every running instance | +Prima carries no logging flags of its own. `DEBUG` in front of a command prints everything the run does — config and browser attachment, every step as it executes, and the debug stream of the agents behind it: + +```bash +DEBUG='explorbot:*' prima do "open the account menu" "switch the theme to dark" +``` + +Without it, a running command writes its current activity to stderr as a single line that each new activity overwrites, erased before the envelope is printed. Piped or captured output is unaffected. + +### Session reports + +Every command is logged to `output/prima/sessions/` as it runs, so `prima report` needs no browser and outlives the session: + +```bash +prima report # the session used most recently +prima report --pw-session my-app # a named playwright-cli session +``` + +It writes one html and one markdown report — each command with its steps, expected outcomes and the proof recorded for them. The log is written in the format the Testomat.io reporter replays, so the same file can be sent as a run: + +```bash +TESTOMATIO=<apiKey> npx @testomatio/reporter replay <the path prima report prints> +``` + ### Without a config file Prima follows the same [configuration ladder](#environment-variables) as every other command, so a provider name in the environment is enough: diff --git a/docs/superpowers/reviews/2026-08-06-prima-vs-playwright-cli.md b/docs/superpowers/reviews/2026-08-06-prima-vs-playwright-cli.md new file mode 100644 index 00000000..c5baecb1 --- /dev/null +++ b/docs/superpowers/reviews/2026-08-06-prima-vs-playwright-cli.md @@ -0,0 +1,268 @@ +# Prima boat vs playwright-cli — field review + +**Date:** 2026-08-06 +**Target:** Testeiya Agent (`localhost:3050`) — sidebar Workflows section + Skill editor +**Setup:** global config (`~/.explorbot/config.js`), no project config +**Models:** `groq/openai/gpt-oss-20b` (model), `openrouter/gpt-5.6-luna` (vision + agentic) +**Comparator:** `@playwright/cli` 0.1.13 +**Method:** drive the feature with prima, fall back to playwright-cli whenever prima stalled + +--- + +## Verdict up front + +Prima is not a replacement for playwright-cli today. It is a strong *complement*: two of its +commands (`verify`, `ask`) do work playwright-cli cannot do at all, and four defects stop it +from owning the driving loop. Recommended split as it stands: + +- **playwright-cli** — attach, navigate, read page state, drive verified locators. +- **prima** — `verify` for assertions, `ask` for visual judgement, `--no-heal` for a failure + report you can retarget from. + +--- + +## The feature under test — it works + +Everything the sidebar Workflow and Skill editor promises checked out. + +**Workflows section** + +- The rail button opens a Workflows panel with five categories: Analysis & Planning, + Test Design & Management, Test Execution & Automation, Reporting/CI-CD & Quality Gates, + Metrics/Release & Analytics. +- Accordion is single-open; the first category is expanded by default and lists its four + prompts (Review requirements, Risk-based focus, Analyze PR requirements, Analyze PR diff). +- "Workflow overview" opens the diagram dialog. Vision check and my own screenshot agree: + a clean left-to-right five-stage pipeline, arrows between stages, nothing clipped or + overlapping. + +**Skill editor** + +- Skills popover lists 42 skills with category filter chips. +- Row actions menu offers "Change skill globally", "Edit for this project", "Disable". +- The editor opens with the skill markdown loaded, Save disabled. +- Typing flips the header to `● unsaved` and enables Save. +- "Close editor" discards: no file appeared under `~/.testeiya/skills/`, and the agent + repo's `git status` stayed clean. + +One cosmetic thing worth a look: the filter chips include single-item categories named +after the skill itself — "Playwright Best Practices Skill 1", "Playwright Cli 1" — next to +real groupings like "Test Management 12". Category derivation looks like it falls back to +the skill name when a skill has no category. + +--- + +## Where prima was better + +**1. `verify` — the standout.** One command turns a sentence into a pass/fail *and* the +assertion code that proved it. + +``` +prima verify "the sidebar Workflows section lists five workflow categories, and the + Analysis & Planning category is expanded showing individual workflow prompts" +→ 14.8s, passed: true + I.seeElement({"role":"button","text":"Analysis & Planning"}); + ... all five categories ... + I.see('Review requirements', '#base-ui-_r_2al_'); ... all four prompts ... +``` + +With playwright-cli the same check is snapshot → read the tree myself → write five +assertions by hand. Prima did it in one call, and the output is reusable test material. +This is the command that justifies the boat. + +**2. `ask` — visual judgement with no image in my context.** + +``` +prima ask "Is the workflow diagram laid out as a readable left-to-right pipeline, + or is anything visually broken, overlapping or cut off?" +→ 8.7s, correct answer +``` + +I checked it against the PNG afterwards and it was right. playwright-cli can only hand me +a file; *I* have to look at it, and the image lands in my context permanently. + +**3. Descriptions outlive refs.** playwright-cli refs (`e13`, `e26`, `e647`) are snapshot-scoped. +This app re-renders constantly, so every interaction meant re-snapshotting for fresh refs — +8k bytes each time. `prima click "the Workflows button in the left vertical sidebar rail"` +needs no prior read at all. + +**4. Artifacts stay on disk.** Every envelope wrote `aria.yml` / `page.html` and cited absolute +paths. `page.html` was 376K and never entered my context. That is the design goal, delivered. + +**5. The `--no-heal` failure envelope is exactly right.** + +``` +prima pw '({page}) => page.click("[data-test=nonexistent-thing]", {timeout:3000})' --no-heal +→ 8.6s, ok: false + error: page.click: Timeout 3000ms exceeded. Call log: waiting for locator(...) + ### Current page (compact ARIA) ← enough to retarget, without the full tree +``` + +Error plus a compact ARIA snapshot in one response is better than what playwright-cli gives +on a failed click, which is the error alone. + +--- + +## Where playwright-cli was better + +**1. Speed — one to two orders of magnitude.** `snapshot` returned in **0.2–0.3s**. Prima's +cheapest command was 6.1s; the median was ~12s; `go` took **3m00s** and one owned-browser +`verify` took **2m32s**. Across ~17 prima commands I spent roughly ten minutes waiting. + +**2. It does exactly what you asked — nothing else.** `prima do` given two instructions +("close the dialog", "open the Skills menu") performed both and then kept going: searched the +skill list for "playwright-cli" and inserted the skill, leaving `/playwright-cli` typed into a +live chat box. Two instructions in, four actions out, `ok: true`. Prima's `go` did the same +during recovery — it clicked "Workspace", "Show panel", pressed F5 and clicked "Cancel" while +merely trying to navigate. playwright-cli has never once done something I did not type. + +**3. Attach just works.** playwright-cli opened and reattached to its own browser with no +ceremony. Prima could not attach at all without hand-patching a file (details below). + +**4. `eval` reaches state prima cannot express.** The one assertion prima got wrong — +"is Save enabled now?" — was a one-liner for playwright-cli: + +``` +$PW eval '() => [...document.querySelectorAll("button")].filter(b=>/Save/.test(b.textContent)) + .map(b=>({t:b.textContent.trim(), disabled:b.disabled}))' +→ [{ "t": "saveSave", "disabled": false }] +``` + +playwright-cli's snapshot also carries `[disabled]`, `[pressed]`, `[active]` inline. Prima's +UI map carries none of that. + +**5. Honest failures.** playwright-cli never told me an action succeeded when it hadn't. +Prima did (see #1 below). + +**6. `research` costs more context than a snapshot, not less.** The one `research` call +returned ~5k tokens of UI-map markdown — versus 8,110 bytes (~2k tokens) for a full +playwright-cli snapshot of the same page. Its locators were also worse: generated Base-UI ids +(`button#base-ui-_R_qcmpbmulb_`) and positional CSS chains +(`div:nth-of-type(1) > div > div:nth-of-type(1) > div`), neither of which survives a re-render. +Prima's own compact-ARIA failure block is the cheaper, better read. + +--- + +## Defects found in prima, worst first + +**1. Heal converts a failed action into a different action and reports success.** + +``` +prima pw '({page}) => page.click("[data-test=nonexistent-thing]", {timeout:3000})' +→ ok: true + healed: true (recovered after 1 attempt) + used: // Click the "New agent" button + I.click({ role: "button", text: "New agent" }); +``` + +The requested element does not exist. Heal did not find another route to the same intent — +there was no such intent to reach. It picked an unrelated control, clicked it, and returned +`ok: true`. An orchestrator that trusts the envelope believes its own action landed. Heal +should only re-route toward the *same* target; when the target cannot be identified, the +correct answer is the `--no-heal` envelope. + +**2. `verify` false-negatives with no diagnostic.** + +``` +prima verify "the Save button in the skill editor is now enabled because the skill + content has unsaved changes" +→ passed: false + evidence: no assertion held on the current page + code: (empty) +``` + +The app was working — `● unsaved` was displayed and Save was enabled. + +The cause is the vocabulary, not the model. `rules/navigator/verification-actions.md` offers +exactly nine assertions — `see`, `seeElement`, `seeInField`, `seeInTitle`, `seeInSource` and +their `dontSee*` counterparts — none of which express enabled/disabled/checked/selected, and +the rules additionally steer away from attribute selectors (`NEVER use ':has-text(...)'`, +"never check value via CSS attribute selectors"). So a claim about interactive state is +unprovable by construction, and prima reports that as the *feature* failing rather than as a +check it could not express. Empty `code:` and a generic evidence line leave no way to tell +"the app is broken" from "I could not phrase this". + +**3. Auto-discovery of a playwright-cli session cannot work; `--endpoint` needs playwright 1.62.** + +> **Corrected 2026-08-07 after re-testing.** The original run was on explorbot's pinned +> playwright **1.60.0** against a 1.61.0-alpha daemon, and concluded attach was broken +> outright. That was a stale-dependency artifact. Re-tested with `playwright@1.62.1` and +> `@playwright/cli@0.1.17` (playwright 1.62.0-alpha): `chromium.connect()` succeeds with +> **our own** playwright, and `prima verify --endpoint <sock>` returns `ok: true` in 16.6s. +> Upgrading the pin fixes the connect half. What remains is discovery. + +- **Version skew (fixed by upgrading).** `playwright@^1.60` could not `connect()` to a + 1.61/1.62 browser server — it timed out. `playwright@1.62.1` connects to both its own + minor and the 1.62.0-alpha daemon. The pin should move to `^1.62`. +- **Missing `workspaceDir` (still broken).** No `@playwright/cli` release writes a + `workspaceDir` field into `~/.cache/ms-playwright/b/browser@<guid>` — verified on 0.1.13 + and 0.1.17. `parseDescriptor` (`boat/prima/src/pw-registry.ts:56`) requires it, so every + descriptor is dropped, `selectDescriptor` filters on a field that never exists, and + auto-discovery finds nothing no matter what version is installed. Prima cannot attach + without `--endpoint` unless it stops keying on `workspaceDir`. +- **The failure message sends you in a circle.** With a session open, discovery fails with + "No browser to drive... Open one first: `playwright-cli open <url>`" — advising exactly + what the user already did. `prima browser list` likewise reports "no browser instances + running" while a session is live. + +**4. A redirect that appends query params is treated as failed navigation.** The app sends +`/` → `/?session=<uuid>&ws=1`. `prima go http://localhost:3050` burned **3 minutes** and eight +attempts before healing; against prima's own browser the same navigation ended in a hard tool +error after **2m32s** — + +``` +error: tool: Navigation to / failed: redirected to /?session=...&ws=1 and could not resolve +``` + +— in an envelope whose own inlined ARIA proves the page had loaded correctly. Session-param +redirects are common enough that this alone blocks unattended use. + +### Smaller issues + +- **`used:` is often not runnable.** `click` concatenated all five ladder attempts, including + invalid JS: `I.click(".sidebar button:has-text("Workflows")")`. `do` emitted a + `// 1. Open dialog...` comment line inside the code. The spec promises "the exact code that + worked" — it should be the winning line only. +- **`### Changes` / ariaDiff never appeared** in any successful envelope. That block is the + envelope's core evidence promise; without it, a successful `click` proves nothing and I had + to spend a playwright-cli snapshot to confirm every action. +- **`network.jsonl` is advertised in every envelope and was 0 bytes in all 18 runs.** +- **`click` reports itself as `do`.** Every `click` envelope printed `command: do "..."`. +- **Every command requires a URL** even when attached to a browser already sitting on the page, + and even when the site is registered. `EXPLORBOT_URL` satisfies config loading but not page + opening on an empty owned browser, which needs `--url` as well. +- **Prima pollutes a shared browser.** After `research`, the visual-annotation overlays + (`Legend`, `e8`, `e10`, …) were still in the live DOM and showed up in the next + playwright-cli snapshot. + +### Environment friction (not prima's design) + +- **`node_modules` is a committed, self-referential symlink.** `git ls-files -s node_modules` + shows mode `120000` pointing at `/home/davert/projects/explorbot/node_modules` — itself. + It was added in `7cc52eb` ("Let EXPLORBOT_* variables win over the global config"). Being + tracked, it overrides the `node_modules/` line in `.gitignore`. Every Node resolution fails + with `ELOOP` / "Too many levels of symbolic links", so `npx tsc`, the build, and the CLI are + all dead on a fresh checkout of this branch. Removing it and running `bun install` fixes it. +- `bun run build:npm` fails with `env: unknown error: execvp failed`; `bash scripts/build-npm.sh` + works. +- The `prima` bin is not exposed by the existing global npm link — `explorbot prima ...` only. +- Prima needs the Node build for browser-server endpoints, so `dist/` must exist before any of + this runs. + +--- + +## What would make prima a replacement + +In priority order: + +1. Heal must never substitute a different target; unresolvable intent → failure envelope. +2. `verify` must distinguish "assertion failed" from "cannot express this assertion", and + `rules/navigator/verification-actions.md` needs state assertions + (enabled/disabled/checked/selected) alongside the nine text/presence ones it has now. +3. Treat a redirect that preserves origin and path as navigation success. +4. Move the playwright pin to `^1.62` (fixes `--endpoint`), and stop keying discovery on + `workspaceDir` — no `@playwright/cli` release emits it. +5. Emit `### Changes` on every action, and reduce `used:` to the winning line. + +With 1–3 fixed, prima could own the assertion and inspection half of a session outright. +Driving would still belong to playwright-cli until the per-command latency comes down. diff --git a/docs/superpowers/specs/2026-08-07-prima-fixes-design.md b/docs/superpowers/specs/2026-08-07-prima-fixes-design.md new file mode 100644 index 00000000..992d4021 --- /dev/null +++ b/docs/superpowers/specs/2026-08-07-prima-fixes-design.md @@ -0,0 +1,394 @@ +# Prima Fixes — Perception Ladder, No Heal, Proof-Carrying Envelopes + +**Date:** 2026-08-07 +**Status:** Draft for review +**Supersedes parts of:** `2026-08-01-prima-boat-design.md` +**Evidence:** `docs/superpowers/reviews/2026-08-06-prima-vs-playwright-cli.md` + +## Problem + +A field run of prima against a live app (Testeiya, `localhost:3050`) found the boat working but +not trustworthy as an executor. Four defects matter: + +- A `pw` call on a selector that does not exist returned `ok: true`, `healed: true`, having + clicked an unrelated control ("New agent"). Heal substituted the target and reported success. +- `### Changes` never appeared in any successful envelope, so a successful action proved + nothing and every step had to be confirmed with a playwright-cli snapshot. +- `do` given two instructions performed four, typing into a live chat box actions nobody asked + for. +- Auto-discovery of a playwright-cli session never matched, while the failure message advised + opening the session the user already had open. + +Underneath the third defect is a perception problem. `action.ts:139` captures +`page.locator('body').ariaSnapshot()` **without** `mode: 'ai'`, so `do` sees roles and text but +no element handles and must invent locators. That is why the `click` tool's schema demands +"multiple commands targeting the SAME element" and why one click produced five ladder attempts, +one of them invalid JavaScript. + +## Goals + +- Prima is the executor for an expensive orchestrator: page data never enters that context on + the happy path, intent is never guessed, and every action carries proof of what changed. +- `do` acts on element handles it was given, not on locators it invented. +- Failures are failures. No code path may reach a different element than the one asked for. +- Attachment to an existing playwright-cli session works without hand-editing files. + +## Non-Goals + +- Replacing playwright-cli. It remains the fallback for direct driving and the tool prima + points back to when it cannot help. +- Reducing per-command latency, and improving `research` locator quality. Separate concerns. +- Changing Tester, Navigator, or Researcher behaviour outside the shared pieces named below. + +--- + +## 1. Perception ladder for `do` + +Four tiers, tried in order. Each tier answers "what can I act on here?" at a different cost. + +| Tier | Source | When | +|---|---|---| +| 1 | Research UI map | a stored map exists for this state hash and the state is well-visited | +| 2 | **ARIA snapshot with refs** | default | +| 3 | Compact HTML tree | a ref action failed, or the control is absent from the ARIA tree | +| 4 | Vision (`visualClick`) | click only, when the target is identifiable solely by appearance | + +### Tier 1 — research map + +Read through `getPreviousResearch(hash)` — a TTL-free disk read. Not `getCachedResearch`, whose +six-hour TTL carries session-scoped semantics that must not be stretched for cross-invocation +reuse. The trigger is the StateManager visit count for the current state hash, the same number +the envelope already prints as `visit #N`: use the map from the **third visit onward**, so a +state has to prove it is worth the map before prima prefers one. Below that, skip to tier 2. +The threshold is a single config field under `ai.agents.prima`, per the one-knob convention. + +Research cost is real — 41s on the vision model in the field run — so tier 1 only pays off +amortized across repeat visits to one state. Prima does not run research on the caller's behalf +inside `do`; it consumes a map that already exists. + +### Tier 2 — ARIA snapshot with refs (the default) + +`page.locator('body').ariaSnapshot({ mode: 'ai' })` emits `[ref=eN]` handles. Playwright resolves +them natively through the `aria-ref=` selector engine, verified live against the running app: + +``` +aria-ref=e1 → 1 match +``` + +No DOM mutation, no attribute stamping. The model is given the ref-bearing tree and names a ref; +resolution to a CodeceptJS command is §1.1. One ref becomes one command and the +multiple-locator fallback ladder collapses. + +**Ref lifetime is one context injection.** Refs are snapshot-scoped and shift between calls — +in the field run the same sidebar button was `ref=e13` in one snapshot and `ref=e284` in a later +one on the same page. Prima re-snapshots whenever the `do` loop re-injects context on state-hash +change (the hook exists at `prima.ts:104`), and the prompt states that refs from an earlier +injection are dead. + +**Ref shape.** Refs are frame-qualified: every ref this app emits is `f1e13`-shaped, not `e13`. +A validator that accepts only `e\d+` rejects every real ref and silently disables the whole tier, +so the accepted shape is `(f\d+)?e\d+`. Refs are never adapted or invented — a ref that does not +parse, or that resolves to nothing, is a failure with its own message. + +### Tier 3 — compact HTML tree + +The current `simplifiedHtml` path. Entered when a ref action fails or the target is not +represented in the accessibility tree. + +### Descending the ladder — `context()` + +The tiers above are only reachable if something can move between them mid-run. The `do` loop +re-injects context between iterations when the state hash changes, which leaves no way to +recover from a ref that died *within* an iteration — the model's only remaining moves would be +to guess a locator, which the prompt forbids, or to stop. + +`do` therefore carries a `context` tool. First call returns the page as it is now with fresh +refs, which replace every ref the model was holding; a later call on the same page drops to +capped markup for elements the accessibility tree does not describe. It is the tier descent, not +a page dump. + +This is deliberately **not** the `context` tool from `createAgentTools`. That one returns +`getInteractiveARIA()` — `compactAriaSnapshot` over `ActionResult.ariaSnapshot`, captured without +`mode: 'ai'`, so it carries **no refs** — plus a 6k-char HTML dump beside it. Offering it here +would hand the model a ref-less tree and push it straight back to guessed locators, undoing +tier 2. + +### Tier 4 — vision + +`visualClick` (`src/ai/tools.ts:791`), already implemented. Click only — a coordinate is not a +handle, so it cannot serve fills, selects, or assertions. + +### 1.1 Acting on a ref — `clickRef` / `hoverRef` in `boat/prima/src/tools.ts` + +CodeceptJS has no `aria-ref` locator, so a ref cannot be handed to `I.click()` directly. It is +resolved to a locator CodeceptJS does understand, using machinery that already exists: + +``` +page.locator(`aria-ref=${ref}`) + → WebElement.fromPlaywrightLocator(...) (web-element.ts:112) + → I.click(<xpath>) +``` + +**Which XPath, and the one that is not available.** `WebElement` declares two — `clickXPath`, +built attribute-first by `buildClickableXPath` (`utils/xpath.ts`), and `xpath`, the absolute +positional path. On this path only `clickXPath` exists: `fromPlaywrightLocator` builds through +`fromRawData`, which hardcodes `xpath: ''` (`web-element.ts:89`). Only `fromXPathMatch`, the +static-HTML path, populates the absolute form. So there is no positional fallback to reach for, +and that is the right outcome anyway — an absolute path is deduplication machinery, fragile the +moment the DOM shifts, and the ref resolution and the click happen on separate round-trips. + +Verified live, ref → `clickXPath` → match count: + +``` +e15 link "Checking the proxy…" //*[self::a and contains(.,"Checking the proxy and the firewall")] → 1 +e19 button "Reload" //*[@id="reload-button"] → 1 +e20 button "Details" //*[@id="details-button"] → 1 +``` + +**Require a unique match.** Because there is no fallback, the resolved `clickXPath` is checked +to match exactly one element before it is used. Zero or many is a failure, reported as such — +never a click on an ambiguous match and never a retreat to guessed locators. + +Ref acting lives in **separate tools in the boat** — `clickRef` and `hoverRef`, from a +`createRefTools` factory in `boat/prima/src/tools.ts`. `click` and `hover` in `src/ai/tools.ts` +are left byte-identical. A prima-only tool belongs to prima; core keeps only what every caller +uses, and lends the boat its result-shaping helpers (`successToolResult`, `failedToolResult`, +`commitNote`) rather than having them copied. + +Adding an optional `ref` to the existing tools looks cheaper and is wrong. A tool's schema and +description are shared with every caller, and Tester never receives ref-bearing snapshots, so it +would be shown a parameter it can only fill by inventing one. Making `commands` optional to +accommodate the new field weakens the contract for Tester too. "Prima enables ref mode, Tester is +untouched" is not achievable through one definition: there is one definition, and Tester sees it. + +Each ref tool takes a ref and nothing else, resolves it, executes one command, and reports that +command as `used:` — real CodeceptJS a generated test can keep. A ref that does not resolve is a +failure, not a cue to fall back to a guessed locator: it means the context is stale. + +`hoverRef` exists for the same reason as `hover` — revealing hover-only controls is a +prerequisite for clicking them. + +Prima keeps the locator tools alongside the ref tools: tier 3 works from markup that carries no +refs, and needs them. Tier 4 keeps the coordinate input. + +--- + +## 2. Heal is deleted + +`heal()` (`prima.ts:428-460`) is removed, along with `--no-heal`, `PrimaOptions.heal`, the +`healed:` / `healNote` envelope fields, the `HealAttempt` type, and the +`### Healing attempts` section. The three call sites — `pw` (`:79`), `do` (`:135`), `go` (`:211`) +— go straight to `failureEnvelope`. + +The failure envelope already does the right thing and becomes the only failure path: `ok: false`, +the exact error with its call log, and compact ARIA inline. Measured at 8.6s in the field run +against 34.8s for the heal path that got the answer wrong. + +This removes the substitution defect at the root. No remaining code path can select an element +other than the one asked for, so no envelope can report success for an action the caller did not +request. Routine obstructions — an overlay covering a button, an element not yet visible — now +return the failure envelope, and the orchestrating model decides. That is the accepted cost: the +compact ARIA block is the one place page data deliberately enters the expensive context. + +--- + +## 3. Proof-carrying envelopes + +Two defects with one cause. `renderOutcome` (`envelope.ts:76`) returns the **first** of +changes / answer / research / verdict, so `### Changes` is structurally impossible alongside +`### Verdict`, `### Answer`, or `### Research`. And `pageChanges` returns `ariaChanges ?? null`, +which renders nothing when the diff is empty or `previousState` is null. + +- `### Changes` renders on every action envelope, showing `no change` explicitly when the tree is + identical. A caller can then tell "nothing happened" from "prima did not say". +- `renderOutcome` stops being mutually exclusive: `### Changes` renders alongside the command's + own outcome section. +- Prima captures an explicit before-snapshot rather than relying on whatever `stateManager` + holds at process start. +- New `### Steps` block for `do`: one line per instruction, each naming the ariaDiff entry that + proves it, or marked `unproven`. + +### Refs must never reach the diff or hash pipeline + +The ref-bearing snapshot is a **context artifact only**. Refs are stripped before hashing and +diffing. Storing the `mode: 'ai'` output in `ActionResult.ariaSnapshot` would poison every +`### Changes` block, because Playwright renumbers refs on each call. Measured on an identical +page whose refs merely shifted: + +``` +ref churn only, page identical → diff count = 6 + added: button "Cancel" [ref=e21], button "Save" [ref=e20], textbox "Name" [ref=e22] + removed: button "Cancel" [ref=e11], button "Save" [ref=e10], textbox "Name" [ref=e12] +``` + +Six phantom entries for a page that did not change. Capture ref-bearing and ref-free variants, +and feed only the ref-free one to `diffAriaSnapshots` and `getStateHash`. + +--- + +## 4. Executor prompt + +`instructionSystemPrompt` (`prima.ts:493`) already says to stop when every instruction is done, +but nothing tracks per-instruction completion, so the loop runs until the model stops calling +tools. Two rules are added, stated as general principles rather than as counter-examples from +any debugging session: + +- Act only on the listed instructions. An adjacent action that appears helpful is out of scope; + report it as an observation instead of performing it. +- For each instruction, cite the observed page change that proves it. An instruction that cannot + be tied to an observed change is reported unproven rather than claimed as done. + +The prompt also states the ref contract: act on refs from the current context injection; refs +from an earlier injection are dead. + +--- + +## 5. Attachment and discovery + +Three separate faults, all confirmed empirically. + +**Connect with the daemon's own build when attached.** `connectDescriptor` (`prima.ts:374`) +tries prima's own playwright first and only falls back to `descriptor.playwrightLib` if connect +fails. Connect *succeeds* across builds, so the fallback is never reached — and then the tier-2 +snapshot breaks: + +| client lib | `connect()` | `ariaSnapshot({mode:'ai'})` | +|---|---|---| +| own playwright 1.62.1 | ok | `locator.ariaSnapshot: timeout: expected float, got undefined` | +| daemon playwright-core 1.62.0-alpha | ok | ok, 3376 bytes, `aria-ref=e1` → 1 match | + +In attached mode, prefer `descriptor.playwrightLib` when present and fall back to our own. Carry +`playwrightLib` through the `--endpoint` path too, which currently hardcodes `''`. + +**Stop keying discovery on `workspaceDir`.** No `@playwright/cli` release writes that field — +verified on 0.1.13 and 0.1.17. `parseDescriptor` (`pw-registry.ts:56`) requires it and +`selectDescriptor` filters on it, so every descriptor is dropped and discovery finds nothing no +matter what is installed. Resolution becomes: `--endpoint` → `--pw-session <title>` → +`PLAYWRIGHT_CLI_SESSION` → live descriptor titled `default` → the single live descriptor → +tool error listing candidate titles. Liveness-probe before selecting. + +**Bump the playwright pin to `^1.62`.** `playwright@^1.60` could not `connect()` to a 1.61/1.62 +browser server at all; 1.62.1 connects. The pin is the reason the original review concluded +attach was broken outright. + +Consequences: `browser list` shows attachable sessions rather than reporting none while one is +live, and the no-browser error stops advising `playwright-cli open <url>` to someone who already +ran it. + +--- + +## 6. `verify` honesty + +`rules/navigator/verification-actions.md` offers nine assertions — `see`, `seeElement`, +`seeInField`, `seeInTitle`, `seeInSource` and their `dontSee*` counterparts — none of which +express enabled, disabled, checked, selected, or expanded. A claim about interactive state is +therefore unprovable by construction, and the field run reported a working feature as failing: + +``` +verify "the Save button in the skill editor is now enabled ..." +→ passed: false, code: (empty), evidence: no assertion held on the current page +``` + +The app was correct — `● unsaved` was displayed and the button was enabled. + +- Add state assertions to the rule. +- `verify` distinguishes **assertion failed** from **could not express this assertion**. The + second is not a test failure and must not be reported as one. +- With §7's compaction fix, `verify` reads state from the same ref-bearing snapshot the tier-2 + ladder produces. + +--- + +## 7. Prerequisite fix — ARIA compaction drops refs and state + +`compactAriaSnapshot` keeps only the first bracket group on a line. Playwright emits state +attributes before `[ref=]`, so every stateful control loses its handle. Measured: + +``` +button "Plain" [ref=e10] → ref KEPT +button "Active" [active] [ref=e13] → ref LOST +button "Disabled" [disabled] [ref=e14] → ref LOST +button "Pressed" [pressed] [ref=e15] → ref LOST +button "Expanded" [expanded] [ref=e16] → ref LOST +checkbox "Checked" [checked] [ref=e17] → ref LOST +button "Cursor" [ref=e18] [cursor=pointer] → ref KEPT +``` + +The controls most worth acting on and asserting about are exactly the ones stripped of their +handle. **Keep every bracket group on a line.** No allow-list, no drop-list — parsing one group +and discarding the rest is the whole bug, and any rule about which groups survive re-creates it +the next time Playwright adds an attribute. This fix is a prerequisite for §1 tier 2 and enables +§6. + +--- + +## 8. Envelope hygiene + +- `used:` carries the winning line only — no concatenated ladder attempts, no comment lines. The + field run produced `I.click(".sidebar button:has-text("Workflows")")`, which is not valid + JavaScript, and a `// 1. Open dialog...` comment inside the code block. +- `click` and `fill` stop labelling themselves `do` in `command:`. +- `network.jsonl` is written and advertised only when requests were actually captured. It was + 0 bytes across all 18 runs while being advertised in every envelope; an artifact line that + points at an empty file is worse than no line. +- Research annotation overlays are removed from the DOM after use. The browser is shared with + playwright-cli, and leftover `Legend` / `e8` / `e10` nodes appeared in its next snapshot. +- Commands stop requiring a URL when attached to a browser already on a page. +- A redirect that preserves origin and path counts as navigation success. The app's + `/` → `/?session=<uuid>&ws=1` redirect cost 3m00s and eight attempts under the old heal path, + and a hard tool failure against a prima-owned browser — in an envelope whose own inlined ARIA + proved the page had loaded. + +--- + +## Testing + +- **Unit** — envelope rendering with `### Changes` always present, including the `no change` + form and the combination with `### Verdict` / `### Answer` / `### Research`; compaction + preserving every bracket group across the §7 combinations and any order of them; ref-free + diffing (the six-phantom case must yield zero); descriptor selection without `workspaceDir`, + including the ambiguous multi-session error. +- **Ref resolution** — a ref resolves to an attribute-based `clickXPath` matching exactly one + element, and still matching after unrelated siblings are added or removed; a ref that resolves + to zero or many fails rather than falling back to a guessed locator. +- **Integration** — `do` prompt behaviour through the existing `@copilotkit/aimock` harness per + `docs/contributing/ai-integration-tests.md`: instructions performed and nothing beyond them, + per-instruction proof citation, unproven reporting. Fictional fixture data only. +- **End-to-end** — against a local fixture: a ref named from the snapshot clicks the element it + names and no other; a failing `pw` returns `ok: false` with compact ARIA and never a + substituted action; attach to a live playwright-cli session with no descriptor editing; a + session-param redirect resolves as success. + +--- + +## Decisions Log + +- Heal is deleted outright rather than constrained to same-target recovery. No flag, no opt-in. +- `do` perception is a four-tier ladder: research map, ARIA with refs, compact HTML, vision for + click. +- Refs come from Playwright's native `aria-ref=` engine. No eidx attribute stamping, so the + shared browser's DOM is not mutated. +- Ref acting lives in `boat/prima/src/tools.ts`, not in core; `click` and `hover` are untouched. + Extending a shared tool would show Tester a `ref` parameter it can only fill by inventing one, + and would weaken `commands` for every caller. +- A ref resolves through `WebElement` to an attribute-based `clickXPath`. The absolute positional + `xpath` is not a fallback — `fromRawData` never populates it on the live-locator path, and it + would be the fragile choice regardless. +- A resolved `clickXPath` must match exactly one element. A dead or ambiguous ref is a failure, + never a fallback to guessed locators. +- ARIA compaction keeps every bracket group. No allow-list of attributes to preserve. +- Refs are a context artifact only, stripped before hashing and diffing. +- Ref lifetime is one context injection; the loop re-snapshots on state-hash change. +- Tier 1 reads maps via `getPreviousResearch`; the trigger is StateManager visit count. +- Discovery matches on title plus liveness; `workspaceDir` is abandoned as a key. +- Attached mode connects with `descriptor.playwrightLib` first, because connect succeeds across + builds but `ariaSnapshot` does not. +- The playwright pin moves to `^1.62`. +- `verify` reports inexpressible assertions as inexpressible, never as failures, and does not + record them as verifications — a claim that could not be checked must not be remembered as + one that failed. +- `do` carries a prima-shaped `context` tool that descends the ladder — fresh refs first, capped + markup on a repeat call. The shared `context` from `createAgentTools` is not reused: it returns + a ref-less tree and an HTML dump. +- The baseline snapshot is captured lazily when a command needs a diff, not in `start()`, which + runs before a page is loaded. diff --git a/package.json b/package.json index 0affdfea..b18b25bc 100644 --- a/package.json +++ b/package.json @@ -113,7 +113,7 @@ "ora-classic": "^5.4.2", "parse5": "^8.0.0", "pixelmatch": "^7.2.0", - "playwright": "^1.60", + "playwright": "^1.62", "pngjs": "^7.0.0", "react": "^19.1.1", "sambanova-ai-provider": "^1.2.2", diff --git a/rules/navigator/verification-actions.md b/rules/navigator/verification-actions.md index e11d7445..91c59847 100644 --- a/rules/navigator/verification-actions.md +++ b/rules/navigator/verification-actions.md @@ -105,9 +105,29 @@ Checks that page source does NOT contain expected text. I.dontSeeInSource('error-class'); </example> +### Asserting the state of a control + +State means disabled, checked, readonly, required, selected, expanded. A presence assertion says +nothing about it, so assert it with an attribute selector: + +<example> + I.seeElement('button[aria-label="Submit"][disabled]'); + I.dontSeeElement('button[aria-label="Submit"][disabled]'); + I.seeElement('input[name="accept"][checked]'); + I.seeElement('button[aria-label="Details"][aria-expanded="true"]'); +</example> + +Use I.seeElement for the state you expect and I.dontSeeElement for the state you expect to be +absent — that pair expresses both directions. + +I.seeAttributesOnElements(<locator>, { disabled: true }) also exists, but only takes a state that +must be PRESENT and does not resolve reliably against a role/text locator. Prefer the selector form above. + <verification_rules> Be strict in assertions to avoid false positives. Prefer I.seeElement() with ARIA locators - most reliable. +For a claim about a control's state, assert it with an attribute selector — presence of the element is not evidence of its state. +If no assertion above can express the claim, say so instead of proposing an assertion that checks something weaker. I.see() and I.dontSee() MUST include context parameter. For input field values, ALWAYS use I.seeInField() — never check value via CSS attribute selectors or I.seeInSource. Prefer text locators (label, name, placeholder) for form fields: I.seeInField('Search', 'value') over I.seeInField('input[name="search"]', 'value'). diff --git a/src/action-result.ts b/src/action-result.ts index 3f688c6c..e5ca7d59 100644 --- a/src/action-result.ts +++ b/src/action-result.ts @@ -30,6 +30,7 @@ interface ActionResultData extends WebPageState { iframeSnapshots?: Array<{ src: string; html: string; id?: string }>; ariaSnapshot?: string | null; ariaSnapshotFile?: string; + focusedElement?: FocusedElement | null; iframeURL?: string; links?: Link[]; } @@ -74,6 +75,7 @@ export class ActionResult implements ActionResultData { private snapshotCache = new TTLCache<string>(); readonly logFile: string | undefined = undefined; readonly ariaSnapshotFile: string | undefined = undefined; + readonly focusedElement: FocusedElement | null = null; private _ariaSnapshot: string | null | undefined = undefined; private _lastExtractedHtml: string | undefined = undefined; notes: string[] = []; @@ -107,6 +109,9 @@ export class ActionResult implements ActionResultData { if (data.ariaSnapshotFile !== undefined) { this.ariaSnapshotFile = data.ariaSnapshotFile; } + if (data.focusedElement !== undefined) { + this.focusedElement = data.focusedElement; + } // Store HTML in a private property if provided if (data.html !== undefined) { @@ -643,3 +648,9 @@ export class Diff { this._ariaChangeCount = ariaDiff.count; } } + +export interface FocusedElement { + role: string; + name: string; + value?: string; +} diff --git a/src/action.ts b/src/action.ts index 6725ad2b..76d303be 100644 --- a/src/action.ts +++ b/src/action.ts @@ -3,7 +3,7 @@ import { join } from 'node:path'; import { context, trace } from '@opentelemetry/api'; import { container, recorder } from 'codeceptjs'; import * as codeceptjs from 'codeceptjs'; -import { ActionResult } from './action-result.js'; +import { ActionResult, type FocusedElement } from './action-result.js'; import { clearActivity, setActivity } from './activity.ts'; import { ConfigParser, outputPath } from './config.js'; import type { ExplorbotConfig } from './config.js'; @@ -32,6 +32,7 @@ class Action { public playwrightHelper: any; public playwrightGroupId: string | null = null; public assertionSteps: Array<{ name: string; args: any[] }> = []; + public lastValue: unknown; private recorder?: PlaywrightRecorder; private recovery: RecoveryRunner; private mainDocumentStatus: number | undefined = undefined; @@ -133,10 +134,12 @@ class Action { let ariaSnapshot: string | null = null; let ariaSnapshotFile: string | undefined = undefined; + let focusedElement: FocusedElement | null = null; try { const page = this.playwrightHelper.page; ariaSnapshot = await page.locator('body').ariaSnapshot(); + focusedElement = await page.evaluate(readFocusedElement); } catch (err) { debugLog('ARIA snapshot failed:', err instanceof Error ? `${err.message}\n${err.stack}` : err); } @@ -160,6 +163,7 @@ class Action { iframeSnapshots, ariaSnapshot, ariaSnapshotFile, + focusedElement, iframeURL: frame ? frame.url?.() || 'iframe' : undefined, }); this.stateManager.updateState(result, codeBlock); @@ -302,9 +306,10 @@ class Action { await playwrightSandbox(page, sanitizedCode); await sleep(this.config.action?.delay || 500); } else { - codeceptJSSandbox(this.actor, sanitizedCode); + const returned = codeceptJSSandbox(this.actor, sanitizedCode); await recorder.add(() => sleep(this.config.action?.delay || 500)); await recorder.promise(); + this.lastValue = await returned; } if (executedSteps.length > 0) { @@ -442,3 +447,23 @@ const detachStepLogger = (listener: StepListener) => { codeceptjs.event.dispatcher.off(codeceptjs.event.step.passed, listener); codeceptjs.event.dispatcher.off(codeceptjs.event.step.failed, listener); }; + +const readFocusedElement = () => { + const el = document.activeElement as any; + if (!el || el === document.body) return null; + + const tag = el.tagName.toLowerCase(); + const textish = new Set(['text', 'search', 'email', 'password', 'url', 'tel', 'number']); + let role = el.getAttribute('role') || tag; + if (tag === 'textarea' || el.isContentEditable) role = 'textbox'; + if (tag === 'input' && textish.has(el.type)) role = 'textbox'; + if (tag === 'select') role = 'combobox'; + if (tag === 'a') role = 'link'; + + const label = el.getAttribute('aria-label') || el.getAttribute('placeholder') || el.labels?.[0]?.textContent || el.textContent || ''; + const focused: { role: string; name: string; value?: string } = { role, name: label.trim().slice(0, 80) }; + + const value = el.value ?? el.textContent; + if (typeof value === 'string' && value) focused.value = value.slice(0, 200); + return focused; +}; diff --git a/src/ai/navigator.ts b/src/ai/navigator.ts index a596f585..94bd1979 100644 --- a/src/ai/navigator.ts +++ b/src/ai/navigator.ts @@ -8,6 +8,7 @@ import type { ExperienceTracker } from '../experience-tracker.js'; import Explorer from '../explorer.ts'; import type { KnowledgeTracker } from '../knowledge-tracker.js'; import { type StateManager, normalizeUrl } from '../state-manager.js'; +import { renderAssertion } from '../playwright-recorder.ts'; import { extractCodeBlocks } from '../utils/code-extractor.js'; import { HooksRunner } from '../utils/hooks-runner.ts'; import { createDebug, pluralize, tag } from '../utils/logger.js'; @@ -620,7 +621,7 @@ class Navigator implements Agent { return suggestion; } - async verifyState(message: string, actionResult: ActionResult): Promise<{ verified: boolean; successfulCodes: string[]; assertionSteps: Array<{ name: string; args: any[] }>; totalAttempted: number }> { + async verifyState(message: string, actionResult: ActionResult): Promise<{ verified: boolean; inexpressible: boolean; results: AssertionResult[]; successfulCodes: string[]; assertionSteps: Array<{ name: string; args: any[] }>; totalAttempted: number }> { tag('info').log('AI Navigator verifying state at', actionResult.url); debugLog('Verification message:', message); @@ -698,6 +699,7 @@ class Navigator implements Agent { let codeBlocks: string[] = []; const successfulCodes: string[] = []; + const results: AssertionResult[] = []; const assertionSteps: Array<{ name: string; args: any[] }> = []; const action = this.explorer.action(); @@ -739,6 +741,8 @@ class Navigator implements Agent { await action.exitIframe(); const verified = await action.attempt(codeBlock, message); + const proof = action.assertionSteps.map(renderAssertion).filter(Boolean); + results.push({ code: codeBlock, passed: verified, proof }); if (verified) { tag('success').log('Verification passed'); @@ -747,12 +751,6 @@ class Navigator implements Agent { } else { failures++; } - - const target = Math.min(codeBlocks.length, this.verifyAttempts); - const majorityNeeded = Math.floor(target / 2) + 1; - if (successfulCodes.length >= majorityNeeded || failures > target - majorityNeeded) { - stop(); - } }, { maxAttempts: this.verifyAttempts, @@ -773,10 +771,16 @@ class Navigator implements Agent { let verified = successfulCodes.length >= majorityNeeded; if (alreadyVerified) verified = true; + const inexpressible = !alreadyVerified && totalAttempted === 0; + if (inexpressible) { + tag('warning').log('No assertion could express this claim'); + return { verified: false, inexpressible, results, successfulCodes, assertionSteps, totalAttempted }; + } + actionResult.addVerification(message, verified); this.stateManager.updateState(actionResult); - return { verified, successfulCodes, assertionSteps, totalAttempted }; + return { verified, inexpressible, results, successfulCodes, assertionSteps, totalAttempted }; } private checkAlreadyVerified(aiResponse: string, actionResult: ActionResult): boolean { @@ -787,4 +791,6 @@ class Navigator implements Agent { } } +export type AssertionResult = { code: string; passed: boolean; proof: string[] }; + export { Navigator }; diff --git a/src/ai/pilot.ts b/src/ai/pilot.ts index 1aa6e09c..a0f8aa1b 100644 --- a/src/ai/pilot.ts +++ b/src/ai/pilot.ts @@ -8,7 +8,7 @@ import type Explorer from '../explorer.ts'; import type { PlaywrightRecorder } from '../playwright-recorder.ts'; import type { StateManager } from '../state-manager.ts'; import { type Test, TestResult } from '../test-plan.ts'; -import { collectInteractiveNodes, detectFocusArea, extractFocusedElement } from '../utils/aria.ts'; +import { collectInteractiveNodes, detectFocusArea } from '../utils/aria.ts'; import { ErrorPageError } from '../utils/error-page.ts'; import { createDebug, tag } from '../utils/logger.ts'; @@ -561,6 +561,55 @@ export class Pilot implements Agent { return text; } + async settleExpectations(task: Test): Promise<Array<{ text: string; status: 'passed' | 'failed' | 'unverified' }>> { + const undecided = task.expected.filter((text) => !task.getCheckedExpectations().includes(text)); + const decided = (text: string): 'passed' | 'failed' => { + if (task.hasAchievedAny() && !task.getRemainingExpectations().includes(text)) return 'passed'; + return 'failed'; + }; + + if (!undecided.length) return task.expected.map((text) => ({ text, status: decided(text) })); + + const schema = z.object({ + outcomes: z.array( + z.object({ + expectation: z.string().describe('The expected outcome, repeated exactly as it was given'), + status: z.enum(['passed', 'failed', 'unverified']).describe('passed = the log shows it happened, failed = the log shows it did not, unverified = the run never established either way'), + }) + ), + }); + + const userContent = dedent` + A test run has finished. Decide, for each expected outcome, what the run established about it. + + <expected_outcomes> + ${undecided.map((text) => `- ${text}`).join('\n')} + </expected_outcomes> + + <run_log> + ${task.notesToString() || 'No steps recorded.'} + </run_log> + + The log is written in the tester's own words, so an outcome can be satisfied by a step that describes it + differently. Judge by what the steps show happened, not by whether the wording matches. + Choose "unverified" only when the log neither shows the outcome happening nor shows it failing — + that is a statement about the run, not about the application. + `; + + const response = await this.provider + .generateObject([{ role: 'user' as const, content: userContent }], schema, this.provider.getAgenticModel('pilot'), { + agentName: 'pilot', + telemetry: { functionId: 'pilot.settleExpectations' }, + }) + .catch(() => null); + + const judged = new Map((response?.object?.outcomes || []).map((outcome: any) => [outcome.expectation, outcome.status])); + return task.expected.map((text) => { + if (!undecided.includes(text)) return { text, status: decided(text) }; + return { text, status: (judged.get(text) as 'passed' | 'failed' | 'unverified') || 'unverified' }; + }); + } + private formatExpectations(task: Test): string { const checked = task.getCheckedExpectations(); const remaining = task.getRemainingExpectations(); @@ -703,7 +752,7 @@ export class Pilot implements Agent { lines.push(`url: ${state.url}`); lines.push(`title: ${state.title || 'unknown'}`); - const focused = extractFocusedElement(state.ariaSnapshot); + const focused = state.focusedElement; if (focused) { const valuePart = focused.value ? ` (value: "${focused.value}")` : ''; lines.push(`focused: ${focused.role} "${focused.name}"${valuePart}`); diff --git a/src/ai/provider.ts b/src/ai/provider.ts index 280037b8..678659f6 100644 --- a/src/ai/provider.ts +++ b/src/ai/provider.ts @@ -326,6 +326,7 @@ export class Provider { try { const response = await withRetry(async () => { const result = await generateText({ messages, ...config }); + this.recordUsage(options.agentName || 'unknown', modelName, result.usage); if (!result.text) { debugLog(result); if (result.finishReason === 'length') { @@ -342,8 +343,6 @@ export class Provider { clearActivity(); responseLog(response.text); - this.recordUsage(options.agentName || 'unknown', modelName, response.usage); - return response; } catch (error: any) { clearActivity(); @@ -378,6 +377,7 @@ export class Provider { try { const response = await withRetry(async () => { const result = (await this.raceWithIdleTimeout((signal) => generateText({ messages, ...config, abortSignal: signal }), config.timeout || 30000)) as any; + this.recordUsage(options.agentName || 'unknown', modelName, result.usage); const hasToolCall = (result.toolCalls?.length || 0) > 0; if (!result.text && !hasToolCall && result.finishReason === 'length') { throw new ContextLengthError('AI response empty: output truncated at maxTokens. Increase maxOutputTokens in config or use a model with higher output capacity.'); @@ -397,8 +397,6 @@ export class Provider { responseLog(response.text); - this.recordUsage(options.agentName || 'unknown', modelName, response.usage); - return response; } catch (error: any) { clearActivity(); diff --git a/src/ai/researcher.ts b/src/ai/researcher.ts index 2dc5e441..a0a321a6 100644 --- a/src/ai/researcher.ts +++ b/src/ai/researcher.ts @@ -102,7 +102,7 @@ export class Researcher extends ResearcherBase implements Agent { const cached = getCachedResearch(stateHash); if (cached) { debugLog('Previous research result found'); - return `!! UI MAP IS CACHED AND MAY NOT REPRESENT CURRENT STATE; REFRESH RESEARCH IF YOU NOTICE ISSUES !!\n\n${cached}`; + return cached; } } diff --git a/src/ai/rules.ts b/src/ai/rules.ts index ab8b6c94..05fac8b7 100644 --- a/src/ai/rules.ts +++ b/src/ai/rules.ts @@ -4,6 +4,10 @@ export const recommendedCodeceptCommands = ['I.click', 'I.type', 'I.fillField', const locatorPriorityRule = dedent` <locator_priority> + When the page context shows the element a ref, such as [ref=e14], there is no locator to select: click it with clickRef + and that ref. A ref names one exact element, so it never matches the wrong one and never has to be narrowed. Everything + below is for elements the context gives no ref for. + Use the following priority when selecting locators: 1. ARIA locators (first choice) - target browser's accessibility tree, most reliable diff --git a/src/ai/tester.ts b/src/ai/tester.ts index 73758421..b160cc6e 100644 --- a/src/ai/tester.ts +++ b/src/ai/tester.ts @@ -11,7 +11,7 @@ import { Observability } from '../observability.ts'; import type { StateTransition } from '../state-manager.ts'; import { Stats } from '../stats.ts'; import { type Test, TestResult, type TestResultType } from '../test-plan.ts'; -import { detectFocusArea, extractFocusedElement } from '../utils/aria.ts'; +import { compactAriaSnapshot, detectFocusArea } from '../utils/aria.ts'; import { ErrorPageError, isErrorPage } from '../utils/error-page.ts'; import { createDebug, tag } from '../utils/logger.ts'; import { loop } from '../utils/loop.ts'; @@ -62,7 +62,6 @@ export class Tester extends TaskAgent implements Agent { private seenUiMapUrls = new Set<string>(); private lastAnalyzedStateHash: string | null = null; private stalledIterations = 0; - private hasSuccessfulAssertion = false; private readonly MAX_STALLED_ITERATIONS = 3; constructor(deps: AgentDeps, researcher: Researcher, navigator: Navigator, agentTools?: any) { @@ -111,7 +110,6 @@ export class Tester extends TaskAgent implements Agent { this.seenUiMapUrls.clear(); this.lastAnalyzedStateHash = null; this.stalledIterations = 0; - this.hasSuccessfulAssertion = false; this.stateManager.clearHistory(); this.resetFailureCount(); this.pilot?.reset(); @@ -119,7 +117,7 @@ export class Tester extends TaskAgent implements Agent { const requestStore = this.requestStore; requestStore.clear(); const offFailedRequest = requestStore.onFailedRequest((r) => { - task.addNote(`Network error: ${r.method} ${r.path} → ${r.status}`, TestResult.FAILED); + task.addObservation(`Network error: ${r.method} ${r.path} → ${r.status}`); }); const initialState = ActionResult.fromState(state); @@ -314,17 +312,9 @@ export class Tester extends TaskAgent implements Agent { const allToolNames = result?.toolExecutions?.map((execution: any) => execution.toolName) || []; const successfulToolNames = result?.toolExecutions?.filter((execution: any) => execution.wasSuccessful)?.map((execution: any) => execution.toolName) || []; const actionPerformed = !!allToolNames.find((toolName: string) => this.ACTION_TOOLS.includes(toolName)); - const successfulActionPerformed = !!successfulToolNames.find((toolName: string) => this.ACTION_TOOLS.includes(toolName)); assertionPerformed = !!successfulToolNames.find((toolName: string) => this.ASSERTION_TOOLS.includes(toolName)); const wasSuccessful = result?.toolExecutions?.every((execution: any) => execution.wasSuccessful); - if (successfulActionPerformed) { - this.hasSuccessfulAssertion = false; - } - if (assertionPerformed) { - this.hasSuccessfulAssertion = true; - } - this.trackToolExecutions(result?.toolExecutions || []); if (this.consecutiveEmptyResults >= 5) { @@ -416,6 +406,7 @@ export class Tester extends TaskAgent implements Agent { if (extensions >= this.MAX_EXTENSIONS) break; extensions++; + this.stalledIterations = 0; tag('info').log(`Pilot extending test (${extensions}/${this.MAX_EXTENSIONS})`); conversation.cleanupTag('page_aria', '...trimmed...', 1); conversation.cleanupTag('page_html', '...trimmed...', 0); @@ -474,13 +465,7 @@ export class Tester extends TaskAgent implements Agent { this.stalledIterations++; if (this.stalledIterations < this.MAX_STALLED_ITERATIONS) return false; - if (this.hasSuccessfulAssertion) { - task.addNote('No further browser progress after successful verification; requesting final review'); - return true; - } - - task.addNote('No browser progress after repeated attempts on unchanged page', TestResult.FAILED); - task.finish(TestResult.FAILED); + task.addNote('No further browser progress on unchanged page; requesting final review'); return true; } @@ -500,7 +485,8 @@ export class Tester extends TaskAgent implements Agent { `; if (task.getPrintableNotes()) { - outcomeStatus = dedent` + outcomeStatus += dedent` + Your current log: <notes> ${task.notesToString()} @@ -529,7 +515,7 @@ export class Tester extends TaskAgent implements Agent { const focusArea = detectFocusArea(currentState.ariaSnapshot); - const focusedElement = extractFocusedElement(currentState.ariaSnapshot); + const focusedElement = currentState.focusedElement; if (focusedElement) { const isTextInput = ['textbox', 'combobox', 'searchbox'].includes(focusedElement.role); context += dedent` @@ -611,7 +597,7 @@ export class Tester extends TaskAgent implements Agent { </page> <page_aria> - ${currentState.getInteractiveARIA()} + ${await this.interactiveAriaWithRefs(currentState)} </page_aria> ${uiMapSection} @@ -648,25 +634,32 @@ export class Tester extends TaskAgent implements Agent { </page> <page_aria> - ${currentState.getInteractiveARIA()} + ${await this.interactiveAriaWithRefs(currentState)} </page_aria> `; } + private async interactiveAriaWithRefs(state: ActionResult): Promise<string> { + const withRefs = await Promise.resolve(this.explorer?.withPage?.((page: any) => page.locator('body').ariaSnapshot({ mode: 'ai' }))).catch(() => null); + if (!withRefs) return state.getInteractiveARIA(); + return compactAriaSnapshot(withRefs, false); + } + private finishTest(task: Test): void { - if (!task.hasFinished) { - task.finish(TestResult.FAILED); + if (!task.result) { + if (task.hasAchievedAll()) task.finish(TestResult.PASSED); + else task.finish(TestResult.FAILED); } if (task.isSuccessful) { tag('success').log(`Successful test: ${task.scenario}`); - } else if (task.isSkipped) { + return; + } + if (task.isSkipped) { tag('warning').log(`Skipped test: ${task.scenario}`); - } else if (task.hasFailed) { - tag('error').log(`Failed test: ${task.scenario}`); - } else { - tag('warning').log(`Test with no result: ${task.scenario}`); + return; } + tag('error').log(`Failed test: ${task.scenario}`); } private async abortStartedTestOnErrorPage(task: Test, actionResult: ActionResult): Promise<{ success: boolean }> { @@ -810,6 +803,9 @@ export class Tester extends TaskAgent implements Agent { ${task.expected.map((e) => `- ${e}`).join('\n')} </expected_results> + An expected result counts as settled only when you record it back word for word as it is written above. + A note in your own wording is a general note and leaves that result unsettled. + Your goal is to perform actions on the web page and verify the expected outcomes. Try to achieve as many goals as possible. If goal is not achievable, log that and skip to next one. @@ -1059,6 +1055,9 @@ export class Tester extends TaskAgent implements Agent { - You unsuccessfully tried multiple iterations and failed - If the expected result was expected to fail, use status="success" instead + When a note settles one of the expected results, that note must repeat the expected result word + for word. Paraphrasing it leaves the expected result unsettled and it is reported as unverified. + Example: - record({ notes: ["clicked login button", "login form appeared", "fill credentials"], status: "success" }) `, @@ -1143,8 +1142,7 @@ export class Tester extends TaskAgent implements Agent { this.stalledIterations++; if (this.stalledIterations < this.MAX_STALLED_ITERATIONS) return false; - task.addNote('No browser progress after repeated execution errors', TestResult.FAILED); - task.finish(TestResult.FAILED); + task.addNote('No browser progress after repeated execution errors; requesting final review'); return true; } diff --git a/src/ai/tools.ts b/src/ai/tools.ts index 526cd2d0..172cee15 100644 --- a/src/ai/tools.ts +++ b/src/ai/tools.ts @@ -4,7 +4,7 @@ import { z } from 'zod'; import { ActionResult, type PageDiff, type ToolResultMetadata } from '../action-result.ts'; import type { ExperienceTracker } from '../experience-tracker.ts'; import { type Task, TestResult } from '../test-plan.js'; -import { LARGE_ARIA_CHANGE_THRESHOLD, extractFocusedElement } from '../utils/aria.ts'; +import { LARGE_ARIA_CHANGE_THRESHOLD } from '../utils/aria.ts'; import { isFatalBrowserError } from '../utils/browser-errors.ts'; import { createDebug, tag } from '../utils/logger.js'; import { pause } from '../utils/loop.js'; @@ -33,6 +33,10 @@ export function createCodeceptJSTools({ explorer, stateManager, ai }: ToolDeps, description: dedent` Click an element by trying multiple CodeceptJS commands in order until one succeeds. + Use this only for elements the page context gives you no ref for. When the element shows a ref such as [ref=e14], + call clickRef with that ref instead — composing a locator for an element that already has a ref is wasted work, + and a locator can match several elements where a ref cannot. + Follow <locator_priority> from system prompt for locator selection. I.click(locator) - click element matching locator @@ -153,6 +157,41 @@ export function createCodeceptJSTools({ explorer, stateManager, ai }: ToolDeps, }, }), + clickRef: tool({ + description: dedent` + Click an element by the ref the page context gave it, e.g. [ref=e14]. + + Prefer this over click() whenever the element you want carries a ref. A ref names one exact element, so it + cannot match several by mistake and never needs disambiguating — it is the fastest way to click. + Only pass a ref that appears in the page context you were given. Never invent or guess one. + If it reports the ref is gone, the page has been rebuilt: get fresh context and use the new ref. + `, + inputSchema: z.object({ + ref: z.string().describe('The ref exactly as it appears in the page context, e.g. "e14"'), + element: z.string().describe('Role and name of the element you are clicking, for the record'), + }), + execute: async ({ ref, element }) => { + const activeNote = task.startNote(`Click ${element}`); + const previousState = ActionResult.fromState(stateManager.getCurrentState()!); + const action = explorer.action(); + const named = await describeRef(explorer, ref); + const run = `I.usePlaywrightTo(${JSON.stringify(`click ${element}`)}, async ({ page }) => page.locator(${JSON.stringify(`aria-ref=${ref}`)}).click())`; + + if (!(await action.attempt(run, `Click ${element}`))) { + activeNote.commit(TestResult.FAILED); + return failedToolResult('clickRef', `Ref ${ref} could not be clicked: ${errorText(action.lastError)}`, { + suggestion: 'The ref may belong to an older version of the page. Get fresh context and use the ref it gives, or fall back to click() with a locator.', + }); + } + + // a ref belongs to this session only, so the run is reported as the locator a later test can replay + const code = named ? `I.click(${JSON.stringify(named)})` : run; + const toolResult = await ActionResult.fromState(stateManager.getCurrentState()!).toToolResult(previousState, code); + await commitNote(activeNote, TestResult.PASSED, toolResult, action); + return successToolResult('clickRef', { ...toolResult, code }, action); + }, + }), + hover: tool({ description: dedent` Move the mouse cursor to an element to reveal hover-only controls. @@ -299,15 +338,11 @@ export function createCodeceptJSTools({ explorer, stateManager, ai }: ToolDeps, const focusFreeKeys = new Set(['Escape', 'Esc', 'Tab', 'F1', 'F2', 'F3', 'F4', 'F5', 'F6', 'F7', 'F8', 'F9', 'F10', 'F11', 'F12']); const needsFocus = !focusFreeKeys.has(keyToUse) && !modifier; - if (needsFocus) { - const currentAriaState = stateManager.getCurrentState()?.ariaSnapshot; - const focused = extractFocusedElement(currentAriaState ?? null); - if (!focused) { - activeNote.commit(TestResult.FAILED); - return failedToolResult('pressKey', `No element is focused. Key '${keyToUse}' requires a focused element.`, { - suggestion: 'Click the target element first, then press the key.', - }); - } + if (needsFocus && !(await hasFocusedElement(explorer))) { + activeNote.commit(TestResult.FAILED); + return failedToolResult('pressKey', `No element is focused. Key '${keyToUse}' requires a focused element.`, { + suggestion: 'Click the target element first, then press the key.', + }); } const previousState = ActionResult.fromState(stateManager.getCurrentState()!); @@ -358,6 +393,8 @@ export function createCodeceptJSTools({ explorer, stateManager, ai }: ToolDeps, Execute raw CodeceptJS code block with multiple commands. USE THIS TOOL for typing text into fields: I.fillField, I.type + Do not put a click on a ref-bearing element in here — clickRef with its ref is cheaper and cannot mis-target. + Follow <actions> from system prompt for available commands. Follow <locator_priority> from system prompt for locator selection. @@ -544,14 +581,15 @@ export function createAgentTools({ explorer, stateManager, ai, researcher, navig const tools: Record<string, any> = { see: tool({ description: dedent` - Check the page contents based on current page state and screenshot. - This tool will trigger visual research to check the page contents on request. - Use it to verify the actions were performed correctly and the page is in the expected state. + Answer a question about the page from a screenshot, for things its structure cannot express: + layout and position, what an image or canvas depicts, colour, and whether something is covered or cut off. + This runs a second model and is the slowest tool here, so reach for it only when the question is genuinely visual. + Do NOT use it to confirm an action landed — every action already reports what changed on the page. Input schema has exactly one field: request. Do not pass text, reason, assertion, or other fields. <example> - request: "Check current state of the Login form" - result: "Login form is visible with username and password fields, username is filled with 'testuser' and password is empty' + request: "Is the save button covered by anything, and does the chart show any plotted data?" + result: "The save button is partly behind a cookie banner at the bottom. The chart area is empty apart from its axes." </example> `, inputSchema: z.object({ @@ -676,6 +714,13 @@ export function createAgentTools({ explorer, stateManager, ai, researcher, navig ); } + if (result.inexpressible) { + return failedToolResult('verify', `No assertion could express this claim: ${assertion}`, { + inexpressible: true, + suggestion: 'This is not evidence the page is wrong — the claim could not be turned into an assertion. Restate it in terms of what is visible or of a control state, or check it with see().', + }); + } + return failedToolResult('verify', `Verification failed: ${assertion}`, { suggestion: 'The assertion could not be verified. Check if the condition is actually present on the page or try a different assertion.', }); @@ -1112,14 +1157,33 @@ function errorText(error: unknown): string { return 'Unknown error occurred'; } -async function commitNote(activeNote: any, result: TestResult, toolResult: any, action: any): Promise<void> { +export async function commitNote(activeNote: any, result: TestResult, toolResult: any, action: any): Promise<void> { if (toolResult?.pageDiff?.ariaChanges || toolResult?.pageDiff?.urlChanged) { activeNote.screenshot = await action.saveScreenshot(); } activeNote.commit(result); } -function successToolResult(action: string, data?: Record<string, any>, source?: { playwrightGroupId?: string | null; assertionSteps?: any[] }) { +async function describeRef(explorer: any, ref: string): Promise<{ role: string; text: string } | null> { + return Promise.resolve( + explorer?.withPage?.((page: any) => + page.locator(`aria-ref=${ref}`).evaluate((el: any) => { + const tag = el.tagName.toLowerCase(); + const roles: Record<string, string> = { a: 'link', button: 'button', select: 'combobox', textarea: 'textbox' }; + const role = el.getAttribute('role') || roles[tag] || tag; + const text = (el.getAttribute('aria-label') || el.innerText || el.value || '').trim().split('\n')[0]; + if (!text) return null; + return { role, text }; + }) + ) + ).catch(() => null); +} + +async function hasFocusedElement(explorer: any): Promise<boolean> { + return explorer.withPage((page: any) => page.evaluate(() => !!document.activeElement && document.activeElement !== document.body)).catch(() => true); +} + +export function successToolResult(action: string, data?: Record<string, any>, source?: { playwrightGroupId?: string | null; assertionSteps?: any[] }) { const result: Record<string, any> = { success: true, action, ...data }; if (source?.playwrightGroupId) { result.playwrightGroupId = source.playwrightGroupId; @@ -1155,7 +1219,7 @@ function hasObservablePageChange(data?: Record<string, any>): boolean { return Array.isArray(data.pageDiff.htmlParts) && data.pageDiff.htmlParts.length > 0; } -async function failedToolResult(action: string, message: string, data?: Record<string, any>, error?: Error | null) { +export async function failedToolResult(action: string, message: string, data?: Record<string, any>, error?: Error | null) { const result: Record<string, any> = { success: false, action, message, ...data }; if (data?.pageDiff) { result.suggestion = data.suggestion ? `${data.suggestion} ${PAGE_DIFF_SUGGESTION}` : PAGE_DIFF_SUGGESTION; diff --git a/src/config.ts b/src/config.ts index 90dfa8b5..2e892069 100644 --- a/src/config.ts +++ b/src/config.ts @@ -6,7 +6,7 @@ import dedent from 'dedent'; import matter from 'gray-matter'; import { type SiteRecord, findGlobalConfig, globalEnvPath, isGlobalConfigPath, registerSite, resolveSiteTarget } from './global-config.js'; import { getCliName } from './utils/cli-name.js'; -import { log } from './utils/logger.js'; +import { log, tag } from './utils/logger.js'; export const PROVIDERS: Record<string, ProviderInfo> = { openai: { envKey: 'OPENAI_API_KEY', load: async () => (await import('@ai-sdk/openai')).createOpenAI() }, @@ -455,7 +455,15 @@ export class ConfigParser { } public resolveTargetPath(target?: string): string { - if (!this.site) return target || '/'; + if (!this.site) { + const configured = this.config?.playwright?.url || this.config?.web?.url; + const targetOrigin = target ? URL.parse(target)?.origin : null; + const baseOrigin = configured ? URL.parse(configured)?.origin : null; + if (targetOrigin && baseOrigin && targetOrigin !== baseOrigin) { + tag('warning').log(`Exploring ${targetOrigin} but base URL is ${baseOrigin}. Relative navigation resolves against the base URL — set web.url to ${targetOrigin} to avoid it.`); + } + return target || '/'; + } if (!target) return this.siteStartPath; const resolved = resolveSiteTarget(target, this.site.url); diff --git a/src/explorbot.ts b/src/explorbot.ts index 63b76910..7278718e 100644 --- a/src/explorbot.ts +++ b/src/explorbot.ts @@ -20,7 +20,7 @@ import { createAgentTools } from './ai/tools.ts'; import { ApiClient } from './api/api-client.ts'; import { RequestStore } from './api/request-store.ts'; import { loadSpec } from './api/spec-reader.ts'; -import type { ExplorbotConfig } from './config.js'; +import type { ExplorbotConfig, ReporterConfig } from './config.js'; import { ConfigParser } from './config.ts'; import { ExperienceTracker } from './experience-tracker.ts'; import Explorer from './explorer.ts'; @@ -49,6 +49,7 @@ export interface ExplorBotOptions { session?: string | boolean; instance?: string; optionalAi?: boolean; + reporter?: ReporterConfig; attachedBrowser?: Browser; applicationSpec?: string; } @@ -128,6 +129,7 @@ export class ExplorBot { if (this.provider) return; this.config = await this.configParser.loadConfig(this.options); if (this.options.session === true) this.options.session = path.join(this.configParser.getOutputDir(), 'session.json'); + if (typeof this.options.session === 'string') this.options.session = path.resolve(this.options.path || '.', this.options.session); if (this.options.optionalAi) return this.bootstrapOptionalProvider(); this.provider = new AIProvider(this.config.ai); await this.provider.validateConnection(); @@ -175,7 +177,7 @@ export class ExplorBot { } reporter(): Reporter { - return (this._reporter ||= new Reporter(this.config.reporter, this.stateManager())); + return (this._reporter ||= new Reporter(this.options.reporter || this.config.reporter, this.stateManager())); } requestStore(): RequestStore { diff --git a/src/explorer.ts b/src/explorer.ts index cf490d1e..2190e8ee 100644 --- a/src/explorer.ts +++ b/src/explorer.ts @@ -721,11 +721,11 @@ class Explorer { if (this.observedTestPages.has(page)) return; this.testPageErrorHandler ||= (err: Error) => { - this._activeTest?.addNote(`Console error: ${err.message}`, TestResult.FAILED); + this._activeTest?.addObservation(`Console error: ${err.message}`); }; this.testConsoleHandler ||= (msg: any) => { if (msg.type() !== 'error') return; - this._activeTest?.addNote(`Console error: ${msg.text()}`, TestResult.FAILED); + this._activeTest?.addObservation(`Console error: ${msg.text()}`); }; this.testDialogHandler ||= (dialog: any) => { const dialogType = dialog.type(); diff --git a/src/playwright-recorder.ts b/src/playwright-recorder.ts index 7f925b2b..71a18364 100644 --- a/src/playwright-recorder.ts +++ b/src/playwright-recorder.ts @@ -287,8 +287,31 @@ function formatSelectOption(options: any): string { return `[${values.map((v) => quote(v)).join(', ')}]`; } +function assertionLocator(target: any): string | null { + if (typeof target === 'string') return `page.locator(${JSON.stringify(target)})`; + if (!target || typeof target !== 'object') return null; + + const role = target.role || target.aria; + const name = target.text ?? target.name ?? target.label; + if (role && name) return `page.getByRole(${JSON.stringify(String(role))}, { name: ${JSON.stringify(String(name))} })`; + if (role) return `page.getByRole(${JSON.stringify(String(role))})`; + if (name) return `page.getByText(${JSON.stringify(String(name))})`; + if (target.css) return `page.locator(${JSON.stringify(String(target.css))})`; + if (target.xpath) return `page.locator(${JSON.stringify(`xpath=${target.xpath}`)})`; + return null; +} + export function renderAssertion(assertion: { name: string; args: any[] }): string { const args = assertion.args; + const target = assertionLocator(args[0]); + + if (target) { + if (assertion.name === 'seeElement') return `await expect(${target}).toBeVisible();`; + if (assertion.name === 'dontSeeElement') return `await expect(${target}).toBeHidden();`; + if (assertion.name === 'seeInField' && args[1] !== undefined) return `await expect(${target}).toHaveValue(${JSON.stringify(String(args[1]))});`; + if (assertion.name === 'dontSeeInField' && args[1] !== undefined) return `await expect(${target}).not.toHaveValue(${JSON.stringify(String(args[1]))});`; + } + if (assertion.name === 'see' && typeof args[0] === 'string') { return `await expect(page).toContainText(${JSON.stringify(args[0])});`; } diff --git a/src/reporter.ts b/src/reporter.ts index 8b780f3b..4d0dd4c6 100644 --- a/src/reporter.ts +++ b/src/reporter.ts @@ -283,13 +283,19 @@ export class Reporter { debugLog(testData); - await this.client.addTestRun(status, testData); + await this.reportTestData(status, testData); debugLog(`Test reported: ${test.scenario} - ${status}`); } catch (error) { debugLog('Failed to report test:', error); } } + async reportTestData(status: string | null, testData: Record<string, unknown>): Promise<void> { + await this.startRun(); + if (!this.isRunStarted) return; + await this.client.addTestRun(status, testData); + } + async finishRun(): Promise<void> { if (!this.isRunStarted) { return; diff --git a/src/state-manager.ts b/src/state-manager.ts index 8918feec..7d9f5e84 100644 --- a/src/state-manager.ts +++ b/src/state-manager.ts @@ -1,4 +1,4 @@ -import { ActionResult } from './action-result.js'; +import { type FocusedElement, ActionResult } from './action-result.js'; import type { ExperienceTracker } from './experience-tracker.js'; import type { Knowledge, KnowledgeTracker } from './knowledge-tracker.js'; import { detectFocusArea } from './utils/aria.js'; @@ -46,6 +46,7 @@ export interface WebPageState { h4?: string; ariaSnapshot?: string | null; ariaSnapshotFile?: string; + focusedElement?: FocusedElement | null; links?: Link[]; verifications?: Record<string, boolean>; } diff --git a/src/test-plan.ts b/src/test-plan.ts index f9f83d2a..98f18eac 100644 --- a/src/test-plan.ts +++ b/src/test-plan.ts @@ -27,6 +27,7 @@ export interface Note { endTime: number; screenshot?: string; log?: string; + observation?: boolean; } export class ActiveNote { @@ -131,6 +132,14 @@ export class Task { this.notes[timestamp] = { message, status, startTime: now, endTime: now, screenshot, log }; } + addObservation(message: string): void { + const isDuplicate = Object.values(this.notes).some((note) => note.message === message); + if (isDuplicate) return; + + const now = performance.now(); + this.notes[`${now}_${this.timestampCounter++}`] = { message, status: TestResult.FAILED, startTime: now, endTime: now, observation: true }; + } + addUrlNote(state: UrlNoteState, prevState?: { title?: string; h1?: string; h2?: string }): void { const fullUrl = state.fullUrl || state.url; if (!fullUrl) return; diff --git a/src/utils/aria.ts b/src/utils/aria.ts index 6d984999..90f10305 100644 --- a/src/utils/aria.ts +++ b/src/utils/aria.ts @@ -85,8 +85,7 @@ const parseLabel = (label: string): { role: string; name?: string; attributes: R } const attributes: Record<string, string | boolean | null> = {}; - const attrMatch = rest.match(/\[([^\]]*)\]/); - if (attrMatch) { + for (const attrMatch of rest.matchAll(/\[([^\]]*)\]/g)) { for (const tok of attrMatch[1].split(/[\s,]+/).filter(Boolean)) { const eq = tok.indexOf('='); if (eq === -1) { @@ -192,7 +191,9 @@ const dropEmpty = (nodes: AriaNode[], opts: { keepNamed?: boolean } = {}): AriaN // ───────────────────────────────────────────────────────────────── // One-line representation of a node. Stable attr order so diff comparisons are deterministic. -const formatNode = (node: AriaNode): string => { +const MAX_INLINE_VALUE = 400; + +const formatNode = (node: AriaNode, offload?: ValueOffload): string => { let line = node.role; if (node.name?.trim()) line += ` "${node.name.trim()}"`; const attrStr = Object.keys(node.attributes) @@ -208,11 +209,19 @@ const formatNode = (node: AriaNode): string => { if (attrStr) line += ` [${attrStr}]`; if (node.value !== undefined && node.value !== null) { const text = String(node.value).trim(); - if (text) line += `: ${text}`; + if (text && text.length <= MAX_INLINE_VALUE) line += `: ${text}`; + if (text && text.length > MAX_INLINE_VALUE) line += `: ${offloadValue(text, offload)}`; } return line; }; +const offloadValue = (text: string, offload?: ValueOffload): string => { + const head = `${text.slice(0, MAX_INLINE_VALUE)}…`; + const reference = offload?.(text); + if (!reference) return `${head} (${text.length} chars, truncated)`; + return `${head} (${text.length} chars) [Full Text: ${reference}]`; +}; + // Group consecutive same-role siblings. [a,a,b,a,a,a] → [[a,a],[b],[a,a,a]] const groupByConsecutiveRole = (nodes: AriaNode[]): AriaNode[][] => nodes.reduce<AriaNode[][]>((groups, node) => { @@ -239,15 +248,15 @@ const collapseGroup = (group: AriaNode[], depth: number): RenderEntry[] => { const collapseSiblingGroups = (nodes: AriaNode[], depth: number): RenderEntry[] => groupByConsecutiveRole(nodes).flatMap((group) => collapseGroup(group, depth)); // Tree → indented YAML text. -const renderTree = (nodes: AriaNode[], depth = 0): string => +const renderTree = (nodes: AriaNode[], depth = 0, offload?: ValueOffload): string => collapseSiblingGroups(nodes, depth) .map((entry) => { if ('placeholder' in entry) return entry.placeholder; const { node } = entry; const indent = ' '.repeat(depth); - const head = `${indent}- ${formatNode(node)}`; + const head = `${indent}- ${formatNode(node, offload)}`; if (node.children.length === 0) return head; - return `${head}:\n${renderTree(node.children, depth + 1)}`; + return `${head}:\n${renderTree(node.children, depth + 1, offload)}`; }) .join('\n'); @@ -382,6 +391,40 @@ const detectToggles = (prev: FlatEntry[], curr: FlatEntry[]): { toggled: string[ return { toggled, togglePaths }; }; +const detectValueChanges = (prev: FlatEntry[], curr: FlatEntry[]): { typed: string[]; typedPaths: Set<string> } => { + const typed: string[] = []; + const typedPaths = new Set<string>(); + const currByPath = new Map(curr.map((e) => [e.path, e])); + + for (const before of prev) { + const after = currByPath.get(before.path); + if (!after) continue; + if (before.entry.role !== after.entry.role) continue; + if (before.entry.name !== after.entry.name) continue; + + const was = valueWord(before.entry.value); + const now = valueWord(after.entry.value); + if (was === now) continue; + + typedPaths.add(before.path); + let label = String(after.entry.role); + const name = after.entry.name; + if (typeof name === 'string' && name.trim()) label += ` "${name.trim()}"`; + typed.push(`${label}: ${was} -> ${now}`); + } + return { typed, typedPaths }; +}; + +const VALUE_EXCERPT = 60; + +const valueWord = (value: unknown): string => { + if (value === undefined || value === null) return 'empty'; + const text = String(value).trim(); + if (!text) return 'empty'; + if (text.length <= VALUE_EXCERPT) return JSON.stringify(text); + return `${JSON.stringify(text.slice(0, VALUE_EXCERPT))}… (${text.length} chars)`; +}; + const TOP_DIFF_ITEMS = 10; const formatDiffSection = (label: string, items: string[]): string[] => { @@ -406,9 +449,13 @@ const formatDiffSection = (label: string, items: string[]): string[] => { return lines; }; -const formatDiff = (added: string[], removed: string[], toggled: string[]): string | null => { - if (added.length === 0 && removed.length === 0 && toggled.length === 0) return null; +const formatDiff = (added: string[], removed: string[], toggled: string[], typed: string[] = []): string | null => { + if (added.length === 0 && removed.length === 0 && toggled.length === 0 && typed.length === 0) return null; const sections = ['ariaDiff:']; + if (typed.length > 0) { + sections.push(' typed:'); + for (const line of typed) sections.push(` - ${line}`); + } if (toggled.length > 0) { sections.push(' toggled:'); for (const line of toggled) sections.push(` - ${line}`); @@ -474,13 +521,15 @@ const findDialogOrModal = (nodes: AriaNode[]): FocusAreaResult | null => { // Public API — pipelines composed visibly, top-to-bottom // ───────────────────────────────────────────────────────────────── -export const compactAriaSnapshot = (snapshot: string | null, keepNamed = false): string => { +export type ValueOffload = (value: string) => string | undefined; + +export const compactAriaSnapshot = (snapshot: string | null, keepNamed = false, offload?: ValueOffload): string => { if (!snapshot) return ''; let tree = parseSnapshot(snapshot); tree = unwrapIgnored(tree); tree = nameIconButtons(tree); tree = dropEmpty(tree, { keepNamed }); - return renderTree(tree); + return renderTree(tree, 0, offload); }; export const diffAriaSnapshots = (previous: string | null, current: string | null): AriaDiff => { @@ -494,15 +543,17 @@ export const diffAriaSnapshots = (previous: string | null, current: string | nul const prevAll = flat(previous); const currAll = flat(current); const { toggled, togglePaths } = detectToggles(prevAll, currAll); - const prev = prevAll.filter((e) => !togglePaths.has(e.path)); - const curr = currAll.filter((e) => !togglePaths.has(e.path)); + const { typed, typedPaths } = detectValueChanges(prevAll, currAll); + const skip = (entry: FlatEntry) => togglePaths.has(entry.path) || typedPaths.has(entry.path); + const prev = prevAll.filter((e) => !skip(e)); + const curr = currAll.filter((e) => !skip(e)); const prevTotals = countBy(prev.map((e) => e.summary)); const currTotals = countBy(curr.map((e) => e.summary)); const byCount = diffByCount(prevTotals, currTotals); const renames = detectRenames(prev, curr, prevTotals, currTotals); const added = [...byCount.added, ...renames.added]; const removed = [...byCount.removed, ...renames.removed]; - return { text: formatDiff(added, removed, toggled), count: added.length + removed.length + toggled.length }; + return { text: formatDiff(added, removed, toggled, typed), count: added.length + removed.length + toggled.length + typed.length }; }; export const detectFocusArea = (snapshot: string | null): FocusAreaResult => { @@ -531,37 +582,6 @@ export const collectInteractiveNodes = (snapshot: string | null): Array<Record<s // Standalone helpers (regex on raw strings — not part of the pipeline) // ───────────────────────────────────────────────────────────────── -export interface FocusedElementInfo { - role: string; - name: string; - value?: string; - attributes?: string[]; -} - -export function extractFocusedElement(ariaSnapshot: string | null): FocusedElementInfo | null { - if (!ariaSnapshot) return null; - - const focusedMatch = ariaSnapshot.match(/-\s*(\w+)\s+"([^"]*)"([^:\n]*)\[focused\](?::\s*(.*))?/); - if (!focusedMatch) return null; - - const [, role, name, attributesStr, value] = focusedMatch; - - const attributes: string[] = []; - if (attributesStr) { - const attrMatches = attributesStr.matchAll(/\[([^\]]+)\]/g); - for (const match of attrMatches) { - if (match[1] !== 'focused') { - attributes.push(match[1]); - } - } - } - - const result: FocusedElementInfo = { role, name }; - if (value) result.value = value.trim(); - if (attributes.length > 0) result.attributes = attributes; - return result; -} - export function parseAriaLocator(ariaStr: string): { role: string; text: string } | null { const trimmed = ariaStr.trim(); if (trimmed === '-' || trimmed === '' || trimmed === '"-"') return null; diff --git a/src/utils/logger.ts b/src/utils/logger.ts index e7e4173e..532ea665 100644 --- a/src/utils/logger.ts +++ b/src/utils/logger.ts @@ -79,9 +79,11 @@ const debugFilter = new DebugFilter(); class ConsoleDestination implements LogDestination { private verboseMode = false; private forceEnabled = false; + private quiet = false; private recentSteps = new RecentStepFilter(); isEnabled(): boolean { + if (this.quiet) return false; return this.forceEnabled || !process.env.INK_RUNNING; } @@ -89,6 +91,10 @@ class ConsoleDestination implements LogDestination { this.forceEnabled = enabled; } + setQuiet(enabled: boolean): void { + this.quiet = enabled; + } + setVerboseMode(enabled: boolean): void { this.verboseMode = enabled; } @@ -136,6 +142,8 @@ class DebugDestination implements LogDestination { setVerboseMode(enabled: boolean): void { this.verboseMode = enabled; + // the debug package reads DEBUG when it loads, so a namespace turned on later needs enabling by hand + if (enabled) debug.enable(process.env.DEBUG || 'explorbot:*'); } write(namespace: string, ...args: any[]): void { @@ -360,6 +368,10 @@ class Logger { this.console.forceEnable(enabled); } + setQuietMode(enabled: boolean): void { + this.console.setQuiet(enabled); + } + isVerboseMode(): boolean { return this.debugDestination.isEnabled(); } @@ -537,6 +549,7 @@ export const startLogCapture = () => logger.captain.startCapture(); export const stopLogCapture = () => logger.captain.stopCapture(); export const setVerboseMode = (enabled: boolean) => logger.setVerboseMode(enabled); export const setPreserveConsoleLogs = (enabled: boolean) => logger.setPreserveConsoleLogs(enabled); +export const setQuietMode = (enabled: boolean) => logger.setQuietMode(enabled); export const isVerboseMode = () => logger.isVerboseMode(); export const setDebugMode = (enabled: boolean) => logger.setDebugMode(enabled); export const isDebugMode = () => logger.isDebugMode(); diff --git a/src/utils/page-readiness.ts b/src/utils/page-readiness.ts index 2a30b830..2f8276d2 100644 --- a/src/utils/page-readiness.ts +++ b/src/utils/page-readiness.ts @@ -4,10 +4,33 @@ export async function waitForPageReadiness(page: any, options: PageReadinessOpti const timeout = options.timeout ?? 6000; await page.waitForLoadState?.('domcontentloaded', { timeout })?.catch(() => {}); - await Promise.race([waitForNetworkIdle(page, timeout), waitForVisibleSpinnersHidden(page, options.spinnerSelectors || [], timeout), sleep(timeout)]).catch(() => {}); + await Promise.race([waitForNetworkIdle(page, timeout), waitForDomQuiet(page, timeout), waitForVisibleSpinnersHidden(page, options.spinnerSelectors || [], timeout), sleep(timeout)]).catch(() => {}); await waitForPageBodyContent(page, timeout); } +const DOM_QUIET_MS = 350; + +function waitForDomQuiet(page: any, timeout: number): Promise<void> { + if (!page?.waitForFunction) return new Promise(() => {}); + + return page + .waitForFunction( + (quiet: number) => { + const store = window as any; + if (!store.__explorbotDomQuiet) { + store.__explorbotDomQuiet = { last: Date.now() }; + new MutationObserver(() => { + store.__explorbotDomQuiet.last = Date.now(); + }).observe(document, { subtree: true, childList: true, attributes: true, characterData: true }); + } + return Date.now() - store.__explorbotDomQuiet.last >= quiet; + }, + DOM_QUIET_MS, + { timeout, polling: 100 } + ) + .catch(() => {}); +} + function waitForNetworkIdle(page: any, timeout: number): Promise<void> { if (!page?.waitForLoadState) return Promise.resolve(); return page.waitForLoadState('networkidle', { timeout }).catch(() => {}); diff --git a/src/utils/url-matcher.ts b/src/utils/url-matcher.ts index ca33146e..1ee6db67 100644 --- a/src/utils/url-matcher.ts +++ b/src/utils/url-matcher.ts @@ -100,6 +100,9 @@ export function matchesNavigationUrl(expected: string, current: string): boolean if (!expectedPath.includes('#')) { currentPath = currentPath.split('#')[0]; } + if (!expectedPath.includes('?')) { + currentPath = currentPath.split('?')[0]; + } const normalize = (value: string) => value.replace(/^\/+|\/+$/g, '').toLowerCase(); return normalize(expectedPath) === normalize(currentPath); } diff --git a/src/utils/web-element.ts b/src/utils/web-element.ts index fc2ce338..32db94e8 100644 --- a/src/utils/web-element.ts +++ b/src/utils/web-element.ts @@ -125,6 +125,15 @@ export class WebElement { return WebElement.fromPlaywrightLocator(page.locator(`[${EXPLORBOT_ATTRS.eidx}="${eidx}"]`)); } + static isAriaRef(ref: string): boolean { + return /^(f\d+)?e\d+$/i.test(ref); + } + + static async fromAriaRef(page: any, ref: string): Promise<WebElement | null> { + if (!WebElement.isAriaRef(ref)) return null; + return WebElement.fromPlaywrightLocator(page.locator(`aria-ref=${ref}`)); + } + static async fromEidxList(page: any, eidxList: string[]): Promise<WebElement[]> { const validEidxList = eidxList.filter((eidx) => /^e\d+$/i.test(eidx)); if (validEidxList.length === 0) return []; diff --git a/src/utils/web-sandbox.ts b/src/utils/web-sandbox.ts index 06382ed7..00a02e35 100644 --- a/src/utils/web-sandbox.ts +++ b/src/utils/web-sandbox.ts @@ -28,13 +28,12 @@ export function playwrightSandbox(page: any, code: string): Promise<any> { return run(page); } -export function codeceptJSSandbox(actor: any, codeOrFn: string | ((...args: any[]) => void)): void { +export function codeceptJSSandbox(actor: any, codeOrFn: string | ((...args: any[]) => void)): any { if (typeof codeOrFn === 'function') { - codeOrFn(actor, tryTo, retryTo, within, hopeThat, step, faker); - return; + return codeOrFn(actor, tryTo, retryTo, within, hopeThat, step, faker); } const run = createSandbox(CODECEPT_ARG_NAMES, codeOrFn); - run(actor, tryTo, retryTo, within, hopeThat, step, faker); + return run(actor, tryTo, retryTo, within, hopeThat, step, faker); } function createSandbox(argNames: string[], body: string): (...args: any[]) => any { diff --git a/tests/integration/pilot-expectations.test.ts b/tests/integration/pilot-expectations.test.ts new file mode 100644 index 00000000..8708b3e1 --- /dev/null +++ b/tests/integration/pilot-expectations.test.ts @@ -0,0 +1,82 @@ +import { LLMock } from '@copilotkit/aimock'; +import { createOpenAI } from '@ai-sdk/openai'; +import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'bun:test'; +import { Pilot } from '../../src/ai/pilot.ts'; +import { Provider } from '../../src/ai/provider.ts'; +import { ConfigParser } from '../../src/config.ts'; +import { Test, TestResult } from '../../src/test-plan.ts'; + +function extractPromptText(entry: any): string { + if (!entry?.body?.messages) return ''; + return entry.body.messages + .map((message: any) => { + if (typeof message.content === 'string') return message.content; + if (Array.isArray(message.content)) { + return message.content + .filter((part: any) => part.type === 'text') + .map((part: any) => part.text || '') + .join('\n'); + } + return ''; + }) + .join('\n'); +} + +describe('Pilot settling expected outcomes', () => { + let mock: LLMock; + let pilot: Pilot; + + beforeAll(async () => { + mock = new LLMock({ port: 0, logLevel: 'silent' }); + await mock.start(); + + const openai = createOpenAI({ baseURL: `${mock.url}/v1`, apiKey: 'test-key', compatibility: 'compatible' }); + ConfigParser.setupTestConfig(); + const provider = new Provider({ model: openai.chat('test-model'), config: {} }); + + pilot = new Pilot({ ai: provider, config: ConfigParser.getInstance().getConfig(), explorer: {}, stateManager: {}, requestStore: {}, playwrightRecorder: {} } as any, {} as any, {} as any); + }); + + beforeEach(() => { + mock.clearRequests(); + mock.resetMatchCounts(); + mock.clearFixtures(); + }); + + afterAll(async () => { + await mock.stop(); + }); + + it('settles an outcome the tester described in its own words', async () => { + const task = new Test('open the editor', 'normal', ['the global editor opens with its markdown loaded'], '/skills'); + task.addNote('Clicked Change skill globally; editor panel appeared showing SKILL.md content', TestResult.PASSED); + + mock.on({}, { content: JSON.stringify({ outcomes: [{ expectation: 'the global editor opens with its markdown loaded', status: 'passed' }] }) }); + + const settled = await pilot.settleExpectations(task); + + expect(settled).toEqual([{ text: 'the global editor opens with its markdown loaded', status: 'passed' }]); + expect(extractPromptText(mock.getRequests()[0])).toContain('editor panel appeared showing SKILL.md content'); + }); + + it('asks nothing when the run already checked every outcome off by name', async () => { + const task = new Test('open the editor', 'normal', ['the editor opens'], '/skills'); + task.addNote('the editor opens', TestResult.PASSED); + + const settled = await pilot.settleExpectations(task); + + expect(settled).toEqual([{ text: 'the editor opens', status: 'passed' }]); + expect(mock.getRequests().length).toBe(0); + }); + + it('leaves an outcome unverified when the model cannot tell from the log', async () => { + const task = new Test('open the editor', 'normal', ['the list refreshes'], '/skills'); + task.addNote('Clicked around the sidebar', TestResult.PASSED); + + mock.on({}, { content: JSON.stringify({ outcomes: [{ expectation: 'the list refreshes', status: 'unverified' }] }) }); + + const settled = await pilot.settleExpectations(task); + + expect(settled).toEqual([{ text: 'the list refreshes', status: 'unverified' }]); + }); +}); diff --git a/tests/integration/prima-do.test.ts b/tests/integration/prima-do.test.ts index c5f7c1bc..3de1b9ac 100644 --- a/tests/integration/prima-do.test.ts +++ b/tests/integration/prima-do.test.ts @@ -20,6 +20,14 @@ function clickCall(id: string, commands: string[], explanation: string) { return { id, name: 'click', arguments: JSON.stringify({ commands, explanation }) }; } +function clickRefCall(id: string, ref: string, element: string) { + return { id, name: 'clickRef', arguments: JSON.stringify({ ref, element }) }; +} + +function completedCall(id: string, numbers: number[], proof: string) { + return { id, name: 'completed', arguments: JSON.stringify({ numbers, proof }) }; +} + function extractPromptText(entry: any): string { if (!entry?.body?.messages) return ''; return entry.body.messages @@ -85,6 +93,8 @@ describe('Prima.do with aimock', () => { requestStore: () => ({ getRequests: () => [] }), getProvider: () => provider, experienceTracker: () => ({ renderExperienceTocFor: () => '' }), + agentResearcher: () => ({}), + agentNavigator: () => ({}), }; }); @@ -95,16 +105,17 @@ describe('Prima.do with aimock', () => { }); it('runs both instructions and collects the executed code', async () => { - mock.on({ sequenceIndex: 0 }, { toolCalls: [clickCall('call-1', ['I.click("Account")'], 'open the account menu')] }); - mock.on({ sequenceIndex: 1 }, { toolCalls: [clickCall('call-2', ['I.click("Settings")'], 'choose the settings entry')] }); + mock.on({ sequenceIndex: 0 }, { toolCalls: [clickRefCall('call-1', 'e5', 'button "Account"'), completedCall('call-2', [1], 'the account menu is open')] }); + mock.on({ sequenceIndex: 1 }, { toolCalls: [clickRefCall('call-3', 'e9', 'link "Settings"'), completedCall('call-4', [2], 'the settings page is shown')] }); mock.on({}, { content: 'Both instructions are done.' }); const envelope = await prima.do(['open the account menu', 'choose the settings entry']); expect(envelope.ok).toBe(true); - expect(envelope.used).toEqual(['I.click("Account")', 'I.click("Settings")']); - expect(executed).toEqual(['I.click("Account")', 'I.click("Settings")']); + expect(executed.every((code) => code.includes('aria-ref='))).toBe(true); + expect(executed).toHaveLength(2); expect(envelope.command).toContain('open the account menu'); + expect(envelope.steps?.map((step) => step.label).filter((label) => label.startsWith('done:'))).toEqual(['done: open the account menu', 'done: choose the settings entry']); }); it('sends every instruction and the page context in one prompt', async () => { @@ -126,6 +137,7 @@ describe('Prima.do with aimock', () => { await prima.do(['open the account menu']); const tools = (mock.getRequests()[0] as any).body.tools.map((entry: any) => entry.function.name); + expect(tools).toContain('clickRef'); expect(tools).toContain('click'); expect(tools).toContain('form'); }); diff --git a/tests/integration/prima-heal.test.ts b/tests/integration/prima-heal.test.ts deleted file mode 100644 index 02e29801..00000000 --- a/tests/integration/prima-heal.test.ts +++ /dev/null @@ -1,160 +0,0 @@ -import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'bun:test'; -import { mkdtempSync, rmSync } from 'node:fs'; -import { tmpdir } from 'node:os'; -import path from 'node:path'; -import { createOpenAI } from '@ai-sdk/openai'; -import { LLMock } from '@copilotkit/aimock'; -import { Prima } from '../../boat/prima/src/prima.ts'; -import { ActionResult } from '../../src/action-result.ts'; -import { Navigator } from '../../src/ai/navigator.ts'; -import { Provider } from '../../src/ai/provider.ts'; -import { ConfigParser } from '../../src/config.ts'; - -const checkoutState = { - url: '/checkout', - title: 'Checkout - Widget Depot', - hash: 'checkout_h1_checkout', - html: '<html><body><h1>Checkout</h1><button id="place-order">Place order</button></body></html>', - ariaSnapshot: '- heading "Checkout" [level=1]\n- button "Place order"', -}; - -const recovery = ['Both routes below reach the same outcome.', '', "```js\nI.click('Confirm purchase')\n```", '', "```js\nI.click('#place-order')\n```"].join('\n'); - -function extractPromptText(entry: any): string { - if (!entry?.body?.messages) return ''; - return entry.body.messages - .map((message: any) => { - if (typeof message.content === 'string') return message.content; - if (Array.isArray(message.content)) { - return message.content - .filter((part: any) => part.type === 'text') - .map((part: any) => part.text || '') - .join('\n'); - } - return ''; - }) - .join('\n'); -} - -describe('Prima heal with aimock', () => { - let mock: LLMock; - let provider: Provider; - let prima: Prima; - let attempted: string[]; - let artifacts: string; - - function buildNavigator(succeedsWith: string): Navigator { - const action: any = { - lastError: null, - actionResult: null, - exitIframe: async () => {}, - attempt: async (code: string) => { - attempted.push(code); - if (code.includes(succeedsWith)) { - action.lastError = null; - return true; - } - action.lastError = new Error('element not visible'); - return false; - }, - getActor: () => ({ wait: async () => {} }), - stateManager: { getCurrentState: () => checkoutState }, - }; - - const experienceTracker = { getSuccessfulExperience: () => [], writeFlow: () => {} }; - return new Navigator({ - explorer: { action: () => action, capture: async () => ActionResult.fromState(checkoutState as any) } as any, - ai: provider, - config: ConfigParser.getInstance().getConfig(), - stateManager: { getCurrentState: () => checkoutState, getVisitCount: () => 1, getExperienceTracker: () => experienceTracker } as any, - knowledgeTracker: { renderRelevantKnowledge: () => '', renderRelevantContext: () => '', getRelevantKnowledge: () => [] } as any, - requestStore: {} as any, - playwrightRecorder: {} as any, - }); - } - - beforeAll(async () => { - mock = new LLMock({ port: 0, logLevel: 'silent' }); - await mock.start(); - - const openai = createOpenAI({ - baseURL: `${mock.url}/v1`, - apiKey: 'test-key', - compatibility: 'compatible', - }); - - ConfigParser.resetForTesting(); - ConfigParser.setupTestConfig(); - provider = new Provider({ model: openai.chat('test-model'), config: {} }); - artifacts = mkdtempSync(path.join(tmpdir(), 'prima-heal-')); - }); - - beforeEach(() => { - mock.clearRequests(); - mock.resetMatchCounts(); - mock.clearFixtures(); - - attempted = []; - prima = new Prima({ instance: 'default' }); - (prima as any).artifactsDir = artifacts; - (prima as any).bot = { - getExplorer: () => ({ - action: () => ({ - execute: async () => { - throw new Error("locator 'text=Confirm' not found"); - }, - }), - capture: async () => ActionResult.fromState(checkoutState as any), - }), - stateManager: () => ({ getCurrentState: () => checkoutState, getVisitCount: () => 1 }), - getCurrentState: () => checkoutState, - requestStore: () => ({ getRequests: () => [] }), - getProvider: () => provider, - }; - }); - - afterAll(async () => { - await mock.stop(); - rmSync(artifacts, { recursive: true, force: true }); - ConfigParser.cleanupAllTestDirectories(); - }); - - it('recovers a failed pw along another route and reports the code that worked', async () => { - mock.on({}, { content: recovery }); - (prima as any).bot.agentNavigator = () => buildNavigator('#place-order'); - - const envelope = await prima.pw("({ page }) => page.click('text=Confirm')"); - - expect(envelope.ok).toBe(true); - expect(envelope.healed).toBe(true); - expect(envelope.healNote).toContain('2 attempts'); - expect(envelope.used).toEqual(["I.click('#place-order')"]); - expect(attempted).toEqual(["I.click('Confirm purchase')", "I.click('#place-order')"]); - }); - - it('asks the model to reach the same outcome another way', async () => { - mock.on({}, { content: recovery }); - (prima as any).bot.agentNavigator = () => buildNavigator('#place-order'); - - await prima.pw("({ page }) => page.click('text=Confirm')"); - - const prompt = extractPromptText(mock.getLastRequest()); - expect(prompt).toContain("({ page }) => page.click('text=Confirm')"); - expect(prompt).toContain("locator 'text=Confirm' not found"); - expect(prompt).toContain('Reach the same outcome on the current page in a different way.'); - expect(prompt).toContain('button "Place order"'); - }); - - it('reports every attempt and its outcome when recovery does not work', async () => { - mock.on({}, { content: recovery }); - (prima as any).bot.agentNavigator = () => buildNavigator('nothing-matches-this'); - - const envelope = await prima.pw("({ page }) => page.click('text=Confirm')"); - - expect(envelope.ok).toBe(false); - expect(envelope.failure?.error).toContain("locator 'text=Confirm' not found"); - expect(envelope.failure?.attempts.map((attempt) => attempt.code)).toContain("I.click('#place-order')"); - expect(envelope.failure?.attempts.every((attempt) => attempt.outcome.includes('element not visible'))).toBe(true); - expect(envelope.failure?.compactAria).toContain('button "Place order"'); - }); -}); diff --git a/tests/integration/prima-smoke.test.ts b/tests/integration/prima-smoke.test.ts index cc35ba8b..8ab7d669 100644 --- a/tests/integration/prima-smoke.test.ts +++ b/tests/integration/prima-smoke.test.ts @@ -127,14 +127,27 @@ describe('Prima drives a real page', () => { expect(envelope.page.url).toContain('note=the+hinge+arrived+bent'); }); - test('pw writes the aria, html and network artifacts to disk', async () => { + test('an action reports a status hash instead of artifact paths, and status resolves it', async () => { const envelope = await prima.pw("({ page }) => page.click('text=Submit')"); - expect(existsSync(envelope.artifacts!.aria)).toBe(true); - expect(existsSync(envelope.artifacts!.html)).toBe(true); - expect(existsSync(envelope.artifacts!.network)).toBe(true); - expect(await Bun.file(envelope.artifacts!.aria).text()).toContain('Thanks for the note'); - expect(await Bun.file(envelope.artifacts!.html).text()).toContain('Thanks for the note'); + expect(envelope.status).toMatch(/^[0-9a-f]{15}$/); + expect(envelope.artifacts).toBeUndefined(); + expect(renderEnvelope(envelope)).toContain(`prima status ${envelope.status}`); + + const status = await prima.status(envelope.status!); + + expect(status.ok).toBe(true); + expect(existsSync(status.artifacts!.aria)).toBe(true); + expect(existsSync(status.artifacts!.html)).toBe(true); + expect(await Bun.file(status.artifacts!.aria).text()).toContain('Thanks for the note'); + expect(await Bun.file(status.artifacts!.html).text()).toContain('Thanks for the note'); + }); + + test('status on an unknown hash is a tool error, not a crash', async () => { + const envelope = await prima.status('000000000000000'); + + expect(envelope.ok).toBe(false); + expect(envelope.failure?.error).toContain('No command was recorded'); }); test('a non-function argument is a tool error and leaves the page alone', async () => { @@ -149,12 +162,11 @@ describe('Prima drives a real page', () => { expect(await page.title()).toBe('Widget Depot Feedback'); }); - test('a failing pw without a usable AI model reports healing as unavailable', async () => { + test('a failing pw without a usable AI model fails with the page inlined', async () => { const envelope = await prima.pw("({ page }) => page.click('text=Ship the order')"); expect(envelope.ok).toBe(false); - expect(envelope.healed).toBe(false); - expect(envelope.healNote).toContain('ai unavailable'); + expect(envelope).not.toHaveProperty('healed'); expect(envelope.failure?.compactAria).toContain('button "Submit"'); }); }); diff --git a/tests/integration/researcher.test.ts b/tests/integration/researcher.test.ts index 77356532..e534fa7b 100644 --- a/tests/integration/researcher.test.ts +++ b/tests/integration/researcher.test.ts @@ -181,13 +181,12 @@ describe('Researcher with aimock', () => { expect(prompt).toContain('/tasks/board'); }); - it('returns cached research without AI call', async () => { + it('returns cached research verbatim and without an AI call', async () => { saveResearch(fakeState.hash!, '## Cached Research\n\nPreviously analyzed page.'); const result = await researcher.research(fakeState, { fix: false }); - expect(result).toContain('Cached Research'); - expect(result).toContain('CACHED AND MAY NOT REPRESENT CURRENT STATE'); + expect(result).toBe('## Cached Research\n\nPreviously analyzed page.'); expect(mock.getRequests().length).toBe(0); }); diff --git a/tests/unit/aria.test.ts b/tests/unit/aria.test.ts index 9b14a675..da6e00f5 100644 --- a/tests/unit/aria.test.ts +++ b/tests/unit/aria.test.ts @@ -148,4 +148,109 @@ describe('aria', () => { expect(result).toContain('link "Keep 9"'); expect(result).not.toContain('omitted'); }); + + it.each([ + ['- button "Plain" [ref=e10]', 'ref=e10'], + ['- button "Active" [active] [ref=e13]', 'active ref=e13'], + ['- button "Disabled" [disabled] [ref=e14]', 'disabled ref=e14'], + ['- button "Pressed" [pressed] [ref=e15]', 'pressed ref=e15'], + ['- button "Expanded" [expanded] [ref=e16]', 'expanded ref=e16'], + ['- checkbox "Checked" [checked] [ref=e17]', 'checked ref=e17'], + ['- button "Cursor" [ref=e18] [cursor=pointer]', 'cursor=pointer ref=e18'], + ])('keeps every bracket group of %p', (snapshot, expected) => { + expect(compactAriaSnapshot(snapshot, true)).toContain(`[${expected}]`); + }); + + it('reports a typed value as a typed change, not as add/remove churn', () => { + const diff = diffAriaSnapshots('- textbox "Title"', '- textbox "Title": Bench probe'); + + expect(diff.text).toBe(['ariaDiff:', ' typed:', ' - textbox "Title": empty -> "Bench probe"', ' added: []', ' removed: []'].join('\n')); + expect(diff.count).toBe(1); + }); + + it('reports a value being replaced', () => { + const diff = diffAriaSnapshots('- textbox "Title": Bench probe', '- textbox "Title": Bench probe renamed'); + + expect(diff.text).toContain('- textbox "Title": "Bench probe" -> "Bench probe renamed"'); + expect(diff.count).toBe(1); + }); + + it('reports a value being cleared', () => { + const diff = diffAriaSnapshots('- textbox "Search": shoes', '- textbox "Search"'); + + expect(diff.text).toContain('- textbox "Search": "shoes" -> empty'); + }); + + it('treats a whitespace-only value as empty', () => { + expect(diffAriaSnapshots('- textbox "Title"', '- textbox "Title": ').text).toBeNull(); + }); + + it('reports an unchanged value as no diff at all', () => { + expect(diffAriaSnapshots('- textbox "Title": same', '- textbox "Title": same').text).toBeNull(); + }); + + it('reports a typed value alongside a toggle in the same diff', () => { + const before = ['- textbox "Title"', '- checkbox "Agree"'].join('\n'); + const after = ['- textbox "Title": Bench probe', '- checkbox "Agree" [checked]'].join('\n'); + + const diff = diffAriaSnapshots(before, after); + + expect(diff.text).toContain('typed:'); + expect(diff.text).toContain('- textbox "Title": empty -> "Bench probe"'); + expect(diff.text).toContain('toggled:'); + expect(diff.text).toContain('- checkbox "Agree": unchecked -> checked'); + expect(diff.count).toBe(2); + }); + + it('keeps a genuinely new field as added rather than typed', () => { + const diff = diffAriaSnapshots('- textbox "Title"', ['- textbox "Title"', '- textbox "Emoji": x'].join('\n')); + + expect(diff.text).toContain('added:'); + expect(diff.text).toContain('textbox "Emoji": x'); + expect(diff.text).not.toContain('typed:'); + }); + + it('excerpts a long typed value instead of printing it whole', () => { + const long = 'x'.repeat(5000); + const diff = diffAriaSnapshots(`- textbox "Editor": ${long}`, `- textbox "Editor": ${long}y`); + + expect(diff.text).toContain('(5000 chars)'); + expect(diff.text).toContain('(5001 chars)'); + expect(diff.text!.length).toBeLessThan(400); + }); + + it('truncates a long value and references the file the caller stored it in', () => { + const long = 'x'.repeat(5000); + const stored: string[] = []; + + const out = compactAriaSnapshot(`- textbox "Editor": ${long}`, true, (value) => { + stored.push(value); + return 'abc123/value-deadbeef.txt'; + }); + + expect(out).toContain('(5000 chars) [Full Text: abc123/value-deadbeef.txt]'); + expect(out.length).toBeLessThan(600); + expect(stored[0]).toBe(long); + }); + + it('truncates a long value with a char count when no store is given', () => { + const out = compactAriaSnapshot(`- textbox "Editor": ${'y'.repeat(5000)}`, true); + + expect(out).toContain('(5000 chars, truncated)'); + expect(out).not.toContain('[Full Text:'); + expect(out.length).toBeLessThan(600); + }); + + it('leaves a short value inline', () => { + expect(compactAriaSnapshot('- textbox "Name": Bench probe', true)).toContain(': Bench probe'); + }); + + it('keeps refs on stateful nodes nested in a tree', () => { + const snapshot = ['- navigation "Panel sections" [ref=e6]:', ' - button "Workspace" [ref=e10]', ' - button "Workflows" [active] [ref=e13]'].join('\n'); + + const result = compactAriaSnapshot(snapshot, true); + + expect(result).toContain('ref=e10'); + expect(result).toContain('ref=e13'); + }); }); diff --git a/tests/unit/page-readiness.test.ts b/tests/unit/page-readiness.test.ts index 51bf8ae5..c46eeccb 100644 --- a/tests/unit/page-readiness.test.ts +++ b/tests/unit/page-readiness.test.ts @@ -25,6 +25,16 @@ describe('page readiness', () => { expect(page.waitedSelectors).toEqual(['.spinner']); }); + it('finishes as soon as the dom goes quiet, without waiting for network idle', async () => { + const page = new FakePage([], 5_000, 5); + + const started = Date.now(); + await waitForPageReadiness(page, { timeout: 5_000 }); + + expect(page.waitedForDomQuiet).toBe(true); + expect(Date.now() - started).toBeLessThan(1_000); + }); + it('does not finish from spinner selectors that are not visible', async () => { const page = new FakePage([], 40); let ready = false; @@ -51,10 +61,12 @@ class FakePage { loadStates: string[] = []; waitedSelectors: string[] = []; waitedForBodyContent = false; + waitedForDomQuiet = false; constructor( private visibleSelectors: string[] = [], - private networkIdleDelay = 0 + private networkIdleDelay = 0, + private domQuietDelay = 10_000 ) {} async waitForLoadState(state: string): Promise<void> { @@ -76,7 +88,12 @@ class FakePage { }; } - async waitForFunction(): Promise<void> { + async waitForFunction(_fn: unknown, arg?: unknown): Promise<void> { + if (typeof arg === 'number') { + await sleep(this.domQuietDelay); + this.waitedForDomQuiet = true; + return; + } this.waitedForBodyContent = true; } } diff --git a/tests/unit/playwright-recorder.test.ts b/tests/unit/playwright-recorder.test.ts index 3352cf4c..f8d9a841 100644 --- a/tests/unit/playwright-recorder.test.ts +++ b/tests/unit/playwright-recorder.test.ts @@ -182,8 +182,18 @@ describe('renderAssertion', () => { expect(renderAssertion(assertion('seeHttpHeader', 'X-Api', 'v1'))).toBe(`// TODO(playwright): seeHttpHeader("X-Api", "v1")`); }); - it('falls back to TODO when seeElement arg is not a string', () => { - expect(renderAssertion(assertion('seeElement', { css: '.x' }))).toMatch(/^\/\/ TODO\(playwright\)/); + it('renders an ARIA locator as getByRole rather than a TODO', () => { + expect(renderAssertion(assertion('seeElement', { role: 'button', text: 'Add workflow' }))).toBe(`await expect(page.getByRole("button", { name: "Add workflow" })).toBeVisible();`); + expect(renderAssertion(assertion('dontSeeElement', { role: 'alert', text: 'Error' }))).toBe(`await expect(page.getByRole("alert", { name: "Error" })).toBeHidden();`); + expect(renderAssertion(assertion('seeInField', { role: 'textbox', text: 'Email' }, 'a@b.c'))).toBe(`await expect(page.getByRole("textbox", { name: "Email" })).toHaveValue("a@b.c");`); + }); + + it('renders a css-keyed object locator without a TODO', () => { + expect(renderAssertion(assertion('seeElement', { css: '.x' }))).toBe(`await expect(page.locator(".x")).toBeVisible();`); + }); + + it('still falls back to TODO when the target cannot be expressed', () => { + expect(renderAssertion(assertion('seeElement', { weird: 1 }))).toMatch(/^\/\/ TODO\(playwright\)/); }); it('falls back to TODO when seeInField is missing the value', () => { diff --git a/tests/unit/provider-usage-retry.test.ts b/tests/unit/provider-usage-retry.test.ts new file mode 100644 index 00000000..9646ffae --- /dev/null +++ b/tests/unit/provider-usage-retry.test.ts @@ -0,0 +1,40 @@ +import { beforeEach, describe, expect, it } from 'bun:test'; +import { MockLanguageModelV3 } from 'ai/test'; +import { Provider } from '../../src/ai/provider.js'; +import { Stats } from '../../src/stats.js'; + +const USAGE = { + inputTokens: { total: 1000, noCache: 1000, cacheRead: 0, cacheWrite: 0 }, + outputTokens: { total: 100, text: 100, reasoning: 0 }, +}; + +function buildFlakyModel(): MockLanguageModelV3 { + let call = 0; + return new MockLanguageModelV3({ + provider: 'test', + modelId: 'flaky-model', + doGenerate: async () => { + call++; + if (call === 1) { + return { content: [], finishReason: 'stop' as const, usage: USAGE, warnings: [] }; + } + return { content: [{ type: 'text' as const, text: 'recovered' }], finishReason: 'stop' as const, usage: USAGE, warnings: [] }; + }, + }); +} + +describe('token accounting across retries', () => { + beforeEach(() => { + Stats.models = {}; + }); + + it('counts tokens burned by an attempt that is later retried', async () => { + const model = buildFlakyModel(); + const provider = new Provider({ model, apiKey: 'test-key', config: {}, vision: false } as any); + + const response = await provider.chat([{ role: 'user', content: 'hi' }], model, { maxRetries: 3, retryDelay: 0 }); + + expect(response.text).toBe('recovered'); + expect(Stats.models['flaky-model']?.total).toBe(2200); + }); +}); diff --git a/tests/unit/tester-focus-scope.test.ts b/tests/unit/tester-focus-scope.test.ts index 6adfeb4e..6f5664fa 100644 --- a/tests/unit/tester-focus-scope.test.ts +++ b/tests/unit/tester-focus-scope.test.ts @@ -2,7 +2,7 @@ import { describe, expect, it } from 'bun:test'; import { ActionResult } from '../../src/action-result.ts'; import { Tester } from '../../src/ai/tester.ts'; import { renderExperienceToc } from '../../src/experience-tracker.ts'; -import { Test } from '../../src/test-plan.ts'; +import { Test, TestResult } from '../../src/test-plan.ts'; function buildTester(): Tester { const provider: any = { @@ -148,17 +148,66 @@ describe('Tester experience context', () => { }); describe('Tester stalled execution', () => { - it('hands a verified scenario to final review without marking it failed', () => { + it('hands a stalled scenario to final review without deciding the verdict itself', () => { const tester = buildTester(); const task = new Test('filter items', 'normal', 'filtered items appear', '/page'); const state = buildState('- main:', '/page'); (tester as any).stateManager.getCurrentState = () => state; - (tester as any).hasSuccessfulAssertion = true; expect((tester as any).shouldStopForStalledExecution(task, state, [])).toBe(false); expect((tester as any).shouldStopForStalledExecution(task, state, [])).toBe(false); expect((tester as any).shouldStopForStalledExecution(task, state, [])).toBe(true); expect(task.hasFinished).toBe(false); - expect(task.getPrintableNotes()).toContain('No further browser progress after successful verification; requesting final review'); + expect(task.result).toBe(null); + expect(task.getPrintableNotes()).toContain('No further browser progress on unchanged page; requesting final review'); + }); + + it('hands repeated execution errors to final review without marking the test failed', () => { + const tester = buildTester(); + const task = new Test('filter items', 'normal', 'filtered items appear', '/page'); + + expect((tester as any).shouldStopAfterStalledLoopError(task)).toBe(false); + expect((tester as any).shouldStopAfterStalledLoopError(task)).toBe(false); + expect((tester as any).shouldStopAfterStalledLoopError(task)).toBe(true); + expect(task.hasFinished).toBe(false); + expect(task.result).toBe(null); + }); +}); + +describe('Tester verdict', () => { + it('passes a test whose expectations were all achieved, instead of leaving it without a result', () => { + const tester = buildTester(); + const task = new Test('filter items', 'normal', ['filtered items appear', 'the count updates'], '/page'); + task.addNote('filtered items appear', TestResult.PASSED); + task.addNote('the count updates', TestResult.PASSED); + + (tester as any).finishTest(task); + + expect(task.result).toBe(TestResult.PASSED); + expect(task.isSuccessful).toBe(true); + }); + + it('fails a test that stopped without achieving every expectation', () => { + const tester = buildTester(); + const task = new Test('filter items', 'normal', ['filtered items appear', 'the count updates'], '/page'); + task.addNote('filtered items appear', TestResult.PASSED); + + (tester as any).finishTest(task); + + expect(task.result).toBe(TestResult.FAILED); + }); +}); + +describe('Tester step instructions', () => { + it('keeps the rules once the log has entries, instead of replacing them with it', async () => { + const tester = buildTester(); + const task = new Test('filter items', 'normal', ['filtered items appear'], '/page'); + task.addNote('clicked the filter button'); + + const instructions = await (tester as any).prepareInstructionsForNextStep(task); + + expect(instructions).toContain('Do not run same tool calls with same parameters again'); + expect(instructions).toContain('clicked the filter button'); + expect(instructions).toContain('filtered items appear'); }); }); diff --git a/tests/unit/url-matcher.test.ts b/tests/unit/url-matcher.test.ts index ad477133..26aa25f4 100644 --- a/tests/unit/url-matcher.test.ts +++ b/tests/unit/url-matcher.test.ts @@ -196,6 +196,19 @@ describe('url-matcher', () => { it('compares absolute and relative URLs by path', () => { expect(matchesNavigationUrl('/users', 'https://example.test/users#details')).toBe(true); }); + + it('accepts a query the app appended when none was requested', () => { + expect(matchesNavigationUrl('/', '/?session=8f2c1e&ws=1')).toBe(true); + expect(matchesNavigationUrl('http://localhost:3050', 'http://localhost:3050/?session=8f2c1e&ws=1')).toBe(true); + }); + + it('still requires the path to match when a query was appended', () => { + expect(matchesNavigationUrl('/billing', '/login?redirect=/billing')).toBe(false); + }); + + it('requires an explicitly requested query', () => { + expect(matchesNavigationUrl('/search?q=shoes', '/search?q=hats')).toBe(false); + }); }); describe('normalizeUrl', () => {