Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1049,7 +1049,7 @@ lyra-uat run examples/uat/demo/run.yaml # one persona, Claude Code plays it
lyra-uat batch examples/uat/demo/batch.yaml # two personas in parallel, one process each
```

A run leaves a directory: `report.json` (and `report.md`), `trace.jsonl`, `events.jsonl`, `captures/`, `network.jsonl`, `console.jsonl`, and the browser's own `audit.jsonl`. Persona, target, brain, hooks and limits are YAML or JSON; `lyra-uat schema run|batch|report` prints the JSON Schemas. A run that should act without anyone to ask sets `target.policy: autonomous`; the default is `closed`, which keeps the run on its trusted sites. Details, the report format and each brain's requirements are in [docs/UAT.md](https://github.com/Intrect-io/lyra-browser/blob/master/docs/UAT.md).
A run leaves a directory: `report.json` (and `report.md`), `trace.jsonl`, `events.jsonl`, `captures/`, `network.jsonl`, `console.jsonl`, and the browser's own `audit.jsonl`. Persona, target, brain, hooks and limits are YAML or JSON; `lyra-uat schema run|batch|report|export` prints the JSON Schemas. Spec `asserts` judge the final page without the model; `lyra-uat export` writes a versioned record for a benchmark and `lyra-uat replay` re-runs a run's own calls to check the judgement holds. A run that should act without anyone to ask sets `target.policy: autonomous`; the default is `closed`, which keeps the run on its trusted sites. Details, the report format and each brain's requirements are in [docs/UAT.md](https://github.com/Intrect-io/lyra-browser/blob/master/docs/UAT.md).

## Headless mode

Expand Down
46 changes: 45 additions & 1 deletion docs/UAT.md
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,7 @@ uat-runs/<run-id>/
observations.json counts, and what the server knows of the browser
browser/ the browser's data dir: audit.jsonl and a profile of its own
harness/ harness brains: prompts, argv, stdout/stderr, MCP config
run.json the spec the run was started from
run.json the spec the run was started from (what replay reads)
```

`report.json` (schema version 1; `lyra-uat schema report`):
Expand All @@ -223,6 +223,50 @@ uat-runs/<run-id>/
that never clicked, an invalid profile, observers that failed to attach.
- `brain` — backend, model, turns, tokens (input, output, cache read/write),
cost where known, session id. `observations`, `hooks`, `artifacts`.
- `asserts` / `asserts_passed` — the spec's assertions on the final page and their
conjunction (below); `asserts_passed` is `null` when the spec has none.

### Assertions: a judgement the model cannot make up

`finish` is what the persona *says* happened. `asserts` in a run or batch spec say what
the page must *show*, and the server checks them itself, once: when `finish` is called,
on the page the persona finished on, or, if it never is, when the runner closes the
browser. A harness that exits without `finish` leaves them `not_evaluated`, and that
counts as not passed.

```yaml
asserts:
- {id: on-receipt, kind: url_matches, value: "/orders/\\d+$"} # regular expression
- {id: title, kind: title_contains, value: "Order complete"}
- {id: thanks, kind: text_present, value: "Thank you", selector: main} # selector optional
- {id: no-error, kind: text_absent, value: "declined"}
- {id: one-receipt, kind: selector_count, selector: ".receipt", min: 1, max: 1}
```

Text is the visible text (`innerText`) of the page or of what `selector` matches. Each
row records what was observed, so a failure says what the page showed instead.

### Export and replay

```bash
lyra-uat export uat-runs/<run-id> [--out run.export.json] # one flat record
lyra-uat schema export # its JSON Schema
lyra-uat replay uat-runs/<run-id> [--out DIR] # same calls, same judgement?
```

`export` writes the stable part of a run that a benchmark or dashboard reads: task
(persona) id, `success` (the assertions' verdict, `null` without any), the outcome the
persona claimed, final URL, steps, tool calls, findings, tokens, cost and every step.
The shape is `lyra-uat-export` version 1; its schema is committed as
`src/lyra_browser/uat/schemas/export.v1.json` and a test keeps it equal to the model, so
a change of shape is a new version. A benchmark's own result format is a mapping from
this record, kept with that benchmark's adapter.

`replay` runs the recorded tool calls again, in order, with the `scripted` brain and the
run's own spec (`run.json`), and compares status, exit reason, outcome, steps used and
every assertion. It prints the comparison and exits 0 when the judgement is the same, 2
when it is not. Values the audit redacts (typed passwords) cannot be replayed as typed:
the result then says `"exact": false`.

`lyra-uat run` exits 0 when the persona called `finish`, 2 when the run did not
complete (budget, wall clock, a brain or hook failure, an invalid profile), and 130
Expand Down
72 changes: 71 additions & 1 deletion scripts/verify_uat_e2e.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,15 @@ def expect(condition: bool, message: str, detail: object = "") -> None:
print(f"PASS {message}")


def spec_for(base: str, out: Path, *, budget: int, script: list[dict], hooks: dict | None = None):
def spec_for(
base: str,
out: Path,
*,
budget: int,
script: list[dict],
hooks: dict | None = None,
asserts: list[dict] | None = None,
):
from lyra_browser.uat.spec import RunSpec

origin = base.rstrip("/")
Expand All @@ -54,6 +62,7 @@ def spec_for(base: str, out: Path, *, budget: int, script: list[dict], hooks: di
},
"brain": {"backend": "scripted", "script": script},
"hooks": hooks or {},
"asserts": asserts or [],
"out_dir": str(out),
"headless": True,
}
Expand Down Expand Up @@ -175,6 +184,66 @@ async def completed_run(base: str, alt_base: str, out: Path) -> None:
expect((run_dir / "browser" / "profile").is_dir(), "run has its own browser profile")


# What the pricing page shows after "Buy Studio": its click sets the title to "bought".
ASSERTS = [
{"id": "on-pricing", "kind": "url_matches", "value": "/pricing$"},
{"id": "bought", "kind": "title_contains", "value": "bought"},
{"id": "button", "kind": "text_present", "value": "Buy Studio", "selector": "#buy"},
{"id": "one-button", "kind": "selector_count", "selector": "#buy", "min": 1, "max": 1},
{"id": "no-free-plan", "kind": "text_absent", "value": "Free forever"},
{"id": "no-cart", "kind": "selector_count", "selector": ".cart-item", "min": 1},
]


async def judged_exported_and_replayed(base: str, out: Path) -> None:
"""Assertions, the export and a replay, on a run of their own: straight to the pricing
page by URL, then "Buy Studio", then finish."""
from lyra_browser.uat.export import committed_schema, export_report
from lyra_browser.uat.replay import replay_run
from lyra_browser.uat.runner import run_spec

script = [
{"tool": "navigate", "args": {"url": base}, "expect": "ok"},
{"tool": "navigate", "args": {"url": f"{base}pricing"}, "expect": "ok"},
{"tool": "click", "args": {"selector": "#buy"}, "expect": "ok"},
{"tool": "finish", "args": {"outcome": "reached_goal", "summary": "Bought."}},
]
report = await run_spec(spec_for(base, out, budget=5, script=script, asserts=ASSERTS))
print(report.one_line())
expect(report.run.exit_reason == "finish", "judged run finished", report.brain.error)

statuses = {a.id: a.status for a in report.asserts}
expect(
statuses
== {
"on-pricing": "passed",
"bought": "passed",
"button": "passed",
"one-button": "passed",
"no-free-plan": "passed",
"no-cart": "failed",
},
"assertions judged on the finished page by the server",
[(a.id, a.status, a.observed, a.error) for a in report.asserts],
)
expect(report.asserts_passed is False, "one failed assertion fails the judgement")
record = export_report(report)
try:
import jsonschema
except ImportError:
print("NOTE jsonschema is not installed: the export is checked by its model only")
else:
jsonschema.validate(record.model_dump(mode="json"), committed_schema())
print("PASS export validates against schemas/export.v1.json")
expect(record.success is False and record.task_id == "e2e", "export record", record.success)
result = await replay_run(Path(report.artifacts.dir))
expect(
result["same"] and result["exact"],
"replay of every step reaches the same judgement",
result,
)


async def budget_run(base: str, out: Path) -> None:
from lyra_browser.uat.runner import run_spec

Expand Down Expand Up @@ -203,6 +272,7 @@ async def main(keep: bool) -> None:
print(f"demo site {base}, runs in {out}")
await completed_run(base, alt_base, out)
await budget_run(base, out)
await judged_exported_and_replayed(base, out)
finally:
server.shutdown()
if keep:
Expand Down
3 changes: 3 additions & 0 deletions src/lyra_browser/tools/uat.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
from fastmcp import FastMCP

from ..context import ServerContext
from ..uat.asserts import evaluate_once
from ..uat.recorder import RunRecorder

SEVERITIES = ("blocker", "major", "minor")
Expand Down Expand Up @@ -128,6 +129,8 @@ async def finish(
return bad
if not summary.strip():
return {"status": "error", "reason": "summary must not be empty."}
# The page the persona finished on is the page the spec's assertions judge.
await evaluate_once(ctx, recorder)
recorder.set_finish(
outcome=outcome,
summary=summary,
Expand Down
119 changes: 119 additions & 0 deletions src/lyra_browser/uat/asserts.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
"""Deterministic checks on the page a run ended on.

The persona's ``finish`` says what it believes happened; an assertion says what the page
shows. Assertions are part of the spec, evaluated by the server — not the model — once:
when ``finish`` is called, on the page the persona finished on, or, for a run that never
calls it, when the runner closes the browser. The same spec on the same page gives the
same answer, which is what makes a run comparable with its replay and with a benchmark's
expected result.
"""

from __future__ import annotations

import re
from typing import Any

from .spec import Assertion

# One call for every text/count assertion: the page's visible text (all of it, or what a
# selector matches) and how many elements a selector matches. ``innerText`` is what a
# person sees; hidden text does not count.
_PAGE_STATE_JS = """
(selector) => {
if (!selector) {
return {text: document.body ? document.body.innerText : "", count: null, error: null};
}
let nodes;
try {
nodes = Array.from(document.querySelectorAll(selector));
} catch (e) {
return {text: "", count: 0, error: String(e && e.message || e)};
}
return {text: nodes.map(n => n.innerText || n.textContent || "").join("\\n"),
count: nodes.length, error: null};
}
"""


def _result(a: Assertion, passed: bool, observed: Any, error: str | None = None) -> dict:
return {
"id": a.id,
"kind": a.kind,
"status": "error" if error else ("passed" if passed else "failed"),
"expected": a.describe(),
"observed": observed,
**({"error": error} if error else {}),
}


async def _state(page: Any, selector: str | None) -> dict[str, Any]:
state = await page.evaluate(_PAGE_STATE_JS, selector)
if not isinstance(state, dict):
raise RuntimeError(f"the page answered {state!r}")
return state


async def evaluate(page: Any, assertions: list[Assertion]) -> list[dict[str, Any]]:
"""Every assertion against ``page``; one that cannot be evaluated is an ``error`` row,
never an exception (the report must still be written)."""
results: list[dict[str, Any]] = []
for a in assertions:
try:
if a.kind == "url_matches":
url = str(page.url)
results.append(_result(a, re.search(a.value, url) is not None, url))
elif a.kind == "title_contains":
title = str(await page.title())
results.append(_result(a, a.value in title, title))
elif a.kind in ("text_present", "text_absent"):
state = await _state(page, a.selector)
if state.get("error"):
results.append(_result(a, False, None, f"selector: {state['error']}"))
continue
found = a.value in str(state.get("text") or "")
results.append(_result(a, found == (a.kind == "text_present"), found))
elif a.kind == "selector_count":
state = await _state(page, a.selector)
if state.get("error"):
results.append(_result(a, False, None, f"selector: {state['error']}"))
continue
count = int(state.get("count") or 0)
ok = (a.min is None or count >= a.min) and (a.max is None or count <= a.max)
results.append(_result(a, ok, count))
except Exception as exc: # noqa: BLE001 - a dead page is a result, not a crash
results.append(_result(a, False, None, f"{type(exc).__name__}: {exc}"[:300]))
return results


async def evaluate_once(ctx: Any, recorder: Any) -> None:
"""Evaluate the run's assertions on the current page and record them, once per run.

Never launches a browser: a run whose browser is gone records every assertion as an
``error`` saying so."""
if not recorder.assertions or recorder.asserts_done:
return
recorder.asserts_done = True
if not getattr(ctx.session, "_launched_once", True):
# ``page()`` would open a browser that was never opened, only to judge it.
recorder.add_asserts(
[
{
"id": a.id,
"kind": a.kind,
"status": "not_evaluated",
"expected": a.describe(),
"observed": None,
"error": "the browser was never opened",
}
for a in recorder.assertions
]
)
return
try:
page = await ctx.session.page(revive=False)
except Exception as exc: # noqa: BLE001 - no page is a result
reason = f"no page to check: {type(exc).__name__}: {exc}"[:300]
rows = [_result(a, False, None, reason) for a in recorder.assertions]
else:
rows = await evaluate(page, recorder.assertions)
recorder.add_asserts(rows)
1 change: 1 addition & 0 deletions src/lyra_browser/uat/batch.py
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,7 @@ def build_run_spec(batch: BatchSpec, persona_id: str, runs_dir: Path) -> RunSpec
hooks=batch.hooks,
limits=batch.limits,
tools=batch.tools,
asserts=batch.asserts,
out_dir=runs_dir,
data_dir=batch.data_dirs.get(persona_id),
headless=batch.headless,
Expand Down
45 changes: 43 additions & 2 deletions src/lyra_browser/uat/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,10 @@
lyra-uat run <spec> [--target FILE] [--brain NAME] [--model M] [--out DIR] ...
lyra-uat batch <batch.yaml>
lyra-uat validate <spec>
lyra-uat schema [run|batch|report]
lyra-uat schema [run|batch|report|export]
lyra-uat render <report.json>
lyra-uat export <run dir | report.json> [--out FILE]
lyra-uat replay <run dir> [--out DIR]

Exit codes of ``run``: 0 when the run completed (findings or not), 2 when it
was cut off (budget, wall clock, harness) or the brain never finished, 130 when
Expand Down Expand Up @@ -66,11 +68,23 @@ def build_parser() -> argparse.ArgumentParser:
validate.add_argument("--target", type=Path)

schema = sub.add_parser("schema", help="Print a JSON Schema.")
schema.add_argument("which", nargs="?", default="report", choices=["run", "batch", "report"])
schema.add_argument(
"which", nargs="?", default="report", choices=["run", "batch", "report", "export"]
)

render = sub.add_parser("render", help="Render report.json as Markdown to stdout.")
render.add_argument("report", type=Path)

export = sub.add_parser("export", help="Write a run as a versioned export record.")
export.add_argument("report", type=Path, help="A run directory or its report.json.")
export.add_argument("--out", type=Path, help="File to write (default: stdout).")

replay = sub.add_parser(
"replay", help="Run a recorded run again with its own tool calls and compare."
)
replay.add_argument("run_dir", type=Path)
replay.add_argument("--out", type=Path, help="Where the replay's run goes (default: beside).")

demo = sub.add_parser(
"demo-site", help="Serve the built-in demo site to run a persona against."
)
Expand Down Expand Up @@ -209,6 +223,10 @@ def cmd_schema(args: argparse.Namespace) -> int:
schema = RunSpec.model_json_schema()
elif args.which == "batch":
schema = BatchSpec.model_json_schema()
elif args.which == "export":
from .export import export_schema

schema = export_schema()
else:
from .report import Report

Expand All @@ -225,6 +243,27 @@ def cmd_render(args: argparse.Namespace) -> int:
return EXIT_OK


def cmd_export(args: argparse.Namespace) -> int:
from .export import export_report, load_report

record = export_report(load_report(args.report)).model_dump_json(indent=2)
if args.out:
args.out.write_text(record, encoding="utf-8")
print(args.out)
else:
print(record)
return EXIT_OK


def cmd_replay(args: argparse.Namespace) -> int:
"""0 when the replay reached the same judgement, 2 when it did not."""
from .replay import replay_run

result = asyncio.run(replay_run(args.run_dir, out_dir=args.out))
print(json.dumps(result, indent=2, ensure_ascii=False))
return EXIT_OK if result["same"] else EXIT_INCOMPLETE


def cmd_demo_site(args: argparse.Namespace) -> int:
from .demo_site import serve_forever

Expand All @@ -238,6 +277,8 @@ def cmd_demo_site(args: argparse.Namespace) -> int:
"validate": cmd_validate,
"schema": cmd_schema,
"render": cmd_render,
"export": cmd_export,
"replay": cmd_replay,
"demo-site": cmd_demo_site,
}

Expand Down
Loading
Loading