diff --git a/README.md b/README.md index e5a57f2..c759a8a 100644 --- a/README.md +++ b/README.md @@ -301,10 +301,24 @@ to get past. It is a page a person has to answer. lyra names the check and stops says not to wait and not to look for another way past, to report the task as blocked (`finish` with `outcome=blocked`) and carry on with what remains. -Nothing solves, waits out or retries a check. Detection reads the frame hosts, the page -title and the first 3000 characters of visible text; an article that quotes a check's -wording further down is not named. A detection in `navigate` is audited as -`challenge_seen`; `read_page` only reports it in its reply. +lyra itself solves, waits out or retries no check. Detection reads the frame hosts, the +page title and the first 3000 characters of visible text; an article that quotes a +check's wording further down is not named. A detection in `navigate` is audited as +`challenge_seen` (with the attempt number); `read_page` only reports it in its reply. + +- **The same wall again.** `navigate` counts each check by vendor and site. The reply + carries `attempts` and `max_attempts`; when the same check stops the agent for the + `LYRA_BROWSER_CHALLENGE_MAX_ATTEMPTS`th time (default 3) the status is + `challenge_abandoned` and the hint says to stop and report it as blocked. Coming back + to the same check is not progress, and a site that sees it is right to harden against + the account. +- **An operator's solver.** `LYRA_BROWSER_CAPTCHA_SOLVER=package.module:function` names + a callable lyra hands an `interactive` check to when nobody is watching (never a + `blocked` one, never with a person at the window). It is called as + `solver(info, page)` with `info = {vendor, kind, evidence, url, attempt}`, may be + async, and whatever it does is its own responsibility; lyra ships none. Its answer is + reported (`solver`), never trusted: the page is checked again, and only a page that is + no longer a check answers `ok` with `challenge_solved`. Audited as `challenge_solver`. ### Hover and scroll @@ -1020,6 +1034,8 @@ throwaway keyring. | `LYRA_BROWSER_TRUSTED_SEND_ORIGINS` | — | Sites where *sending* is also pre-approved — `SUBMIT` and `UPLOAD` — for acceptance runs against your own product. Same entry forms, **separate list**: a site in `TRUSTED_ORIGINS` is not send-trusted by being there. Never `PUBLISH` or `DOWNLOAD`; each approval stays single-use, and it steps aside during a takeover. Audited as `consent_channel=trusted_send`. | | `LYRA_BROWSER_FOLLOW_BYPASS` | on | Claude Code client only: while Claude Code runs in bypassPermissions mode, consent requests are answered without a dialog (`consent_channel=bypass`). The mode arrives from the plugin's PreToolUse hook, never from the model; see docs/CLAUDE_CODE_PLUGIN.md (Bypass mode). | | `LYRA_BROWSER_DENIED_ORIGINS` | — | Sites refused whatever else says yes: the trusted lists, a live grant and the `autonomous` channel all step aside. Same entry forms. | +| `LYRA_BROWSER_CHALLENGE_MAX_ATTEMPTS` | `3` | How many times the same bot check (vendor at a site) may stop `navigate` before it answers `challenge_abandoned`. | +| `LYRA_BROWSER_CAPTCHA_SOLVER` | — | `module:function` of a solver an interactive bot check is handed to when nobody is watching. None is shipped; see [Bot checks](#bot-checks-challenge). | | `LYRA_BROWSER_CONSENT_TIMEOUT` | `300` | Seconds a prompt waits for an answer before it counts as a denial (`timed out waiting for the user`). The tool call is held that long, so for an unattended Hermes gateway set it lower — e.g. `60`. | | `LYRA_BROWSER_RELEASE_GRACE` | `2` | Seconds before a finished action's unused single-use scope is reclaimed. | | `LYRA_BROWSER_OWNER_IDLE_TIMEOUT` | `900` | Seconds a session may hold the browser without using it. Handing it on closes the window. | diff --git a/docs/UAT.md b/docs/UAT.md index 569c63e..c4ef748 100644 --- a/docs/UAT.md +++ b/docs/UAT.md @@ -207,7 +207,8 @@ uat-runs// - `run` — `status` (`completed`, `incomplete`, `error`), `exit_reason` (`finish`, `budget_exhausted`, `wall_timeout`, `brain_error`, `browser_error`, - `harness_exit`, `hook_failed`, `no_finish`, `interrupted`), timing. + `harness_exit`, `hook_failed`, `no_finish`, `interrupted`, `challenge_abandoned`), + timing. - `outcome` — what `finish` said (`reached_goal`, `partial`, `blocked`) or `unknown` when it never did; steps used against the budget; call count. - `findings` — severity (`blocker`, `major`, `minor`), URL, step, expected, @@ -257,6 +258,47 @@ This is POSIX code (process groups, signals). It was run on Linux only: where th `/proc` (macOS) it falls back to `killpg` alone for the liveness check, which has not been run, and Windows is not supported. +## Logins, one-time codes and bot checks + +A persona that has to sign in gets the login **by name**. The spec says where each secret +lives, never the secret: + +```yaml +credentials: + shop: + origin: https://app.example.com # the one site it may be typed into + username: tester@example.com # or username_env + password_env: SHOP_TEST_PASSWORD # or password_file + totp_file: ~/.config/shop-totp-seed # or totp_env: a base32 seed, as sites show it +``` + +With `credentials` set the persona is offered `fill_credential(name, field, selector, +submit=false)`, where `field` is `username`, `password` or `totp` (the code of this +moment, RFC 6238, computed when it is typed). The server types the value through +`type_text`, so the same consent, enforcement and audit redaction apply, and only on the +credential's own `origin`: anywhere else the answer is `wrong_origin` and nothing is +typed. A secret that cannot be read is `credential_unavailable`, naming the variable or +file. The value is in no argument, result, trace row or report: the persona, the run +file and the report hold the name and the field. + +Every password and TOTP seed of the spec is also held by a scrubber: a page that echoes +it, a result that quotes it, or a POST body `network_capture` keeps is written (and shown +to the model) with `` in its place, in its plain, URL-encoded and JSON forms. +Usernames and the six-digit codes are not scrubbed. Screenshots are pixels: a password +field shows dots, a code field shows the code of that moment. + +Bot checks are named as in [the README](../README.md#bot-checks-challenge). In a run the +third meeting with the same check (`challenge_abandoned`) ends the persona's browsing: +every later browser tool is refused, the policy event `challenge_abandoned` is recorded, +and without a `finish` the report ends `incomplete` with exit reason +`challenge_abandoned`. `target.captcha_solver` (`module:function`) installs a solver for +the run's server; none is shipped. + +`scripts/verify_unattended_e2e.py` is the gate: on the demo site (`/login` -> `/2fa` -> +`/account`, fresh canary password and seed per run) a scripted persona signs in with +`fill_credential` in a real headless Chrome, meets `/gate` three times, and every file +of both runs is searched for the password and the seed. + ## Hooks `hooks.before` and `hooks.after` are shell commands. A failing `before` (non-zero) diff --git a/scripts/verify_unattended_e2e.py b/scripts/verify_unattended_e2e.py new file mode 100644 index 0000000..a5f1f31 --- /dev/null +++ b/scripts/verify_unattended_e2e.py @@ -0,0 +1,187 @@ +#!/usr/bin/env python3 +"""Real-browser gate: an unattended sign-in with a password and a one-time code, a bot +check that is given up on, and no secret anywhere in what the run leaves. + +The built-in demo site serves ``/login`` -> ``/2fa`` -> ``/account`` (the password and +the TOTP seed are fresh canaries for each run) and ``/gate``, a page that reads as a bot +check. A scripted brain plays the persona through the real runner and a real headless +Chrome: + +1. sign-in run: ``fill_credential`` types the email, the password and the current code + (the persona never sees them), the forms are sent, and the run ends on ``/account``. + ``network_capture`` keeps the two POSTs whole, the way an analytics check would. +2. challenge run: ``/gate`` three times -> ``challenge``, ``challenge``, + ``challenge_abandoned``; the next browser call is refused, and the report ends + ``challenge_abandoned``. + +Then every file of both runs (trace, events, audit, report, run file, network and console +logs, observations, harness and capture metadata) is searched for the password and the +seed: there must be none. + + python scripts/verify_unattended_e2e.py [--keep] +""" + +from __future__ import annotations + +import argparse +import asyncio +import base64 +import os +import secrets +import shutil +import sys +import tempfile +from pathlib import Path + +REPO = Path(__file__).resolve().parent.parent +sys.path.insert(0, str(REPO / "src")) + + +def expect(condition: bool, message: str, detail: object = "") -> None: + if not condition: + raise AssertionError(f"{message}: {detail}") + print(f"PASS {message}") + + +def spec_for(base: str, out: Path, seed_file: Path, script: list[dict]): + from lyra_browser.uat.spec import RunSpec + + origin = base.rstrip("/") + return RunSpec.model_validate( + { + "persona": { + "id": "signin", + "name": "Returning customer", + "entry_url": f"{base}login", + "goal": "sign in and open the account page", + "step_budget": 12, + }, + "target": { + "trusted_origins": [origin], + "trusted_send_origins": [origin], + "network_capture": ["/login", "/2fa"], + }, + "brain": {"backend": "scripted", "script": script}, + "credentials": { + "demo": { + "origin": origin, + "username": "demo@example.com", + "password_env": "LYRA_GATE_DEMO_PASSWORD", + "totp_file": str(seed_file), + } + }, + "out_dir": str(out), + "headless": True, + } + ) + + +def fill(field: str, selector: str, **extra) -> dict: + return { + "tool": "fill_credential", + "args": {"name": "demo", "field": field, "selector": selector, **extra}, + "expect": "ok", + } + + +async def signin_run(base: str, out: Path, seed_file: Path): + from lyra_browser.uat.runner import run_spec + + script = [ + {"tool": "navigate", "args": {"url": f"{base}login"}, "expect": "ok"}, + fill("username", "#user"), + fill("password", "#pass"), + {"tool": "click", "args": {"selector": "#signin", "submits": True}, "expect": "ok"}, + {"tool": "wait_for", "args": {"text": "authenticator", "timeout_ms": 8000}}, + fill("totp", "#code", submit=True), + {"tool": "wait_for", "args": {"text": "Signed in as", "timeout_ms": 8000}}, + {"tool": "get_url", "args": {}}, + {"tool": "finish", "args": {"outcome": "reached_goal", "summary": "Signed in."}}, + ] + report = await run_spec(spec_for(base, out, seed_file, script)) + print(report.one_line()) + expect(report.run.exit_reason == "finish", "sign-in run finished", report.brain.error) + fills = [s for s in report.trace if s.tool == "fill_credential"] + expect([s.status for s in fills] == ["ok", "ok", "ok"], "three fields typed", fills) + expect( + all(set(s.args) <= {"name", "field", "selector", "submit"} for s in fills), + "the trace holds the credential's name and field only", + [s.args for s in fills], + ) + last = report.trace[-2] + expect(last.url_after.endswith("/account"), "the run ended signed in on /account", last) + return report + + +async def challenge_run(base: str, out: Path, seed_file: Path): + from lyra_browser.uat.runner import run_spec + + script = [{"tool": "navigate", "args": {"url": f"{base}login"}, "expect": "ok"}] + script += [{"tool": "navigate", "args": {"url": f"{base}gate"}} for _ in range(3)] + script += [{"tool": "read_page", "args": {}}, {"tool": "note", "args": {"text": "walled"}}] + report = await run_spec(spec_for(base, out, seed_file, script)) + print(report.one_line()) + statuses = [s.status for s in report.trace] + expect( + statuses + == ["ok", "challenge", "challenge", "challenge_abandoned", "challenge_abandoned", "ok"], + "the same check: named twice, abandoned the third time, then refused", + statuses, + ) + expect( + report.run.exit_reason == "challenge_abandoned" and report.run.status == "incomplete", + "the run ends as challenge_abandoned", + report.run, + ) + return report + + +def leaks(run_dir: Path, needles: list[str]) -> list[str]: + found = [] + for path in run_dir.rglob("*"): + if not path.is_file(): + continue + data = path.read_bytes() + found += [f"{path}: {n[:4]}…" for n in needles if n.encode() in data] + return found + + +async def main(keep: bool) -> None: + from lyra_browser.uat import demo_site + + password = "gate-" + secrets.token_urlsafe(12) + seed = base64.b32encode(secrets.token_bytes(20)).decode().rstrip("=") + demo_site.Handler.password = password + demo_site.Handler.totp_seed = seed + os.environ["LYRA_GATE_DEMO_PASSWORD"] = password + server, base, _alt = demo_site.start() + out = Path(tempfile.mkdtemp(prefix="lyra-unattended-e2e-")) + seed_file = out / "seed.txt" + seed_file.write_text(seed) + try: + print(f"demo site {base}, runs in {out}") + signed = await signin_run(base, out / "runs", seed_file) + walled = await challenge_run(base, out / "runs", seed_file) + for report in (signed, walled): + run_dir = Path(report.artifacts.dir) + found = leaks(run_dir, [password, seed]) + expect(not found, f"no password or seed in any file of {run_dir.name}", found) + captured = (Path(signed.artifacts.dir) / "network.jsonl").read_text(encoding="utf-8") + expect( + "/login" in captured and "" in captured, + "the captured sign-in POST is kept, with the password replaced", + ) + finally: + server.shutdown() + if keep: + print(f"kept {out}") + else: + shutil.rmtree(out, ignore_errors=True) + + +if __name__ == "__main__": + ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + ap.add_argument("--keep", action="store_true", help="Keep the run directories.") + args = ap.parse_args() + asyncio.run(main(args.keep)) + print("ALL PASS") diff --git a/src/lyra_browser/challenge.py b/src/lyra_browser/challenge.py index ef7fc76..5d1ecbe 100644 --- a/src/lyra_browser/challenge.py +++ b/src/lyra_browser/challenge.py @@ -8,12 +8,27 @@ ``classify`` is the judgement and takes plain values, so it is tested without a browser. ``observe`` collects those values from a page. Nothing here solves, waits out -or retries a challenge. +or retries a challenge by itself. + +Two things sit around that judgement: + +- ``ChallengeLedger`` counts how often the same check (vendor, at an origin) has stopped + the agent. Past ``Config.challenge_max_attempts`` the answer is ``challenge_abandoned``: + coming back to the same wall is not progress, and a site that sees it is right to + harden its stance against the account. +- A *solver hook* (``Config.captcha_solver``, ``"module:function"``) the operator may + install. lyra ships none. When one is configured and nobody is attending, an + interactive check is handed to it once per attempt; whatever it does is its own + responsibility (a human-in-the-loop service, a test double), and the page is observed + again afterwards to see whether the check is gone. """ from __future__ import annotations -from dataclasses import asdict, dataclass +import importlib +import inspect +from collections.abc import Awaitable, Callable +from dataclasses import asdict, dataclass, field from typing import Any INTERACTIVE = "interactive" @@ -124,6 +139,57 @@ async def observe(page: Any, http_status: int | None = None) -> Challenge | None ) +@dataclass +class ChallengeLedger: + """How many times each check (vendor at an origin) has been met in this session.""" + + seen: dict[tuple[str, str], int] = field(default_factory=dict) + + def record(self, origin: str, challenge: Challenge) -> int: + key = (origin, challenge.vendor) + self.seen[key] = self.seen.get(key, 0) + 1 + return self.seen[key] + + def clear(self, origin: str, challenge: Challenge) -> None: + self.seen.pop((origin, challenge.vendor), None) + + +Solver = Callable[[dict[str, Any], Any], Awaitable[Any] | Any] +"""``solver(info, page)``: ``info`` is the challenge with ``url`` and ``attempt``; +``page`` is the live page. Its return value is reported, never trusted: the page is +observed again to see whether the check is gone.""" + + +def load_solver(spec: str) -> Solver: + """``"package.module:function"`` to the callable it names.""" + module_name, sep, attr = spec.partition(":") + if not sep or not module_name or not attr: + raise ValueError(f"captcha_solver must be 'module:function', got {spec!r}") + target = getattr(importlib.import_module(module_name), attr) + if not callable(target): + raise ValueError(f"captcha_solver {spec!r} is not callable") + return target + + +async def run_solver(solver: Solver, info: dict[str, Any], page: Any) -> str: + """Call the solver; what it said, as a short string for the envelope and the audit.""" + try: + answer = solver(info, page) + if inspect.isawaitable(answer): + answer = await answer + except Exception as exc: # noqa: BLE001 - a broken solver is reported, not raised + return f"error: {type(exc).__name__}: {exc}"[:200] + return str(answer)[:200] + + +def abandon_hint(challenge: Challenge, attempts: int) -> str: + return ( + f"The same {challenge.vendor} check has stopped this {attempts} times. Do not " + "try again and do not look for another way past it: report it as blocked " + "(finish with outcome=blocked) and continue with what remains of the task." + ) + + def hint_for(challenge: Challenge, attended: bool) -> str: """What the agent should do next, depending on whether a person is watching.""" if challenge.kind == BLOCKED: diff --git a/src/lyra_browser/config.py b/src/lyra_browser/config.py index f6c7c53..79f9319 100644 --- a/src/lyra_browser/config.py +++ b/src/lyra_browser/config.py @@ -517,6 +517,12 @@ class Config: # denial. Five minutes suits a person at a screen; a gateway nobody is # watching wants it short (docs/HERMES_INTEGRATION.md). consent_timeout_s: float = 300.0 + # How many times the same bot check (vendor at an origin) may stop the agent before + # navigate answers ``challenge_abandoned`` instead of ``challenge``. + challenge_max_attempts: int = 3 + # ``"module:function"`` of an operator-installed solver an interactive bot check is + # handed to when nobody is attending. None: checks are named and handed over only. + captcha_solver: str | None = None # Sites the operator has pre-approved for NAVIGATE (and the INTERACT it # implies), so one they always use is not asked about again. Entry forms: # see parse_trusted_origins. An answer, not a consent channel, and never a @@ -674,6 +680,10 @@ def from_env(cls, client: str | None = None) -> Config: # substitutes an empty string, and that must not read as "off". follow_bypass=_as_optional_bool(_env("LYRA_BROWSER_FOLLOW_BYPASS")) is not False, consent_timeout_s=_as_float(_env("LYRA_BROWSER_CONSENT_TIMEOUT"), default=300.0), + challenge_max_attempts=max( + 1, _as_int(_env("LYRA_BROWSER_CHALLENGE_MAX_ATTEMPTS"), default=3) + ), + captcha_solver=_env("LYRA_BROWSER_CAPTCHA_SOLVER") or None, trusted_origins=parse_trusted_origins(_env("LYRA_BROWSER_TRUSTED_ORIGINS")).entries, trusted_send_origins=parse_trusted_origins( _env("LYRA_BROWSER_TRUSTED_SEND_ORIGINS") diff --git a/src/lyra_browser/context.py b/src/lyra_browser/context.py index 48ecc6b..761829e 100644 --- a/src/lyra_browser/context.py +++ b/src/lyra_browser/context.py @@ -9,6 +9,7 @@ from .approval import CollaborationState from .audit import AuditLog +from .challenge import ChallengeLedger from .config import Config from .consent import ConsentBroker from .devtools import DevtoolsRecorder @@ -42,6 +43,8 @@ class ServerContext: """Judges every file a tab is offered, and saves the ones asked for. See downloads.py.""" devtools: DevtoolsRecorder | None = None """The console and network log every tab feeds. See devtools.py.""" + challenges: ChallengeLedger | None = None + """How often each bot check has stopped the agent. See challenge.py.""" def __post_init__(self) -> None: if self.perms is None: @@ -52,6 +55,8 @@ def __post_init__(self) -> None: self.guard = NavigationGuard(self.config, self.perms, self.audit, collab=self.collab) if self.claim_lock is None: self.claim_lock = asyncio.Lock() + if self.challenges is None: + self.challenges = ChallengeLedger() if self.downloads is None: self.downloads = Downloads( self.config, diff --git a/src/lyra_browser/scrub.py b/src/lyra_browser/scrub.py new file mode 100644 index 0000000..171e5ac --- /dev/null +++ b/src/lyra_browser/scrub.py @@ -0,0 +1,48 @@ +"""Keep known secret values out of what a run writes and what a model reads. + +The audit already redacts the arguments that carry typed values. A secret can still come +back another way: a page that echoes a field, a captured POST body, a tool result that +quotes the form. A ``Scrubber`` holds the values that must never appear (a password, a +TOTP seed) and replaces them, as they are and as a form or URL encodes them, wherever +text is written or returned. Values shorter than four characters are not held: replacing +them would mangle ordinary text and they protect nothing. +""" + +from __future__ import annotations + +import json +from typing import Any +from urllib.parse import quote, quote_plus + +MARK = "" +MIN_LEN = 4 + + +class Scrubber: + def __init__(self, secrets: list[str] | tuple[str, ...] = ()) -> None: + self._needles: list[str] = [] + for secret in secrets: + self.add(secret) + + def add(self, secret: str) -> None: + if not secret or len(secret) < MIN_LEN: + return + for form in {secret, quote(secret, safe=""), quote_plus(secret), json.dumps(secret)[1:-1]}: + if form not in self._needles: + self._needles.append(form) + self._needles.sort(key=len, reverse=True) + + def __bool__(self) -> bool: + return bool(self._needles) + + def text(self, value: str) -> str: + for needle in self._needles: + if needle in value: + value = value.replace(needle, MARK) + return value + + def data(self, value: Any) -> Any: + """A JSON-shaped value with every string scrubbed.""" + if not self._needles: + return value + return json.loads(self.text(json.dumps(value, ensure_ascii=False, default=str))) diff --git a/src/lyra_browser/tools/navigation.py b/src/lyra_browser/tools/navigation.py index b287e26..97d2920 100644 --- a/src/lyra_browser/tools/navigation.py +++ b/src/lyra_browser/tools/navigation.py @@ -5,7 +5,7 @@ from fastmcp import FastMCP from ..approval import TakeoverActive -from ..challenge import hint_for, observe +from ..challenge import abandon_hint, hint_for, load_solver, observe, run_solver from ..context import ( ServerContext, acquire_page, @@ -18,7 +18,7 @@ from ..origin import parse_origin from ..permission import Capability from ..profile_import import read_sidecar -from .scope import released, require +from .scope import page_origin, released, require # How long a navigation waits, by Playwright's own names. ``domcontentloaded`` # stays the default: the document is parsed and usable, and a site that holds a @@ -319,15 +319,7 @@ async def navigate( result["hint"] = _NOT_A_FILE_HINT elif found := await observe(page, result["http_status"]): # The page answered, but it is a bot check, not the page asked for. - result["status"] = "challenge" - result["challenge"] = found.as_dict() - result["hint"] = hint_for(found, ctx.config.attended) - ctx.audit.record( - "challenge_seen", - {"vendor": found.vendor, "kind": found.kind}, - status=found.kind, - **audit_kw, - ) + await _met_challenge(ctx, page, found, result, audit_kw) return result @mcp.tool @@ -438,3 +430,50 @@ async def reload_page( return refused_envelope(page.url, redirected_to=hop) ctx.audit.record("reload_page", origin=current.describe()) return {"status": "ok", "url": page.url, **_response_facts(response)} + + +async def _met_challenge(ctx, page, found, result: dict, audit_kw: dict) -> None: + """Fill ``result`` for a page that is a bot check: name it, count it, hand it to the + operator's solver when there is one and nobody is watching, give up on it once the + same check has stopped the agent ``challenge_max_attempts`` times.""" + from ..challenge import INTERACTIVE + + origin = page_origin(page).describe() + attempts = ctx.challenges.record(origin, found) + limit = ctx.config.challenge_max_attempts + result["challenge"] = found.as_dict() + result["attempts"], result["max_attempts"] = attempts, limit + ctx.audit.record( + "challenge_seen", + {"vendor": found.vendor, "kind": found.kind, "attempt": attempts}, + status=found.kind, + **audit_kw, + ) + if attempts >= limit: + result["status"] = "challenge_abandoned" + result["hint"] = abandon_hint(found, attempts) + ctx.audit.record("challenge_abandoned", {"vendor": found.vendor}, **audit_kw) + return + if ctx.config.captcha_solver and found.kind == INTERACTIVE and not ctx.config.attended: + info = {**found.as_dict(), "url": page.url, "attempt": attempts} + try: + solver = load_solver(ctx.config.captcha_solver) + except Exception as exc: # noqa: BLE001 - a missing solver is reported + said = f"error: {type(exc).__name__}: {exc}"[:200] + else: + said = await run_solver(solver, info, page) + result["solver"] = said + cleared = await observe(page) + ctx.audit.record( + "challenge_solver", + {"vendor": found.vendor, "said": said, "cleared": cleared is None}, + **audit_kw, + ) + if cleared is None: + ctx.challenges.clear(origin, found) + result["status"] = "ok" + result["challenge_solved"] = found.vendor + result["url"], result["title"] = page.url, await page.title() + return + result["status"] = "challenge" + result["hint"] = hint_for(found, ctx.config.attended) diff --git a/src/lyra_browser/tools/uat.py b/src/lyra_browser/tools/uat.py index 8f9b740..189fe0d 100644 --- a/src/lyra_browser/tools/uat.py +++ b/src/lyra_browser/tools/uat.py @@ -144,3 +144,64 @@ async def finish( "verdicts": len(recorder.verdicts), "missing_verdicts": recorder.missing_verdicts(), } + + +def register_credentials(mcp: FastMCP, ctx: ServerContext, store, typer: dict) -> None: + """``fill_credential``: registered only when the spec names credentials. + + ``typer["type_text"]`` is the raw ``type_text`` body, set once the server has listed + its tools; typing through it keeps every gate of ``type_text`` (actionability, + consent for ``submit``, the navigation guard, audit redaction).""" + from ..uat.credentials import fill_envelope + from .scope import page_origin + + @mcp.tool + async def fill_credential( + name: str, + field: str, + selector: str, + submit: bool = False, + reason: str = "", + confirm: bool = False, + ) -> dict: + """Type one part of a stored login into ``selector`` without seeing it. + + ``name`` is a credential your brief lists. ``field`` is ``username``, + ``password`` or ``totp`` (the current one-time code, computed now). It is typed + only on the credential's own site: anywhere else the answer is ``wrong_origin`` + and nothing is typed. ``submit=true`` presses Enter afterwards and asks to send + the form, as ``type_text`` does. The value never appears in your results, the + trace or the report; what you get back is whether it was typed. + """ + found, refused = fill_envelope(store, name, field) + if refused is not None: + ctx.audit.record("fill_credential", {"name": name, "field": field}, status="refused") + return refused + cred, value = found + try: + page = await ctx.session.page(revive=False) + here = page_origin(page).describe() + except Exception as exc: # noqa: BLE001 - no page is an answer, not a crash + return {"status": "error", "reason": f"no page: {type(exc).__name__}"} + if here != cred.origin: + ctx.audit.record( + "fill_credential", {"name": name, "field": field}, status="wrong_origin" + ) + return { + "status": "wrong_origin", + "credential": name, + "field": field, + "page_origin": here, + "credential_origin": cred.origin, + "hint": "Only type a credential on its own site. Navigate there first.", + } + result = await typer["type_text"]( + selector=selector, value=value, submit=submit, reason=reason, confirm=confirm + ) + status = result.get("status") if isinstance(result, dict) else "error" + ctx.audit.record( + "fill_credential", + {"name": name, "field": field, "selector": selector, "submit": submit}, + status=status, + ) + return {**(result if isinstance(result, dict) else {}), "credential": name, "field": field} diff --git a/src/lyra_browser/totp.py b/src/lyra_browser/totp.py new file mode 100644 index 0000000..f1668ed --- /dev/null +++ b/src/lyra_browser/totp.py @@ -0,0 +1,56 @@ +"""Time-based one-time passwords (RFC 6238), from the standard library alone. + +What an authenticator app shows for a shared secret: HMAC of the number of ``period`` +second steps since the epoch, dynamically truncated (RFC 4226) to ``digits`` digits. +Secrets are base32 as sites hand them out (spaces and lower case accepted, padding +optional). +""" + +from __future__ import annotations + +import base64 +import hashlib +import hmac +import struct +import time + +_DIGESTS = {"sha1": hashlib.sha1, "sha256": hashlib.sha256, "sha512": hashlib.sha512} + + +def decode_secret(secret: str) -> bytes: + cleaned = "".join(secret.split()).upper() + cleaned += "=" * (-len(cleaned) % 8) + try: + return base64.b32decode(cleaned, casefold=True) + except (ValueError, TypeError) as exc: # binascii.Error is a ValueError + raise ValueError("a TOTP secret must be base32") from exc + + +def hotp(key: bytes, counter: int, *, digits: int = 6, algorithm: str = "sha1") -> str: + digest = hmac.new(key, struct.pack(">Q", counter), _DIGESTS[algorithm]).digest() + offset = digest[-1] & 0x0F + code = struct.unpack(">I", digest[offset : offset + 4])[0] & 0x7FFFFFFF + return str(code % 10**digits).zfill(digits) + + +def totp( + secret: str | bytes, + *, + at: float | None = None, + period: int = 30, + digits: int = 6, + algorithm: str = "sha1", +) -> str: + """The code for ``secret`` at time ``at`` (now by default).""" + key = decode_secret(secret) if isinstance(secret, str) else secret + now = time.time() if at is None else at + return hotp(key, int(now // period), digits=digits, algorithm=algorithm) + + +def verify(secret: str, code: str, *, at: float | None = None, window: int = 1) -> bool: + """Whether ``code`` is valid now, or within ``window`` steps either side (clock skew).""" + now = time.time() if at is None else at + return any( + hmac.compare_digest(totp(secret, at=now + step * 30), code) + for step in range(-window, window + 1) + ) diff --git a/src/lyra_browser/uat/batch.py b/src/lyra_browser/uat/batch.py index c4be353..c5ebc5a 100644 --- a/src/lyra_browser/uat/batch.py +++ b/src/lyra_browser/uat/batch.py @@ -119,6 +119,7 @@ def build_run_spec(batch: BatchSpec, persona_id: str, runs_dir: Path) -> RunSpec hooks=batch.hooks, limits=batch.limits, tools=batch.tools, + credentials=batch.credentials, out_dir=runs_dir, data_dir=batch.data_dirs.get(persona_id), headless=batch.headless, diff --git a/src/lyra_browser/uat/credentials.py b/src/lyra_browser/uat/credentials.py new file mode 100644 index 0000000..3aba657 --- /dev/null +++ b/src/lyra_browser/uat/credentials.py @@ -0,0 +1,124 @@ +"""Logins a persona can use without ever seeing them. + +A run spec names a credential and where its secrets live (an environment variable or a +file), never the secrets. The persona calls ``fill_credential(name, field, selector)``; +the server resolves the value, checks that the page is on the credential's own origin, +and types it through ``type_text`` — so the same consent, enforcement and audit +redaction apply as to anything typed. The tool's arguments, the trace, the report and +the result the model reads carry the name and the field only. ``totp`` is computed at +the moment it is typed from the seed (RFC 6238); the seed itself is never typed. + +Everything resolved here is also handed to the run's ``Scrubber``, so a page that echoes +a password, or a captured POST body that carries it, is written with the value replaced. +""" + +from __future__ import annotations + +import os +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Literal + +from ..origin import parse_origin +from ..totp import decode_secret, totp +from .spec import CredentialRef + +Field = Literal["username", "password", "totp"] +FIELDS: tuple[str, ...] = ("username", "password", "totp") + + +@dataclass +class Credential: + name: str + origin: str # parse_origin(...).describe() + username: str + password: str | None + totp_secret: str | None + problem: str | None = None # why a part could not be read + + def value(self, field: str) -> str | None: + if field == "username": + return self.username or None + if field == "password": + return self.password + if field == "totp": + return totp(self.totp_secret) if self.totp_secret else None + return None + + +def _read(env: str | None, file: Path | None, what: str) -> tuple[str | None, str | None]: + if env: + value = os.environ.get(env) + if not value: + return None, f"{what}: environment variable {env} is not set" + return value, None + if file: + try: + return file.expanduser().read_text(encoding="utf-8").strip(), None + except OSError as exc: + return None, f"{what}: cannot read {file}: {exc.strerror}" + return None, None + + +def resolve(name: str, ref: CredentialRef) -> Credential: + """Read the secrets now. A missing one is a ``problem`` reported when it is asked + for, not an exception: the run goes on and the persona is told what is missing.""" + problems = [] + username, problem = _read(ref.username_env, None, "username") + if problem: + problems.append(problem) + password, problem = _read(ref.password_env, ref.password_file, "password") + if problem: + problems.append(problem) + seed, problem = _read(ref.totp_env, ref.totp_file, "totp") + if problem: + problems.append(problem) + if seed: + try: + decode_secret(seed) + except ValueError: + problems.append("totp: the seed is not base32") + seed = None + return Credential( + name=name, + origin=parse_origin(ref.origin).describe(), + username=username or ref.username, + password=password, + totp_secret=seed, + problem="; ".join(problems) or None, + ) + + +class CredentialStore: + def __init__(self, refs: dict[str, CredentialRef]) -> None: + self.credentials = {name: resolve(name, ref) for name, ref in refs.items()} + + def secrets(self) -> list[str]: + """What must never be written or shown: passwords and TOTP seeds (not usernames, + and not the six-digit codes, which would match ordinary numbers).""" + out = [] + for cred in self.credentials.values(): + out += [s for s in (cred.password, cred.totp_secret) if s] + return out + + +def fill_envelope(store: CredentialStore, name: str, field: str) -> tuple[Any, dict | None]: + """The value to type, or the envelope that refuses (never containing a value).""" + cred = store.credentials.get(name) + if cred is None: + return None, { + "status": "error", + "reason": "unknown credential", + "known": sorted(store.credentials), + } + if field not in FIELDS: + return None, {"status": "error", "reason": f"field must be one of {list(FIELDS)}"} + value = cred.value(field) + if not value: + return None, { + "status": "credential_unavailable", + "credential": name, + "field": field, + "reason": cred.problem or f"credential {name!r} has no {field}", + } + return (cred, value), None diff --git a/src/lyra_browser/uat/demo_site.py b/src/lyra_browser/uat/demo_site.py index f8af76a..63b24b0 100644 --- a/src/lyra_browser/uat/demo_site.py +++ b/src/lyra_browser/uat/demo_site.py @@ -6,8 +6,10 @@ site (something the server must refuse), an analytics-style hit, a console error and a page error — and a verification page whose code exists only as pixels on a canvas, so a run can prove whether a screenshot really reached its -model. ``lyra-uat demo-site`` serves it; the real-browser gate -(``scripts/verify_uat_e2e.py``) runs against it too. +model. ``/login`` → ``/2fa`` → ``/account`` is a sign-in with a password and a +one-time code (RFC 6238), for the unattended-login gate +(``scripts/verify_unattended_e2e.py``). ``lyra-uat demo-site`` serves it; the +real-browser gate (``scripts/verify_uat_e2e.py``) runs against it too. """ from __future__ import annotations @@ -16,7 +18,9 @@ import socket import threading from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer -from urllib.parse import urlparse +from urllib.parse import parse_qs, urlparse + +from ..totp import verify as verify_totp HOME = """Demo home

Tune — clean up your tracks

@@ -75,8 +79,35 @@ """ +LOGIN = """Sign in +

Sign in

+
+ + + +
__ERROR__""" + +TWO_FA = """Two-step verification +

Enter the code from your authenticator app

+
+ + +
__ERROR__""" + +ACCOUNT = """Your account +

Your account

Signed in as __USER__

""" + + class Handler(BaseHTTPRequestHandler): alt_base = "" + # The demo account. The gate sets these; ``lyra-uat demo-site`` prints them. + user = "demo@example.com" + password = "" + totp_seed = "" + # Sign-in state per cookie value: "password" after the first step, "in" after 2FA. + sessions: dict[str, str] = {} # Drawn on the canvas of /verify and nowhere in the page's text or markup that a # tool reads: only a model that was shown the screenshot can repeat it. visual_code = "000000" @@ -105,12 +136,62 @@ def do_GET(self) -> None: # noqa: N802 self._send("", 204, "text/plain") elif path == "/checkout": self._send("checkout

Checkout

") + elif path == "/login": + self._send(LOGIN.replace("__ERROR__", "")) + elif path == "/2fa": + if self._stage() != "password": + self._redirect("/login") + else: + self._send(TWO_FA.replace("__ERROR__", "")) + elif path == "/account": + if self._stage() != "in": + self._redirect("/login") + else: + self._send(ACCOUNT.replace("__USER__", self.user)) else: self.send_error(404) + def _sid(self) -> str: + for part in (self.headers.get("Cookie") or "").split(";"): + key, _, value = part.strip().partition("=") + if key == "sid": + return value + return "" + + def _stage(self) -> str: + return self.sessions.get(self._sid(), "") + + def _redirect(self, where: str, cookie: str | None = None) -> None: + self.send_response(303) + self.send_header("Location", where) + if cookie: + self.send_header("Set-Cookie", f"sid={cookie}; Path=/; HttpOnly; SameSite=Lax") + self.send_header("Content-Length", "0") + self.end_headers() + + def _sign_in(self, path: str, form: dict[str, list[str]]) -> None: + def field(name: str) -> str: + return (form.get(name) or [""])[0] + + if path == "/login": + if self.password and field("user") == self.user and field("pass") == self.password: + sid = secrets.token_hex(8) + self.sessions[sid] = "password" + self._redirect("/2fa", cookie=sid) + else: + self._send(LOGIN.replace("__ERROR__", '

Wrong email or password.

')) + elif self._stage() == "password" and verify_totp(self.totp_seed, field("code")): + self.sessions[self._sid()] = "in" + self._redirect("/account") + else: + self._send(TWO_FA.replace("__ERROR__", '

That code did not work.

')) + def do_POST(self) -> None: # noqa: N802 - self.rfile.read(int(self.headers.get("Content-Length") or 0)) - if urlparse(self.path).path == "/collect": + body = self.rfile.read(int(self.headers.get("Content-Length") or 0)) + path = urlparse(self.path).path + if path in ("/login", "/2fa"): + self._sign_in(path, parse_qs(body.decode("utf-8", "replace"))) + elif path == "/collect": self._send("", 204, "text/plain") else: self._send("submitted

Thanks, we got it.

") diff --git a/src/lyra_browser/uat/observe.py b/src/lyra_browser/uat/observe.py index 2d65f39..06cbec8 100644 --- a/src/lyra_browser/uat/observe.py +++ b/src/lyra_browser/uat/observe.py @@ -40,7 +40,11 @@ class PageObservers: """Listeners for every tab of a run. ``attach`` goes on ``BrowserSession.page_observers``; the session calls it for each tab it adopts.""" - def __init__(self, run_dir: Path, *, capture_patterns: Iterable[str] = ()) -> None: + def __init__( + self, run_dir: Path, *, capture_patterns: Iterable[str] = (), scrub: Any = None + ) -> None: + # Known secrets (a password a captured POST carries) are replaced before writing. + self._scrub = scrub self.network_path = run_dir / "network.jsonl" self.console_path = run_dir / "console.jsonl" self._patterns = [re.compile(p) for p in capture_patterns] @@ -172,7 +176,8 @@ def flush(self) -> None: pending, self._pending = self._pending, [] by_path: dict[Path, list[str]] = {} for path, row in pending: - by_path.setdefault(path, []).append(json.dumps(row, ensure_ascii=False, default=str)) + line = json.dumps(row, ensure_ascii=False, default=str) + by_path.setdefault(path, []).append(self._scrub.text(line) if self._scrub else line) for path, lines in by_path.items(): path.parent.mkdir(parents=True, exist_ok=True) with path.open("a", encoding="utf-8") as fh: diff --git a/src/lyra_browser/uat/recorder.py b/src/lyra_browser/uat/recorder.py index 55acc3c..9ef8054 100644 --- a/src/lyra_browser/uat/recorder.py +++ b/src/lyra_browser/uat/recorder.py @@ -29,6 +29,7 @@ from mcp.types import TextContent from ..audit import _redact as redact_args +from ..scrub import Scrubber from ..vision import payload_of from .spec import UAT_TOOLS @@ -39,6 +40,7 @@ "reload_page", "click", "type_text", + "fill_credential", "press_key", "hover", "scroll", @@ -70,6 +72,9 @@ "budget_exhausted", "finished", "blocked_by_uat_policy", + "wrong_origin", + "credential_unavailable", + "challenge_abandoned", } ) """Statuses of an action that never reached the page. They are recorded but do @@ -176,6 +181,11 @@ def __init__( self.finding_count = 0 self.verdicts: dict[str, str] = {} self.finished = False + # Secret values (a credential's password, a TOTP seed) never written or returned. + self.scrub = Scrubber() + # Set when the same bot check stopped the run ``challenge_max_attempts`` times: + # every browser tool after it is refused, as after finish. + self.abandoned: str | None = None self.last_url = "" self._entry_done = False self._system_depth = 0 @@ -209,6 +219,12 @@ def guard(self, tool: str, args: dict[str, Any]) -> dict | None: """The envelope that stands in for the call, or None to let it through.""" if tool in UAT_TOOLS: return None + if self.abandoned: + return { + "status": "challenge_abandoned", + "reason": self.abandoned, + "hint": "Report what you saw and call finish with outcome=blocked.", + } if self.finished: return { "status": "finished", @@ -309,6 +325,12 @@ def complete( step.counted = step.action and not step.system and status not in NO_ATTEMPT if step.counted: self.steps_used += 1 + if status == "challenge_abandoned" and not self.abandoned: + vendor = (envelope.get("challenge") or {}).get("vendor", "a bot check") + self.abandoned = ( + f"the same {vendor} check stopped the run {envelope.get('attempts')} times" + ) + self.policy_event("challenge_abandoned", self.abandoned) if step.tool == "navigate" and step.system and not self._entry_done and not step.error: self._entry_done = status not in NO_ATTEMPT self._append(self.trace_path, asdict(step)) @@ -327,8 +349,9 @@ async def take_screenshot(self) -> str | None: # --- events ----------------------------------------------------------------- def _append(self, path: Path, row: dict[str, Any]) -> None: + line = json.dumps(row, ensure_ascii=False, default=str) with path.open("a", encoding="utf-8") as fh: - fh.write(json.dumps(row, ensure_ascii=False, default=str) + "\n") + fh.write((self.scrub.text(line) if self.scrub else line) + "\n") def _event(self, event_type: str, **data: Any) -> dict[str, Any]: row = {"type": event_type, "ts": _now(), "step": len(self.steps), **data} @@ -421,6 +444,23 @@ def summary(self) -> dict[str, Any]: } +def scrub_result(result: ToolResult, scrub: Scrubber) -> ToolResult: + """The result with every known secret replaced, before the trace or the model sees it.""" + if not scrub: + return result + content = [ + TextContent(type="text", text=scrub.text(item.text)) + if isinstance(item, TextContent) + else item + for item in (result.content or []) + ] + structured = getattr(result, "structured_content", None) + return ToolResult( + content=content, + structured_content=scrub.data(structured) if structured is not None else None, + ) + + def _refusal(envelope: dict) -> ToolResult: return ToolResult( content=[TextContent(type="text", text=json.dumps(envelope, ensure_ascii=False))], @@ -444,7 +484,7 @@ async def on_call_tool(self, context: MiddlewareContext, call_next) -> ToolResul return _refusal(refusal) step = self.recorder.begin(name, args) try: - result = await call_next(context) + result = scrub_result(await call_next(context), self.recorder.scrub) except Exception as exc: self.recorder.complete( step, {"status": "error", "error": type(exc).__name__}, is_error=True diff --git a/src/lyra_browser/uat/report.py b/src/lyra_browser/uat/report.py index b0e0080..85ebb80 100644 --- a/src/lyra_browser/uat/report.py +++ b/src/lyra_browser/uat/report.py @@ -34,6 +34,7 @@ "hook_failed", "no_finish", "interrupted", + "challenge_abandoned", ] OutcomeStatus = Literal["reached_goal", "partial", "blocked", "unknown"] @@ -398,7 +399,14 @@ def build_report( else: if exit_reason is None or exit_reason == "finish": budget_hit = any(p.kind == "budget_exhausted" for p in policy_events) - reason = "budget_exhausted" if budget_hit else "no_finish" + walled = any(p.kind == "challenge_abandoned" for p in policy_events) + reason = ( + "challenge_abandoned" + if walled + else "budget_exhausted" + if budget_hit + else "no_finish" + ) else: reason = exit_reason status = ( diff --git a/src/lyra_browser/uat/server.py b/src/lyra_browser/uat/server.py index 84a0c79..3ec038b 100644 --- a/src/lyra_browser/uat/server.py +++ b/src/lyra_browser/uat/server.py @@ -31,6 +31,7 @@ from ..tools import register_all from ..tools import uat as uat_tools from ..vision import VisionMiddleware +from .credentials import CredentialStore from .observe import PageObservers from .recorder import RunRecorder, UatMiddleware from .spec import UAT_TOOLS, RunSpec @@ -81,6 +82,7 @@ def build_config(spec: RunSpec, run_dir: Path) -> Config: driver=spec.driver, # Every capture is evidence; nothing is pruned during a run. capture_keep=0, + captcha_solver=spec.target.captcha_solver, ) cfg.capture_dir = run_dir / "captures" return cfg @@ -138,7 +140,12 @@ async def build_uat_server( ) # Requests, console and page errors of every tab, for the report. The real # session calls each observer per adopted tab; a test double has no such list. - observers = PageObservers(run_dir, capture_patterns=spec.target.network_capture) + store = CredentialStore(spec.credentials) + for secret in store.secrets(): + recorder.scrub.add(secret) + observers = PageObservers( + run_dir, capture_patterns=spec.target.network_capture, scrub=recorder.scrub + ) page_observers = getattr(ctx.session, "page_observers", None) if isinstance(page_observers, list): page_observers.append(observers.attach) @@ -147,6 +154,9 @@ async def build_uat_server( mcp = FastMCP("lyra-browser", instructions=instructions_for(cfg) + _UAT_INSTRUCTIONS) register_all(mcp, ctx) uat_tools.register(mcp, ctx, recorder) + typer: dict[str, Any] = {} + if spec.credentials: + uat_tools.register_credentials(mcp, ctx, store, typer) offered = set(spec.offered_tools()) | set(UAT_TOOLS) tools = await mcp.list_tools() @@ -156,6 +166,9 @@ async def build_uat_server( raise ValueError(f"tools.allow names tools this server does not have: {missing}") mcp.disable(keys=[tool.key for tool in tools if tool.name not in offered]) + # fill_credential types through type_text's own body (gates, audit redaction), not + # through the middleware: the persona's call is the step, the typing is not another. + typer["type_text"] = by_name["type_text"].fn # The capture for the trace, bypassing the middleware so it is not a step. shoot = by_name["screenshot"].fn diff --git a/src/lyra_browser/uat/spec.py b/src/lyra_browser/uat/spec.py index e318c1a..0881c44 100644 --- a/src/lyra_browser/uat/spec.py +++ b/src/lyra_browser/uat/spec.py @@ -132,6 +132,9 @@ class Target(Strict): # Requests whose URL matches one of these regexes are kept in full # (query string, POST body) in ``network.jsonl``; everything else is a summary. network_capture: list[str] = Field(default_factory=list) + # ``"module:function"``: an interactive bot check is handed to it (the run is never + # attended). lyra ships none; see ``lyra_browser.challenge``. + captcha_solver: str | None = None @model_validator(mode="after") def _closed_needs_a_site(self) -> Target: @@ -213,6 +216,30 @@ class ToolPolicy(Strict): deny: list[str] = Field(default_factory=list) +class CredentialRef(Strict): + """A login the server types for the persona, by name. The spec says where each secret + lives (an environment variable or a file), never the secret: the run file, the trace + and the report hold this and nothing more. ``origin`` is the one site it may be typed + into (``https://app.example.com``); ``totp`` is a base32 seed the code is computed from + when it is typed.""" + + origin: str + username: str = "" + username_env: str | None = None + password_env: str | None = None + password_file: Path | None = None + totp_env: str | None = None + totp_file: Path | None = None + + @model_validator(mode="after") + def _one_source_each(self) -> CredentialRef: + if self.password_env and self.password_file: + raise ValueError("give password_env or password_file, not both") + if self.totp_env and self.totp_file: + raise ValueError("give totp_env or totp_file, not both") + return self + + class PriorFinding(Strict): """A finding from an earlier round the persona re-checks and judges.""" @@ -263,6 +290,8 @@ class RunSpec(Strict): hooks: Hooks = Field(default_factory=Hooks) limits: Limits = Field(default_factory=Limits) tools: ToolPolicy = Field(default_factory=ToolPolicy) + # Logins the persona may use through ``fill_credential``, by name. + credentials: dict[str, CredentialRef] = Field(default_factory=dict) # Where run directories are created: ``//``. out_dir: Path = Path("uat-runs") # The browser profile. None: a fresh one inside the run directory. Set: an @@ -303,6 +332,8 @@ def offered_tools(self) -> list[str]: allowed = list(self.tools.allow) if self.tools.allow is not None else list(DEFAULT_TOOLS) if self.persona.uploads_allowed_dir is not None and "upload_file" not in allowed: allowed.append("upload_file") + if self.credentials and "fill_credential" not in allowed: + allowed.append("fill_credential") denied = set(self.tools.deny) return [name for name in allowed if name not in denied] @@ -332,6 +363,7 @@ class BatchSpec(Strict): hooks: Hooks = Field(default_factory=Hooks) limits: Limits = Field(default_factory=Limits) tools: ToolPolicy = Field(default_factory=ToolPolicy) + credentials: dict[str, CredentialRef] = Field(default_factory=dict) out_dir: Path = Path("uat-runs") # Persona id -> browser profile (``LYRA_BROWSER_DATA_DIR`` layout) that already holds # its login. Required for ``preseeded`` personas; others get a fresh profile. diff --git a/tests/test_challenge.py b/tests/test_challenge.py index 05f509a..3a00d04 100644 --- a/tests/test_challenge.py +++ b/tests/test_challenge.py @@ -7,6 +7,8 @@ from __future__ import annotations +import json + import pytest from lyra_browser import challenge as ch @@ -212,3 +214,151 @@ async def nothing(page, http_status=None): assert "challenge" not in result assert "hint" not in result + + +# -------------------------------------------------------------------------- +# navigate: the challenge branch, the same check again and again, the solver hook +# -------------------------------------------------------------------------- + + +def _always(found): + async def observe(page, http_status=None): + return found + + return observe + + +async def test_navigate_names_the_check_and_counts_it(make_ctx, tools_of, page, monkeypatch): + from lyra_browser.tools import navigation + + monkeypatch.setattr( + navigation, "observe", _always(ch.Challenge("cloudflare", INTERACTIVE, "title")) + ) + ctx = make_ctx(headless=True) + navigate = (await tools_of(ctx))["navigate"] + + first = await navigate(url=page.url) + + assert first["status"] == "challenge" and first["challenge"]["vendor"] == "cloudflare" + assert first["attempts"] == 1 and first["max_attempts"] == 3 + assert "nobody is at this window" in first["hint"] + rows = [json.loads(line) for line in ctx.config.audit_path.read_text().splitlines()] + assert any(r["tool"] == "challenge_seen" for r in rows) + + +async def test_the_same_check_three_times_is_abandoned(make_ctx, tools_of, page, monkeypatch): + from lyra_browser.tools import navigation + + monkeypatch.setattr( + navigation, "observe", _always(ch.Challenge("datadome", INTERACTIVE, "frame")) + ) + navigate = (await tools_of(make_ctx(headless=True)))["navigate"] + + results = [await navigate(url=page.url) for _ in range(3)] + + assert [r["status"] for r in results] == ["challenge", "challenge", "challenge_abandoned"] + assert [r["attempts"] for r in results] == [1, 2, 3] + assert "Do not try again" in results[-1]["hint"] + + +async def test_the_limit_is_configurable_and_counts_per_vendor( + make_ctx, tools_of, page, monkeypatch +): + from lyra_browser.tools import navigation + + found = {"c": ch.Challenge("cloudflare", INTERACTIVE, "t")} + + async def observe(page, http_status=None): + return found["c"] + + monkeypatch.setattr(navigation, "observe", observe) + ctx = make_ctx(headless=True) + ctx.config.challenge_max_attempts = 2 + navigate = (await tools_of(ctx))["navigate"] + + assert (await navigate(url=page.url))["status"] == "challenge" + found["c"] = ch.Challenge("hcaptcha", INTERACTIVE, "frame") + assert (await navigate(url=page.url))["status"] == "challenge" # another check + assert (await navigate(url=page.url))["status"] == "challenge_abandoned" + + +async def test_a_solver_that_clears_the_check_lets_the_page_through( + make_ctx, tools_of, page, monkeypatch +): + from lyra_browser.tools import navigation + + state = {"solved": False, "calls": []} + + async def observe(page, http_status=None): + return None if state["solved"] else ch.Challenge("recaptcha", INTERACTIVE, "frame") + + async def solver(info, page): + state["calls"].append(info) + state["solved"] = True + return "solved by the test double" + + monkeypatch.setattr(navigation, "observe", observe) + monkeypatch.setattr(navigation, "load_solver", lambda spec: solver) + ctx = make_ctx(headless=True) + ctx.config.captcha_solver = "tests.double:solve" + navigate = (await tools_of(ctx))["navigate"] + + result = await navigate(url=page.url) + + assert result["status"] == "ok" and result["challenge_solved"] == "recaptcha" + assert result["solver"] == "solved by the test double" + assert state["calls"][0]["vendor"] == "recaptcha" and state["calls"][0]["attempt"] == 1 + assert ctx.challenges.seen == {}, "a cleared check starts its count again" + + +async def test_a_solver_that_fails_leaves_the_challenge_envelope( + make_ctx, tools_of, page, monkeypatch +): + from lyra_browser.tools import navigation + + def solver(info, page): + raise RuntimeError("no balance") + + monkeypatch.setattr( + navigation, "observe", _always(ch.Challenge("recaptcha", INTERACTIVE, "frame")) + ) + monkeypatch.setattr(navigation, "load_solver", lambda spec: solver) + ctx = make_ctx(headless=True) + ctx.config.captcha_solver = "tests.double:solve" + navigate = (await tools_of(ctx))["navigate"] + + result = await navigate(url=page.url) + + assert result["status"] == "challenge" + assert result["solver"].startswith("error: RuntimeError: no balance") + + +async def test_no_solver_is_called_when_someone_is_watching_or_the_page_is_blocked( + make_ctx, tools_of, page, monkeypatch +): + from lyra_browser.tools import navigation + + called = [] + monkeypatch.setattr(navigation, "load_solver", lambda spec: lambda i, p: called.append(i)) + monkeypatch.setattr( + navigation, "observe", _always(ch.Challenge("recaptcha", INTERACTIVE, "frame")) + ) + attended = make_ctx(headless=False) + attended.config.captcha_solver = "tests.double:solve" + assert (await (await tools_of(attended))["navigate"](url=page.url))["status"] == "challenge" + + monkeypatch.setattr( + navigation, "observe", _always(ch.Challenge("unidentified", BLOCKED, "text")) + ) + blocked = make_ctx(headless=True) + blocked.config.captcha_solver = "tests.double:solve" + assert (await (await tools_of(blocked))["navigate"](url=page.url))["status"] == "challenge" + assert called == [] + + +def test_load_solver_wants_module_colon_function(): + import pytest + + with pytest.raises(ValueError): + ch.load_solver("no_colon_here") + assert ch.load_solver("json:dumps")({"a": 1}) == '{"a": 1}' diff --git a/tests/test_totp.py b/tests/test_totp.py new file mode 100644 index 0000000..283c41c --- /dev/null +++ b/tests/test_totp.py @@ -0,0 +1,51 @@ +"""TOTP against the RFC 6238 appendix B test vectors.""" + +from __future__ import annotations + +import base64 + +import pytest + +from lyra_browser.totp import decode_secret, totp, verify + +# RFC 6238 appendix B: the seeds are the ASCII strings below, 8-digit codes. +SEEDS = { + "sha1": b"12345678901234567890", + "sha256": b"12345678901234567890123456789012", + "sha512": b"1234567890123456789012345678901234567890123456789012345678901234", +} +VECTORS = [ + (59, "94287082", "46119246", "90693936"), + (1111111109, "07081804", "68084774", "25091201"), + (1111111111, "14050471", "67062674", "99943326"), + (1234567890, "89005924", "91819424", "93441116"), + (2000000000, "69279037", "90698825", "38618901"), + (20000000000, "65353130", "77737706", "47863826"), +] + + +@pytest.mark.parametrize(("at", "sha1", "sha256", "sha512"), VECTORS) +def test_rfc6238_vectors(at, sha1, sha256, sha512): + for algorithm, expected in (("sha1", sha1), ("sha256", sha256), ("sha512", sha512)): + assert totp(SEEDS[algorithm], at=at, digits=8, algorithm=algorithm) == expected + + +def test_a_base32_secret_as_sites_print_it(): + secret = base64.b32encode(SEEDS["sha1"]).decode().rstrip("=").lower() + spaced = " ".join(secret[i : i + 4] for i in range(0, len(secret), 4)) + assert decode_secret(spaced) == SEEDS["sha1"] + assert totp(spaced, at=59, digits=8) == "94287082" + assert len(totp(spaced)) == 6 + + +def test_verify_allows_one_step_of_skew_and_no_more(): + secret = base64.b32encode(SEEDS["sha1"]).decode() + code = totp(secret, at=1_000_000) + assert verify(secret, code, at=1_000_000) + assert verify(secret, code, at=1_000_000 + 30) + assert not verify(secret, code, at=1_000_000 + 90) + + +def test_a_secret_that_is_not_base32_is_refused(): + with pytest.raises(ValueError): + decode_secret("not base32 at all!") diff --git a/tests/uat/test_uat_credentials.py b/tests/uat/test_uat_credentials.py new file mode 100644 index 0000000..59eb85a --- /dev/null +++ b/tests/uat/test_uat_credentials.py @@ -0,0 +1,198 @@ +"""Stored logins, secret scrubbing and giving up on a bot check, in a UAT run. + +The run goes through the real runner and server on the fake page, so what is checked is +what a model and the run's files would hold: the persona names a credential, the server +types it, and the value is nowhere but in the field it was typed into. +""" + +from __future__ import annotations + +import base64 +from pathlib import Path + +import pytest + +from conftest import FakePage, FakeSession +from lyra_browser import challenge as ch +from lyra_browser.scrub import MARK, Scrubber +from lyra_browser.totp import totp +from lyra_browser.uat import runner as runner_mod +from lyra_browser.uat import server as server_mod +from lyra_browser.uat.spec import RunSpec + +ENTRY = "https://start.example/" +PASSWORD = "canary-Pw-7f3a9c!" +SEED = base64.b32encode(b"canary-seed-0123456789").decode().rstrip("=") + + +@pytest.fixture +def fake_browser(monkeypatch, page: FakePage): + async def build(spec, run_dir, **kw): + kw.setdefault("session_factory", lambda cfg: FakeSession(page)) + mcp, ctx, recorder = await server_mod.build_uat_server(spec, run_dir, **kw) + ctx.config.download_settle_s = 0.0 + return mcp, ctx, recorder + + monkeypatch.setattr(runner_mod, "build_uat_server", build) + return page + + +@pytest.fixture +def secrets_env(monkeypatch, tmp_path): + monkeypatch.setenv("CANARY_PASSWORD", PASSWORD) + seed_file = tmp_path / "seed.txt" + seed_file.write_text(SEED + "\n") + return seed_file + + +def spec_for(tmp_path: Path, script: list[dict], seed_file: Path, **extra) -> RunSpec: + return RunSpec.model_validate( + { + "persona": { + "id": "p1", + "name": "Tester", + "entry_url": ENTRY, + "goal": "sign in", + "step_budget": 10, + }, + "target": { + "trusted_origins": ["start.example"], + "trusted_send_origins": ["start.example"], + }, + "brain": {"backend": "scripted", "script": script}, + "credentials": { + "demo": { + "origin": "https://start.example", + "username": "demo@example.com", + "password_env": "CANARY_PASSWORD", + "totp_file": str(seed_file), + }, + "elsewhere": {"origin": "https://other.example", "password_env": "CANARY_PASSWORD"}, + "missing": {"origin": "https://start.example", "password_env": "NOT_SET_ANYWHERE"}, + }, + "out_dir": str(tmp_path / "runs"), + **extra, + } + ) + + +def fill(name: str, field: str, selector: str) -> dict: + return {"tool": "fill_credential", "args": {"name": name, "field": field, "selector": selector}} + + +def all_text(run_dir: Path) -> str: + return "\n".join( + p.read_text(encoding="utf-8", errors="replace") for p in run_dir.rglob("*") if p.is_file() + ) + + +@pytest.mark.asyncio +async def test_a_login_is_typed_and_its_secrets_are_nowhere_in_the_run( + fake_browser, tmp_path, secrets_env, monkeypatch +): + monkeypatch.delenv("NOT_SET_ANYWHERE", raising=False) + script = [ + {"tool": "navigate", "args": {"url": ENTRY}, "expect": "ok"}, + fill("demo", "username", "#u"), + fill("demo", "password", "#p"), + fill("demo", "totp", "#c"), + fill("elsewhere", "password", "#p"), + fill("missing", "password", "#p"), + fill("nobody", "password", "#p"), + {"tool": "finish", "args": {"outcome": "reached_goal", "summary": "Signed in."}}, + ] + spec = spec_for(tmp_path, script, secrets_env) + assert "fill_credential" in spec.offered_tools() + report = await runner_mod.run_spec(spec) + + fills = [c for c in fake_browser.calls if c[0] == "fill"] + assert fills[0] == ("fill", "#u", "demo@example.com") + assert fills[1] == ("fill", "#p", PASSWORD) + assert fills[2][1] == "#c" and fills[2][2] in {totp(SEED, at=t) for t in _now_window()} + assert len(fills) == 3, "nothing is typed on another site or for a missing secret" + + statuses = [s.status for s in report.trace if s.tool == "fill_credential"] + assert statuses == ["ok", "ok", "ok", "wrong_origin", "credential_unavailable", "error"] + unavailable = next(s for s in report.trace if s.status == "credential_unavailable") + assert "NOT_SET_ANYWHERE" in unavailable.result_head + + run_dir = Path(report.artifacts.dir) + everything = all_text(run_dir) + assert PASSWORD not in everything and SEED not in everything + assert "CANARY_PASSWORD" in (run_dir / "run.json").read_text(), "only the reference is kept" + + +def _now_window(): + import time + + now = time.time() + return (now - 30, now, now + 30) + + +@pytest.mark.asyncio +async def test_a_page_that_echoes_the_password_is_scrubbed_before_the_model_reads_it( + fake_browser, tmp_path, secrets_env, monkeypatch +): + from lyra_browser.uat import recorder as recorder_mod + + seen: list[str] = [] + original = recorder_mod.scrub_result + + def spy(result, scrub): + out = original(result, scrub) + seen.extend(getattr(item, "text", "") for item in out.content) + return out + + monkeypatch.setattr(recorder_mod, "scrub_result", spy) + fake_browser._title = f"Welcome {PASSWORD}" + script = [ + {"tool": "navigate", "args": {"url": ENTRY}, "expect": "ok"}, + {"tool": "get_url", "args": {}}, + {"tool": "finish", "args": {"outcome": "partial", "summary": "Looked."}}, + ] + report = await runner_mod.run_spec(spec_for(tmp_path, script, secrets_env)) + assert seen and all(PASSWORD not in text for text in seen) + assert any(MARK in text for text in seen) + assert PASSWORD not in all_text(Path(report.artifacts.dir)) + + +@pytest.mark.asyncio +async def test_the_same_check_three_times_ends_the_run_as_challenge_abandoned( + fake_browser, tmp_path, secrets_env, monkeypatch +): + from lyra_browser.tools import navigation + + async def observe(page, http_status=None): + return ch.Challenge("cloudflare", ch.INTERACTIVE, "title") + + monkeypatch.setattr(navigation, "observe", observe) + script = [{"tool": "navigate", "args": {"url": ENTRY}} for _ in range(3)] + [ + {"tool": "click", "args": {"selector": "#a"}}, + {"tool": "note", "args": {"text": "walled"}}, + ] + report = await runner_mod.run_spec(spec_for(tmp_path, script, secrets_env)) + statuses = [s.status for s in report.trace] + walled = "challenge_abandoned" + assert statuses == ["challenge", "challenge", walled, walled, "ok"] + assert report.run.status == "incomplete" and report.run.exit_reason == "challenge_abandoned" + assert any(p.kind == "challenge_abandoned" for p in report.policy.events) + + +def test_the_scrubber_replaces_every_encoding_of_a_secret(): + scrub = Scrubber([PASSWORD, "abc"]) # too short to hold + assert scrub.text(f"pw={PASSWORD}") == f"pw={MARK}" + assert scrub.text("pass=canary-Pw-7f3a9c%21") == f"pass={MARK}" + assert scrub.text("abc stays") == "abc stays" + assert scrub.data({"a": [f"x {PASSWORD}"]}) == {"a": [f"x {MARK}"]} + assert not Scrubber([]) + + +def test_a_credential_spec_names_one_source_per_secret(): + from pydantic import ValidationError + + from lyra_browser.uat.spec import CredentialRef + + with pytest.raises(ValidationError): + CredentialRef(origin="https://a.example", password_env="X", password_file="/tmp/x") + with pytest.raises(ValidationError): + CredentialRef(origin="https://a.example", totp_env="X", totp_file="/tmp/x")