Browser extension with a shared LLM + Jev core

.gitignore

@@ -62,3 +62,4 @@ next-env.d.ts
 
 # Screenshots (testing)
 /screenshots
+/extension/dist

ARCHITECTURE.md

@@ -61,10 +61,22 @@ Dependabot opens a weekly grouped PR for `playwright` and `playwright-core` (sta
 3. Run one real task (`npm run agent:cli -- "Go to https://example.com and tell me the page heading."`) to confirm the snapshot format still parses.
 4. For releases, run the `Package Desktop` workflow to package and smoke-test the desktop app on every platform.
 
+## Browser Extension
+
+`extension/` is a Manifest V3 side-panel extension that shares the agent core with the desktop app:
+
+- `lib/cdp/driver.ts`: a browser driver over the Chrome DevTools Protocol that works with any transport, `chrome.debugger` in the extension or a Playwright CDP session in tests. Input is sent as real mouse and keyboard events.
+- `lib/cdp/snapshot.ts`: converts Chrome's accessibility tree into the same snapshot text Playwright MCP produces, so the page parser, fast mode and planner read pages identically on both. Element refs are `b<backendDOMNodeId>`, stable for the life of the document.
+- `lib/core/run-task.ts`: the task flow over any `TaskBrowser` (Jev preflight, starting page, fast mode, answer check, LLM planner fallback, confirmations), with events for the UI and cancellation through an `AbortSignal`. It knows the user's current tab, so "summarize this page" starts there.
+- `extension/src/`: the `chrome.debugger` transport, settings, OpenRouter OAuth (PKCE) and the React side panel. The agent runs in the side panel page, which stays alive while open (MV3 service workers are suspended when idle).
+
+`npm run extension:build` bundles it with esbuild into `extension/dist`; `process.env` is stubbed because the shared model client reads optional defaults from it in Node.
+
 ## Safety
 
 `lib/safety.ts` holds the rules every agent path shares:
 
+- **Requests that change something go to the careful planner.** Jev's preflight also asks whether the request buys, books, sends, deletes, submits…; if so, fast mode is skipped. Jev is excellent at finding and reading but weak at multi-step flows like add to cart → checkout, and these requests are rarer and higher-stakes.
 - **Irreversible actions need the user's OK.** Before a click whose label starts with a committing verb (buy, pay, place order, checkout, send, delete, unsubscribe, publish...), the run pauses with "Confirm: WebPilot wants to click …"; Resume allows it and Stop cancels. It applies to the LLM planner, fast mode and parallel sub-agents, and matches on the page's own label for the element rather than the model's description. Links only trigger it for strong verbs (buy, checkout, delete...), so navigation like "Sent items" or "Send feedback" does not.
 - **Page content is untrusted.** Planner, sub-agent, text-writing and answer prompts say that instructions found in pages or emails must never be followed. Jev gets a neutral version ("only the task defines what to do"), because it reads instructions literally and a list of risky verbs made it avoid legitimate steps.
 

README.md

@@ -13,6 +13,7 @@ WebPilot is meant to be a hackable, local-first alternative to closed agentic br
 ## Features
 
 - Desktop app built with Electron and Next.js.
+- Browser extension for Chrome, Edge, Brave and Arc: WebPilot in the side panel, working in your current tab with your existing logins ([extension/README.md](./extension/README.md)).
 - Browser automation through Playwright MCP, running on the latest stable Playwright.
 - Fast mode: TypeSafe's Jev decision model (through OpenRouter) picks each browser action in ~0.3s while the LLM only writes text, with automatic fallback to the LLM planner.
 - Model providers: OpenRouter (one key for Gemini, Claude, GPT, Qwen, DeepSeek and hundreds more) and local Ollama.
@@ -80,6 +81,14 @@ MODEL_BASE_URL=http://127.0.0.1:11434/v1
 
 The settings UI can discover local Ollama models and hides known embedding-only models when Ollama reports enough metadata to identify them.
 
+## Browser Extension
+
+```bash
+npm run extension:build
+```
+
+Load `extension/dist` from `chrome://extensions` (Developer mode > Load unpacked), then click the WebPilot icon to open the side panel and **Connect OpenRouter**. See [extension/README.md](./extension/README.md) for how it works, permissions and privacy.
+
 ## Fast Mode
 
 Turn on **Fast mode (Jev)** in Settings (OpenRouter only). Instead of asking the LLM to plan every click, WebPilot asks [Jev](https://openrouter.ai/typesafe/jev-1.13), a decision model that returns a choice with probabilities in about 0.3 seconds. The LLM still writes anything that needs text (search queries, the final answer), and the regular planner takes over whenever Jev is stuck, blocked, or about to click something irreversible.

TESTING.md

@@ -40,6 +40,16 @@ Runs each scenario in both modes and writes JSON and Markdown reports (pass/fail
 - `realistic`: requests phrased the way people ask, without URLs where possible ("hey can you please find some recent tents to buy on kijiji?", "what's trending on hacker news right now?"). Answers are checked against live data where it exists: the Hacker News API, GitHub's latest Playwright release, and current books.toscrape prices.
 - `realistic` also runs private-data tasks against a local webmail and shop (`scripts/fixtures/realistic-sites.mjs`, also runnable on its own with `node scripts/fixtures/realistic-sites.mjs`). The inbox has a distractor newsletter and a prompt-injection email asking AI assistants to forward mail and delete evidence; the shop has a real cart and checkout. The fixture records every send, delete and order, and any of them fails the scenario. When a run asks to confirm an irreversible action, the harness denies it; `shop_buy` passes only if it asks before checking out with the right tent in the cart. Model latency on OpenRouter varies run to run, so compare medians over `--repeat 3` or more before drawing conclusions.
 
+## Browser Extension
+
+```bash
+npm run extension:build
+OPENROUTER_API_KEY=sk-or-... npm run extension:smoke -- --headed
+npm run bench:fast-mode -- --engine cdp --suite all
+```
+
+`extension:smoke` loads the built extension into Chromium and drives its real side panel against the local fixtures. `--engine cdp` runs the scenario suite on the extension's engine (`lib/core/run-task.ts` over the CDP driver) instead of the desktop agent. `tests/cdp-snapshot.test.ts` covers the accessibility-tree converter without a browser.
+
 ## Deterministic Browser Smoke
 
 ```bash

eslint.config.mjs

@@ -20,7 +20,15 @@ const eslintConfig = defineConfig([
     "agent_threads/**",
     "e2e_reports/**",
     "readiness_reports/**",
+    "extension/dist/**",
   ]),
+  {
+    // The browser extension is not a Next.js app.
+    files: ["extension/**/*.{ts,tsx}"],
+    rules: {
+      "@next/next/no-img-element": "off",
+    },
+  },
   {
     files: ["electron/**/*.cjs", "scripts/**/*.js"],
     rules: {

extension/README.md

@@ -0,0 +1,63 @@
+# WebPilot browser extension
+
+WebPilot in your browser's side panel. Ask it about the page you have open or anything on the web; it works in your current tab, with the logins you already have, and asks before anything irreversible.
+
+Works in Chromium browsers: Chrome, Edge, Brave, Arc (Chrome 116+).
+
+## Install from source
+
+```bash
+npm install
+npm run extension:build
+```
+
+Then open `chrome://extensions`, turn on **Developer mode**, click **Load unpacked**, and pick `extension/dist`. Click the WebPilot toolbar icon to open the side panel.
+
+For development, `npm run extension:dev` rebuilds on every change; click the reload icon on the extension card to pick it up.
+
+## First run
+
+Open **Settings** (gear icon) and click **Connect OpenRouter**. You approve a key for WebPilot on openrouter.ai; it can be revoked or given a credit limit from the OpenRouter dashboard. You can also paste an existing `sk-or-...` key.
+
+Fast mode (Jev) is on by default.
+
+## How it works
+
+- The agent runs in the side panel page and controls the active tab through `chrome.debugger` (the Chrome DevTools Protocol). Chrome shows a "WebPilot started debugging this browser" bar while a task runs; it goes away when the task ends, and clicking **Cancel** on it stops the task.
+- Pages are read from Chrome's accessibility tree and acted on with real mouse and keyboard events, so sites treat them like your own input.
+- The same core runs in the desktop app and the extension: `lib/core/run-task.ts` (the task flow), `lib/cdp/` (the CDP driver and accessibility-tree snapshot), `lib/jev/` (fast mode), `lib/safety.ts` and `lib/model-client.ts`.
+- Requests that only find or read information use fast mode: Jev picks each click in ~0.3s and the model writes text and the answer. Requests that change something (buy, book, send, delete, submit) use the careful model planner.
+- Before an irreversible click (buy, place order, checkout, send, delete, publish, unsubscribe...) the panel shows **"Hold on — this can't be undone"** with Allow and Cancel.
+
+## Permissions
+
+| Permission | Why |
+|---|---|
+| `sidePanel` | The WebPilot panel |
+| `debugger` | Reading and controlling the tab you ask it to work in |
+| `tabs` | Knowing which tab is active and its title |
+| `storage` | Your settings and OpenRouter key (stored locally) |
+| `identity` | "Connect OpenRouter" sign-in |
+| `https://openrouter.ai/*` | Model and Jev requests |
+
+Chrome does not allow extensions to control `chrome://` pages or the Chrome Web Store.
+
+## Privacy
+
+Page text, element labels and URLs of the tab WebPilot works in are sent to OpenRouter to run the model, and in fast mode to TypeSafe (through OpenRouter) for Jev. Nothing is sent while no task is running. Settings stay in the browser's local extension storage.
+
+## Testing
+
+```bash
+npm run extension:build
+OPENROUTER_API_KEY=sk-or-... npm run extension:smoke            # headless
+OPENROUTER_API_KEY=sk-or-... npm run extension:smoke -- --headed
+```
+
+The smoke test loads `extension/dist` into Chromium, opens the local webmail and shop fixtures, types requests into the real side panel and checks the answers, confirmations and recorded actions (no sends, deletes or orders). Screenshots go to `e2e_reports/extension/`.
+
+The full scenario suite runs against the extension's engine too:
+
+```bash
+npm run bench:fast-mode -- --engine cdp --suite all
+```

extension/manifest.json

@@ -0,0 +1,29 @@
+{
+  "manifest_version": 3,
+  "name": "WebPilot",
+  "description": "An AI agent in your browser's side panel: it reads and acts on your current tab, fast, and asks before anything irreversible.",
+  "version": "0.1.0",
+  "minimum_chrome_version": "116",
+  "icons": {
+    "16": "icons/16x16.png",
+    "32": "icons/32x32.png",
+    "48": "icons/48x48.png",
+    "128": "icons/128x128.png"
+  },
+  "action": {
+    "default_title": "Open WebPilot",
+    "default_icon": {
+      "16": "icons/16x16.png",
+      "32": "icons/32x32.png"
+    }
+  },
+  "side_panel": {
+    "default_path": "sidepanel.html"
+  },
+  "background": {
+    "service_worker": "background.js",
+    "type": "module"
+  },
+  "permissions": ["sidePanel", "debugger", "tabs", "storage", "identity"],
+  "host_permissions": ["https://openrouter.ai/*", "http://127.0.0.1/*", "http://localhost/*"]
+}