Extension: work alongside password managers, never touch passwords, keep run traces

.gitignore

@@ -64,3 +64,4 @@ next-env.d.ts
 /screenshots
 /extension/dist
 /extension/.dev-bridge-token
+/extension/.dev-build-id

extension/README.md

@@ -21,6 +21,7 @@ Then open `chrome://extensions`, turn on **Developer mode**, click **Load unpack
 - It's a conversation: follow-ups like "what about the second one?" or "open it" continue from the previous answers. Each turn also keeps hidden notes (pages visited, where it ended, what it clicked) so later messages can pick up from there, even after the tab has moved on.
 - **History** (clock icon) lists past conversations. Opening one continues it in the current tab; WebPilot doesn't restore the old pages but knows what it did and navigates back if needed. **+** starts a new conversation.
 - Enter sends, Shift+Enter adds a line, Esc stops a running task.
+- Rate any answer with 👍/👎 and add a note, or type `/feedback <what was off>` to attach a note to the last answer. Notes are saved locally with that run's trace (below), so problems can be looked into later.
 
 ## First run
 
@@ -31,29 +32,32 @@ Fast mode (Jev) is on by default.
 ## How it works
 
 - WebPilot controls the tab you ask it to through `chrome.debugger` (the Chrome DevTools Protocol), attaching only when a task needs the page. Chrome shows a "WebPilot started debugging this browser" bar while a task runs; it goes away when the task ends, and clicking **Cancel** on it stops the task.
+- Chrome refuses the debugger on pages that contain another extension's frame, which password managers (LastPass, 1Password...) and tools like Grammarly add to many pages, including Gmail, and detaches it when such a frame appears mid-task. It also can't attach while another tool is debugging the tab. On those pages WebPilot switches to page scripts (`src/dom-browser.ts`, `src/page-scripts.ts`): it reads the visible page into the same snapshot format and acts with DOM events. The flight log notes the switch; each navigation tries the debugger again.
 - Pages are read from Chrome's accessibility tree and acted on with real mouse and keyboard events, so sites treat them like your own input.
 - The same core runs in the desktop app and the extension: `lib/core/run-task.ts` (the task flow), `lib/cdp/` (the CDP driver and accessibility-tree snapshot), `lib/jev/` (fast mode), `lib/safety.ts` and `lib/model-client.ts`.
 - The agent runs in the extension's background service worker (`src/engine.ts`), one session per tab; panels are views connected over a port. While a task runs, the worker keeps itself alive.
 - Requests about the open page that reading alone can answer ("summarize this", "what does this say about X") skip navigation and clicks and stream the answer straight from the page. Other requests that only find or read information use fast mode: Jev picks each click in ~0.3s and the model writes text and the answer. Requests that change something (buy, book, send, delete, submit) use the careful model planner.
 - Answers stream into the panel as they are written; Jev then checks them against the page, and an unsupported answer is redone (first by Jev looking closer, then by the model).
 - Before an irreversible click (buy, place order, checkout, send, delete, publish, unsubscribe...) the panel shows **"Hold on — this can't be undone"** with Allow and Cancel. Cancel ends the task there; say "go ahead" to continue.
+- WebPilot never types into password, one-time-code or card fields and never makes up usernames or other personal details. On a sign-in page your password manager has already filled in, clicking "Log in" asks first (**"Hold on — sign in?"**); otherwise it asks you to sign in yourself and then continue.
+- A follow-up like "no, it's on Gmail" or "try again" continues a request that didn't finish.
 
 ## Permissions
 
 | Permission | Why |
 |---|---|
 | `sidePanel` | The WebPilot panel |
 | `debugger` | Reading and controlling the tab you ask it to work in |
+| `scripting` + all sites | Page scripts, for pages where Chrome doesn't allow the debugger (see How it works) |
 | `tabs` | Knowing which tab is active and its title |
 | `storage` | Your settings, OpenRouter key and conversation history (stored locally) |
 | `identity` | "Connect OpenRouter" sign-in |
-| `https://openrouter.ai/*` | Model and Jev requests |
 
 Chrome does not allow extensions to control `chrome://` pages or the Chrome Web Store.
 
 ## Privacy
 
-Page text, element labels and URLs of the tab WebPilot works in are sent to OpenRouter to run the model, and in fast mode to TypeSafe (through OpenRouter) for Jev. Nothing is sent while no task is running. Settings and conversation history (at most the 100 most recent conversations) stay in the browser's local extension storage on this device; they are never synced or uploaded. History has Clear all.
+Page text, element labels and URLs of the tab WebPilot works in are sent to OpenRouter to run the model, and in fast mode to TypeSafe (through OpenRouter) for Jev. Nothing is sent while no task is running. Settings and conversation history (at most the 100 most recent conversations) stay in the browser's local extension storage on this device; they are never synced or uploaded. Each answer also keeps a short technical trace there (routing decisions, steps and timings, errors) and any feedback you left, for troubleshooting. History has Clear all, which removes these too.
 
 ## Testing
 
@@ -80,7 +84,13 @@ node scripts/build-extension.mjs --dev --out ~/WebPilot-extension   # load this
 npx tsx scripts/extension-bridge.ts ping chrome                      # or edge, brave...
 npx tsx scripts/extension-bridge.ts reload edge                      # pick up a rebuild without clicking
 npx tsx scripts/extension-bridge.ts run chrome "summarize this page" --url https://example.com
+npx tsx scripts/extension-bridge.ts run edge "find X in my email" --then "no, it's on gmail"   # a conversation
+npx tsx scripts/extension-bridge.ts last edge [n]            # your most recent conversation (n-th most recent), with traces
+npx tsx scripts/extension-bridge.ts feedback edge            # answers you rated or commented on, with traces
+npx tsx scripts/extension-bridge.ts targets chrome https://mail.google.com   # what blocks the debugger on a page
 npm run bench:fast-mode -- --engine extension --browser chrome --suite realistic --modes fast
 ```
 
+The smoke test also loads a stand-in for a password manager (`scripts/fixtures/frame-injector`) that puts its frame into pages opened with `?pm=1`, so the page-script path is tested too.
+
 The extension polls `http://127.0.0.1:4466` and only accepts commands from a server that knows the random secret generated for that build (`extension/.dev-bridge-token`, gitignored). Release builds (`npm run extension:build`) compile the bridge out and drop its permissions (`alarms`, localhost).

extension/manifest.json

@@ -24,12 +24,13 @@
   "permissions": [
     "sidePanel",
     "debugger",
+    "scripting",
     "tabs",
     "storage",
     "identity"
   ],
   "host_permissions": [
-    "https://openrouter.ai/*"
+    "<all_urls>"
   ],
   "commands": {
     "_execute_action": {

extension/sidepanel.css

@@ -143,6 +143,31 @@ button { font: inherit; color: inherit; }
 .answer strong { font-weight: 650; }
 .answer.error { color: var(--wp-error); }
 
+/* ---------- feedback ---------- */
+.feedback { margin-top: 4px; }
+.feedback-row { display: flex; align-items: center; gap: 2px; opacity: 0.55; transition: opacity 0.15s; }
+.turn:hover .feedback-row, .feedback-row:focus-within { opacity: 1; }
+.icon-button.small { width: 24px; height: 24px; }
+.icon-button.on.up { color: var(--wp-success); opacity: 1; }
+.icon-button.on.down { color: var(--wp-accent); opacity: 1; }
+.link-button { background: none; border: 0; padding: 2px 6px; color: var(--wp-text-faint); font: inherit; font-size: 11.5px; cursor: pointer; }
+.link-button:hover { color: var(--wp-text-secondary); }
+.feedback-note { margin: 4px 0 0 4px; padding-left: 8px; border-left: 2px solid var(--wp-accent); color: var(--wp-text-secondary); font-size: 12px; white-space: pre-wrap; }
+.feedback-form { display: flex; gap: 6px; margin-top: 6px; }
+.feedback-form input {
+  flex: 1;
+  min-width: 0;
+  padding: 6px 9px;
+  border: 1px solid var(--wp-border);
+  border-radius: var(--wp-radius-sm);
+  background: var(--wp-surface);
+  color: var(--wp-text);
+  font: inherit;
+  font-size: 12.5px;
+}
+.feedback-form input:focus { outline: none; border-color: var(--wp-accent); }
+.feedback-form .btn { padding: 5px 10px; font-size: 12px; }
+
 /* ---------- flight log ---------- */
 .log {
   position: relative;

extension/src/debugger-transport.ts

@@ -12,21 +12,40 @@ export function isRestrictedUrl(url: string | undefined): boolean {
         || /^https:\/\/chrome\.google\.com\/webstore/i.test(url);
 }
 
+/** Commands that change the page; see `actionsSent`. */
+const ACTION_METHODS = /^(Input\.|Page\.navigate|Runtime\.callFunctionOn$)/;
+
+/**
+ * Attach errors that mean the page is fine but the debugger can't be used on it:
+ * another extension's frame is in the page (password managers, Grammarly...),
+ * or another tool is already debugging the tab.
+ */
+export function isDebuggerBlocked(error: unknown): boolean {
+    const message = error instanceof Error ? error.message : String(error);
+    return /chrome-extension:\/\/ URL of different extension|Another debugger is already attached/i.test(message);
+}
+
 /**
  * CDP transport over `chrome.debugger` for one tab. While attached, Chrome shows
  * a "WebPilot started debugging this browser" bar; `detach()` removes it.
  */
 export class DebuggerTransport implements CdpTransport {
     private listeners = new Set<(method: string, params: Record<string, unknown>) => void>();
-    private detachedReason: string | null = null;
+    /** Why Chrome detached the debugger, once it has. */
+    detachedReason: string | null = null;
+    /** Commands sent that change the page (input, navigation), to tell whether an interrupted action took effect. */
+    actionsSent = 0;
 
     private readonly onEventHandler = (source: chrome.debugger.Debuggee, method: string, params?: object) => {
         if (source.tabId !== this.tabId) return;
         for (const listener of this.listeners) listener(method, (params || {}) as Record<string, unknown>);
     };
 
     private readonly onDetachHandler = (source: chrome.debugger.Debuggee, reason: string) => {
-        if (source.tabId === this.tabId) this.detachedReason = reason;
+        if (source.tabId !== this.tabId) return;
+        this.detachedReason = reason;
+        // Lets waits in progress (e.g. for a page load) stop instead of timing out.
+        for (const listener of this.listeners) listener("Inspector.detached", { reason });
     };
 
     private constructor(public readonly tabId: number) {}
@@ -45,6 +64,7 @@ export class DebuggerTransport implements CdpTransport {
                 ? "Browser control was cancelled from Chrome's debugging bar."
                 : `Lost control of the tab (${this.detachedReason}).`);
         }
+        if (ACTION_METHODS.test(method)) this.actionsSent++;
         return await chrome.debugger.sendCommand({ tabId: this.tabId }, method, params) as T;
     }
 

extension/src/dev-bridge.ts

@@ -12,9 +12,11 @@
  */
 
 import type { TaskEvent } from "../../lib/core/run-task";
+import { listConversations, loadConversation } from "./conversations";
 import type { Engine } from "./engine";
 
 declare const __DEV_BRIDGE_TOKEN__: string;
+declare const __DEV_BUILD_ID__: string;
 
 const BRIDGE_URL = "http://127.0.0.1:4466";
 const RETRY_MS = 3000;
@@ -27,9 +29,16 @@ interface RunCommand {
     confirm?: "allow" | "deny";
     fastMode?: boolean;
     keepOpen?: boolean;
+    /** Follow-up messages sent after the first answer, in the same tab and conversation. */
+    followUps?: string[];
 }
 
-type Command = RunCommand | { id: string; type: "reload" } | { id: string; type: "ping" };
+type Command = RunCommand
+    | { id: string; type: "reload" }
+    | { id: string; type: "ping" }
+    | { id: string; type: "targets"; url: string }
+    | { id: string; type: "last"; index?: number }
+    | { id: string; type: "feedback"; limit?: number };
 
 function browserName(): string {
     const brands = (navigator as Navigator & { userAgentData?: { brands?: Array<{ brand: string }> } }).userAgentData?.brands || [];
@@ -72,25 +81,34 @@ async function runCommand(engine: Engine, command: RunCommand) {
         return;
     }
     if (command.url) await waitForTabComplete(tabId);
+    const hooks = {
+        overrides: typeof command.fastMode === "boolean" ? { fastMode: command.fastMode } : undefined,
+        // The task reports the confirmation itself (a "confirm" event); just answer it.
+        confirm: async () => command.confirm === "allow",
+        onEvent: (event: TaskEvent) => {
+            if (event.type === "answer-delta") return;
+            post("/bridge/event", { id: command.id, event }).catch(() => {});
+        },
+    };
     const started = Date.now();
     try {
-        const turn = await engine.run(tabId, command.goal, {
-            overrides: typeof command.fastMode === "boolean" ? { fastMode: command.fastMode } : undefined,
-            // The task reports the confirmation itself (a "confirm" event); just answer it.
-            confirm: async () => command.confirm === "allow",
-            onEvent: (event: TaskEvent) => {
-                if (event.type === "answer-delta") return;
-                post("/bridge/event", { id: command.id, event }).catch(() => {});
-            },
-        });
+        const turns = [];
+        for (const message of [command.goal, ...(command.followUps || [])]) {
+            if (turns.length) post("/bridge/event", { id: command.id, event: { type: "status", message: `Follow-up: ${message}` } }).catch(() => {});
+            const turnStarted = Date.now();
+            const turn = await engine.run(tabId, message, hooks);
+            turns.push({ user: message, answer: turn.answer || "", error: turn.error, mode: turn.mode, stats: turn.stats, durationMs: Date.now() - turnStarted });
+        }
+        const last = turns[turns.length - 1];
         await post("/bridge/result", {
             id: command.id,
-            answer: turn.answer || "",
-            error: turn.error,
-            mode: turn.mode,
-            stats: turn.stats,
-            steps: turn.steps.length,
+            answer: last.answer,
+            error: last.error,
+            mode: last.mode,
+            stats: last.stats,
+            steps: turns.length,
             durationMs: Date.now() - started,
+            ...(turns.length > 1 ? { turns } : {}),
         });
     } catch (error) {
         await post("/bridge/result", { id: command.id, error: error instanceof Error ? error.message : String(error) });
@@ -99,6 +117,36 @@ async function runCommand(engine: Engine, command: RunCommand) {
     }
 }
 
+/**
+ * Opens `url` in a new window and reports the debug targets that appeared with
+ * it (frames, workers), to see what stops chrome.debugger on a page.
+ */
+async function targetsCommand(command: { id: string; url: string }) {
+    const key = (target: chrome.debugger.TargetInfo) => `${target.type} ${target.id}`;
+    const before = new Set((await chrome.debugger.getTargets()).map(key));
+    const window = await chrome.windows.create({ url: command.url, focused: false, width: 1280, height: 860 });
+    const tabId = window?.tabs?.[0]?.id;
+    try {
+        if (tabId !== undefined) await waitForTabComplete(tabId);
+        await new Promise((resolve) => setTimeout(resolve, 3000));
+        const targets = (await chrome.debugger.getTargets())
+            .filter((target) => !before.has(key(target)) || target.tabId === tabId)
+            .map(({ type, url, title, attached, tabId: targetTab, extensionId }) => ({ type, url: url.slice(0, 160), title, attached, tabId: targetTab, extensionId }));
+        let attachError = "";
+        if (tabId !== undefined) {
+            try {
+                await chrome.debugger.attach({ tabId }, "1.3");
+                await chrome.debugger.detach({ tabId });
+            } catch (error) {
+                attachError = error instanceof Error ? error.message : String(error);
+            }
+        }
+        await post("/bridge/result", { id: command.id, targets, attachError });
+    } finally {
+        if (window?.id !== undefined) await chrome.windows.remove(window.id).catch(() => {});
+    }
+}
+
 let polling = false;
 
 async function pollLoop(engine: Engine) {
@@ -122,12 +170,34 @@ async function pollLoop(engine: Engine) {
             }
             const command = await response.json() as Command;
             if (command.type === "ping") {
-                await post("/bridge/result", { id: command.id, client, version: chrome.runtime.getManifest().version }).catch(() => {});
+                await post("/bridge/result", { id: command.id, client, version: chrome.runtime.getManifest().version, build: __DEV_BUILD_ID__ }).catch(() => {});
             } else if (command.type === "reload") {
                 await post("/bridge/result", { id: command.id, reloading: true }).catch(() => {});
                 chrome.runtime.reload();
             } else if (command.type === "run") {
                 runCommand(engine, command).catch(() => {});
+            } else if (command.type === "last") {
+                // The user's most recent conversations, to debug a run they did by hand.
+                const summaries = await listConversations();
+                const summary = summaries[command.index || 0];
+                const conversation = summary ? await loadConversation(summary.id) : null;
+                await post("/bridge/result", { id: command.id, recent: summaries.slice(0, 5), conversation }).catch(() => {});
+            } else if (command.type === "feedback") {
+                // Answers the user rated or commented on, newest first, with their traces
+                // and the turns just before them for context.
+                const items = [];
+                for (const summary of await listConversations()) {
+                    const conversation = await loadConversation(summary.id);
+                    for (const [index, turn] of (conversation?.turns || []).entries()) {
+                        if (!turn.feedback) continue;
+                        const earlier = conversation!.turns.slice(Math.max(0, index - 2), index).map((item) => ({ user: item.user, answer: item.answer?.slice(0, 400), error: item.error }));
+                        items.push({ conversationId: summary.id, conversationTitle: summary.title, earlier, turn });
+                    }
+                }
+                items.sort((left, right) => (right.turn.feedback?.at || 0) - (left.turn.feedback?.at || 0));
+                await post("/bridge/result", { id: command.id, items: items.slice(0, command.limit || 20) }).catch(() => {});
+            } else if (command.type === "targets") {
+                targetsCommand(command).catch((error) => post("/bridge/result", { id: command.id, error: String(error) }).catch(() => {}));
             }
         }
     } finally {