Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,12 @@

## Unreleased

- **Semantic page map.** `snapshotTab` reads a tab from `listTabs` without the
debugger and without changing focus. Each entry has a role, an accessible
name, text, state, visibility, and a stable ref. `actOnRef` clicks or types
that ref with the same untrusted path as `softClick` and `softType`, then
reports what changed: the URL, new console errors, and new failed network
requests.
- **Non-scriptable pages are reported the same way everywhere.** `chrome://`,
DevTools, `view-source:`, the Chrome Web Store, extension pages, and the
built-in PDF viewer come back as metadata only (`access: "metadata-only"`,
Expand Down
8 changes: 6 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,9 +101,11 @@ setup and a silent one look the same.
| `listBrowserTabs` | Tabs with DevTools open, addressed by a DevTools-connected `tabId` |
| `listTabs` | Tabs `chrome.tabs` returns (not every open tab), addressed by `chromeTabId` |
| `readTab` | Headings, excerpt, and a snapshot of a `chromeTabId`, without the debugger |
| `snapshotTab` | Semantic page map of a `chromeTabId`: role, name, text, state, visibility, and a stable ref |
| `navigateBrowser` | Load a URL in a DevTools-connected tab, without focusing it |
| `navigateTab` | Load a URL in the background from `listTabs`, visible window only |
| `softClick` / `softType` | Untrusted click or type. Result is `worked` or `needs a real gesture` |
| `actOnRef` | Click or type a `snapshotTab` ref, then report URL, console, and failed-request changes |
| `takeScreenshot` | Screenshot returned **as an image**, plus its file path |
| `refreshBrowser` | Reloads the inspected tab |
| `getBrowserStorage` | localStorage, sessionStorage and cookies (values gated) |
Expand All @@ -123,13 +125,15 @@ clients can auto-approve them safely.

### Tabs without DevTools

`listTabs`, `readTab`, `navigateTab`, `softClick`, and `softType` use `chrome.tabs` and `chrome.scripting`. They do not attach the debugger, so they do not show the debugging banner, and they do not activate a tab or focus a window.
`listTabs`, `readTab`, `snapshotTab`, `navigateTab`, `softClick`, `softType`, and `actOnRef` use `chrome.tabs` and `chrome.scripting`. They do not attach the debugger, so they do not show the debugging banner, and they do not activate a tab or focus a window.

`chromeTabId` comes from `listTabs`. `tabId` on the other tools comes from `listBrowserTabs`. They are different id spaces. Pass each id only to the tool that issued it.

`listTabs` reports the tabs `chrome.tabs` returns in the windows it returns. Other Arc Spaces, Little Arc, pinned favorites, split-view partners, and slept tabs may be missing. Incognito is skipped. This is not every open tab. Background navigation stays in the visible window until that is measured on Arc.

The popup switch has to be on. Navigation and soft input also need `--allow-input` on `browser-tools-mcp`, or `--allow-actions` on the `browser-tools` CLI. The same origin rules as other input apply. An asleep tab is reported and not reloaded. A soft action says `worked` or `needs a real gesture` and does not claim a failed action worked.
The popup switch has to be on. Navigation, soft input, and `actOnRef` also need `--allow-input` on `browser-tools-mcp`, or `--allow-actions` on the `browser-tools` CLI. The same origin rules as other input apply. An asleep tab is reported and not reloaded. A soft action says `worked` or `needs a real gesture` and does not claim a failed action worked.

`snapshotTab` returns a semantic page map for one `chromeTabId`: role, accessible name, text, state, visibility, and a stable ref. `actOnRef` uses a ref from that map and the same untrusted click or type path as `softClick` and `softType`. After the action it reports a short delta: whether the URL changed, console errors that appeared, and network requests that failed. Neither call attaches the debugger or changes which tab is active.

### Several tabs at once

Expand Down
5 changes: 3 additions & 2 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,8 +127,9 @@ be, both are worth reporting.
- Chrome's "started debugging this browser" banner is a browser safety
control. It is intentional, and this project never hides it, suppresses it,
or documents a setup step that does. Features that can run without the
debugger do so. `listTabs`, `readTab`, `navigateTab`, `softClick`, and
`softType` use ordinary extension APIs and do not call `chrome.debugger`,
debugger do so. `listTabs`, `readTab`, `snapshotTab`, `navigateTab`,
`softClick`, `softType`, and `actOnRef` use ordinary extension APIs and do
not call `chrome.debugger`,
`tabs.update({active:true})`, or `windows.update({focused:true})`. When a
feature attaches the debugger, the banner stays visible for that attachment,
and dismissing it stops the attachment.
Expand Down
5 changes: 5 additions & 0 deletions browser-tools-mcp/src/connector/api-policy.ts
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,11 @@ const AUDIT_TOOL: Record<AuditCategory, string> = {
export const API_TOOL_NAMES: readonly string[] = [
"listTabs",
"readTab",
"snapshotTab",
"navigateTab",
"softClick",
"softType",
"actOnRef",
"listBrowserTabs",
"getConsoleLogs",
"getConsoleErrors",
Expand Down Expand Up @@ -104,9 +106,11 @@ function toolsForRequest(method: string, path: string, query: Record<string, unk
if (method === "GET" && path === "/tabs") return ["listBrowserTabs"];
if (method === "GET" && path === "/chrome-tabs") return ["listTabs"];
if (method === "POST" && path === "/chrome-tabs/read") return ["readTab"];
if (method === "POST" && path === "/chrome-tabs/snapshot") return ["snapshotTab"];
if (method === "POST" && path === "/chrome-tabs/navigate") return ["navigateTab"];
if (method === "POST" && path === "/chrome-tabs/soft-click") return ["softClick"];
if (method === "POST" && path === "/chrome-tabs/soft-type") return ["softType"];
if (method === "POST" && path === "/chrome-tabs/act") return ["actOnRef"];
if (method === "POST" && path === "/screenshot") return ["takeScreenshot"];
if (method === "POST" && path === "/refresh") return ["refreshBrowser"];
if (method === "POST" && path === "/navigate") return ["navigateBrowser"];
Expand Down Expand Up @@ -148,6 +152,7 @@ export function isStateChangingRequest(method: string, path: string): boolean {
path === "/chrome-tabs/navigate" ||
path === "/chrome-tabs/soft-click" ||
path === "/chrome-tabs/soft-type" ||
path === "/chrome-tabs/act" ||
path === "/settings" ||
path === "/tabs/select")
);
Expand Down
114 changes: 108 additions & 6 deletions browser-tools-mcp/src/connector/connector.ts
Original file line number Diff line number Diff line change
Expand Up @@ -73,16 +73,20 @@ import {
RestrictedPageError,
} from "../restricted-page.js";
import type {
ActOnRefResult,
ChromeNavigateResult,
ChromeTabListing,
ChromeTabRead,
SemanticSnapshot,
SoftInputResult,
} from "../tabs/contract.js";
import {
assertTabActionSafe,
normalizeChromeTabListing,
parseActOnRef,
parseChromeNavigate,
parseChromeTabRead,
parseSemanticSnapshot,
parseSoftInput,
} from "../tabs/listing.js";

Expand Down Expand Up @@ -284,6 +288,14 @@ export interface Connector {
text: string;
submit?: boolean;
}): Promise<SoftInputResult>;
snapshotChromeTab(options: { chromeTabId: number }): Promise<SemanticSnapshot>;
actOnChromeTabRef(options: {
chromeTabId: number;
ref: string;
op: "click" | "type";
text?: string;
submit?: boolean;
}): Promise<ActOnRefResult>;
close(): Promise<void>;
}

Expand Down Expand Up @@ -622,6 +634,19 @@ export async function createConnector(config: ConnectorConfig = {}): Promise<Con
}
});

api.post("/chrome-tabs/snapshot", async (req: Request, res: Response) => {
try {
const chromeTabId = req.body?.chromeTabId;
if (typeof chromeTabId !== "number") {
res.status(400).json({ error: "chromeTabId is required" });
return;
}
res.json(await snapshotChromeTab({ chromeTabId }));
} catch (error) {
respondWithError(res, error);
}
});

api.post("/chrome-tabs/navigate", async (req: Request, res: Response) => {
try {
const url = typeof req.body?.url === "string" ? req.body.url.trim() : "";
Expand Down Expand Up @@ -677,6 +702,37 @@ export async function createConnector(config: ConnectorConfig = {}): Promise<Con
}
});

api.post("/chrome-tabs/act", async (req: Request, res: Response) => {
try {
const chromeTabId = req.body?.chromeTabId;
const ref = req.body?.ref;
const op = req.body?.op;
const text = req.body?.text;
if (typeof chromeTabId !== "number" || typeof ref !== "string") {
res.status(400).json({ error: "chromeTabId and ref are required" });
return;
}
if (op !== "click" && op !== "type") {
res.status(400).json({ error: "op must be click or type" });
return;
}
if (op === "type" && typeof text !== "string") {
res.status(400).json({ error: "text is required" });
return;
}
res.json(
await actOnChromeTabRef({
chromeTabId,
ref,
op,
...(op === "type" && typeof text === "string" ? { text, submit: req.body?.submit === true } : {}),
})
);
} catch (error) {
respondWithError(res, error);
}
});

api.post("/navigate", async (req: Request, res: Response) => {
try {
const url = typeof req.body?.url === "string" ? req.body.url.trim() : "";
Expand Down Expand Up @@ -1225,7 +1281,9 @@ export async function createConnector(config: ConnectorConfig = {}): Promise<Con
case "navigate-chrome-result":
case "chrome-tabs-result":
case "chrome-tab-read-result":
case "chrome-tab-snapshot-result":
case "soft-input-result":
case "act-ref-result":
case "storage-result":
case "capability-probe-result":
resolvePending(connection, message);
Expand Down Expand Up @@ -1769,6 +1827,17 @@ export async function createConnector(config: ConnectorConfig = {}): Promise<Con
return parseChromeTabRead(options.chromeTabId, response["read"]);
}

async function snapshotChromeTab(options: { chromeTabId: number }): Promise<SemanticSnapshot> {
const gate = await checkCapability("readTab", { tabId: options.chromeTabId });
if (!gate.allowed) throw new ExtensionRequestError(gate.reason);
const response = await requestFromExtension(requireWorker(), "snapshot-chrome-tab", {
chromeTabId: options.chromeTabId,
approvedUrl: gate.liveUrl,
});
assertTabActionSafe(response);
return parseSemanticSnapshot(options.chromeTabId, response["snapshot"]);
}

async function navigateChromeTab(options: {
url: string;
chromeTabId?: number;
Expand Down Expand Up @@ -1821,6 +1890,27 @@ export async function createConnector(config: ConnectorConfig = {}): Promise<Con
return parseSoftInput(options.chromeTabId, response);
}

async function actOnChromeTabRef(options: {
chromeTabId: number;
ref: string;
op: "click" | "type";
text?: string;
submit?: boolean;
}): Promise<ActOnRefResult> {
const gate = await checkCapability(options.op === "type" ? "softType" : "softClick", {
tabId: options.chromeTabId,
});
if (!gate.allowed) throw new ExtensionRequestError(gate.reason);
const response = await requestFromExtension(requireWorker(), "act-chrome-tab", {
op: options.op,
chromeTabId: options.chromeTabId,
ref: options.ref,
approvedUrl: gate.liveUrl,
...(options.op === "type" ? { text: options.text ?? "", submit: options.submit === true } : {}),
});
return parseActOnRef(options.chromeTabId, options.ref, response);
}

async function close(): Promise<void> {
clearInterval(heartbeat);
for (const { timer, reject } of pending.values()) {
Expand Down Expand Up @@ -1875,6 +1965,8 @@ export async function createConnector(config: ConnectorConfig = {}): Promise<Con
navigateChromeTab,
softClickChromeTab,
softTypeChromeTab,
snapshotChromeTab,
actOnChromeTabRef,
readStorage,
runAudit,
close,
Expand Down Expand Up @@ -1985,7 +2077,13 @@ function parseList(value: unknown): string[] | undefined {
return undefined;
}

/** Binds the requested port, walking forward if it is taken. */
/**
* Binds the requested port, walking forward if it is taken.
*
* A failed listen used to call listen() again without closing. Each failure
* left its listening callback attached, and on macOS that stack could sit
* there until the caller timed out.
*/
function listen(
server: http.Server,
host: string,
Expand All @@ -1998,21 +2096,25 @@ function listen(

const tryListen = () => {
const onError = (error: NodeJS.ErrnoException) => {
server.removeListener("listening", onListening);
if (error.code === "EADDRINUSE" && remaining > 1 && startPort !== 0) {
remaining -= 1;
port += 1;
setImmediate(tryListen);
server.close(() => tryListen());
return;
}
reject(error);
};

server.once("error", onError);
server.listen(port, host, () => {
server.off("error", onError);
const onListening = () => {
server.removeListener("error", onError);
const address = server.address();
resolve(typeof address === "object" && address ? address.port : port);
});
};

server.once("error", onError);
server.once("listening", onListening);
server.listen(port, host);
};

tryListen();
Expand Down
38 changes: 38 additions & 0 deletions browser-tools-mcp/src/mcp/client.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,11 @@ import {
type RestrictedPageKind,
} from "../restricted-page.js";
import type {
ActOnRefResult,
ChromeNavigateResult,
ChromeTabListing,
ChromeTabRead,
SemanticSnapshot,
SoftInputResult,
} from "../tabs/contract.js";

Expand Down Expand Up @@ -103,6 +105,14 @@ export interface ConnectorClient {
text: string;
submit?: boolean;
}): Promise<SoftInputResult>;
snapshotChromeTab(options: { chromeTabId: number }): Promise<SemanticSnapshot>;
actOnChromeTabRef(options: {
chromeTabId: number;
ref: string;
op: "click" | "type";
text?: string;
submit?: boolean;
}): Promise<ActOnRefResult>;
storage(kinds: string[], options?: { tabId?: TabId }): Promise<Record<string, unknown>>;
audit(
category: AuditCategory,
Expand Down Expand Up @@ -213,6 +223,20 @@ export class InProcessConnectorClient implements ConnectorClient {
return this.#connector.softTypeChromeTab(options);
}

snapshotChromeTab(options: { chromeTabId: number }): Promise<SemanticSnapshot> {
return this.#connector.snapshotChromeTab(options);
}

actOnChromeTabRef(options: {
chromeTabId: number;
ref: string;
op: "click" | "type";
text?: string;
submit?: boolean;
}): Promise<ActOnRefResult> {
return this.#connector.actOnChromeTabRef(options);
}

async storage(
kinds: string[],
options: { tabId?: TabId } = {}
Expand Down Expand Up @@ -366,6 +390,20 @@ export class HttpConnectorClient implements ConnectorClient {
return this.#request("/api/chrome-tabs/soft-type", { method: "POST", body: options });
}

snapshotChromeTab(options: { chromeTabId: number }): Promise<SemanticSnapshot> {
return this.#request("/api/chrome-tabs/snapshot", { method: "POST", body: options });
}

actOnChromeTabRef(options: {
chromeTabId: number;
ref: string;
op: "click" | "type";
text?: string;
submit?: boolean;
}): Promise<ActOnRefResult> {
return this.#request("/api/chrome-tabs/act", { method: "POST", body: options });
}

async storage(
kinds: string[],
options: { tabId?: TabId } = {}
Expand Down
2 changes: 2 additions & 0 deletions browser-tools-mcp/src/mcp/tool-invoke.ts
Original file line number Diff line number Diff line change
Expand Up @@ -324,8 +324,10 @@ export const INVOKABLE_TOOL_NAMES: readonly string[] = [
"listTabs",
"readTab",
"navigateTab",
"snapshotTab",
"softClick",
"softType",
"actOnRef",
"listBrowserTabs",
"getConsoleLogs",
"getConsoleErrors",
Expand Down
6 changes: 6 additions & 0 deletions browser-tools-mcp/src/runtime.ts
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,12 @@ class UnavailableConnectorClient implements ConnectorClient {
softTypeChromeTab(): Promise<never> {
return this.#fail();
}
snapshotChromeTab(): Promise<never> {
return this.#fail();
}
actOnChromeTabRef(): Promise<never> {
return this.#fail();
}
storage(): Promise<never> {
return this.#fail();
}
Expand Down
Loading
Loading