Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -322,6 +322,12 @@ jobs:
cargo build -p bsk --locked
pnpm --filter @browser-skill/extension build
pnpm --filter @wxg-prc-cpg/browser-skill-dsh-plugin build
- name: Validate task termination on tab detachment
working-directory: apps/extension
env:
BSK_TAB_DETACH_CHROME: ${{ env.BSK_VIDEO_CHROME }}
BSK_TAB_DETACH_REPORT: ${{ env.BSK_VIDEO_ARTIFACTS }}/tab-detachment.json
run: pnpm exec vitest run src/session-manager/__tests__/tab-detachment.browser.test.ts
- name: Validate capture, playback, recovery and CLI/DSH export
working-directory: apps/extension
env:
Expand Down
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,8 @@ Use browser-skill to open https://example.com, summarize the page, and end the b

The agent opens its own window in the background, reads the page, returns the summary, and stops its task. External links or manually created tabs that land in the task window are moved to your most recently used normal window; a user window is created if none remain. Routing happens after tab creation, so a tab may briefly appear in the task window. Page-opened tabs and human assistance or recording are preserved. Harnesses that support skill slash commands can also use `/browser-skill`.

Moving any controlled tab out of its original Agent Window ends the entire browser session, preserves every page in place, and removes its control indicators. Borrowed tabs also stay where they are. Moving a tab back does not resume the task; starting again requires a user decision. Reordering tabs within the same window is unaffected. Normal session stop still returns borrowed tabs and cleans up task pages.

Chrome can still activate a window when a page opens a native popup. For observed task popups, the extension restores the previous Chrome window or minimized state if you have not switched windows. Chrome's public extension APIs cannot guarantee zero transient activation or restore focus to another application.

<details>
Expand Down
2 changes: 2 additions & 0 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,8 @@ bsk doctor

Agent 默认在后台打开独立代理窗口、读取页面、返回总结,并结束任务。误入代理窗口的外部链接或手动新建标签会移回最近使用的普通用户窗口;没有用户窗口时会新建一个。分流发生在标签创建之后,可能短暂显示在代理窗口中。页面打开的子标签、人类协助和录制流程会保留。支持 skill 斜杠命令的 Agent,也可以通过 `/browser-skill` 调用。

将任一受控标签移出原代理窗口会结束整个浏览器会话,保留全部页面并清除控制标识;借用页也留在当前位置。拖回标签不会恢复任务,需要由用户决定是否重新开始。同窗口内调整标签顺序不受影响,正常结束会话仍按原规则归还借用页、清理任务页。

网页打开原生弹窗时,Chrome 仍可能激活窗口。对于观察到的任务弹窗,若用户没有切换窗口,扩展会恢复之前的 Chrome 用户窗口或最小化状态。Chrome 公开扩展 API 无法保证完全没有瞬时激活,也无法恢复其他应用的焦点。

<details>
Expand Down
56 changes: 56 additions & 0 deletions apps/extension/src/browser-driver/__tests__/chromium-cdp.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,62 @@ afterEach(() => {
});

describe("ChromiumCdp", () => {
it.each([
"abort",
"replacement",
] as const)("does not downgrade %s during an OOPIF tree read to a missing child", async (ending) => {
const { api, onEvent } = fakeApi();
let finishChild!: (value: object) => void;
let childStarted!: () => void;
const started = new Promise<void>((resolve) => {
childStarted = resolve;
});
const childReply = new Promise<object>((resolve) => {
finishChild = resolve;
});
vi.mocked(api.sendCommand).mockImplementation(async (target, method) => {
if (method === "Page.getFrameTree") {
if (target.sessionId === "child-session") {
childStarted();
return childReply;
}
return {
frameTree: {
frame: { id: "main" },
childFrames: [{ frame: { id: "child", parentId: "main" } }],
},
};
}
return {};
});
const cdp = new ChromiumCdp(api);
const signal = new AbortController();
try {
await cdp.ensureAttached(4);
onEvent.fire({ tabId: 4 }, "Target.attachedToTarget", {
sessionId: "child-session",
targetInfo: { type: "iframe", targetId: "child" },
});
const graph = cdp.getFrameGraph(4, signal.signal);
await started;
if (ending === "abort") signal.abort();
else {
await cdp.detach(4);
await cdp.ensureAttached(4);
}
const callsBeforeReply = vi.mocked(api.sendCommand).mock.calls.length;
finishChild({ frameTree: { frame: { id: "child" } } });
await expect(graph).rejects.toMatchObject({
name: ending === "abort" ? "AbortError" : "CdpAttachmentChangedError",
});
expect(api.sendCommand).toHaveBeenCalledTimes(callsBeforeReply);
expect(api.attach).toHaveBeenCalledTimes(ending === "abort" ? 1 : 2);
} finally {
await cdp.detachAll();
cdp.dispose();
}
});

it("isolates a known child tree timeout and keeps it gated until recovery", async () => {
vi.useFakeTimers();
vi.spyOn(console, "warn").mockImplementation(() => {});
Expand Down
47 changes: 39 additions & 8 deletions apps/extension/src/browser-driver/chromium-cdp.ts
Original file line number Diff line number Diff line change
Expand Up @@ -421,18 +421,50 @@ export class ChromiumCdp {
}
}

async getFrameGraph(tabId: number): Promise<CdpFrameGraph> {
async getFrameGraph(tabId: number, signal?: AbortSignal): Promise<CdpFrameGraph> {
signal?.throwIfAborted();
await this.ensureAttached(tabId);
const attachmentId = this.getAttachmentId(tabId);
const check = () => {
signal?.throwIfAborted();
if (attachmentId === undefined || this.getAttachmentId(tabId) !== attachmentId) {
const error = new Error("Debugger attachment changed during frame discovery");
error.name = "CdpAttachmentChangedError";
throw error;
}
};
// Frame discovery belongs to this attachment. An old tree reply must not
// reconnect a released tab, or continue against another reader's connection.
const read = async <T>(
target: CdpTarget,
method: string,
params: object,
timeoutMs?: number,
): Promise<T> => {
check();
try {
const result = await this.command(target, method, params, timeoutMs, check);
check();
return result as T;
} catch (error) {
check();
throw error;
}
};
check();
// Ordinary configuration failures leave the graph to the root tree. A
// timed-out read means the renderer is not answering: stop here instead of
// queueing the tree, owner and snapshot reads behind the stuck command.
await this.enableFrameDiscovery({ tabId }).catch(rethrowReadTimeout);
check();
this.retryChildFrameDiscovery(tabId);
await this.drainFrameAttachTasks(tabId);
check();

const sources: CdpFrameTreeSource[] = [];
const unavailable = new Map<string, CdpTarget>();
const isolateChildTimeout = (error: unknown, target: CdpTarget) => {
check();
if (!(error instanceof CdpReadTimeoutError)) return;
const frameId =
target.sessionId && this.frameDiscovery.get(tabId)?.frameIds.get(target.sessionId);
Expand All @@ -442,7 +474,7 @@ export class ChromiumCdp {
throw error;
unavailable.set(frameId, target);
};
const root = await this.sendToTarget<{ frameTree?: CdpFrameTreeNode }>(
const root = await read<{ frameTree?: CdpFrameTreeNode }>(
{ tabId },
"Page.getFrameTree",
{},
Expand All @@ -455,7 +487,7 @@ export class ChromiumCdp {
sessions.map(async (sessionId): Promise<CdpFrameTreeSource | null> => {
const target = { tabId, sessionId };
try {
const reply = await this.sendToTarget<{ frameTree?: CdpFrameTreeNode }>(
const reply = await read<{ frameTree?: CdpFrameTreeNode }>(
target,
"Page.getFrameTree",
{},
Expand Down Expand Up @@ -484,18 +516,17 @@ export class ChromiumCdp {
const parent = frameById.get(frame.parentFrameId);
if (!parent) return;
try {
const owner = await this.sendToTarget<{ backendNodeId?: number }>(
parent.target,
"DOM.getFrameOwner",
{ frameId: frame.frameId },
);
const owner = await read<{ backendNodeId?: number }>(parent.target, "DOM.getFrameOwner", {
frameId: frame.frameId,
});
if (owner.backendNodeId !== undefined) frame.ownerBackendNodeId = owner.backendNodeId;
} catch (err) {
// The frame may have navigated between tree capture and owner lookup.
isolateChildTimeout(err, parent.target);
}
}),
);
check();
return omitFrameSubtrees(discovered, unavailable);
}

Expand Down
39 changes: 39 additions & 0 deletions apps/extension/src/cooperation/controller.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2145,6 +2145,45 @@ describe("tab management respects human control of the task window", () => {
expect(other.windowId).toBe(200);
});

it.each([
"input",
"hold",
] as const)("rechecks return after %s during the source tab lookup", async (intervention) => {
const f = await fixture();
f.ctx.agentCreatedTabs.delete(7);
f.ctx.borrowedTabs.set(7, { tabId: 7, originalWindowId: 200, originalIndex: 0 });
const move = vi.fn(async (_id, props) => ({ ...f.tab, ...props }));
const get = vi.fn(async () => {
if (intervention === "input") f.change({ human: 1, editing: true, lastInput: 10000 });
else await f.controller.control(7, f.controller.view(7)!.revision, "hold");
return { ...f.tab } as chrome.tabs.Tab;
});
const params = { session_id: "shared", tab_id: 7 };
expect(
await f.controller.run(
{ id: "return", method: "tool.tab_return", params },
f.cdp,
f.abort.signal,
(cdp, signal) =>
handleTabReturn(f.sessions, params, {
cdp,
signal,
tabs: { get, move, create: vi.fn(), remove: vi.fn(), update: vi.fn() },
windows: {
get: async () => ({ id: 200 }) as chrome.windows.Window,
getLastFocused: async () => ({ id: 200 }) as chrome.windows.Window,
create: vi.fn(),
remove: vi.fn(),
},
agentOverlayReset: { resetAgentOverlays: async () => {} },
}),
),
).toMatchObject({ data: { reason: "cooperation_sync_required", effect_state: "none" } });
expect(get).toHaveBeenCalled();
expect(move).not.toHaveBeenCalled();
expect(f.ctx.borrowedTabs.has(7)).toBe(true);
});

it("rechecks return after locating the original window", async () => {
const f = await fixture();
f.ctx.agentCreatedTabs.delete(7);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ function event<T extends unknown[] = []>() {
async function fixture() {
const overlay = new OverlayController();
const onDetached = event<[number]>();
const onAttached = event<[number, chrome.tabs.TabAttachInfo]>();
const onCreated = event<[chrome.tabs.Tab]>();
const onCreatedNavigationTarget = event<[{ tabId: number; sourceTabId: number }]>();
const sendMessage = vi.fn(
Expand All @@ -96,7 +97,7 @@ async function fixture() {
onActivated: event(),
onUpdated: event(),
onCreated,
onAttached: event(),
onAttached,
onRemoved: event(),
},
debugger: { onDetach: event() },
Expand All @@ -119,9 +120,33 @@ async function fixture() {
(background as unknown as () => void)();
const deps = vi.mocked(ToolDispatcher).mock.calls.at(-1)![0];
const task = await deps.sessions.start("one");
return { overlay, onDetached, onCreated, onCreatedNavigationTarget, sendMessage, deps, task };
return {
overlay,
onDetached,
onAttached,
onCreated,
onCreatedNavigationTarget,
sendMessage,
deps,
task,
};
}

it("refreshes released-page controls and groups after attachment completes", async () => {
const f = await fixture();
f.overlay.applyAgentControlMode("one", "control");
f.deps.sessions.release("one");
f.onAttached.emit(10, { newWindowId: 90, newPosition: 0 });
await Promise.resolve();
expect(f.overlay.snapshot().controlVisible).toBe(false);
expect(f.sendMessage).toHaveBeenCalledWith(
10,
expect.objectContaining({ sessionId: null, mode: "hidden" }),
);
const groups = vi.mocked(AgentWindowTabGroups).mock.instances.at(-1)!;
expect(groups.sync).toHaveBeenCalledOnce();
});

it("refreshes native window identification after session changes and explicit tab claims", async () => {
const f = await fixture();
const groups = vi.mocked(AgentWindowTabGroups).mock.instances.at(-1)!;
Expand Down
29 changes: 22 additions & 7 deletions apps/extension/src/entrypoints/background.ts
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ import {
attachRecordFinishListener,
attachRecordQueryListener,
attachRecordStepListener,
clearRecordingForSession,
hasActiveRecording,
type RecordRuntimeDeps,
} from "@/tools/record";
Expand Down Expand Up @@ -92,6 +93,21 @@ export default defineBackground(() => {
});
const sessions = new SessionManager({ remote: () => remoteEndpoint !== null });
const windowTabGroups = new AgentWindowTabGroups(sessions);
// Capture the original owner before cooperation releases a detached page.
attachSessionEventHandler({
manager: sessions,
transport,
cdp: {
detachSession: (id) => cdp.detachSession(id),
releaseSessionTab: (id, tabId) => cdp.releaseSessionTab(id, tabId),
},
onSessionReleased: (ctx) => {
dispatcher.cancelSession(ctx.sessionId);
cooperation.release(ctx.sessionId);
clearRecordingForSession(ctx.sessionId);
},
onSessionsChanged: onOverlaySessionStateChanged,
});
const cooperation = attachCooperation(
sessions,
(tabId) => {
Expand Down Expand Up @@ -357,6 +373,12 @@ export default defineBackground(() => {
if (!sessions.findByWindowId(tab.windowId)) return;
void pushOverlayStateForTab(tab.id, tab.windowId);
});
chrome.tabs.onAttached.addListener((tabId, { newWindowId }) => {
// Reconcile again after Chrome finishes the move; editing a group or
// messaging a page can fail transiently while it is being dragged.
void pushOverlayStateForTab(tabId, newWindowId);
windowTabGroups.sync();
});
chrome.tabs.onDetached.addListener((tabId) => {
void video.stopTab(tabId, "session_ended").catch(() => {});
debug.releaseTab(tabId);
Expand Down Expand Up @@ -493,13 +515,6 @@ export default defineBackground(() => {
"[browser-skill] chrome.notifications.onButtonClicked unavailable; borrow Allow/Deny buttons will be inactive",
);
}
attachSessionEventHandler({
manager: sessions,
transport,
cdp,
onSessionsChanged: onOverlaySessionStateChanged,
});

// MV3 service worker keepalive + reconnect supervisor (review M4/M5
// C3 + C4). Every 30s the alarm wakes the SW; if the transport is
// not connected we force a fresh connect attempt so we never sit on
Expand Down
Loading
Loading