Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions docs/v2.1/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# v2.1 design notes

Design and spike plans for the tab-awareness follow-ups. They do not change
runtime behavior, and they do not block the 2.1.0 code. Nothing here attaches
the debugger, launches a browser, or publishes a package.

Banner policy, from [SECURITY.md](../../SECURITY.md):

> Chrome's "started debugging this browser" banner is a browser safety
> control. It is intentional, and this project never hides it, suppresses it,
> or documents a setup step that does. Features that can run without the
> debugger do so. When a feature attaches the debugger, the banner stays
> visible for that attachment, and dismissing it stops the attachment.

The same rule is in the README. These notes keep it.

Order of operations, matching the
[Oct 23 findings](https://linear.app/agentdesk/document/findings-using-teds-browser-without-the-debugging-banner-oct-23-2026-f49d757c2c4d):

| Need | Path | Banner in Ted's browser |
| --- | --- | --- |
| See tabs, read them, soft click and type | [AGE-584](https://linear.app/agentdesk/issue/AGE-584) default. No debugger. | No |
| UI that only works once the page is drawn | [AGE-613](age-613-never-focused-window.md) spike. Still no debugger. | No |
| A short batch of real (trusted) input in Ted's signed-in browser | [AGE-614](age-614-batched-debugger-attach.md). Opt-in. Banner stays up. | Yes, for the batch plus about 5 seconds after the last detach |
| Real input on a job that should not touch Ted's session | [AGE-615](age-615-agent-profile-lane.md). A different browser profile. | No |

[AGE-617](age-617-upstream-watchlist.md) is the monthly check on the two
upstream changes that could shrink that table. [AGE-616](https://linear.app/agentdesk/issue/AGE-616)
still has to rule on the bypass list before any debugger fallback ships.
This set does not ask for any bypass to be turned on.
102 changes: 102 additions & 0 deletions docs/v2.1/age-613-never-focused-window.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
# AGE-613: never-focused agent window

[AGE-613](https://linear.app/agentdesk/issue/AGE-613). Spike plan and the
numbers that decide whether AGE-584 v2 should build it. This note does not
record a measurement. Every cell below stays `unmeasured` until the spike
runs on Chrome stable, on a machine where someone is actually working.

No debugger, no banner, no `tabs.update({ active: true })`, no
`windows.update({ focused: true })`. The private macOS display API that
agent-lanes uses for an invisible screen is out of scope. Do not port it.

## What the spike creates

A normal Chrome window, created once with
`chrome.windows.create({ focused: false, type: "normal", state: "normal", ... })`
and never updated to focused, minimized, maximized, or fullscreen.
Minimized windows stop drawing. The agent's tab is the active tab inside
that window only.

Suggested home when someone runs it: `spikes/age-613-agent-window/`, same
shape as `spikes/age-618-chrome-tabs/`. Permissions: `tabs`, `windows`,
`storage`. Not `debugger`. The spike must not read page contents beyond a
same-origin fixture it opened itself.

Placement to try first (question 5): 320×240, left at
`screen.availWidth - 8`, top at `screen.availHeight - 8`, so a sliver stays
on the display Ted is not using. Title prefix `BrowserTools agent`. If he
closes it, recreate once and log the close. A second close ends the spike
and reports `closed-by-user`. An off-screen window (fully past
`availWidth`) is a second arm, because occlusion may stop drawing there.

## Sampler

Sample from 100ms before each create until 2s after each close.

| Signal | How | Valid run |
| --- | --- | --- |
| Frontmost app | A local sampler on a 20ms loop. Record the actual interval. | Median interval ≤ 25ms and no gap over 40ms. A slower run is invalid, not a pass. |
| Chrome's focused window | The extension, `chrome.windows.onFocusChanged` plus `windows.getLastFocused` | Every cycle |
| Active tab in Ted's window | `chrome.tabs.query` on the window id captured before the cycle | Unchanged |
| Space | Automatic window-focus signal, plus a person watching 10 of the 50 cycles | The 10 watched cycles are written down |

AppleScript once per sample will miss the interval. Use a loop that can
actually hit 20ms. Do not call a private window-server API to get there.

The with-flag arm (question 2) relaunches Chrome with
`--disable-backgrounding-occluded-windows`. That arm is a separate Chrome
process, not a silent change to Ted's daily launch. The flag is a drawing
flag. It is still a launch change, so it stays opt-in even if the numbers
say drawing needs it.

## Score sheet

Run Chrome stable first. Arc is a separate sheet. Fill `result` with `yes`,
`no`, or `only-with-flag`. Leave `unmeasured` until then.

| # | Question | Pass | Fail | Result |
| --- | --- | --- | --- | --- |
| 1 | Does the unfocused window stay behind? | 0 focus steals in 50 create/close cycles. Ted's active tab id unchanged on all 50. Frontmost app unchanged, or Chrome stays front and the focused window id stays Ted's. 0 Space changes on the 10 watched cycles. | Any steal, any active-tab change, any watched Space change. | unmeasured |
| 2 | Is the occlusion flag required to draw? | Without the flag, over 2s while Ted's window covers the agent window: requestAnimationFrame count ≥ 30, a lazy-loaded fixture node appears, and `captureVisibleTab(agentWindowId)` is not a blank frame. | Those checks fail without the flag and pass with it → `only-with-flag`. They fail both ways → `no`. | unmeasured |
| 3 | Do external opens land in the agent window? | 0 of 20 `open <url>` launches, 0 of 5 Mail links, and 0 of 5 Slack links have `windowId` equal to the agent window. If any do, the mitigation (below) moves 20/20 back with 0 focus steals. | A landed tab stays in the agent window, or the move changes focus. | unmeasured |
| 4 | Does Arc pass question 1? | Same bar as question 1. | Any API failure, Little Arc, or a Space switch. Record `no`. Chrome can still pass. | unmeasured |
| 5 | Can the window sit where Ted will not close it, and still draw? | The question-5 placement passes question 2, and two intentional closes stop the spike (one recreate, then stop). | Drawing only works fully on-screen in the middle of the work, or the spike keeps recreating. | unmeasured |

Fixture page for question 2, served by the spike: a rAF counter, a
`content-visibility` or below-the-fold image, and a menu that opens on the
second animation frame. The page origin is the fixture only.

Question 3 mitigation, if the first counts are non-zero: on
`chrome.tabs.onCreated`, when `windowId` is the agent window and the opener
is not the agent's own tab, `chrome.tabs.move` the new tab into Ted's
window with the moved tab not active. Do not call `windows.update`. Also
rewrite `window.open` from the agent's own page into a same-tab navigation,
so a popup cannot pull the agent window forward.

Record the browser version, the screen layout, the sampler intervals, and
the cycle counts next to the sheet. Origins only if a note has to mention a
URL. No page titles.

## Recommendation rule

Build it into AGE-584 v2 only when all of these are true:

- Question 1 is `yes` on Chrome stable.
- Question 2 is `yes`, or `only-with-flag` and the flag stays an opt-in
launch for that window's Chrome, documented as a drawing flag.
- Question 3 is `yes` with or without the move-back mitigation.
- Question 5 is `yes`.
- Question 4 is recorded. `no` on Arc means the feature is Chrome-only,
which is an acceptable ship note. A missing Arc row is not.

Drop it when question 1 is `no`. A window that takes focus fails Ted's ask,
and no flag repairs that.

Until the sheet is filled, the recommendation is to run the spike and to
keep it out of 2.1.0. There is no measured `yes` to build on.

## Out of scope

Trusted input and any debugger attach. That is AGE-614. Logged-out jobs
that need real input throughout are AGE-615. Both can run later against a
window this spike proved, and neither is part of the measurement.
197 changes: 197 additions & 0 deletions docs/v2.1/age-614-batched-debugger-attach.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,197 @@
# AGE-614: brief batched debugger attach

[AGE-614](https://linear.app/agentdesk/issue/AGE-614). Design only. The
implementation waits on this note, the tests listed below, and security-lead
sign-off ([AGE-483](https://linear.app/agentdesk/issue/AGE-483),
[AGE-616](https://linear.app/agentdesk/issue/AGE-616)).

## Decision

Real input is a short, explicit fallback behind the input switch. The
extension attaches `chrome.debugger` to one tab, runs the batch, and detaches
before the tool returns. The Chrome banner stays visible for that attachment.
Dismissing the banner stops the batch and blocks another attach until Ted
turns the panel switch off and back on.

Default work stays on the AGE-584 path and never attaches.

## When an attach is allowed

All of these are already enforced by `decideCapability` in
`browser-tools-mcp/src/gates/capability.ts` and `confirmBeforeAct` in
`chrome-extension/gates.js`. An attach happens only after both have allowed
`debuggerFallback`.

| Gate | Today | Attach? |
| --- | --- | --- |
| `--allow-input` off (the default) | `inputOff` | Never |
| DevTools panel checkbox off (the default, session key `btmcpPanelInput`) | `panelOff` | Never |
| `hideBanner: true` | `banner` ("banner bypass is disabled") | Never |
| Live tab URL missing, non-http(s), excluded, or a public origin not in `--allow-input-origins` | url / exclude reasons | Never |
| Local origin, or an origin listed in `--allow-input-origins`, with both switches on | `allowed` | Yes, for one batch |

The toolbar popup switch does not arm this action. `debuggerFallback` is a
panel action. Soft click, soft type, and navigate stay on the popup switch
and still do not attach.

The panel label says "on this tab". The bit is one session-storage flag for
the whole browser session. Before this ships, either the label should say
that, or the flag should become per-tab. Until then the origin check is what
limits which tab can be attached, and it is re-read from `chrome.tabs.get`
immediately before the attach.

Two further gates belong in the same function when this is built:

- `bannerDismissed`, after Ted presses Cancel, until the panel checkbox goes
off and then on again in this browser session.
- A live URL re-check after attach, before the first input command. A
navigation between the check and the command aborts the batch.

`chrome.debugger.getTargets()` is not a probe. Call it for nothing. An
unsettled field report says the call itself can raise the banner, and the
extension can track its own sessions without it.

## Who owns the session

Chrome gives this extension one debugger client per tab. The service worker
and the DevTools page cannot both attach to that tab.

| Situation | Owner | What the other side does |
| --- | --- | --- |
| DevTools is open on that tab and capture mode is `debugger` (devtools.js has already called `attach`) | The DevTools page | The worker sends the batch over the `btmcp-debugger` port. It does not call `attach` or `detach`. |
| DevTools is closed, or capture mode is `inject` | The service worker | It attaches, runs the batch, undoes focus emulation, and detaches. |
| `attach` fails because a client is already there, and no port answers within 300ms | Nobody | The batch fails with `owner-unavailable`. No second attempt. |

`devtools.js` already owns the long-lived capture session
(`attachDebugger` around the inspected `tabId`, protocol `1.3`). The worker
today does not attach; the comment at the top of `worker.js` says so. This
fallback is the first worker attach, and only in the second row.

The DevTools page registers the port when it attaches and clears it when it
detaches or unloads. After a service-worker restart the map is empty. A
failed attach plus a silent port is `owner-unavailable`, not a retry.

One batch at a time for the whole extension. A second `debuggerFallback`
while one is running returns `busy`. It does not queue. A queue would hold
the session open.

## Batching rule

A batch is one tool call: at most 8 steps, one `tabId`, wall clock 5 seconds
for the commands and a hard stop at 8 seconds. The worker (or the DevTools
page, if it owns the tab) attaches at the start and detaches in a `finally`
block before the call returns. There is no idle timer and no "keep it warm".

Steps that may be in the batch: a trusted click, a trusted key or text
insert, and the gesture that opens a file picker, a popup, or fullscreen.
The file picker then stops. The tool reports `needsUser: "file-chooser"` and
Ted picks the file. This design does not supply a filesystem path.

Soft input runs first. The attach happens after a step comes back
`needsRealGesture`, or when the caller asked for a known gesture action
(clipboard, file picker, popup, fullscreen, a site that checks `isTrusted`).
A step on a different tab ends the call. The agent sends a new call, which
is a new attach, after the first detach.

Protocol version stays `1.3`, the version capture already uses, so a borrowed
DevTools session and a worker session match.

Commands inside the window:

1. `Emulation.setFocusEmulationEnabled` with `{ enabled: true }`.
2. The input commands for the steps.
3. `Page.setWebLifecycleState` with `{ state: "active" }` only if a step
still failed because the page was frozen, and only for the rest of this
batch. Record `wokeTab: true`. Discarded tabs stay skipped on the default
path; waking one is part of the result, not a surprise.
4. In `finally`, before detach: focus emulation back to `{ enabled: false }`.
Leave lifecycle `active` if it was woken, and say so.

`chrome.tabs.update({ active: true })` and `chrome.windows.update({ focused: true })`
are not called. Ted's active tab, window, and Space stay as they were.

Side effect for the security review: the page is told it is focused, and
maybe that it is visible. Read receipts, presence, and autoplay can fire
while the batch runs. The result includes `focusEmulated: true` when step 1
ran.

## The banner stays, and Cancel stops the work

Attaching shows Chrome's "started debugging this browser" banner. Facts
taken from Chromium's extension infobar (Amplifier's
[DEBUGGER_BANNER.md](https://github.com/microsoft/amplifier-browser-bridge/blob/main/docs/DEBUGGER_BANNER.md)
cites the source; this project has not re-timed them):

- One banner per extension, on every tab in every window of that profile,
not only the tab being driven.
- Chromium clears it about 5 seconds after the last detach. It does not
clear while any session of this extension is still attached, and a
navigation does not clear it.
- Cancel detaches every debugger session this extension holds, including
console capture on other tabs.

So the "action plus about 5 seconds" line is true only when this batch is
the extension's only debugger session. Capture mode `debugger` on any open
DevTools holds a session for as long as that DevTools is open, and the
banner stays for that whole time. The result says `bannerCleared: false`
and `bannerReason: "other-session"` in that case. The brief-banner promise
needs capture mode `inject`, or DevTools closed, on every tab.

Dismiss handling:

1. Listen for `chrome.debugger.onDetach` before `attach`.
2. Reason `canceled_by_user` sets `cancelled: true`, sets the
`bannerDismissed` latch, and skips every step not already sent.
3. No code path calls `attach` again until the panel checkbox is turned off
and on. A reconnect, a capture restart, or the next agent tool call does
not clear the latch.
4. The tool returns the partial result. It does not swap in a soft click
and report success.

`target_closed` also stops the batch. It does not set the latch.

## Result

Each call returns:

| Field | Meaning |
| --- | --- |
| `trusted` | `true` only when a CDP input command completed for that step |
| `steps[]` | One entry per step: name, `trusted`, `ok`, optional `error` |
| `attachedMs` | Time from attach acknowledgement to detach. `0` when an existing DevTools session was borrowed |
| `bannerShown` | `true` when this extension attached or already held a session |
| `bannerAlreadyUp` | `true` when the DevTools page already owned the tab |
| `bannerCleared` | `true` only if this detach was the last session |
| `cancelled` | Ted pressed Cancel |
| `detached` | This call released the session. `false` when DevTools still owns capture |
| `focusEmulated` | Focus emulation was turned on |
| `wokeTab` | Lifecycle was forced to `active` |
| `focusStealing` | Always `false`. Present so a caller can assert it |

## Tests to add with the implementation

These are not in this change.

| Case | Expected |
| --- | --- |
| `--allow-input` omitted | `debuggerFallback` is `inputOff`. `chrome.debugger.attach` is not called. |
| `hideBanner: true`, every other switch on | Reason `banner`. `attach` is not called. |
| Both switches on, allowlisted origin | One `attach`, the steps, focus emulation undone, one `detach`. No `tabs.update` active and no `windows.update` focused. |
| `onDetach` with `canceled_by_user` after step 1 of 3 | Steps 2 and 3 are not sent. No second `attach`. `cancelled: true`. The next call is `bannerDismissed` until the panel switch cycles. |
| DevTools already attached to that tab | The worker does not call `attach`. Commands go to the port. The fallback does not call `detach`. |
| Second call while the first batch is open | `busy`. Still one attach. |

Unit coverage for `hideBanner` already exists in
`browser-tools-mcp/test/unit/capability-gates.test.ts` and
`chrome-extension` gate tests. Keep it.

## For the security review

In scope to approve or reject: a worker-owned attach of at most 8 seconds,
banner visible, Cancel latched, focus emulation with the visibility side
effect, same origin rules as other input.

Out of scope, and not designed here: `--silent-debugger-extension-api`,
policy force-install, shielding this extension by disabling others,
SkyLight or AppleScript input, copying Ted's profile or cookies, and any
setup step that hides the banner.
Loading
Loading