Skip to content

Late navigations, streamed planner answers, and declines that skip instead of stall - #45

Merged
ahamsel merged 2 commits into
mainfrom
claude/late-navigation-streaming
Sep 28, 2026
Merged

ahamsel merged 2 commits into
mainfrom
claude/late-navigation-streaming

Conversation

@ahamsel

@ahamsel ahamsel commented Sep 28, 2026

Copy link
Copy Markdown
Owner

Why

Real runs showed four problems:

  • Late navigations. Sites often navigate a moment after an action (Kijiji submits its search once suggestions load), so Jev's next click landed on the page being left.
  • Planner answers arrived all at once at the end. Gemini delivers tool arguments, including finish({result}), in one piece.
  • Facebook Marketplace, "find me rent downtown under $1000":
    • Jev tried to click "Send message to " three times.
    • Each decline was honored at the click, so nothing was sent. But fast mode's step loop swallowed the decline as an ordinary failed step, so it moved on and asked again, and the flight log showed plain "click" steps.
    • It then ran out of fast-mode steps and handed off to the planner: 67s, 2 listings.
  • Walmart: picked walmart.com for a user in Canada.

What changed

Late navigations

  • Both drivers notice when the page starts changing after the last snapshot.
  • Instead of acting on the old page, they wait for the new one and ask for a fresh snapshot.
  • Same-document navigations and subframe loads don't hold clicks up.

Streaming

  • The planner replies with its answer as plain text, which streams.
  • Tool-chat turns are streamed and rebuilt exactly (content, tool calls, reasoning_details) for the next turn.
  • Text written before a tool call is withdrawn.

Declines and irreversible actions

  • Skip replaces Cancel. The action isn't done and isn't asked about again during the task; the task carries on, and the answer says what was skipped. The stop button ends the task.
  • Find-only requests never send, buy or delete. Those controls aren't offered to Jev, and the planner refuses them without asking.
  • A Stop during a confirmation ends the task.
  • The flight log shows "declined:", "skipped:" and "failed:" steps.

Fast mode

  • A field the task gives nothing to type into is skipped instead of handing off.
  • When it runs out of steps with pages read, it answers from them (checked by Jev) before calling the planner.
  • Option-finding answers list the best several matches.
  • Pages keep 4k characters of text for the answer; Jev still sees 1.5k.

Resilience

  • A model turn that fails mid-stream (an upstream abort hit a real run) is withdrawn and retried once.
  • The same goes for errors inside 200 responses.

Locale

  • The site picker, router and planner know the user's browser language and time zone (walmart.ca in Canada).

Traces

  • Each Jev decision records its page (title, URL, start of text) and its top four options with probabilities.

Results

  • Same Marketplace request (Edge): 16.6s, no prompts. A detailed best downtown match ($625, available October, link) plus 15 more listings under $1000. The answer starts showing at 12.2s. Before: 67s, 3 prompts, 2 listings.
  • User's MUN request (Edge, from the MUN account): 15.9s, correct. The answer streams about 1.4s before the end.
  • Kijiji (Chrome): 10.8s, no stale clicks.
  • npm test: 71/71. New tests cover:
    • late-navigation driver behavior;
    • streamed tool turns (rebuilt calls and reasoning; withdrawn text);
    • retry after a mid-stream failure;
    • declined clicks skipped and not re-offered;
    • find-only requests never offering or clicking buy/send;
    • fields with nothing to type skipped.
  • npm run extension:smoke: 11/11. Read-only cases now also fail on any confirmation prompt.
  • Realistic suite in the user's browsers: Chrome 11/11, median 9.4s; Edge (see comment).
  • npm run build passes.

🤖 Generated with Claude Code

ahamsel and others added 2 commits September 28, 2026 19:59
- Late navigations: sites often navigate a moment after an action (a search
  that submits once its suggestions load). Both drivers now notice when the
  page starts changing after the last snapshot and, instead of clicking or
  typing on the page being left, wait for the new page and ask for a fresh
  snapshot. Same-document navigations and subframe loads don't hold clicks up.
  Fast mode doesn't count such steps against its no-progress limit.
- Streaming: the planner replies with its answer as plain text instead of a
  finish() call (Gemini delivers tool arguments in one piece at the end), and
  tool-chat turns are streamed and rebuilt exactly (content, tool calls,
  reasoning_details) for the next turn. Text written before a tool call is
  withdrawn. The dev bridge reports when an answer starts showing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ests never send or buy

From a real Facebook Marketplace run ("find me rent downtown under $1000"):
Jev tried to click "Send message to <seller>" three times. Each decline was
honored at the click (nothing was clicked), but fast mode's step loop swallowed
the decline as an ordinary failed step, so it moved on and asked again; the
flight log showed the steps as plain clicks. 67s, 2 listings.

- Decline = skip: the action isn't done, isn't asked about again during the
  task (task-wide list), and the task carries on; the answer says what was
  skipped. The stop button ends the task. The card says Allow / Skip.
- Requests that only find or read never send, buy or delete: those controls
  aren't offered to Jev, and the planner refuses them without asking.
- A Stop during a confirmation ends the task instead of failing one step.
- Flight log: "declined:", "skipped:", "failed:" steps.
- A field the task gives nothing to type into ("Send seller a message") is
  skipped instead of handing the task to the planner.
- Out of steps with pages read, fast mode answers from them (checked) before
  the planner; option-finding answers list the best several matches; pages
  keep 4k characters of text for the answer (Jev still sees 1.5k).
- A model turn that fails mid-stream (provider abort) is withdrawn and retried
  once; so are errors inside 200 responses.
- Sites are picked for the user's country (browser language and time zone).
- Traces record each Jev decision's page (title, URL, text start) and top four
  options with probabilities.

Same Marketplace request now: 16.6s, no prompts, a detailed best match plus
15 more listings under $1000.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ahamsel

ahamsel commented Sep 28, 2026

Copy link
Copy Markdown
Owner Author

Edge realistic suite: 11/11 passed, median 10.5s (mail_two_parts 14.6s, books_two_parts 8.1s, shop_buy 19.1s with 1 confirmation).

@ahamsel
ahamsel merged commit 7ef31e9 into main Sep 28, 2026
1 check passed
@ahamsel
ahamsel deleted the claude/late-navigation-streaming branch September 29, 2026 00:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant