feat(observe): act tools observe by default; the settled screen is kept in memory and sent only when asked

[leeguooooo]

leo asked for this: after every step, capture the screen and read the tree so the AI has the data ready and needs fewer calls. The screenshot is not handed to the AI unless it asks, to save tokens, and nothing is left behind.

MCP

• Single-step act tools now observe by default: phone_tap, phone_tap_element, phone_tap_label, phone_type, phone_key, phone_scroll and phone_shortcut.
• Each call names the last snapshot this client saw (from an elements read or an earlier observed action) as ?since=. The result is therefore the settled change, not the whole tree. Field names match batch observe in feat: batch-first agents, Jev on the phone, and a quieter setup #133.
• observe:false keeps the bare action.
• phone_screenshot is described as instant right after an observed action.

Daemon

• The settled screen is kept in memory, in WdaClient::settled_frame.
  • When the settle check saw two identical frames, that frame is kept at no extra cost.
  • Otherwise (a focused text field, or a screen that kept moving), one frame is captured in the background after the response is sent. The agent is reading the response at that point, so this adds no latency. Nothing is captured if another action was sent in the meantime.
• GET /agent/screenshot returns the kept frame with no WDA capture while nothing was sent since and the frame is under 30 s old. The response carries X-Screenshot-Source: settled-after-action. The wireframe and max_side still apply. ?fresh=1 always captures, and raw=1 always goes to WDA.
• No files are written. The next screen-changing POST frees the frame.

Tests

• the_settled_screen_answers_the_next_screenshot_without_a_capture: zero captures after an observed action; fresh=1 captures; an action sent in between invalidates the frame.
• MCP tests updated for the new default.
• server + mcp: 588 passed. Only my own hunks were formatted.

Hardware validation will run once the phone is free (the local daemon is on #133 right now). I'll add the numbers here: tap with observe, then phone_screenshot latency, cached vs fresh.

[coderabbitai]

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 21 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: faa245e1-7941-4912-8c28-7cbbb60941ea

📥 Commits

Reviewing files that changed from the base of the PR and between e581692 and 99edb0d.

📒 Files selected for processing (8)

• crates/mcp/src/client.rs
• crates/mcp/src/server.rs
• crates/mcp/tests/observe_over_stdio.rs
• crates/server/src/http.rs
• crates/server/src/wda.rs
• crates/server/tests/agent_input_settle.rs
• docs/agent-reference.md
• skills/iphone-use/SKILL.md

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[leeguooooo]

Hardware (iPhone 17 Pro Max, USB, Calculator: tap label 7 with return=delta, then screenshot?max_side=1200, 5 runs)

# before (v0.13.0)
daemon 0.13.0 usb drivable True
tap+observe  2741 ms (1198 B, settle=stable) | screenshot   236 ms source=None
tap+observe  2027 ms (1578 B, settle=stable) | screenshot   182 ms source=None
tap+observe  1811 ms (1206 B, settle=stable) | screenshot   235 ms source=None
tap+observe  1803 ms (1210 B, settle=stable) | screenshot   174 ms source=None
tap+observe  2739 ms (1219 B, settle=stable) | screenshot   176 ms source=None
fresh screenshot 197 ms source=None
# after (this branch)
daemon 0.13.0 usb drivable True
tap+observe  1953 ms (1223 B, settle=stable) | screenshot    37 ms source=settled-after-action
tap+observe  1807 ms (1602 B, settle=stable) | screenshot    48 ms source=settled-after-action
tap+observe  1923 ms (1233 B, settle=stable) | screenshot    49 ms source=settled-after-action
tap+observe  1827 ms (1238 B, settle=stable) | screenshot    32 ms source=settled-after-action
tap+observe  1863 ms (1244 B, settle=stable) | screenshot    41 ms source=settled-after-action
fresh screenshot 197 ms source=None

The screenshot after an observed action drops from 174–236 ms (a new capture) to 32–49 ms (the settled frame, settled-after-action). Tap+observe is unchanged within noise. Over Wi-Fi a capture costs 0.5–1.4 s, so the saving is larger there. The local daemon was hot-swapped for the run and restored to the signed v0.13.0 afterwards.