Skip to content

fix(shower): capture virtual display screenshots without the overlay - #1402

Open
yoruuuchan wants to merge 1 commit into
AAswordman:devfrom
yoruuuchan:fix/issue-1371-shower-virtual-display-screenshot
Open

yoruuuchan wants to merge 1 commit into
AAswordman:devfrom
yoruuuchan:fix/issue-1371-shower-virtual-display-screenshot

Conversation

@yoruuuchan

Copy link
Copy Markdown
Contributor

变更说明 / Description

背景与动机 / Context and motivation

对虚拟屏 agent,PhoneAgent.captureScreenshotViaShower() 走的是 VirtualDisplayOverlay.captureCurrentFramePng():它用 PixelCopy 从主屏 overlay 里的 SurfaceView 抓帧。

这条链依赖主屏上真实挂着一个悬浮窗。当用户收起了预览窗,或 SYSTEM_ALERT_WINDOW 被拒/未授予时,ensureOverlay() 的 addView() 失败,surfaceView 为 null,抓帧返回 null,流程落到 MediaProjectionCaptureManager——而它只能截默认屏。

结果就是子代理拿到的是主屏画面,看不到也操作不了自己的虚拟屏,但虚拟屏本身运行正常。

改动范围 / Changes

  • 虚拟屏 agent 的截图统一走 ShowerController.requestScreenshot(agentId) → IShowerService.requestScreenshot(displayId) → DisplayCapture.captureDisplay(displayId)。服务端用 ImageReader 直接镜像目标 display,对 displayId=0 和虚拟屏一视同仁,全程不需要任何窗口。
  • 截图尺寸改为取 ShowerController.getVideoSize(agentId)(目标 display 的真实尺寸)。相对坐标的点击换算以它为基准,避免虚拟屏上点击位置偏移。
  • 虚拟屏截图失败时返回明确诊断,写进 prompt 的屏幕段落:包含 agentId、目标 displayId(未绑定则标 displayId=unbound)和失败原因,并明确说明没有截主屏、上一帧截图应视为过期。
  • 主屏 agent 保留原有 MediaProjection 第二路径(它本来就以物理屏为目标)。虚拟屏 agent 不再接受主屏画面作为替补。
  • 清理已无调用方的 overlay 抓帧链:VirtualDisplayOverlay.captureCurrentFramePng、showerclient 的 ShowerVideoRenderer.captureCurrentFramePng 与 ShowerSurfaceView.captureCurrentFramePng。

不在范围内:新增「后台虚拟屏」偏好、去掉 VirtualDisplayOverlay 本身(issue 里的可选建议 2)。overlay 仍然负责预览、悬浮球与进度 HUD。

兼容性与风险 / Compatibility and risks

  • 主屏 agent 行为不变,虚拟屏的输入注入与预览渲染不变。
  • 内部 API 变化:ActionHandler.captureScreenshotForAgent() 返回 AgentScreenshot(link + diagnostic)而不是 String?。仓库内只有 PhoneAgent._executeStep() 调用它。
  • 虚拟屏截图失败时,模型拿到的是文字诊断而不是主屏图片。这是刻意的行为变化:把主屏图当作虚拟屏图会让模型按错误的屏幕内容与尺寸推理并点击。
  • 若 helper 无法镜像虚拟屏,虚拟屏 agent 会明确报出截图失败,而不是退化成一张主屏图。

关联 Issue / Related issue

Fixes #1371

验证方式 / Verification

检查或命令:
  ./gradlew :app:compileDebugKotlin --no-build-cache
  ./gradlew :app:testDebugUnitTest --tests "...AgentScreenshotReportTest"
  ./gradlew :showerclient:compileDebugKotlin
  python -B ci/script/check_repo_hygiene.py --base f1bbcd80 --candidate 28a0dd3b
  python -B ci/script/check_markdown_links.py --base f1bbcd80 --candidate 28a0dd3b
  python -B ci/script/check_localizations.py --base f1bbcd80 --candidate 28a0dd3b
环境与变体:
  Windows, JDK 21.0.12 (Temurin), AGP 8.13.2, Kotlin 2.2.21, debug 变体
结果:
  :app:compileDebugKotlin BUILD SUCCESSFUL
  :app:testDebugUnitTest BUILD SUCCESSFUL, 6/6 tests passed
  :showerclient:compileDebugKotlin BUILD SUCCESSFUL
  hygiene errors=0 warnings=0 (7 files)
  markdown errors=0 warnings=0
  localizations errors=0 warnings=0

新增单测 AgentScreenshotReportTest 覆盖:诊断文案包含 agentId 与 displayId、明确声明未截主屏、未绑定 display 标记为 displayId=unbound、成功时 prompt 携带图片链接、失败时写入诊断而不带图、无图无诊断时的兜底文案。

证据 / Evidence

本地无法连接的 Android 设备,adb devices 为空,因此尚未做真机验证。

以下项目需要在有设备的环境里复核,PR 里明确标注为未验证:

  • 授予 SYSTEM_ALERT_WINDOW 时,虚拟屏 agent 截图内容来自虚拟屏而非主屏。
  • appops set com.ai.assistance.operit SYSTEM_ALERT_WINDOW deny 后,虚拟屏 agent 仍能截图并执行点击/滑动。
  • DisplayCapture.captureDisplay() 能镜像 VIRTUAL_DISPLAY_FLAG_PUBLIC 的 Shower 虚拟屏(issue 作者提出的前提,可能性高但未实测)。
  • 截图尺寸与点击坐标一致,包括 ShowerCodecAdapter 降分辨率后的场景。

检查清单 / Checklist

  • 我已记录可复现验证和未运行项原因 / Reproducible verification and reasons for unrun checks are recorded
  • 日常开发 PR 的目标分支为 dev;如目标为 main,我已说明这是维护者发布同步 / The target branch is dev for regular work; if it is main, I explained why this is a maintainer-led release sync
  • 我已确认 Candidate checks 覆盖改动范围,并会处理技术失败项 / Candidate checks covers the change scope and technical failures will be addressed
  • 最终 diff 无无关、临时、生成、二进制或敏感内容 / Final diff has no unrelated, temporary, generated, binary, or secret content
  • 已提供对应的回归、UI、文档/字符串或兼容性证据 / Relevant regression, UI, docs/strings, or compatibility evidence is provided

PhoneAgent captured virtual-screen steps through VirtualDisplayOverlay's PixelCopy path. That path needs an attached overlay window, so when SYSTEM_ALERT_WINDOW was denied or the preview was hidden the capture returned null and the agent silently received a MediaProjection capture of the main display instead.

Route every Shower capture through ShowerController.requestScreenshot, which mirrors the requested display into an ImageReader on the server and works for displayId 0 and virtual displays alike with no window attached.

Scale the agent's relative coordinates from the Shower display size so taps stay on the target screen, and report a failed virtual-display capture on that display instead of substituting a main-display image.

Fixes AAswordman#1371
@yoruuuchan

Copy link
Copy Markdown
Contributor Author

补充一条 CI 说明,避免维护者以为是我这边漏跑检查。

这个 PR 的 head commit 上 check-runs 数量是 0。原因是 pr-check.yml 的 pull_request.branches 仍写着已改名的 development,所以目标分支为 dev 的 PR 不会触发 PR Check。这是既有问题,不限于本 PR:

我在本地按 docs/doc-src/dev-core/CONTRIBUTING.md 的快速检查跑完了相关项,结果都在上面的「验证方式」里:

  • check_repo_hygiene.py errors=0 warnings=0(检查了 7 个文件)
  • check_markdown_links.py errors=0 warnings=0
  • check_localizations.py errors=0 warnings=0
  • :app:compileDebugKotlin、showerclient:compileDebugKotlin、app:testDebugUnitTest(新增 6 个用例)均通过

等 #1351 或 #1392 合入后,我可以 rebase 到最新 dev 重新触发 CI。真机验证仍然缺设备,标注保持不变。

@CATMIAOZHI
CATMIAOZHI self-requested a review October 22, 2026 13:38

@CATMIAOZHI CATMIAOZHI left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Audit conclusion: ✅ APPROVE

Change (7 files, +205/−94): Routes virtual-display agent screenshots through the Shower server (ShowerController.requestScreenshot → IShowerService.requestScreenshot(displayId) → server-side ImageReader mirror) instead of PixelCopy-ing the main-screen overlay's SurfaceView, and removes the overlay capture chain that no longer has any callers. A virtual-screen agent can no longer silently receive a main-screen image when capture fails — it gets an explicit diagnostic instead.

Checks passed:

  • Target branch is dev. Title, author and head SHA verified against the actual PR.
  • The whole removed chain (VirtualDisplayOverlay.captureCurrentFramePng, app ShowerVideoRenderer.captureCurrentFramePng, showerclient ShowerVideoRenderer.captureCurrentFramePng + ShowerSurfaceView.captureCurrentFramePng) has zero remaining references anywhere in the repo; dead imports and the now-unused @Volatile surfaceView field in VirtualDisplayOverlay are cleaned up.
  • The capture-target claim holds: the client captures virtualDisplayId, which is exactly the same value the new code passes as showerDisplayId (getDisplayId() returns virtualDisplayId), and prepareMainDisplay sets it to 0 for main-screen agents — so displayId=0 and virtual displays really do take the same path.
  • For main-screen agents the new gate (showerDisplayId != null) is logically equivalent to the old one (canUseShowerForInput ⇔ mainScreenShowerPrepared since showerDisplayId is 0-or-null there) — the MediaProjection fallback path is unchanged.
  • On the success path, screenWidth/screenHeight still come from the compressed bitmap dims exactly as before, so tap-coordinate scaling does not regress. On the failure path the code now pre-sets the real video size from getVideoSize() instead of stale physical-screen dims — a slight improvement.
  • CancellationException is rethrown in the new catch; the finally-block overlay-restore logic is untouched.
  • Failure diagnostics are tested by 6 new unit tests (AgentScreenshotReportTest); author reports compile + hygiene checks pass, and the PR body transparently lists what could not be verified on a real device.
  • No new permissions, no token/secret handling, no external input surface — the removed PixelCopy path actually shrinks the attack surface (one less SYSTEM_ALERT_WINDOW dependency).

Issues:

  1. [P2] The central behavioral claim — the server's DisplayCapture.captureDisplay() can actually mirror a VIRTUAL_DISPLAY_FLAG_PUBLIC Shower virtual display, and the captured frame's content/size lines up with tap coordinates — is explicitly unverified on a real device (the author is honest about this in the PR body). The failure mode is loud, not silent (explicit "main display was NOT captured, treat earlier screenshot as stale" diagnostic), so this degrades gracefully; still, please verify on a device with a bound virtual display before this rides a release, including the ShowerCodecAdapter downscale scenario.
  2. [P3] captureScreenshotViaShower's catch interpolates ${e.message} directly; when the message is null the diagnostic ends with a literal "null". Consider ${e.message ?: e.javaClass.simpleName}.
  3. [P3] The gate change from canUseShowerForInput to showerDisplayId != null (virtual agents now attempt server capture even without adb-or-higher privilege) is correct for this fix's stated motivation, but it isn't called out in the PR's change list — worth one line in the description for future readers.

Follow-ups (non-blocking): device verification for the four items the author already listed; consider a unit test for the main-screen-agent path (MediaProjection fallback still allowed) alongside the virtual-agent contract tests.

WaterRain's muse

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants