Repository navigation
fix(shower): capture virtual display screenshots without the overlay - #1402
yoruuuchan wants to merge 1 commit into
Conversation
PhoneAgent captured virtual-screen steps through VirtualDisplayOverlay's PixelCopy path. That path needs an attached overlay window, so when SYSTEM_ALERT_WINDOW was denied or the preview was hidden the capture returned null and the agent silently received a MediaProjection capture of the main display instead. Route every Shower capture through ShowerController.requestScreenshot, which mirrors the requested display into an ImageReader on the server and works for displayId 0 and virtual displays alike with no window attached. Scale the agent's relative coordinates from the Shower display size so taps stay on the target screen, and report a failed virtual-display capture on that display instead of substituting a main-display image. Fixes AAswordman#1371
|
补充一条 CI 说明,避免维护者以为是我这边漏跑检查。 这个 PR 的 head commit 上
我在本地按
等 #1351 或 #1392 合入后,我可以 rebase 到最新 |
CATMIAOZHI
left a comment
There was a problem hiding this comment.
Audit conclusion: ✅ APPROVE
Change (7 files, +205/−94): Routes virtual-display agent screenshots through the Shower server (ShowerController.requestScreenshot → IShowerService.requestScreenshot(displayId) → server-side ImageReader mirror) instead of PixelCopy-ing the main-screen overlay's SurfaceView, and removes the overlay capture chain that no longer has any callers. A virtual-screen agent can no longer silently receive a main-screen image when capture fails — it gets an explicit diagnostic instead.
Checks passed:
- Target branch is
dev. Title, author and head SHA verified against the actual PR. - The whole removed chain (
VirtualDisplayOverlay.captureCurrentFramePng, appShowerVideoRenderer.captureCurrentFramePng,showerclientShowerVideoRenderer.captureCurrentFramePng+ShowerSurfaceView.captureCurrentFramePng) has zero remaining references anywhere in the repo; dead imports and the now-unused@Volatile surfaceViewfield inVirtualDisplayOverlayare cleaned up. - The capture-target claim holds: the client captures
virtualDisplayId, which is exactly the same value the new code passes asshowerDisplayId(getDisplayId()returnsvirtualDisplayId), andprepareMainDisplaysets it to 0 for main-screen agents — so displayId=0 and virtual displays really do take the same path. - For main-screen agents the new gate (
showerDisplayId != null) is logically equivalent to the old one (canUseShowerForInput⇔mainScreenShowerPreparedsinceshowerDisplayIdis 0-or-null there) — the MediaProjection fallback path is unchanged. - On the success path,
screenWidth/screenHeightstill come from the compressed bitmap dims exactly as before, so tap-coordinate scaling does not regress. On the failure path the code now pre-sets the real video size fromgetVideoSize()instead of stale physical-screen dims — a slight improvement. CancellationExceptionis rethrown in the new catch; the finally-block overlay-restore logic is untouched.- Failure diagnostics are tested by 6 new unit tests (
AgentScreenshotReportTest); author reports compile + hygiene checks pass, and the PR body transparently lists what could not be verified on a real device. - No new permissions, no token/secret handling, no external input surface — the removed PixelCopy path actually shrinks the attack surface (one less
SYSTEM_ALERT_WINDOWdependency).
Issues:
- [P2] The central behavioral claim — the server's
DisplayCapture.captureDisplay()can actually mirror aVIRTUAL_DISPLAY_FLAG_PUBLICShower virtual display, and the captured frame's content/size lines up with tap coordinates — is explicitly unverified on a real device (the author is honest about this in the PR body). The failure mode is loud, not silent (explicit "main display was NOT captured, treat earlier screenshot as stale" diagnostic), so this degrades gracefully; still, please verify on a device with a bound virtual display before this rides a release, including theShowerCodecAdapterdownscale scenario. - [P3]
captureScreenshotViaShower's catch interpolates${e.message}directly; when the message is null the diagnostic ends with a literal "null". Consider${e.message ?: e.javaClass.simpleName}. - [P3] The gate change from
canUseShowerForInputtoshowerDisplayId != null(virtual agents now attempt server capture even without adb-or-higher privilege) is correct for this fix's stated motivation, but it isn't called out in the PR's change list — worth one line in the description for future readers.
Follow-ups (non-blocking): device verification for the four items the author already listed; consider a unit test for the main-screen-agent path (MediaProjection fallback still allowed) alongside the virtual-agent contract tests.
WaterRain's muse
变更说明 / Description
背景与动机 / Context and motivation
对虚拟屏 agent,
PhoneAgent.captureScreenshotViaShower()走的是VirtualDisplayOverlay.captureCurrentFramePng():它用PixelCopy从主屏 overlay 里的SurfaceView抓帧。这条链依赖主屏上真实挂着一个悬浮窗。当用户收起了预览窗,或
SYSTEM_ALERT_WINDOW被拒/未授予时,ensureOverlay()的addView()失败,surfaceView为 null,抓帧返回 null,流程落到MediaProjectionCaptureManager——而它只能截默认屏。结果就是子代理拿到的是主屏画面,看不到也操作不了自己的虚拟屏,但虚拟屏本身运行正常。
改动范围 / Changes
ShowerController.requestScreenshot(agentId)→IShowerService.requestScreenshot(displayId)→DisplayCapture.captureDisplay(displayId)。服务端用ImageReader直接镜像目标 display,对displayId=0和虚拟屏一视同仁,全程不需要任何窗口。ShowerController.getVideoSize(agentId)(目标 display 的真实尺寸)。相对坐标的点击换算以它为基准,避免虚拟屏上点击位置偏移。displayId=unbound)和失败原因,并明确说明没有截主屏、上一帧截图应视为过期。MediaProjection第二路径(它本来就以物理屏为目标)。虚拟屏 agent 不再接受主屏画面作为替补。VirtualDisplayOverlay.captureCurrentFramePng、showerclient的ShowerVideoRenderer.captureCurrentFramePng与ShowerSurfaceView.captureCurrentFramePng。不在范围内:新增「后台虚拟屏」偏好、去掉
VirtualDisplayOverlay本身(issue 里的可选建议 2)。overlay 仍然负责预览、悬浮球与进度 HUD。兼容性与风险 / Compatibility and risks
ActionHandler.captureScreenshotForAgent()返回AgentScreenshot(link+diagnostic)而不是String?。仓库内只有PhoneAgent._executeStep()调用它。关联 Issue / Related issue
Fixes #1371
验证方式 / Verification
新增单测
AgentScreenshotReportTest覆盖:诊断文案包含 agentId 与 displayId、明确声明未截主屏、未绑定 display 标记为displayId=unbound、成功时 prompt 携带图片链接、失败时写入诊断而不带图、无图无诊断时的兜底文案。证据 / Evidence
本地无法连接的 Android 设备,
adb devices为空,因此尚未做真机验证。以下项目需要在有设备的环境里复核,PR 里明确标注为未验证:
SYSTEM_ALERT_WINDOW时,虚拟屏 agent 截图内容来自虚拟屏而非主屏。appops set com.ai.assistance.operit SYSTEM_ALERT_WINDOW deny后,虚拟屏 agent 仍能截图并执行点击/滑动。DisplayCapture.captureDisplay()能镜像VIRTUAL_DISPLAY_FLAG_PUBLIC的 Shower 虚拟屏(issue 作者提出的前提,可能性高但未实测)。ShowerCodecAdapter降分辨率后的场景。检查清单 / Checklist
dev;如目标为main,我已说明这是维护者发布同步 / The target branch isdevfor regular work; if it ismain, I explained why this is a maintainer-led release syncCandidate checks覆盖改动范围,并会处理技术失败项 /Candidate checkscovers the change scope and technical failures will be addressed