Add live device screen streaming to the Screen Capture window by todi1856 · Pull Request #224 · Unity-Technologies/com.unity.mobile.logcat

todi1856 · 2026-09-23T12:51:29Z

https://jira.unity3d.com/browse/PLAT-22403

Type of change:

Description

The main logcat window is not affected by this change, only screen capture window is affected

How to test? Go to https://yamato.ds.unity3d.com/project/277/branch/livestream/jobDefinition/.yamato%2Fwrench%2Fpackage-pack-jobs.yml%23package_pack_-_mobile_android-logcat/recent-jobs , take latest artifacts, extract them, and add com.unity.mobile.android-logcat-1.5.0.tgz package to Unity. tgz file will have the necessary jar inside.

https://www.youtube.com/watch?v=YmHoV16Ld8Q

You can now watch the device's screen live in the Screen Capture window, and click, scroll and type into it from the Editor. The window opens from Window > Analysis > Android Screen Capture, so it no longer has to be reached through the Logcat window.

Capturing needs something on the device: External/UnityLogcatServer is a small Gradle project that builds an 18 KB jar, run through app_process as the shell user. It mirrors a display into an ImageReader, JPEG-encodes each frame and writes it over an abstract socket that adb forward exposes. Frames are only produced when the screen changes, so an idle device costs almost nothing. The jar is a build output rather than a committed binary, and a Yamato job builds it before the package is packed — that half is also up on its own as #226, cherry-picked from this branch to make it reviewable.

Input travels back up the same socket: touch, scroll wheel, keys through the device's own keyboard layout, and Ctrl/Cmd + A, C, V for select-all/copy/paste on the device. Back / Home / Overview buttons sit beside the image, which is the only way in on gesture-navigation devices, and below them Device Rotation (Auto, 0°, 90°, 180°, 270°) and, on a foldable, Device Fold (Auto, Fold, Half, Unfold) drive the device into a state without touching it. Take Screenshot is there too, so a screenshot is taken from where you are watching, at full device resolution whatever the stream is scaled to.

The window was reorganised around it. Screenshots used to overwrite one file in Temp; they are now kept, listed down the left under a Live toggle button, and each one is saved with a .json details file beside it recording the device, its Android version and the display size at the time. Where they go is a setting — Captures Folder, Library/AndroidLogcat/Screenshots by default.

The list takes a selection of more than one row (Shift and Ctrl click, Ctrl+A), and arrow keys, Home and End move through it. F2 renames, Del deletes, and right-click offers Show in Explorer / Reveal in Finder, Open, Copy To..., Rename, Delete and Select All, with the multi-row entries saying how many they will act on. Double-click opens a capture in whatever the OS associates with it. Ctrl+Shift+S (Cmd+Shift+S on macOS) takes a screenshot while the live view is showing.

A status bar along the bottom says where the last capture was written and what the stream is doing — the device, the display size, what it is scaled to and the frame rate cap. Stream size, JPEG quality and frame rate cap, and the captures folder, are in Preferences > Analysis > Android Logcat Settings, reachable from the window's own toolbar menu.

Improved screenshot window

Settings

Checklist for PR maker

Have you added a backport label? (new feature, no backport intended — confirm)
Have you updated the Changelog?
Have you added or updated the Documentation to your PR?

Testing status

Existing or new automation tests - what automation was added, changed
New tests are covering
- start/stop live steam
- frame delivery
- touch, keys
- scroll
- restart
- streaming against a device that was asleep
- screenshot list, delete, rename and external-change tests.

Devices:

Pixel 2 (Android 10)
Pixel 9 Pro Fold (Android 16)
- Folding/Unfolding while viewing
CPH1831 (Android 8.1).

Testing checklist

Built and run editor Locally or Yamato
Run on device Android/iOS (if applicable)
All items have tooltips?

A small Java server, built with Gradle into a dex jar and pushed to the device, that mirrors a display into an ImageReader, encodes each frame as JPEG and writes it to an abstract unix socket. It also reads control messages back up the same socket and injects touch, scroll, key and text events. The jar is a build output and is not committed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The window now lists saved screenshots beside a Live row that mirrors the device. Selecting Live starts the server on the device, shows its frames, and forwards touch, scroll, keyboard and text back to it. Along with it: a shared zoom and pan viewer for the live view and screenshot previews, a details file written beside each capture, a stats column, device rotation, and settings for the stream's size, quality and frame rate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Integration tests covering the stream end to end: frames arriving, injected touch, scroll, keys and text taking effect on the device, streaming a device that was asleep, and an unknown display id. Edit mode coverage for the details file written beside a screenshot. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The jar the live stream pushes to the device is a build output and is not committed, so a clone has none. A build_server_jar job produces it and the pack job now depends on it, which is also what puts it in the tarball the test jobs install. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

todi1856 · 2026-09-23T12:51:42Z

/review

u-pr · 2026-09-23T12:51:49Z

ℹ️ Skipped — a harness review for this commit (5aced5d) already ran or is in flight. Same-commit re-runs are paused for 10 min; push a new commit, or re-comment /harness-review in ~10 min to force a fresh one.

_{🤖 Helpful? 👍/👎}

u-pr

💡 Harness Review

The streaming pipeline is thoughtfully integrated, but several reachable UI, persistence, and startup edge cases can leave the displayed or saved content inconsistent or prevent streaming.

_{Reviewed commit 5aced5d}

_{🤖 Helpful? 👍/👎}

todi1856 · 2026-09-23T13:21:02Z

/review

u-pr

💡 Harness Review

The new capture workflow has state-management issues when device selection changes or a live-stream reader does not terminate promptly.

_{Reviewed commit 6508758}

_{🤖 Helpful? 👍/👎}

todi1856 · 2026-09-23T13:36:39Z

/review

u-pr

💡 Harness Review

The new live-view and persistent screenshot workflow has correctness gaps on reachable edge paths.

_{Reviewed commit f1d921f}

_{🤖 Helpful? 👍/👎}

todi1856 · 2026-09-23T13:48:18Z

/review

u-pr

💡 Harness Review

The live-stream feature is broadly cohesive, but keyboard shortcut forwarding can emit an unmatched key-down event for a normal release order.

_{Reviewed commit c58dc0f}

_{🤖 Helpful? 👍/👎}

todi1856 · 2026-09-23T14:29:11Z

/review

u-pr

💡 Harness Review

The streaming and screenshot-management implementation is broadly cohesive, but saving metadata-less screenshots can preserve incorrect provenance.

_{Reviewed commit b5bf917}

_{🤖 Helpful? 👍/👎}

todi1856 · 2026-09-24T08:29:07Z

/design-consistency

IGuscin

Thanks, that's an awesome improvement. Very useful for testing purposes.

Tested the functionality on Samsung Galaxy S23+ Android 13 and Pixel 9 Pro XL Android 17.
Live capture works fine, doesn't lag even on highest available settings, mouse, keyboard input works as expected as well.
Tomas quickly addressed couple small issues I've noticed. The only remaining issue I've seen related to rotation of Unity apps not working reproduces in Android Studio as well, so probably is a generic Unity issue, rather than Logcat.

anjalin14

Thanks for the docs! I've left quite a few suggestions but those are mostly minor nits around conversational tone, formatting, and passive voice fixes. I've also added a couple of comments for clarification. I'm approving now but happy to re-review if you'd like!

anjalin14 · 2026-10-21T10:22:18Z

+Select **Browse...** to pick a folder. A folder inside the project is stored relative to it, and one outside it is stored as an absolute path.
+
+The default, `AndroidLogcat/Captures`, is a folder in your project, beside `Assets`. It is outside `Assets`, so captures are never imported and are not part of a build, and it survives deleting `Library`, so captures stay until you delete them. It is not covered by Unity's own `.gitignore`, so add it to yours if you do not want captures in source control.


Suggested change

Select **Browse...** to pick a folder. A folder inside the project is stored relative to it, and one outside it is stored as an absolute path.

The default, `AndroidLogcat/Captures`, is a folder in your project, beside `Assets`. It is outside `Assets`, so captures are never imported and are not part of a build, and it survives deleting `Library`, so captures stay until you delete them. It is not covered by Unity's own `.gitignore`, so add it to yours if you do not want captures in source control.

To choose a folder, select **Browse...**. If the folder is inside the project, Unity stores it as a relative path. If the folder is outside the project, Unity stores it as an absolute path.

The default, `AndroidLogcat/Captures`, is a folder in your project, beside `Assets`. It is outside `Assets`, so Unity doesn't import captures or include them in a build. Deleting the `Library` folder doesn't remove the captures folder. Captures in this folder remain until you delete them. Unity's own `.gitignore` file doesn't cover the captures folder. If you don't want captures in source control, add the captures folder to your own `.gitignore` file.

Applied in 59f276f, thanks!

anjalin14 · 2026-10-21T10:25:54Z

+
+The default, `AndroidLogcat/Captures`, is a folder in your project, beside `Assets`. It is outside `Assets`, so captures are never imported and are not part of a build, and it survives deleting `Library`, so captures stay until you delete them. It is not covered by Unity's own `.gitignore`, so add it to yours if you do not want captures in source control.
+
+Changing the folder takes effect immediately. Captures already written stay where they are, and the [Capture list](screen-capture-window-reference.md#capture-list) shows what is in the new folder.


can you clarify this a bit. Does this mean, if the captures are moved from A to B, captures still remain in folder A and the captures list display only items from folder B?

Yes, exactly. Existing captures aren't moved: they stay in folder A, and the list shows only what's in B. I reworded the paragraph to say that explicitly, and that you can move old captures (with their .json files) yourself. Fixed in 59f276f.

anjalin14 · 2026-10-21T10:37:06Z

+
+The live view mirrors the device's display into the Unity Editor. Unlike a screenshot or a video, there is nothing to save: the view is live and stops when you select something else.
+
+## View the screen


Suggested change

## View the screen

## View the screen

To view the device screen live:

adding a lead line for steps

Applied in 59f276f, thanks!

anjalin14 · 2026-10-21T14:04:39Z

+| **Vol +**   | Sends the Volume Up key.                                     |
+| **Vol -**   | Sends the Volume Down key.                                   |
+
+Below them, **Device Rotation** turns the device's screen:


Suggested change

Below them, **Device Rotation** turns the device's screen:

**Device Rotation** turns the device's screen:

Applied in 59f276f, thanks!

anjalin14 · 2026-10-21T14:07:57Z

+
+On a foldable, **Device Fold** follows: **Fold**, **Unfold**, and **Half** on a device that reports half open as a state of its own, hold the device that way whatever its hinge is doing - which is how the other display is reached without touching the device. **Auto** hands it back to the hinge. The row does not appear for a device that does not fold.
+
+**Take Screenshot**, at the bottom, captures the device screen and adds it to the [Capture list](#capture-list). The screenshot is taken on the device rather than copied from the stream, so it is full resolution whatever the stream is scaled down to. Ctrl+Shift+S (Cmd+Shift+S on macOS) does the same while this window has focus and the **Live** row is selected - the shortcut follows the button, so it does nothing while a saved capture is on screen.


Suggested change

**Take Screenshot**, at the bottom, captures the device screen and adds it to the [Capture list](#capture-list). The screenshot is taken on the device rather than copied from the stream, so it is full resolution whatever the stream is scaled down to. Ctrl+Shift+S (Cmd+Shift+S on macOS) does the same while this window has focus and the **Live** row is selected - the shortcut follows the button, so it does nothing while a saved capture is on screen.

Under *Device Capture**, **Take Screenshot** captures the device screen and adds it to the [Capture list](#capture-list). The screenshot is taken on the device rather than copied from the stream, so it is full resolution regardless of how the stream is scaled. **Ctrl+Shift+S** (macOS: **Cmd+Shift+S**) does the same while this window has focus and the **Live** row is selected. The shortcut has no effect while a saved capture is on screen.

Applied in 59f276f (fixed the missing * in Device Capture, and "while a saved capture is selected").

anjalin14 · 2026-10-21T14:16:36Z

@@ -5,10 +5,17 @@ This page explains how to use the [Screen Capture tool](screen-capture.md) to ca
 1. Open the [Device Screen Capture window](screen-capture-window-reference.md).
 2. In the [Toolbar](screen-capture-window-reference.md#toolbar), use **Device Selector** to specify to device to take a screenshot of.


Suggested change

2. In the [Toolbar](screen-capture-window-reference.md#toolbar), use **Device Selector** to specify to device to take a screenshot of.

2. In the [Toolbar](screen-capture-window-reference.md#toolbar), use **Device Selector** to specify the device to take a screenshot of.

Applied in 59f276f, thanks!

anjalin14 · 2026-10-21T14:17:17Z

    * [Capture a screenshot](screen-capture-screenshot.md)
    * [Capture a video](screen-capture-video.md)
+    * [View the device screen live](screen-capture-live-stream.md)
  * [Stacktace Utility](stacktrace-utility.md)


Suggested change

* [Stacktace Utility](stacktrace-utility.md)

* [Stacktrace Utility](stacktrace-utility.md)

Applied in 59f276f, thanks!

anjalin14

Thanks for the docs! I've left quite a few suggestions but those are mostly minor nits around conversational tone, formatting, and passive voice fixes. I've also added a couple of comments for clarification. I'm approving now but happy to re-review if you'd like!

anjalin14 · 2026-10-21T16:01:41Z

pls ignore the duplicate comment due to a Github glitch

…settings.md Co-authored-by: anjalin14 <anjali.nabar@unity3d.com>

…window-reference.md Co-authored-by: anjalin14 <anjali.nabar@unity3d.com>

Address review suggestions on the Screen Capture and Android Logcat Settings pages: tone, passive voice, key formatting, no directional wording, Device Fold table, and correct the stale Library folder note. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

todi1856 · 2026-10-22T08:16:20Z

Will merge, trunk related issues will be addressed separately in #228

todi1856 and others added 5 commits September 23, 2026 15:40

Document the live stream and update the changelog

5aced5d

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

u-pr Bot reviewed Sep 23, 2026

View reviewed changes

todi1856 added 5 commits September 23, 2026 16:06

fix preview going out of bounds

5db42b1

handle virtualDisplay being null

f87a4dd

Fix save as for screenshot

7123543

shorten changelog

076c7de

Try to fix panning on OSX

6508758

u-pr Bot reviewed Sep 23, 2026

View reviewed changes

Comment thread com.unity.mobile.android-logcat/Editor/AndroidLogcatScreenCaptureWindow.cs Resolved

Comment thread com.unity.mobile.android-logcat/Editor/AndroidLogcatLiveStream.cs Resolved

todi1856 added 3 commits September 23, 2026 16:35

update screenshot

b2e6c2f

apply review fix

58fd75e

apply review fix

f1d921f

u-pr Bot reviewed Sep 23, 2026

View reviewed changes

Comment thread com.unity.mobile.android-logcat/Editor/AndroidLogcatCaptureScreenshot.cs Resolved

Comment thread com.unity.mobile.android-logcat/Editor/AndroidLogcatLiveStream.cs Resolved

Apply review fixes

c58dc0f

u-pr Bot reviewed Sep 23, 2026

View reviewed changes

Comment thread com.unity.mobile.android-logcat/Editor/AndroidLogcatLiveStream.cs Resolved

Track keys which were held

b5bf917

u-pr Bot reviewed Sep 23, 2026

View reviewed changes

Comment thread com.unity.mobile.android-logcat/Editor/AndroidLogcatScreenshotInfo.cs Resolved

todi1856 added 3 commits September 23, 2026 17:36

show live view when opening the window

68f3993

Decouple layout viewer from screen capture

638bbef

LayoutView->LayoutViewer

a280502

todi1856 added 2 commits October 7, 2026 10:46

Added side buttons too

3f985a4

fix stats ui being clipped when phone is in landscape

10f8767

BorisShir approved these changes Oct 7, 2026

View reviewed changes

todi1856 added 6 commits October 16, 2026 11:47

prefer using newer api for setting rotation on Android

1883433

Turn display, to visually display rotation

b8c2755

preserve rotation when switching between screenshot and live

608fe87

select entry when right clicking

ee0c3d0

implement deselection

821e3ae

move stream shutdown to thread

82e453e

todi1856 requested a review from anjalin14 October 20, 2026 07:13

todi1856 added 9 commits October 20, 2026 10:21

forgot some control docs

a34d758

better image

f91bbbc

don't squeze buttons

e972c1e

Turn the screenshot to match live view

c587c30

don't stretch the live image when rotating

c924dd9

don't resize screenshot view when rotated

8d86a71

Move default location to AndroidLogcat/Captures

b50a17f

update ProjectRelativePathTests

fe1ae65

revert image resize when rotating

be0d85d

IGuscin approved these changes Oct 21, 2026

View reviewed changes

anjalin14 approved these changes Oct 21, 2026

View reviewed changes

todi1856 and others added 4 commits October 22, 2026 10:30

Update com.unity.mobile.android-logcat/Documentation~/android-logcat-…

bf38f49

…settings.md Co-authored-by: anjalin14 <anjali.nabar@unity3d.com>

Update com.unity.mobile.android-logcat/Documentation~/android-logcat-…

0c0f938

…settings.md Co-authored-by: anjalin14 <anjali.nabar@unity3d.com>

Update com.unity.mobile.android-logcat/Documentation~/screen-capture-…

b89f917

…window-reference.md Co-authored-by: anjalin14 <anjali.nabar@unity3d.com>

todi1856 merged commit 69abcc6 into master Oct 22, 2026
19 of 23 checks passed

todi1856 deleted the livestream branch October 22, 2026 08:16

		Select Browse... to pick a folder. A folder inside the project is stored relative to it, and one outside it is stored as an absolute path.

		The default, `AndroidLogcat/Captures`, is a folder in your project, beside `Assets`. It is outside `Assets`, so captures are never imported and are not part of a build, and it survives deleting `Library`, so captures stay until you delete them. It is not covered by Unity's own `.gitignore`, so add it to yours if you do not want captures in source control.


		The default, `AndroidLogcat/Captures`, is a folder in your project, beside `Assets`. It is outside `Assets`, so captures are never imported and are not part of a build, and it survives deleting `Library`, so captures stay until you delete them. It is not covered by Unity's own `.gitignore`, so add it to yours if you do not want captures in source control.

		Changing the folder takes effect immediately. Captures already written stay where they are, and the [Capture list](screen-capture-window-reference.md#capture-list) shows what is in the new folder.


		The live view mirrors the device's display into the Unity Editor. Unlike a screenshot or a video, there is nothing to save: the view is live and stops when you select something else.

		## View the screen

	## View the screen
	## View the screen
	To view the device screen live:

	Below them, Device Rotation turns the device's screen:
	Device Rotation turns the device's screen:


		On a foldable, Device Fold follows: Fold, Unfold, and Half on a device that reports half open as a state of its own, hold the device that way whatever its hinge is doing - which is how the other display is reached without touching the device. Auto hands it back to the hinge. The row does not appear for a device that does not fold.

		Take Screenshot, at the bottom, captures the device screen and adds it to the [Capture list](#capture-list). The screenshot is taken on the device rather than copied from the stream, so it is full resolution whatever the stream is scaled down to. Ctrl+Shift+S (Cmd+Shift+S on macOS) does the same while this window has focus and the Live row is selected - the shortcut follows the button, so it does nothing while a saved capture is on screen.

		@@ -5,10 +5,17 @@ This page explains how to use the [Screen Capture tool](screen-capture.md) to ca
		1. Open the [Device Screen Capture window](screen-capture-window-reference.md).
		2. In the [Toolbar](screen-capture-window-reference.md#toolbar), use Device Selector to specify to device to take a screenshot of.

	* [Stacktace Utility](stacktrace-utility.md)
	* [Stacktrace Utility](stacktrace-utility.md)

Conversation

todi1856 commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Type of change:

Description

Improved screenshot window

Settings

Checklist for PR maker

Testing status

Devices:

Testing checklist

Uh oh!

todi1856 commented Sep 23, 2026

Uh oh!

u-pr Bot commented Sep 23, 2026

Uh oh!

u-pr Bot left a comment

Choose a reason for hiding this comment

💡 Harness Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

todi1856 commented Sep 23, 2026

Uh oh!

u-pr Bot left a comment

Choose a reason for hiding this comment

💡 Harness Review

Uh oh!

Uh oh!

Uh oh!

todi1856 commented Sep 23, 2026

Uh oh!

u-pr Bot left a comment

Choose a reason for hiding this comment

💡 Harness Review

Uh oh!

Uh oh!

Uh oh!

todi1856 commented Sep 23, 2026

Uh oh!

u-pr Bot left a comment

Choose a reason for hiding this comment

💡 Harness Review

Uh oh!

Uh oh!

todi1856 commented Sep 23, 2026

Uh oh!

u-pr Bot left a comment

Choose a reason for hiding this comment

💡 Harness Review

Uh oh!

Uh oh!

todi1856 commented Sep 24, 2026

Uh oh!

IGuscin left a comment

Choose a reason for hiding this comment

Uh oh!

anjalin14 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

todi1856 commented Sep 23, 2026 •

edited

Loading