Skip to content

Add WinUI DevTools: live inspection, binding diagnosis and source-anchored UI comments - #943

Draft
Nikola Metulev (nmetulev) wants to merge 44 commits into
mainfrom
nmetulev-devtools
Draft

Nikola Metulev (nmetulev) wants to merge 44 commits into
mainfrom
nmetulev-devtools

Conversation

@nmetulev

@nmetulev Nikola Metulev (nmetulev) commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

Includes #942's commits. This PR targets main but is built on #942 (nmetulev-agent-friendly-ui-help), whose compact help renderer winapp devtools reuses, so #942's changes appear in this diff until it merges. After #942 merges, this branch will be rebased onto main with only its own commits.

Adds WinUI DevTools to winapp. winapp run --devtools launches a WinUI 3 app with an in-app overlay so a developer can pick an element, tweak it, and leave comments anchored to the XAML that produced it. The new winapp devtools command group exposes the same live tree to the terminal and to agents: inspect, search, read/set properties, diagnose bindings, and read or resolve the comments.

Usage

# Developer: run with the overlay, pick elements, leave comments
winapp run . --devtools

# Agent: read the comments and act on them
winapp devtools comments list

# Agent: inspect the live app
winapp devtools inspect --depth 3
winapp devtools search Button
winapp devtools get-property <selector> Text
winapp devtools diagnose-binding <selector> IsEnabled

What's in it

  • In-app overlay: pick, quick edit, comments, inspector window
  • winapp devtools command group (inspect, search, get-property, set-property, diagnose-binding, comments, ...)
  • Source-anchored comments stored in .winapp/ui-comments.json
  • Windows Sandbox support (--on sandbox)
  • winapp devtools --help with the workflow, examples for every command, and a root-help Start here pointer; winapp ui prints a tip when its target has DevTools
  • Docs: docs/guides/devtools.md and devtools-advanced.md, plus updated usage docs and agent skills

Triage fixes in this PR

Fixed (details and before/after in the PR comments):

  • Picking text with no source selects the authored control; unanchored comments say "Not linked to source"
  • diagnose-binding handles non-public, page-owned and window-owned x:Bind
  • Unique declarations and x:Names are confirmed comment anchors; cleaner anchor output
  • set-property warns when it replaces a binding
  • A rerun while the app is running names its PID; stale-build source warnings collapse to one line
  • Smaller output and consistency fixes

Not a bug: the "quick-edit panel off-screen" report mixed physical and logical coordinates; the panel stays inside the monitor work area (see comments).

Validation

CI is green. Each fix has a regression test, and fixes were checked live against AI Dev Gallery and samples/winui-app.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ilters

- -a falls back to title matching (preferring ApplicationFrameHost frames) when the
  matched process owns no top-level window, and fails instead of reporting an empty target
- inspect --interactive includes Document and writable-value elements
- --type/--root/--class-name on every single-selector ui command; invoke accepts them
  without --action

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…d examples

- `winapp ui --help` renders plain text: golden path, -a/<selector> definitions, and
  commands grouped by task (2,741 bytes redirected, down from 6,538)
- every ui command gets plain-text help with 1-3 placeholder examples and a one-line
  global options pointer; examples are data (IHelpExamples) and are parse-tested
- `winapp ui <unknown>` fails with exit 1 (even with --help) before sandbox routing,
  suggesting up to two commands; --json emits the error envelope with `suggestions`
- `find` and `tree` are aliases for `search` and `inspect`
- root help points agents at `winapp ui --help`
- a missing option value ends with a pointer to --help instead of the full help
- inspect's footer example uses set-value for editable-only elements

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ghten unknown-command detection

- `-a` falls back only to ApplicationFrameHost frames; a process with no window yet keeps the
  process-scoped target so `wait-for` and `status` behave as before
- bare `winapp ui` still prints the group help after "Required command was not provided"
- only the first unmatched token counts as a command, so `ui -a Notepad` and `ui -- inspect`
  report the normal parse error instead of an unknown command

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Comment on lines +392 to +393
{
}
decimal.TryParse(requested, NumberStyles.Integer, CultureInfo.InvariantCulture, out var b) && a == b,
"Double" => double.TryParse(actual, NumberStyles.Float, CultureInfo.InvariantCulture, out var a) &&
double.TryParse(requested, NumberStyles.Float, CultureInfo.InvariantCulture, out var b) &&
double.IsFinite(a) && double.IsFinite(b) && a == b,
@github-actions

github-actions Bot commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

⏳ Build in progress — metrics below are from a previous commit and will update when the current build finishes.

Build Metrics Report

Validation did not pass. Artifacts were uploaded before validation finished; check the workflow run before using them.

Binary Sizes

Artifact Baseline Current Delta
CLI (ARM64) 57.29 MB 66.48 MB 📈 +9.18 MB (+16.03%)
CLI (x64) 57.34 MB 65.15 MB 📈 +7.81 MB (+13.63%)
MSIX (ARM64) 23.79 MB 26.37 MB 📈 +2.58 MB (+10.83%)
MSIX (x64) 25.26 MB 27.95 MB 📈 +2.69 MB (+10.64%)
NPM Package 49.63 MB 54.91 MB 📈 +5.28 MB (+10.63%)
NuGet Package 49.74 MB 55.04 MB 📈 +5.30 MB (+10.66%)

.NET Test Results (TRX reports)

Other suites are reflected in the overall validation status above.

✅ 9361 passed, 41 skipped out of 9402 tests in 1431.8s (+1486 tests, +81.5s vs. baseline)

Test Coverage

✅ 86% line coverage, 80.1% branch coverage · ⚠️ -0.3% vs. baseline

CLI Startup Time

67ms median (x64, winapp --version) · ✅ +5ms vs. baseline

Try This Build

Installs the MSIX for your architecture, replacing any previously installed build. Needs the GitHub CLI — the command offers to install it and sign you in if it is missing.

& ([scriptblock]::Create((irm https://raw.githubusercontent.com/microsoft/winappCli/main/scripts/winapp-pr.ps1))) 943
Switching between builds often?

Put the tool on your PATH once:

& ([scriptblock]::Create((irm https://raw.githubusercontent.com/microsoft/winappCli/main/scripts/winapp-pr.ps1))) -AddToPath

Then this build is just:

winapp-pr 943

Run winapp-pr with no arguments to pick from a list of open PRs.


Updated 2026-09-30 15:27:12 UTC · commit dad673e · workflow run

@nmetulev

Copy link
Copy Markdown
Member Author

Triage fixes (groups 1 and 2)

Seven commits on top of the original squash. Each fix has a regression test that fails without it. Live checks ran against AI Dev Gallery and samples/winui-app (from a temp copy). "Before" is pre-fix code (the installed build, or this branch before that fix), except where marked "reported".

Fixed

Picking text with no source picks the authored control. Hover and click share the path; Just my XAML off keeps the raw hit.

before  Internal.pick on "Click Me"      -> Microsoft.UI.Xaml.Controls.TextBlock
after   Internal.pick on "Click Me"      -> Microsoft.UI.Xaml.Controls.Button  CounterButton
after   Internal.pick on "Enable feature" -> Microsoft.UI.Xaml.Controls.CheckBox FeatureCheckBox

A comment on an element that still has no source says Saved. Not linked to source. and is weak / requiresConfirmation in JSON.

diagnose-binding on x:Bind. Private members, page-owned and window-owned (static App.MainWindow) x:Bind now evaluate. Reported before: bad-segment / unavailable.

[ModelsExpander] SettingsExpander.ItemsSource   state: evaluated  path: cachedModels             source: SettingsPage
[MRUView] ItemsView.ItemsSource                 state: evaluated  path: mostRecentlyUsedItems    source: HomePage
[titleBar] TitleBar.IsBackButtonVisible         state: evaluated  path: NavFrame.CanGoBack       source: MainWindow

Comment anchors. A unique declaration or x:Name is confirmed, whatever the runtime instance count; via type candidates are dropped when a declaration or name matches; the historical line appears only when there is no current match, and never as file:.

after  SubmitButton: anchorConfirmed=true requiresConfirmation=false (via declaration)

set-property on a bound property warns, and JSON has replacedBinding. The chain's Default "0" for String is now empty.

✅ [textblock-1efceef664] TextBlock.Text: C:\Users\…\cpu-int4-rtn-block-32-acc-level-4 -> C:\edited
⚠ This overrode {x:Bind Path} with a local value until the binding updates again.

Rerun while the app is running

before  … Access was denied. Check permissions on the app's input and staging files. If a previous app instance is still running, close it explicitly before retrying; access denied alone does not identify a lock owner.
after   … The app is still running from this build (PID 16732), so its files cannot be replaced. Close it, then run again.

Identical sourceWarnings collapse into one entry with count, and the entry says to rebuild. I could not recreate the 94-file case without modifying Gallery; unit-tested.

Smaller fixes

search "Phi 3 Medium"   before  No element matches … not an element's rendered text.
                        after   ⚠ Nothing in your XAML matches "Phi 3 Medium"; these are generated or framework elements, such as items created from data.  (+ NavigationViewItem)
list --status resolved  before  2 comments total (1 resolved — hidden, use --all).
                        after   2 comments total (1 open hidden; use --all).
inspect --filter … --depth 1
                        before  No elements matched. Drop --filter to see the whole tree.
                        after   No elements matched within --depth 1; deeper elements were not searched. Rerun with --depth 2, or use `winapp devtools search "submitbutton"`.
comments get --json     before  "declaration": "\u003CButton … x:Name=\u0022SubmitButton\u0022
                        after   "declaration": "<Button … x:Name=\"SubmitButton\"
run --json devTools     before  "NodeCount", "OverlayShown", "Comments"
                        after   "nodeCount", "overlayShown", "comments"

Also fixed: the comments pane says "not linked to source" for unanchored comments and pluralizes the footnote correctly; Layout adorners is a UIA toggle (ToggleState: On/Off) with a state-neutral tooltip; "propert(ies)" is gone.

Group 2: -a on comments add/update/delete; --on hidden on comments list/get; get-property <selector> Text and set-property <selector> Width 200 (-p still works); run --detach prints Next: winapp devtools inspect -a <pid>; human output drops running bytes are not verified and the healthy-binding reason (both stay in JSON); the inspector's stretch note appears only for a non-default alignment.

Not reproduced

Quick-edit panel off-screen. On a 150% single monitor, the panel stays inside the work area for a full-width TextBox and a Gallery expander, with the Popup unconstrained. The existing placement tests already cover the flip, shift and height cap. I added the element and viewport to the overlay.sel shown log line. A repro (resolution, scale, monitors, window state) would unblock it.

Also

  • Rebased on main. Hide MSBuild property JSON from winapp run --aot output #936 fixed the same --aot property-JSON leak, so this branch now uses its result-file approach (plus the build-pass flag DevTools needs) and drops the stdout filter.
  • Pre-existing: the first Internal.pick of a session returns no-hit (old build too).

@nmetulev

Copy link
Copy Markdown
Member Author

Quick-edit panel off-screen: not a bug. I reproduced the reported setup: a window as wide as the monitor, then picks on a full-width element and on one near the bottom, at 150% (monitor 3840×2160 px, work area 3840×2088 px). The panel stays inside the work area:

full-width TextBox  Quick edits (3229,1260 600x815)   Open in DevTools (3248,2022 190x35)
Submit near bottom  Quick edits (194,1260 600x815)    Open in DevTools (214,2022 190x35)

The original report compared physical coordinates with a logical screen size. winapp ui reports physical pixels (panel x=1751, "Open in DevTools" y=1206), while "1843×1229" was a DPI-unaware read of a 2764×1843 px screen. In physical pixels both points are on-screen; they are only outside the app window, which the guide allows. No change made, apart from logging the element and viewport on overlay.sel shown.

@nmetulev

Copy link
Copy Markdown
Member Author

Group 3 decisions implemented

Four commits. Each change has a test that fails without it. "Before" is the pre-fix build; "after" is from this branch, run on samples/winui-app.

Pin (kept) is a toolbar toggle. It uses the same accent on-state as Pick and Layout adorners, and the corner dot is gone.

before  (old toolbar.xaml) DevToolsProtoPin: Button (no toggle state) + DevToolsProtoPinDot, a 7px accent dot at the bar corner
after   DevToolsProtoPin: ClassName ToggleButton, ToggleState On -> invoke -> Off; accent background when pinned

Theme toggle (kept on the toolbar). Tooltip "Toggle the inspected app between Light and Dark themes" is now "Switch app theme".

Inspector Properties pane. Order is now: type and name, ancestors, source line + Open, then the filter and the grid. The box model and the offered/took explanation move into a Layout section after the grid. It is collapsed by default, and its state is remembered across runs (devtools-InspectorLayoutOpen.setting flipped 1 -> 0 as I toggled it). Source-verification notes are one line, with the full reason on hover and in the UIA HelpText. The new guide screenshot shows the reordered pane: devtools-inspector.png.

set-property on a bound property now ends with a restart hint:

⚠ This replaced the binding {Binding Title}; its source no longer updates Text. Restart the app to restore the binding.

No devtools reset and no automatic capture.

pid is now processId in all winapp devtools JSON (attach, list, live commands, error payloads). run --json keeps ProcessId, and --pid/-a are unchanged. The native protocol's DevTools.negotiate still reports pid; that's the wire protocol, not a CLI payload.

before  devtools list --json -> "pid": 23652        get-property … --json -> "pid": 0
after   devtools list --json -> "processId": 23652  get-property CounterButton Content --json -> "processId": 29736

Guide split. devtools.md starts with five numbered steps and two screenshots (16 KB and 28 KB, in docs/images/). Then it covers reviewing comments with an agent, finding and inspecting, diagnosing bindings, live property changes, refresh, continuing from a terminal, and failures. devtools-advanced.md covers Sandbox, attaching (with the visual UI prerequisites), window and subtree targeting, comment storage and authoring, and the protocol (call, Binding.walk, capture/restore). The two pages are cross-linked. usage.md, sandbox-execution.md, the docs index and the skills now point at the right page, and the ProcessId-vs-pid note is gone.

Unchanged: --all, and Layout adorners (already off by default and remembered).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…hored UI comments

`winapp run --devtools` injects an inspection agent into a WinUI 3 app: an
in-app overlay (pick, quick edit, comments), an inspector window, and a
`winapp devtools` command group for agents (inspect, search, get/set
property, get-source, diagnose-binding, comments).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Keep walking past a nearer generated scope that does not reference the target
  (a UserControl whose content the page supplies) instead of reporting unavailable.
- Enrol windows held in static App fields (App.MainWindow { get; private set; }).
- Resolve non-public members for {x:Bind}, which compiles against its owner.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…omments

A hit with no source (a Button's string content, a TextBox placeholder, a nav
item label) now promotes to its nearest ancestor with authored source, the same
as a template part. Hover and click share this path; Just my XAML off keeps the
raw hit. A comment saved on an element that still has no source reports
'Saved. Not linked to source.' in the panel.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- A declaration that matches exactly one source declaration, or a unique
  x:Name, is a confirmed anchor; runtime instance counts no longer weaken it.
- Drop 'via type' candidates when a declaration or x:Name match exists.
- Show the historical creation location only when there is no current match,
  and never print an empty line number.
- A comment without a source file is weak, requires confirmation, and says it
  is not linked to source.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Comments pane: an unanchored comment says 'not linked to source' instead of
  'not on this screen' and is not counted as off screen; the footnote agrees in number.
- Layout adorners is a ToggleButton, so UI Automation exposes its state; the tooltip
  no longer says 'Show' while it is on.
- The precedence chain shows a null default as empty instead of the raw handle '0'.
- The inspector's extra-width note appears only when HorizontalAlignment is not Stretch.
- Log the quick-edit panel's element and viewport for placement triage.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…arnings

- When staging fails because the app is still running, name its PID(s) by matching
  the process image to the staged path, instead of a generic access-denied error.
- Collapse identical source-location warnings into one entry with a count, and say
  to rebuild when the build output does not match the XAML sources.
- run --devtools --detach ends with 'Next: winapp devtools inspect -a <pid>' and no
  longer prints 'running bytes are not verified'.
- devTools fields in run --json are camelCase like the rest of the new fields.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- set-property warns when it replaces a {Binding} or overrides an x:Bind (replacedBinding in JSON).
- get-property/set-property take the property positionally; -p still works.
- search falls back to the whole tree when nothing in your XAML matches (items created
  from data), and no longer claims displayed text is not searched.
- inspect --filter says when matches may be below --depth.
- comments: -a alias; list --status footer counts what was hidden; --on is hidden
  where it is always rejected.
- Relaxed JSON escaping so captured XAML is readable; 'properties' instead of 'propert(ies)'.
- Hide the healthy-binding scope disclaimer from human diagnose-binding output.
- Binding-owner e2e: static-held windows diagnose; unique x:Name comments confirm.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Pinned uses the same accent on-state as Pick and Layout adorners and is a
  ToggleButton, so UI Automation reports its state. The corner dot, which read as a
  notification badge, is removed.
- The theme button's tooltip is 'Switch app theme' so it is not clipped at the edge.
- Move the settings helpers to DevToolsSettings.h so the inspector can use them.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The Properties pane shows the header (type and name, ancestors, source line and
Open), then the filter and the grid. The box model and the offered/took explanation
move into a Layout section after the grid, collapsed by default and remembered
across runs. Source-verification notes are one line; the full reason is on hover
and exposed to screen readers.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…inding

- winapp devtools JSON (attach, list, live commands, errors) uses processId like
  winapp ui and winapp target. run --json keeps ProcessId; --pid and -a are unchanged.
- The set-property binding warning ends with 'Restart the app to restore the binding.'

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
devtools.md starts with numbered steps and two screenshots, then covers reviewing
comments with an agent, finding elements, bindings, live changes, refreshing, the
terminal workflow and failures. devtools-advanced.md covers Sandbox, attaching to a
running app, window and subtree targeting, comment storage and the protocol.
Links in usage, Sandbox docs and skills point at the right page.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
DevTools.negotiate (and the legacy hello reply) report processId instead of pid,
matching the CLI payloads. The protocol schema and the CLI's negotiate parser
change with it; there is no alias because the protocol has not shipped.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Generalize #942's ui help renderer and unknown-command handling to any compact
  help group; winapp ui output is unchanged apart from one pointer line.
- winapp devtools --help: purpose, the main workflow, what -a and <selector> mean,
  commands grouped by task, and when to use winapp ui instead. Every devtools
  command has examples, stored as data and parsed in tests. Unknown devtools
  commands suggest the closest one.
- Root help gets a short 'Start here' block for winapp ui and winapp devtools.
- A winapp ui command whose target has the DevTools agent prints one tip line on
  stderr (not with --json or --quiet).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
#942 added the options to ui record; the hand-written argument builder did not
forward them, which failed the generated-options sync test.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The winui-devtools skill now matches requests to change, translate or restyle a
running UI without editing source; winapp-ui-automation says it reads, clicks and
types in any app and defers live WinUI edits to winui-devtools. The UI automation
and DevTools docs mention the new help and the ui tip.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@nmetulev
Nikola Metulev (nmetulev) changed the base branch from main to nmetulev-agent-friendly-ui-help September 30, 2026 03:44
A Type argument cannot be serialized for discovery, so the two rows were discovered
as one case and ran as two. CI's CLI shard 2 requires the reported total to match
discovery exactly, which failed on #942 and here.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@nmetulev

Copy link
Copy Markdown
Member Author

Round 4 skeptical retest: AI Dev Gallery

Build: installed winapp --version → 0.7.1-nmetulev-devtools.16 (package winapp-dev_0.7.1.16).

Setup:

  • App: AI Dev Gallery (92e89340, unmodified).
  • Its comment store was restored byte-identical, the repo-root .winapp I created is deleted, and git status is clean.
  • The edit → rebuild check ran on a throw-away copy of samples/winui-app with a renamed package identity. That package has since been unregistered.

Picking:

  • Mostly done through the protocol pick (Internal.pick), which takes the same path as a real click.
  • Real input was limited to a few hover/click checks, each preceded by a check that the point was inside Gallery's own window.

Claims

# Claim Result Evidence
1 Pick promotion for sourceless text; JMX off keeps the raw hit; hover agrees with click; unanchored comment says "Not linked to source" ⚠️ Partial Promotion ✅. JMX on: search placeholder → AutoSuggestBox SearchBox; Settings "Change" button text → Button (SettingsPage.xaml:64). The same point with JMX off → TextBlock. A real click agrees (panel Button, XAML source: SettingsPage.xaml:64). Unanchored comment ✅. Panel shows Saved. Not linked to source.; JSON has "weak":true, anchorConfirmed=False, requiresConfirmation=True; list shows ⚠ not linked to source; no source candidates were found. Hover ❓. With JMX on, one hover over "Change" showed the raw TextBlock 48 × 19 highlight, but clicking the same point selected Button. A second hover showed the Button outline. See N3. NavigationViewItem ❓. A pick on the "Home" item returns Border Backplate (from the app's own Styles/NavigationView.xaml) in both JMX states, never the NavigationViewItem. See N4.
2 x:Bind: private field, page-owned, WindowEx-owned ✅ [ModelsExpander] SettingsExpander.ItemsSource state: evaluated path: cachedModels source: SettingsPage. [IndexStorageExpander] … state: evaluated path: indexStores. [MRUView] ItemsView.ItemsSource state: evaluated path: mostRecentlyUsedItems source: HomePage. [titleBar] TitleBar.IsBackButtonVisible state: evaluated path: NavFrame.CanGoBack source: MainWindow … resolved value: False.
3 Unique declaration or x:Name is strong; no via type noise; no empty historical line; edit → rebuild ⚠️ Partial Live capture ✅: an unnamed heading → TextBlock (current: Pages\HomePage.xaml:32) with no warning. SamplesBtn → current: …GettingStartedSection.xaml:91. Offline unique x:Name ❌: comments add --file Pages\HomePage.xaml --name MRUView → Candidate: Pages\HomePage.xaml:39:29 (weak, via x:Name), requiresConfirmation: true, and a line-less Created at (historical): Pages\HomePage.xaml. There is only one x:Name="MRUView" in the project. Edit → rebuild ❌: see N1.
4 set-property on a bound property warns; replacedBinding; no Default "0" ✅ ✅ [textblock-1efceef664] TextBlock.Text: …cpu-int4-rtn-block-32-acc-level-4 -> C:\edited then ⚠ This overrode {x:Bind Path} with a local value until the binding updates again. Restart the app to restore the binding. JSON: "replacedBinding": "{x:Bind Path}". The chain's Default entry is "value": "". Nit: a later get-property still labels the value Text: C:\edited [{x:Bind Path}].
5 Rerun names the PID; identical source warnings collapse ⚠️ Partial With a build ✅: …The app is still running from this build (PID 7824), so its files cannot be replaced. Close it, then run again. (Win32 5, HRESULT 0x80070005). --no-build ❌: [ERROR] - The launched process exited before inspection. An existing instance has not been confirmed to receive this launch's environment. Exit code: 0. No existing instance was adopted. Check App execution aliases, or launch normally and use 'winapp devtools attach --pid <pid>' …. It doesn't name the PID, and it's jargon (N2). Warnings: one stale file gives one readable line: [WARNING] - Source locations are unavailable for Pages\HomePage.xaml: The build output does not match the XAML sources. Rebuild (run without --no-build) to get source locations. I couldn't reproduce the N>1 collapse without modifying more files.
6 Smaller fixes ✅ mostly search: search "Semantic Kernel Chat" → ItemContainer/TextBlock Pages/HomePage.xaml. search "Phi 3 Medium" → ⚠ Nothing in your XAML matches "Phi 3 Medium"; these are generated or framework elements, such as items created from data. That hint only appears after about 20 s; before that, two calls printed ⚠ The agent has not finished classifying…. The zero-match message is fine. Comments pane: not linked to source / not on this screen, plus 1 of these isn't on this screen right now, so it has no marker… and 2 of these aren't … they have no marker…. Layout adorners: ClassName: ToggleButton ToggleState: Off→On→Off, matching Overlay.getState layoutAdornersOn: False→True→False. Footer: 3 comments total (2 open hidden; use --all). inspect --filter: No elements matched within --depth 3; deeper elements were not searched. Rerun with --depth 6, or use \winapp devtools search "tilegallery"`.**Properties:**Showing 24 of 292 properties. **JSON escaping:** "xaml": "<HyperlinkButton\n x:Name="SamplesBtn"…, no \u003C. **run --json:** "devTools": { "nodeCount": 581, "overlayShown": true, "comments": "local" }`.
7 Consistency ✅ mostly -a works on comments add/update ✅. On comments list it's rejected: Unrecognized command or argument '-a' (exit 1). --on is hidden from list/get help, and list --on sandbox exits 1 with a clear message ✅. Positional get-property <sel> Text and -p Text give the same output; set-property SamplesBtn Width 200 -p Height → Error: Pass the property once… ✅. run --detach ends Next: winapp devtools inspect -a 35956 ✅. The healthy-binding reason is gone from human output but present in JSON; the launch line is now ✅ Compiled XAML resources matched on disk. ✅. Stretch note: only the negative case is verified (Center-aligned heading, no note).
8 Group 3 decisions ✅ Pin: ClassName: ToggleButton ToggleState: Off→On, accent on-state, no corner dot (screenshot 1). Theme tooltip: reads "Switch app theme" and is not clipped (screenshot 1). Its UIA Name is still Toggle app theme (N5). Inspector: properties come first, and the verification notice is one line, Authored values unavailable (why?) (screenshot 2). Layout is collapsed by default; toggling writes %ProgramData%\winapp\devtools-InspectorLayoutOpen.setting 0→1→0. processId: present in devtools list, attach, inspect, search, get-layout, get-source, diagnose-binding, call and error payloads ("ok": false, "processId": 9064, "error": {…}), with no pid key anywhere. run --json keeps ProcessId.
9 Regressions ✅ Source mapping: a build launch printed zero source warnings, and get-source SamplesBtn → Authored declaration: …GettingStartedSection.xaml:91:13 (disk-matched). Marker auto-refresh: update --status resolved without -a → live Comment.list total 4 → 3. %TEMP%: during an attached run there is one winapp-source-inventory-*.json (448,201 bytes). Stop-Process -Force on the CLI → tmp: 0, the app stays alive, and get-source/diagnose-binding still work. stop: winapp stop exits 1 and doesn't appear in help. Sandbox: not tested.

New issues, by severity

N1 (High for the agent loop): a comment on an unnamed element needs confirmation again after the agent makes the requested change.

Repro on a copy of samples/winui-app:

  1. Add <TextBlock FontSize="18" Text="Repro heading" />.
  2. Save a comment on it (current: MainWindow.xaml:39, no warning).
  3. Do what the comment asks: set FontSize="24", and add two comment lines above it so it moves.
  4. Rebuild and relaunch.

Result:

cmt_d9374b6de819 open  TextBlock (candidate: MainWindow.xaml:41)
   ⚠ current source candidates require explicit confirmation; the creation location is historical
  Candidate: MainWindow.xaml:41:17 (weak, via treePath) <TextBlock FontSize="24" Text="Repro heading" />
  Candidate: MainWindow.xaml:34:17 (weak, via type) …PageSubtitle…
  Candidate: MainWindow.xaml:29:17 (weak, via type) …PageTitle…
  Candidate: MainWindow.xaml:53:21 (weak, via type) …CounterText…
  Candidate: MainWindow.xaml:97:17 (weak, via type) …ResultText…
  • The live marker still attaches correctly after the relaunch (placed: True).
  • In the same run, the named CounterButton comment stays current: MainWindow.xaml:47 ✅.
  • The top candidate is right, but agents are told to ask the user whenever confirmation is required. So the agent stops on exactly the comment it just implemented.
  • Four via type candidates are listed alongside a treePath match.

N2 (Medium): rerunning with --no-build while the app is running gives a jargon error, and rerun behaviour depends on the launch mode.

  • devtools --no-build over a running devtools instance: fails with the "exited before inspection … not been confirmed to receive this launch's environment" error quoted in claim 5. It doesn't name the PID. With --json, the reported ProcessId (35236) is a process that has already exited.
  • devtools --no-build over a running plain instance: succeeds, and the old PID (22512) is gone afterwards.
  • plain winapp run --detach --no-build over a running devtools instance: succeeds, and the old PID (7884) is gone afterwards.
  • In neither successful case is the replaced instance mentioned in the output.

N3 (Medium, unconfirmed): hover and click disagreed once with Just my XAML on.

  • The hover highlight on the Settings "Change" button showed TextBlock 48 × 19, but a click at the same point selected Button.
  • A later hover with real relative mouse moves showed the Button outline.
  • The first attempt moved the cursor with SetCursorPos only, so the difference may be in how hover handles a jump versus a move. The PR says hover and click share a path, so this is worth one deterministic test.

N4 (Low-Medium): a pick on a NavigationViewItem label selects the template's Border Backplate, not the item.

  • Backplate comes from the app's own Styles/NavigationView.xaml, so it counts as "your XAML" and promotion stops there. That's correct by the rule, but surprising: someone clicking "Home" expects NavigationViewItem (MainWindow.xaml:91).
  • Any app that ships a restyled control template will see the same result.

N5 (Low):

  • The theme button's tooltip says "Switch app theme", but its UIA Name is still Toggle app theme.
  • get-source for a file excluded as stale still prints the old long message (The authored markup is not shown: the compiler output, original source and deployed XAML could not be verified together…), while the launch warning now says Rebuild (run without --no-build).
  • diagnose-binding NavView IsBackButtonVisible (a literal) prints only state: none, with no reason.
  • comments list -a <pid> is a parse error while add/update accept -a; an agent will reuse -a everywhere.
  • Piped devtools list still hard-wraps (… v1 581 \nnodes (mutation)).
  • After navigating, search/inspect show ⚠ The agent has not finished classifying… and search the whole tree for about 20 s.

Toolbar: Pin on with accent, no dot; "Switch app theme" tooltip

Inspector: properties first, one-line verification notice

Bottom line: Every round-3 High is fixed for the main flows:

  • private, page-owned and WindowEx-owned x:Bind evaluate
  • pick promotion works for placeholders and button text
  • live-captured comments on unnamed and named elements are strong
  • the %TEMP% inventory leaves nothing behind after a hard kill

What still breaks the agent loop is N1: an agent that implements a comment on an unnamed element is then asked to confirm it. Behind that are the offline --name anchor that stays weak, and the --no-build rerun error (N2).

@nmetulev

Copy link
Copy Markdown
Member Author

DevTools discoverability from the CLI

This PR is now stacked on #942, and winapp devtools uses its compact help renderer. winapp ui --help output is byte-identical to #942's apart from one new pointer line. "Before" means #942 + DevTools before this change; "after" is this head. Sizes are bytes of the captured output.

Help Before After
winapp --help 6294 6434
winapp devtools --help 1668 2179
winapp devtools set-property --help 1937 2059
winapp ui --help 2741 2822

winapp --help (the #942 line becomes a block)

Before:

 Driving an app's UI from an agent or script? Start with 'winapp ui --help'.

After (bare winapp prints the same):

 Start here:
   Drive any app's UI (click, type, read)              winapp ui --help
   Inspect or change a running WinUI app's XAML live   winapp devtools --help
   (tree, properties, bindings; no source edits)

winapp devtools --help

Before
Description:
  Inspect live WinUI elements, edit properties, and review UI comments.

Usage:
  winapp devtools [command] [options]

Options:
  -?, -h, --help  Show help and usage information
  --cli-schema    Output the complete CLI command structure as JSON for tooling, scripting, and LLM integration. Includes all commands, options, arguments, and their descriptions.
  --on <on>       Run on local (default) or managed Windows Sandbox (sandbox); never falls back to local.

Commands:
  inspect <selector>                          Show the app's live XAML tree and reusable element selectors.
  search <query>                              Find live XAML elements and print their reusable selectors.
  get-property <selector> <property>          Read an element's live property values and where they come from.
  get-layout <selector>                       Read an element's size, position, and parent layout.
  get-source <selector>                       Find an element's XAML declaration, when source information is available.
  diagnose-binding <selector> <property>      Explain a live binding's path, source, and any failure.
  set-property <selector> <property> <value>  Change a live property and read it back; source files stay unchanged.
  call <method> <params>                      Call an advertised DevTools protocol method (advanced).
  comments                                    Manage saved UI review comments anchored to app source.
  list                                        List apps with DevTools attached.
  attach                                      Enable DevTools in a running WinUI 3 app until that app exits.

After:

winapp devtools - Inspect and change a running WinUI 3 app's XAML live. Changes are not
written to source.

  winapp run . --devtools --detach                     launch your app with DevTools
  winapp devtools attach --pid <pid>                   or add DevTools to a running app
  winapp devtools list                                 apps with DevTools attached
  winapp devtools search <text> -a <app>               find elements by text or x:Name
  winapp devtools get-property <selector> -a <app>     read its live properties
  winapp devtools set-property <selector> <prop> <value> -a <app>
                                                       change a property live
  winapp devtools comments list                        comments left in the app

  -a <app>     Process name, window title, or PID. Optional when only one app has
               DevTools attached.
  <selector>   The name in brackets from search or inspect, an x:Name, or a handle.

Use 'winapp ui' to click, type, and read values in any app. Use 'winapp devtools' for a
WinUI app's XAML tree, properties, bindings, and live edits.

Usage: winapp devtools <command> [options]    Details: winapp devtools <command> --help

Discover
  list              List attached DevTools apps or running WinUI attach candidates
  attach            Enable DevTools in a running app
  inspect           View a running app's XAML visual tree
  search            Find elements in a running app's XAML visual tree

Read
  get-property      Read an element's live properties and their value sources
  get-layout        Read an element's measured/arranged layout
  get-source        Read the XAML file and line an element was declared at

Change live
  set-property      Change a live property and read back what took effect

Bindings
  diagnose-binding  Explain a live binding's state, path, and failure

Comments
  comments          Manage source-anchored UI review comments

Protocol
  call              Call any advertised DevTools protocol method (advanced)

Options:
  --on <target>     Run on 'sandbox' (Windows Sandbox) or 'local' (default)
  -h, --help        Show help

winapp devtools set-property --help

Before
Description:
  Change a live property and read it back; source files stay unchanged.

Usage:
  winapp devtools set-property [<selector> [<property> [<value>]]] [options]

Arguments:
  <selector>  Element to change: the selector printed in brackets, an x:Name, or a handle.
  <property>  The property to change, e.g. Width. Same as --property.
  <value>     The new value, e.g. 200, false, #FF0067C0, or "Save changes".

Options:
  -a, --app <app>            Target app by process name, window title, or PID
  -w, --window <hwnd>        Target window by handle (printed by -a and list-windows; overrides --app)
  --root <root>              Constrain execution to a live visual-tree root or subtree handle from DevTools inspect or Surface.list.
  --attach                   Authorize attaching DevTools if the target is not already attached.
  --json                     Format output as JSON
  -p, --property <property>  Dependency property name (e.g. Width, IsEnabled, Background).
  --type <type>              XAML type to create the value as (e.g. Double, Boolean, String, Thickness). Inferred from the value when omitted.
  --of-type <of-type>        Match an exact XAML runtime type; short names must be unambiguous.
  --with <with>              Property<operator>Literal predicate. Repeat for AND; quote the whole argument.
  --value <value>            New literal value for a query-targeted set; omit the positional selector.
  -v, --verbose              Enable verbose output
  -q, --quiet                Suppress progress messages
  -?, -h, --help             Show help and usage information
  --cli-schema               Output the complete CLI command structure as JSON for tooling, scripting, and LLM integration. Includes all commands, options, arguments, and their descriptions.
  --on <on>                  Run on local (default) or managed Windows Sandbox (sandbox); never falls back to local.

After:

winapp devtools set-property - Change a live property and read back what took effect.
Change a live property and read it back; source files stay unchanged.

Usage: winapp devtools set-property <selector> <property> <value> [-a <app> | -w <hwnd>] [options]

Examples:
  winapp devtools set-property <selector> Text "<text>" -a <app>
  winapp devtools set-property <selector> Width 200 -a <app>
  winapp devtools set-property <selector> Background "#FF0067C0" -a <app>

Arguments:
  <selector>        Element to change: the selector printed in brackets, an x:Name, or a
                    handle.
  <property>        The property to change, e.g. Width. Same as --property.
  <value>           The new value, e.g. 200, false, #FF0067C0, or "Save changes".

Options:
  -a, --app <app>         Target app by process name, window title, or PID
  -w, --window <hwnd>     Target window by handle (printed by -a and list-windows;
                          overrides --app)
  --root <root>           Constrain execution to a live visual-tree root or subtree
                          handle from DevTools inspect or Surface.list.
  --attach                Authorize attaching DevTools if the target is not already
                          attached.
  -p, --property <property>
                          Dependency property name (e.g. Width, IsEnabled, Background).
  --type <type>           XAML type to create the value as (e.g. Double, Boolean,
                          String, Thickness). Inferred from the value when omitted.
  --of-type <of-type>     Match an exact XAML runtime type; short names must be
                          unambiguous.
  --with <with>           Property<operator>Literal predicate. Repeat for AND; quote the
                          whole argument.
  --value <value>         New literal value for a query-targeted set; omit the
                          positional selector.
  --json                  Format output as JSON

Global options: -v, -q, --on <target>, --cli-schema   (see winapp --help)

winapp ui --help: one added line

Changing a WinUI app's properties or text live? See 'winapp devtools --help'.

winapp ui on an app with DevTools attached (stderr, once; never with --json/--quiet)

> winapp ui inspect -a Daylight --depth 1        # 13 lines of tree on stdout, then on stderr:
Tip: this app has WinUI DevTools attached. 'winapp devtools --help' can read and change its XAML live.

No tip for an app without the agent (checked against Windows Terminal). The check lists the local \\.\pipe\ namespace and makes no connection.

Unknown devtools command

> winapp devtools set-text
Unknown command 'set-text'. Did you mean 'set-property'?
Commands: list, attach, inspect, search, get-property, set-property, diagnose-binding, comments, ...
Run 'winapp devtools --help' for the full list.

With --json it's a DevTools error (token: unknown-command). devtools comments <typo> gets the same handling.

Skills

  • winui-devtools now matches "change, translate, or restyle the running UI", "set text or any property at runtime", "without modifying source", and "inspect the live XAML visual tree, properties and bindings".
  • winapp-ui-automation says it's for reading, clicking and typing in any app, and points to winui-devtools for live WinUI edits.

Also fixed, because they broke #942's CI and would break this PR's:

Pull-request CI doesn't run while this PR's base isn't main, so Build and Package was started manually: run 36668649610 is green at 17a20661 (samples don't run on manual dispatch).

After an agent implements a comment on an element without x:Name, the
declaration no longer matches exactly. When the element keeps its tree
position and resembles the captured declaration more than any other
element of its type, it is now a strong match, and type-only candidates
are dropped.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
comments add --file X --name Y now captures the authored identity and
line when Y names exactly one element in X, so the comment is strong
and prints no line-less historical location.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A --devtools run over a running instance either failed at staging, or
launched an instance the running one absorbed, with a jargon error and
the PID of an exited process. It now stops before registering, names
the running PID, and says to close it. When re-registering a changed
package closes a running instance, run says so in one line.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@nmetulev
Nikola Metulev (nmetulev) changed the base branch from nmetulev-agent-friendly-ui-help to main September 30, 2026 05:34
A pick on a NavigationViewItem restyled in the app's own XAML selected
the template's Border, because it has authored source. Promotion now
also skips declarations inside a <ControlTemplate>, up to the element
the developer declared. Hover re-picks when Just my XAML changes, so a
cached highlight no longer disagrees with the click.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Source info was backfilled 100 elements per request, so search and
inspect said the agent had not finished classifying for several calls
after navigating. Backfill now continues in UI-thread batches of 100
until done or 250 ms have passed.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Agents reuse -a across devtools commands, but list and get rejected it
as a parse error. With -a <pid>, they now read the comments of that
running DevTools app's project.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- The theme button's accessible name matches its tooltip.
- get-source on an unverified build says to rebuild in plain language.
- diagnose-binding on a literal says the property has no binding.
- Piped devtools list keeps one line per app.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@nmetulev

Copy link
Copy Markdown
Member Author

Round-4 fixes (head 3d937849)

This responds to the round-4 report. Each fix has a test that fails without it. All live checks ran on a temporary copy of samples/winui-app with a renamed package identity, which I unregistered and deleted afterwards.

N1: a comment on an unnamed element survives the requested edit

Repro: comment on <TextBlock FontSize="18" Text="Repro heading" />, then change it to FontSize="24", add two comment lines above it, and rebuild.

Before (round 4):

cmt_d9374b6de819 open  TextBlock (candidate: MainWindow.xaml:41)
   ⚠ current source candidates require explicit confirmation; the creation location is historical
  Candidate: MainWindow.xaml:41:17 (weak, via treePath) <TextBlock FontSize="24" Text="Repro heading" />
  Candidate: MainWindow.xaml:34:17 (weak, via type) …PageSubtitle…
  …3 more via type

After:

cmt_49a46044d2bc open  TextBlock (current: MainWindow.xaml:41)
   Make the heading bigger

The JSON has anchorConfirmed: true, requiresConfirmation: false, and a single hit, MainWindow.xaml:41 treePath strong. The live marker is placed: true.

The rule: an element without x:Name stays confirmed if it keeps its tree position and matches the captured declaration's attributes more closely than any other element of its type. If a new sibling now sits at the old position and looks less like the original, the comment still needs confirmation (there is a test for this). Type-only candidates are dropped once there is a strong match.

Claim 3: offline --name with a unique x:Name

> winapp devtools comments add --file MainWindow.xaml --name CounterText --text "Offline note"
> winapp devtools comments list
cmt_f8fc4a7a4792 open  TextBlock #CounterText (current: MainWindow.xaml:59)

The comment is strong and has its line number. Nothing prints Created at (historical) without a line. A duplicate name stays weak.

Claim 5 / N2: reruns while the app is running

A --devtools rerun now stops before registering and names the running process. --no-build, a build, and --json all give the same result:

❌ The app is already running (PID 23956). DevTools needs to start it, so close it, then run again.

When re-registering a changed package closes the running instance (for example, a plain run over a DevTools run), run now says so:

✅ R4DevToolsWinUI_md30f2v49kz6j launched (PID: 2768)
📝 Closed the running instance (PID 23956) to update its registration.

If an app that allows only one instance takes over a launch anyway, the error names the running PID and reports it as ProcessId, not the PID of the process that exited. No new command was added.

Claim 1: picking inside a control template the app restyles

A pick now skips declarations inside a <ControlTemplate>, including templates in the app's own XAML, up to the element the developer declared. On a Button whose template (in the app's XAML) has a Border x:Name="Backplate", both the text and the Border's padding resolve to the Button:

{"handle":"3196901195624","nodeType":"Microsoft.UI.Xaml.Controls.Button","name":"StyledButton",…}

Hover and click already share PickTarget. The hover highlight was cached by the raw hit, so it went stale when Just my XAML changed. It now refreshes when that setting changes. I haven't reproduced the hover mismatch live, to keep real input to a minimum.

N5

  • The theme button's UIA name is now Switch app theme, matching its tooltip.
  • get-source on an unverified build: DevTools could not confirm that MainWindow.xaml matches the running build. Rebuild and run again (without --no-build); if this persists, check the original XAML before editing.
  • diagnose-binding on a literal:
    [CounterText] TextBlock.VerticalAlignment
      state: none
      No binding: the XAML sets this property to a literal value.
    
  • comments list -a <pid> and comments get <id> -a <pid> now read that app's project. --on is still rejected.
  • Piped devtools list prints one line per app.
  • Classification after navigating: one request now backfills new elements in batches until it finishes or 250 ms pass, instead of 100 elements per request.

Not changed: get-property still labels a value that set-property overrode [{x:Bind Path}]. x:Bind writes local values, so the runtime reports the same value source before and after the override; the set-property warning is where this is surfaced.

PasswordBox.Password was readable in plain text through get-property,
the precedence chain, query fields, the inspector and binding diagnosis.
Every property-chain read now replaces the value of a property named
*Password with <redacted> before any caller sees it. Property rows mark
it redacted and read-only, and JSON keeps the key with "redacted": true.
Binding diagnosis redacts its source, resolved and target values, and
writes to such a property are refused so nothing echoes it back.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The diagnostics chain reports a null value, including an empty String
such as AccessKey="", as the handle text "0". Property rows showed that
"0" while the precedence chain showed "". Rows and the quick-peek read
now show "" and keep valueState "null".

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Nothing has shipped, so DevTools.negotiate now reports protocolVersion "0"
alongside experimental: true, matching the legacy hello reply. It moves
to "1" together with dropping experimental once the protocol is stable.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A refused whole-app query write reported only truncated: true. The query
result now carries truncatedBy (for example tree-changed, stale-node,
unknown-child or time-budget), and the human summary prints it.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The inspector keeps its snapshot model, but re-reads live values when the
selected element is picked or clicked again, when a property row or its
editor is opened, and when a quick-peek text editor gains focus. Nothing
is re-read over an open editor, typed text or an unconfirmed write.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@nmetulev

Copy link
Copy Markdown
Member Author

Presentation-prep findings (head c6940bab)

Each fix below has a test that fails without it. I checked everything live on a temporary copy of the demo app, with a PasswordBox and an AccessKey="" Button added. The copy is unregistered and deleted.

1. Passwords are redacted (security)

Before:

[SecretBox] PasswordBox MainWindow.xaml
  Password: hunter2 [Local]

JSON before: "value": "hunter2", "authored": "hunter2", and chain value "hunter2".

After:

[SecretBox] PasswordBox MainWindow.xaml
  Password: <redacted> [Local]

JSON after: "value": "<redacted>", "authored": "<redacted>", "redacted": true, and the chain value is "<redacted>". get-property --all, inspect --fields Password, search hunter2 and diagnose-binding don't contain the secret.

Writes are refused:

> winapp devtools set-property SecretBox Password leaked --type String -a <pid>
Error: 'Password' holds a secret; DevTools does not read or write it.

How it works

  • Scope: any property whose name ends in Password.
  • Native reads: every read of an element's property values goes through one function that redacts these values. That function feeds the CLI, the inspector, the quick peek, queries and search. Rows are also redacted after the live getter and binding evaluation.
  • Bindings: binding diagnosis (native and managed) redacts the source, resolved and target values. Binding capture, restore and write-through are refused for these properties.
  • No opt-in flag was added.

2. An empty or null value shows as empty, not "0"

AccessKey="": get-property --json returned "value": "0" and now returns "value": "" with "valueState": "null", matching the chain. Rows and the quick-peek read share the fix. The earlier Default "0" fix only covered the chain.

3. Whole-app query write refused as incomplete: not reproduced

I couldn't reproduce this on this head or on a local build of the reported d25242bf. Both applied the write fresh, after opening and closing the inspector (Window.open/Window.close), and after a protocol pick with the highlight showing.

I couldn't open the quick peek with a real click, because winapp ui click refused to move focus to the app. So I didn't loosen any completeness check without a failing case. Instead, the result now says why the scope was incomplete (truncatedBy: tree-changed, stale-node, unknown-child, time-budget, classifying, ...), and the human summary prints it, for example incomplete (tree-changed). The next repro will name the cause.

4. The protocol version is "0" while experimental

"protocolVersion": "0",
"experimental": true,

devtools list shows v0. The legacy hello reply already said "0".

5. The inspector re-reads values when you look at them again

The snapshot model stays. Values are read live when you:

  • pick or click the element that is already selected
  • open a property row or its editor
  • focus a text editor in the quick peek

Nothing is re-read over an open editor, text you've typed, or an unconfirmed write. The window tests cover row opening and both guards. I didn't check this live, because it needs clicks inside the app.

6. Docs

  • The guide and the skill don't make claims about ResolveResource, handle-valued chain entries, or SrcInfo columns, so nothing needed correcting.
  • I added a short Performance section to devtools-advanced.md, based on the prep session's measurements.

The toolbar, highlight, selection chrome, pick catcher and comment markers
now live in IXamlDiagnostics2::GetUiLayerForXamlRoot's layer for each
XamlRoot. They draw above app content, in-window popups and dialogs, and
stay out of the app's tree and layout. Each chrome Popup remains the
fallback host, chosen per XamlRoot when any step of the layer lookup
fails.

- UIA does not walk into the layer. An unparented zero-size Popup per
  XamlRoot holds a proxy whose automation peer reports the layer content,
  so screen readers and winapp ui keep the same tree, names, roles and
  toggle states.
- The quick peek's windowed Popup opens unparented on the XamlRoot, and
  the pick catcher keeps only a border around it: windowed popup input
  is hit-tested against the layer too.
- Tab cycles within the toolbar and Esc reaches the pick handler, as they
  did inside a Popup. The chrome follows the app content's theme, which
  the layer does not inherit.
- Overlay.getState reports host (uiLayer, popup or none). The internal
  Internal.overlayHost forces the fallback for tests.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Lifting the windowed quick-peek Popup out of the selection markup broke
FindName lookups for its status and error text, so the comment failure
status never showed (e2e probe). The pick catcher's cut-out around the
open peek is what lets the windowed popup receive input, so the Popup now
stays where it was and is only seeded out of the census.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
… open

A windowed Popup whose parent is in the diagnostics UI layer makes the
framework list the layer's chrome under that popup's window, so UIA
clients resolved the toolbar to the peek and 'winapp ui focus' on it
failed with foreground_not_target (e2e probe). The peek's Popup now opens
unparented on the XamlRoot again; names inside it are looked up from its
content when the selection root no longer reaches them.

The peek's comment status was looked up from the binding-diagnosis root,
which is only set when the element has binding rows, so unbound elements
never showed Saving/Saved/Unsaved status. It now uses the selection root.
The probe checks that status on an unbound element.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@nmetulev

Copy link
Copy Markdown
Member Author

In-app overlay now hosted in the XAML diagnostics UI layer

The toolbar, highlight, selection tag, pick catcher, comment markers and layout adorners now live in IXamlDiagnostics2::GetUiLayerForXamlRoot's layer, one per XamlRoot. The quick peek stays a windowed popup so it can still extend past the window. The old Popup-in-outermost-panel host remains as the fallback for any XamlRoot where the layer lookup fails. Overlay.getState now reports host as uiLayer, popup or none.

Before (Popup host): the toolbar sits under a ContentDialog's smoke.
Before: toolbar dimmed under the dialog smoke

After (UI layer): the toolbar draws above the dialog and stays usable.
After: toolbar above the dialog

Comment markers and the toolbar on the layer path, with a comment badge and marker:
Layer path: comments mode with marker and badge

Accessibility

UI Automation does not walk into the layer. To keep tree-walk access, each XamlRoot gets one unparented, zero-size Popup holding a proxy element. The proxy's automation peer reports the layer content as its children. It is torn down when the XamlRoot closes.

I compared the same toolbar through managed UIA on both hosts:

  • Popup host: DevToolsProtoPick Button 'Select element', DevToolsProtoLayout Button 'Layout adorners' toggle=Off, DevToolsProtoPin Button 'Pin toolbar open' toggle=Off, …
  • Layer host: the same ControlType, Name, focusability and toggle state for every toolbar action. The only difference is the parent, which is the DevTools pane instead of the window root.
  • Tab cycles within the toolbar on both hosts. Without a fix it left into the app on the layer; the layer content now uses TabFocusNavigation=Cycle, as a Popup does.
  • Entering from the app: Tab does not reach the toolbar on either host. There is no existing toolbar focus shortcut (I didn't find a Ctrl+Shift+P), so I added none. ui focus DevToolsProtoRailL reaches DevToolsProtoPick on both.

Fixed while moving

  • Quick-peek clicks: windowed-popup input is hit-tested against the layer too, so the full-size pick catcher swallowed clicks on the quick peek. While the peek is open, the catcher now keeps only a transparent border around it.
    • Evidence: before, clicking the comment box logged pick click outside the open panel -> dismiss. After, focus lands on DevToolsSelComment.
  • Esc from the layer reaches the pick handler.
  • Toolbar resolved to the quick peek's window: a windowed Popup whose parent is in the layer makes WinUI list the layer chrome under that popup's window. While the peek was open, UIA clients resolved the toolbar to the peek's window, and winapp ui focus DevToolsProtoRailL failed with foreground_not_target (caught by the CI probe). The peek's Popup now opens unparented on the XamlRoot. The toolbar stays under the app window, and ui focus reaches DevToolsProtoPick again.
  • Comment status on elements without bindings (existed before this change): "Unsaved changes." / "Saving..." / "Saved." never appeared in the quick peek for an element with no binding rows, because the status was looked up from the binding-diagnosis root. It now uses the selection root, and the probe checks it on an unbound element.
  • Theme: the chrome follows the app content's effective theme, including theme changes the app makes itself. The layer on its own follows the Application theme.

Checklist (samples/winui-app on 1.8, SpikeApp on 1.8 and 2.3.1)

Area Result
Pick, Just my XAML promotion (placeholder → TextBox) ✅ same result on both hosts
Quick peek: 5 row peers once, comment editor, save, delete ✅
Comment markers, badge ✅
Layout adorners, pin, top-left/top-right corners with title-bar avoidance, resize ✅
Theme toggle (inverse toolbar), app-driven theme change ✅
Inspector open and close ✅
ContentDialog open: toolbar above the smoke, UIA-reachable ✅ on 1.8 and 2.3.1
App Flyout and MenuFlyout light-dismiss with the bridge Popup open ✅ both dismiss on an outside click
Navigation, Window.Content replace ✅ toolbar stays hosted, host: uiLayer
Second window: its own layer root and bridge; torn down on close (uiLayer root=… closed), and picking recovers in window 1 ✅
Read-only posture (WINAPP_DEVTOOLS_MUTATION=deny): host: uiLayer, writes refused ✅
Forced fallback (Internal.overlayHost {"force":"popup"}): host: popup, toolbar reachable ✅
C++ WinUI sample (no XamlControlsResources) Unchanged: the overlay reports the missing resource, and headless inspection works
Native suites dispatcher 401/401, window 69/69, overlay geometry 599/599

Perf (demo app, same harness as the presentation run)

Before After
Idle overlay, UI-thread work none none
Highlight, mean 1.63 ms 1.51 ms
Inspector open, UI-thread stall 0.21 s 0.16 s
Private memory: overlay / inspector open +2.2 / +20 MB +2.4 / +22 MB
Headless attach 0.7–1.1 s 0.3 s (not a hosting change; attach doesn't build the overlay)

Known limits

  • A menu or drop-down that opens in its own window (MenuFlyout, ComboBox) draws over the chrome. Where the two overlap, a click goes to our chrome, and while pick mode is armed the full-size catcher takes those clicks too.
  • My harness can't deliver keystrokes into the windowed quick peek on either host, so keyboard entry into the peek is covered only by the CI probe's set-value and invoke.

While the toolbar is shown, Ctrl+Shift+F12 in the app expands it and
focuses its first action; Esc from the toolbar returns focus to the app
element that had it (a following Esc still leaves pick mode). The chord is
ignored while the toolbar is hidden, including --no-overlay. Plain F12 is
avoided because Windows reserves it to break into an attached debugger.

The overlay probe focuses a fixture button, sends the chord and Esc, and
checks where UIA reports focus. The toolbar carries the shortcut as its
AcceleratorKey and tooltip; the guide and skill document it.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Esc did not return focus in the CI probe, and it did not reproduce locally.
The shortcut captured the return target only when the toolbar's focus
counter read zero, which depends on every LostFocus arriving. The shortcut
and Esc now check whether the focused element is inside the toolbar
instead. The overlay probe keeps the app's DevTools log as evidence so a
repeat failure shows which step failed.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant