Agent-first winapp ui help, unknown-command suggestions, and hosted-app targeting

[Nikola Metulev (nmetulev)]

Description

Agents driving an app with winapp ui had to read a 6.5 KB help page with no starting point. They guessed command names that don't exist (ui dump, ui tree), and then hit two dead ends on the most common path:

• -a calculator found nothing, but still reported success.
• inspect --interactive hid Notepad's editor.

This PR makes the first minutes with winapp ui work (Stage 1 of the agent-guidance spec):

• Compact winapp ui --help: plain text that starts with the golden path (inspect --interactive → invoke/set-value → get-value). It defines -a and <selector>, then lists the commands grouped by task: Discover, Act, Read and wait, Capture, Gestures, Workflow coordination. Edge-case commands say when to use them. It is 2,741 bytes redirected, down from 6,538.
• Per-command help for every ui command, also plain text: summary, usage, 1–3 examples (placeholders only), arguments, options, and one Global options: line. Examples are stored as data and a test parses each one against its command. Help for other command groups is unchanged.
• Unknown commands fail with suggestions. winapp ui dump exits 1 with Did you mean 'inspect'?, a short command list, and a pointer to winapp ui --help. This holds even with --help, and it happens before --on sandbox routing, so no sandbox starts. With --json it emits the existing UI error envelope plus error.suggestions.
• find and tree are aliases for search and inspect. They are hidden from ui --help.
• Root help gains one line: Driving an app's UI from an agent or script? Start with 'winapp ui --help'.
• --type / --root / --class-name on every single-selector command (set-value, click, focus, hover, scroll, scroll-into-view, screenshot, record, touch, pen, and inspect with a selector). With filters, the selector must match exactly one element. invoke now accepts the filters without --action; filtered invoke skips the ancestor fallback. drag is unchanged.
• Hosted apps via -a. When the matched process owns no top-level window, as with Calculator hosted by ApplicationFrameHost, -a now uses the ApplicationFrameHost frame whose title matches. Only frames qualify, so a window like myapp - Visual Studio Code is never picked. A process with no window yet, such as an app that is still starting, keeps the process-scoped target, so wait-for still polls for its window. list-windows -a falls back to title matches too.
• inspect --interactive shows editable elements. Documents, such as Notepad's Text editor, and elements with a writable Value pattern now appear, and --json elements carry isEditable: true. The footer example suggests set-value for such elements.
• Missing option value (winapp ui invoke Cancel -w) now prints the error and Run 'winapp ui invoke --help' for usage. instead of the full help.

Usage Example

Observed with the built CLI (win-arm64 NativeAOT):

> winapp ui search Seven -a calculator
Before (from the bug report): Window: (none) / Found 0 matches   (exit 0)
After:  'CalculatorApp' (PID 46004) has no top-level window of its own; using the app frame titled like 'calculator' instead.
          num7Button Button "Seven" (14,1158 116x74)
        Found 1 matches                       (exit 0)

> winapp ui dump --help
Unknown command 'dump'. Did you mean 'inspect'?
Commands: status, list-windows, inspect, search, invoke, set-value, send-keys, get-value, wait-for, ...
Run 'winapp ui --help' for the full list.
(exit 1)

> winapp ui dump --help --json --on sandbox
{"error":{"code":"invalid_arguments","message":"Unknown command \u0027dump\u0027.","suggestions":["inspect"],"recoveryHint":"Run \u0027winapp ui --help\u0027 to list commands."}}
(exit 1, no sandbox started)

Notepad golden path, run against a Notepad window the test launched itself (-w <hwnd>):

> winapp ui inspect -w <hwnd> --interactive     -> includes: doc-texteditor-4ff3 Document "Text editor"
> winapp ui set-value "Text editor" "hello from winapp" -w <hwnd> --type Document   -> Set value on doc-texteditor-4ff3
> winapp ui get-value "Text editor" -w <hwnd> --type Document                      -> hello from winapp

New winapp ui --help (full output, 2,741 bytes):

winapp ui - Drive any running Windows app through UI Automation (WinUI 3, WPF, WinForms,
Win32, UWP, Electron).

  winapp ui inspect -a <app> --interactive           see what you can act on
  winapp ui invoke <selector> -a <app>               press buttons, menu items, tabs, toggles
  winapp ui set-value <selector> "<text>" -a <app>   fill text boxes and documents
  winapp ui get-value <selector> -a <app>            check the result

  -a <app>     Process name, window title, or PID. Targets the app's active window,
               including an open dialog. It prints the window's -w <hwnd>; use that
               if it picked the wrong window.
  <selector>   A visible label ("Save as"), an AutomationId (stable), or a slug from
               inspect (changes when the element is recreated). If a label matches
               several elements, narrow it: "Save" --type Button, or --root <selector>.
  After an action changes the UI (a dialog opens, a page loads), inspect again.

Usage: winapp ui <command> [options]    Details: winapp ui <command> --help

Discover
  inspect           Show an app's elements and their selectors
  search            Find elements by text; narrow with --type and --root
  list-windows      List an app's windows, when -a picks the wrong one
  get-focused       Show the element that has keyboard focus
  status            Check that winapp can connect to an app

Act
  invoke            Activate an element (Invoke, Toggle, Select, Expand)
  set-value         Set the text or value of an element
  send-keys         Keyboard shortcuts, or text where set-value is not supported
                    (shortcuts need --via send-input)
  click             Mouse click, when invoke is not supported
  focus             Move keyboard focus to an element
  scroll            Scroll a container element
  scroll-into-view  Scroll an element into the visible area

Read and wait
  get-value         Read an element's text or value
  get-property      Read UIA properties from an element
  wait-for          Wait for an element to appear, disappear, or reach a value

Capture
  screenshot        Capture a window or element as PNG
  record            Record a window or element region to MP4

Gestures
  hover             Move the mouse to an element (tooltips, hover states)
  drag              Drag from one element or point to another
  touch             Inject touch gestures (tap, swipe, pinch)
  pen               Inject pen input (taps and ink strokes)

Workflow coordination
  yield             Release this workflow's UI turn now

Options:
  --on <target>     Run on 'sandbox' (Windows Sandbox) or 'local' (default)
  -h, --help        Show help

Related Issue

Stacked on #932 (scoped --type/--root/--class-name targeting for ui invoke). Retarget to main after #932 merges.

Type of Change

• 🐛 Bug fix
• ✨ New feature
• 💥 Breaking change
• 📝 Documentation
• 🧪 Test update

Checklist

• New tests added for new functionality (if applicable)
• Tested locally on Windows
• docs/usage.md updated (if CLI commands changed)
• Shipped skills updated in plugins/winapp/skills/ (if CLI commands/workflows changed)

Screenshots / Demo

Not applicable: these are text-only CLI changes. See the observed output above.

Additional Notes

Compatibility: winapp ui <unknown> --help changed from exit 0 (it printed the ui help) to exit 1 with an error. This is the 0.7.0 behavior, and no known consumer depends on it. ui invoke with --type/--root/--class-name but without --action was rejected on #932 and is now accepted. That behavior was never released.

Validation:

• .\scripts\build-cli.ps1 -SkipTests succeeded. It regenerated docs/cli-schema.json, src/winapp-npm/src/winapp-commands.ts, and docs/npm-usage.md. .\scripts\validate-plugin-package.ps1 passes.
• Tests: all UI, help, Program, and execution-target tests pass (2,004). The rest of the CLI suite ran 5,042 tests with 17 failures, all environmental on this machine: NuGet feed access, Node E2E wrappers, and one ARM64-vs-x64 crash dump analysis. WinApp.UIAutomation.Tests pass.
• New tests:
  • Every ui command is in exactly one help category.
  • Every example parses against its command.
  • Every command with a selector argument accepts the three filters.
  • Help size and format.
  • Unknown-command text and JSON envelopes (including --help, --on sandbox, --json=false).
  • The aliases, and the hosted-app resolver fallback.
• End to end with the built CLI: Calculator via -a (search, invoke, get-value, inspect, --json), and a self-launched Notepad window via -w (inspect --interactive, set-value/get-value/focus with --type Document, tree/find aliases). Also checked: bare winapp ui still shows the group help, and winapp ui -a Notepad reports the normal parse error rather than Unknown command 'Notepad'.

Limitations: A process that has no window and no matching app frame still gets a process-scoped target, as before, so read commands like search can report 0 matches instead of an error. Failing there would break wait-for while an app starts. ui invoke --help is about 1.6 KB, over the spec's ~1 KB target, because it keeps the full action and filter explanations. JSON error strings escape ' as \u0027, the same as existing UI errors.

AI Description

This section is auto-generated by AI when the PR is opened or updated. To opt out, delete this entire section including the marker comments.