Skip to content

feat(computer-use): rebuild scoped control and batched native input - #3191

Merged
bobleer merged 2 commits into
GCWing:mainfrom
bobleer:bob/computer-use-control-rework
Sep 22, 2026
Merged

bobleer merged 2 commits into
GCWing:mainfrom
bobleer:bob/computer-use-control-rework

Conversation

@bobleer

@bobleer bobleer commented Sep 22, 2026

Copy link
Copy Markdown
Collaborator

Summary

Rebuild Computer Use around scoped control sessions, typed batched actions, persistent window observation, and native platform input adapters. Application tasks default to background control; delegated prompts cannot silently authorize foreground takeover.

Type and Areas

Feature, bug fix, refactor, tests and CI dependencies. Rust contracts/core, Desktop native adapters, agent prompts, AI image transport and Web UI control feedback.

Motivation / Impact

Computer Use previously interrupted foreground work, rejected ordinary background pointer actions, lost visual context and encouraged slow single-action loops. This change adds shared single/batch execution with explicit receipts, observation freshness and cancellation, preserves multimodal screenshots, and supplies compact batch examples to the model.

macOS uses persistent window capture and window-directed input with target-local focus, explicit modifiers and process-identity cleanup. Windows uses window capture and UIA/window-message adapters; Ubuntu uses AT-SPI and negotiated portal/PipeWire capabilities. Unsupported operations report their actual limits instead of silently taking over the foreground. The UI exposes control status, stop and pointer feedback. No application-specific WeChat workflow is added.

Verification

  • cargo build --locked -p openbitfun-desktop: passed locally on macOS.
  • cargo test --locked -p openbitfun-desktop --lib computer_use: 143 passed, 9 native fixtures intentionally ignored in the unit invocation.
  • Focused Core Computer Use tool tests: 45 passed; prompt catalog tests: 8 passed.
  • node scripts/test-macos-input-controls.mjs: passed against dedicated native test apps.
  • node scripts/test-macos-input-controls.mjs --orchestration: passed through the production Tool pipeline. A five-action batch on an occluded non-AX canvas completed in 466 ms including final observation, excluding model reasoning. It preserved the observer's foreground/key/stacking state and physical cursor, delivered one content click, typed into the correct destination, executed a real menu shortcut, scrolled and cleaned up on Stop.
  • Repository hygiene, GitHub configuration, core boundaries, i18n and all 23 theme/color surfaces: passed.
  • Capability catalog check and 15 capability contract tests: passed after registering status/preview/stop and regenerating remote command maps.
  • Frontend production build: passed. Full Web UI suite using CI's Node 22.23.2: 850 files / 9229 tests passed. The local Node 26 run exposed storage-environment differences; no test expectations were weakened.
  • Final head f98bbb29cc64149a0ffd3c637310db489ed2eddb: all 15 PR checks passed, including three-platform Rust/CLI, frontend, Linux x64/ARM64 binary and installer validation. CI run.

Reviewer Notes

Architecture and acceptance boundaries are documented in docs/architecture/computer-use-control.md. Native macOS focus uses dynamically resolved private APIs and needs broader application/OS compatibility testing; capability or permission failures must not silently activate apps. The macOS fixtures are local native evidence, not Windows/Ubuntu native or remote-workspace, remote-control, peer-device or detached-dispatch acceptance. Existing Windows/Linux compile/unit evidence is recorded in the spec.

Checklist

  • This PR is focused and does not include secrets, temporary prompts, generated scratch files, or unrelated artifacts.
  • Relevant verification is recorded above, or skipped checks are explained.
  • User-facing strings, docs, and locales are updated where applicable.

@bobleer
bobleer merged commit 0be594d into GCWing:main Sep 22, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant