feat(computer-use): rebuild scoped control and batched native input - #3191
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Rebuild Computer Use around scoped control sessions, typed batched actions, persistent window observation, and native platform input adapters. Application tasks default to background control; delegated prompts cannot silently authorize foreground takeover.
Type and Areas
Feature, bug fix, refactor, tests and CI dependencies. Rust contracts/core, Desktop native adapters, agent prompts, AI image transport and Web UI control feedback.
Motivation / Impact
Computer Use previously interrupted foreground work, rejected ordinary background pointer actions, lost visual context and encouraged slow single-action loops. This change adds shared single/batch execution with explicit receipts, observation freshness and cancellation, preserves multimodal screenshots, and supplies compact batch examples to the model.
macOS uses persistent window capture and window-directed input with target-local focus, explicit modifiers and process-identity cleanup. Windows uses window capture and UIA/window-message adapters; Ubuntu uses AT-SPI and negotiated portal/PipeWire capabilities. Unsupported operations report their actual limits instead of silently taking over the foreground. The UI exposes control status, stop and pointer feedback. No application-specific WeChat workflow is added.
Verification
cargo build --locked -p openbitfun-desktop: passed locally on macOS.cargo test --locked -p openbitfun-desktop --lib computer_use: 143 passed, 9 native fixtures intentionally ignored in the unit invocation.node scripts/test-macos-input-controls.mjs: passed against dedicated native test apps.node scripts/test-macos-input-controls.mjs --orchestration: passed through the production Tool pipeline. A five-action batch on an occluded non-AX canvas completed in 466 ms including final observation, excluding model reasoning. It preserved the observer's foreground/key/stacking state and physical cursor, delivered one content click, typed into the correct destination, executed a real menu shortcut, scrolled and cleaned up on Stop.f98bbb29cc64149a0ffd3c637310db489ed2eddb: all 15 PR checks passed, including three-platform Rust/CLI, frontend, Linux x64/ARM64 binary and installer validation. CI run.Reviewer Notes
Architecture and acceptance boundaries are documented in
docs/architecture/computer-use-control.md. Native macOS focus uses dynamically resolved private APIs and needs broader application/OS compatibility testing; capability or permission failures must not silently activate apps. The macOS fixtures are local native evidence, not Windows/Ubuntu native or remote-workspace, remote-control, peer-device or detached-dispatch acceptance. Existing Windows/Linux compile/unit evidence is recorded in the spec.Checklist