Clarify keyboard-mouse profile output semantics #41

User and workflow

A game-agent developer choosing OpenController's keyboard-mouse profile may expect desktop keyboard events, cursor movement, or coordinate-based clicks. The current profile name and README shorthand do not explain which names become gamepad report controls and which remain only in controller state.

Direct evidence

• Codex issue #35510, opened 2026-07-26, reports that Computer Use said it sent Return but a Pygame application received no event; a comment reports similar key failures in Satisfactory.
• Unity MCP issue #1408, opened 2026-09-21, says an agent could inspect a Unity game but lacked Play Mode key and mouse input tools; its requested mouse operation uses screen coordinates.

These concern other products and do not show that an OpenController user misunderstood the profile. They show that “keyboard/mouse” can mean target OS events and pointer coordinates, not merely named controller controls.

Repository gap and exact current behavior

packages/core/src/profiles/keyboard-mouse.ts accepts these canonical button names: KEY_SPACE, KEY_ESCAPE, KEY_E, KEY_F, KEY_W, KEY_A, KEY_S, KEY_D, MOUSE_LEFT, MOUSE_RIGHT, MOUSE_MIDDLE, and the four DPAD_* directions. It accepts MOUSE_LEFT and MOUSE_RIGHT as trigger names as well.

The profile's toUniversal map includes:

• KEY_SPACE → SOUTH (alias A)
• KEY_ESCAPE → EAST (alias B)
• KEY_F → WEST (alias X)
• KEY_E → NORTH (alias Y)
• MOUSE_RIGHT → LEFT_TRIGGER (alias LT)
• MOUSE_LEFT → RIGHT_TRIGGER (alias RT)
• DPAD_UP, DPAD_DOWN, DPAD_LEFT, and DPAD_RIGHT → their corresponding universal D-pad controls

KEY_W, KEY_A, KEY_S, KEY_D, and MOUSE_MIDDLE are accepted and can appear in state/replay data, but have no toUniversal entry. The XInput and generic HID encoders skip controls without a universal mapping, so those five names are omitted from those encoded reports. MOUSE_LEFT/MOUSE_RIGHT are mapped trigger controls; they are not screen-coordinate clicks or OS mouse events. KEY_* names likewise do not cause OS keyboard injection.

packages/core/src/bridge/native.ts:createNativeBridgeStateMessage emits XInput and generic HID gamepad reports. The public docs/controller-profiles.md currently lists keyboard-mouse but does not describe the mapping/omission boundary; the README calls it a keyboard/mouse-style profile.

Proposed outcome

Document the current mapping exactly and add profile-specific tests that keep the documented supported-report subset aligned with the encoder behavior. This is a contract-clarity and test gap, not a proposal to add desktop input injection.

Acceptance criteria

• Document the six non-D-pad mappings and their aliases listed above.
• Document that KEY_W/A/S/D and MOUSE_MIDDLE are currently accepted into state but have no universal mapping and are omitted from XInput/generic HID reports.
• Document that the profile does not emit OS keyboard events, screen coordinates, cursor movement, coordinate clicks, or window targeting.
• Add tests that assert the mapped names produce the expected universal XInput/HID controls, the five unmapped names remain in state but are omitted from those reports, and aliases resolve as listed.
• Keep the profile name and public behavior unchanged in this slice; do not relabel unsupported names as working keyboard/mouse output.

Tests and validation

Put profile and native-bridge serialization assertions in a new focused packages/core/src/tests/keyboard-mouse-profile.test.ts file, not controller.test.ts; run core tests, typecheck, and docs link checks. No host driver or game launch should be required.

Non-goals

• No desktop keyboard/mouse injection, screen coordinates, cursor movement, drag gestures, app/window selection, or new native driver.
• No rename or breaking change to the public keyboard-mouse profile.
• No claim that a generic HID gamepad report is consumed as keyboard/mouse by every host or game.

Confidence and limits

Low to medium. Two dated agent-tool reports distinguish game-window key/mouse interaction from ordinary controller input. No source states that an OpenController user was misled by this profile; the gap is inferred from the public profile name and current mapping/documentation.

Ownership and dependencies

Lane: packages/core/src/profiles/keyboard-mouse.ts tests, packages/core/src/hid/xinput.ts/HID encoder tests, docs/controller-profiles.md, and README. Independent of the MCP example and timed-press cancellation work. One agent can own this documentation/contract slice.