Skip to content

Feat/windows camera vision host input - #533

Open
power246 wants to merge 8 commits into
OctoSense-org:mainfrom
woshuoduijiushidui:feat/windows-camera-vision-host-input
Open

power246 wants to merge 8 commits into
OctoSense-org:mainfrom
woshuoduijiushidui:feat/windows-camera-vision-host-input

Conversation

@power246

@power246 power246 commented Oct 13, 2026 •

Copy link
Copy Markdown

Team: 我说对就是队

Why

Contained apps on Windows could import an image, but could not complete the camera-to-model flow: camera preview and host camera consent were unavailable, model.complete accepted text only, and Windows pointer presses could not approve host-owned sheets. This combines the changes from the group fork's PR #1 and PR #2. The motivating app is PantrySteward PR #45.

How

  • Add a pinned Makepad Windows camera-source patch. It feeds Media Foundation frames into the existing CameraPreview/Video path, with a bounded latest-frame pool; the shell advertises the existing camera permission methods on Windows. Capture returns an app-local photo path through the host API.
  • Add model.complete@2 on Windows with one optional {image: {app_path}}. After caller admission, the host resolves the file inside that app's storage, checks its JPEG/PNG/WebP signature and 10 MiB limit, and sends it through a configured OpenAI-compatible provider. Provider credentials, provider choice and model name stay with the host. Other platforms keep @1.
  • Let desktop files.import stage up to 64 MiB without moving bytes through the script heap. files.pick_photo keeps its 1 MiB limit, and model image input keeps its separate 10 MiB limit. Apps can use runtime.describe to discover model.complete@2 and offer a manual path when it is absent.
  • Add a pinned Makepad Windows input-source patch. During native pointer down/up dispatch, GetCurrentInputMessageSource marks only mouse/touch/pen/touchpad messages reported as IMO_HARDWARE trusted; posted, IMO_INJECTED and IMO_SYSTEM messages remain untrusted. This enables existing host review/consent checks on Windows. Update the English and Chinese boundary documentation.

Supported behavior and limits

  • OBS Virtual Camera preview, still capture and a MiniMax image request were exercised by the user in an isolated Windows host. There is no in-app camera-source chooser in this change; physical webcams, multiple-source selection, remote-desktop input and macOS device behavior were not verified here.
  • runtime.describe reports host image support, not whether a chosen provider accepts images; providers may still refuse them. The image must already be in the calling app's storage and fit within 10 MiB.
  • IMO_HARDWARE is an OS classification, not proof of a physical person: Windows also classifies UIAccess-injected input this way. The trust change applies to Windows host-owned approvals beyond the camera sheet. Linux approval remains unsupported. Windows gallery export and video recording remain unsupported.

Verification

  • cargo test --locked -p octosense-llm-service --test complete --quiet: 23 passed.
  • cargo test --locked -p octosense-shell --features app-hub platform_services::tests --quiet: 10 passed.
  • cargo test --locked -p makepad-platform only_os_hardware_class_pointer_messages_are_trusted: 1 passed.
  • python tools/setup.py --check --cargo --hub <local sources hub> and git diff --check: passed on the combined tree.
  • Full CI and the remaining device matrix were not run locally.

Split Windows trusted pointer handling into its own parent PR, restore strict runtime tree verification, and expose generic versioned image input without app or model-name coupling. Update API docs and platform assertions.
Document the UIAccess injection caveat in both language variants, rename the source test, and restamp the locked Makepad patch.
Use the updated trusted-input baseline and its exact locked tree alongside the camera preview patch.
Preserve the upstream runtime patches and append the Windows input provenance patch with a verified locked tree.
feat: Windows 相机、图片识别与宿主点击授权

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant