openclaw-computer-use

A reusable macOS visual computer-use skill for OpenClaw.

English | 简体中文

Turn OpenClaw into a practical visual computer-use agent for native macOS apps.

It gives OpenClaw a lightweight native-app control layer with:

• full-screen and region screenshots
• OCR via Apple Vision
• annotated OCR boxes
• text search and OCR-driven clicking
• frontmost app/window inspection
• mouse move, click, drag, scroll
• typing and key combos
• a small helper bridge for foreground screenshot capture when needed

This project is designed to be cloned as a single skill directory and dropped into an OpenClaw workspace or managed skills folder.

Why this exists

Most OpenClaw workflows are strong on browser automation, but weaker when the target is a native desktop window.

This repo fills that gap with a minimal visual-control layer for macOS:

• screen capture
• OCR
• annotated visual grounding
• coordinate and text-driven mouse actions
• keyboard input
• frontmost app and window inspection

Use it when the task is happening in Terminal, Finder, Codex Desktop, Electron apps, or any other native UI that does not expose reliable DOM refs.

Who this is for

Use this if you want OpenClaw to inspect and control native macOS apps such as Codex Desktop, Terminal, Finder, or Electron apps.

For websites, keep using openclaw browser ... whenever possible. DOM refs are still more reliable than pixel clicks.

Requirements

• macOS
• Python 3.9+
• osascript
• swiftc
• OpenClaw installed and working
• Accessibility permission for the process that runs OpenClaw gateway
• Screen Recording permission for the process that runs OpenClaw gateway

Python dependency:

python3 -m pip install -r requirements.txt

Quick install

Clone the repo first:

git clone https://github.com/your-name/openclaw-computer-use.git
cd openclaw-computer-use

Option 1: install into the current workspace.

./install.sh --workspace /path/to/your/workspace

Option 2: install into OpenClaw managed skills.

./install.sh --managed

That copies this repository into one of these locations:

• workspace mode: /path/to/your/workspace/skills/openclaw-computer-use
• managed mode: ~/.openclaw/skills/openclaw-computer-use

Repo layout

openclaw-computer-use/
├── SKILL.md
├── install.sh
├── requirements.txt
├── scripts/
│   ├── computer_use.py
│   ├── capture_bridge.py
│   └── smoke_test.py
└── references/
    └── architecture.md

Permission model

The key macOS permissions must be granted to the app or process that actually runs OpenClaw gateway.

Typical setups:

• If you run gateway in Terminal: grant Terminal
• If you run gateway in iTerm: grant iTerm
• If you run gateway as a launchd service: the safest practical setup is often to run gateway in a foreground Terminal session instead

You usually need:

• Screen Recording
• Accessibility
• sometimes Automation for System Events

Core commands

python3 scripts/computer_use.py permissions
python3 scripts/computer_use.py app-info
python3 scripts/computer_use.py screen --path /tmp/openclaw-cu.png
python3 scripts/computer_use.py ocr --path /tmp/openclaw-cu.png --annotate
python3 scripts/computer_use.py find-text "Settings" --path /tmp/openclaw-cu.png
python3 scripts/computer_use.py click-text "Settings"
python3 scripts/computer_use.py move 800 500
python3 scripts/computer_use.py click 800 500
python3 scripts/computer_use.py drag 800 500 1100 500
python3 scripts/computer_use.py scroll 900 700 down --amount 300
python3 scripts/computer_use.py type "hello"
python3 scripts/computer_use.py press "cmd+a"

Recommended OpenClaw workflow

1. Inspect the frontmost app/window.
2. Capture a screenshot.
3. Run OCR and inspect annotated boxes.
4. Use click-text or coordinates.
5. Re-capture after important steps.

Optional screenshot helper bridge

Some macOS setups can capture screenshots only from a foreground user session. This repo includes scripts/capture_bridge.py as a tiny local HTTP helper.

Run it in a foreground Terminal if needed:

python3 scripts/capture_bridge.py

Then computer_use.py can use it automatically through:

export OPENCLAW_CU_HELPER_URL=http://127.0.0.1:8765

Smoke test

The smoke test exercises:

• permissions
• screenshot capture
• OCR
• annotated OCR output
• text matching, nearest-text, and crop
• browser-assisted open / snapshot / fill flow
• drag and scroll primitives

Run:

python3 scripts/smoke_test.py

Notes:

• It expects OpenClaw browser tooling to be available.
• It is intended for a local macOS desktop session.

Security and privacy

This public repo is intentionally designed to avoid shipping machine-specific secrets.

It should not contain:

• local absolute home-directory paths in docs or scripts, except generic runtime discovery
• API keys, gateway tokens, or certificates
• personal logs, screenshots, or OCR artifacts
• local OpenClaw session state
• machine-specific .openclaw or .codex home-directory paths

Before publishing, you can run a simple audit:

python3 scripts/release_audit.py .

License

MIT