Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 0 additions & 10 deletions .skillsrc
Original file line number Diff line number Diff line change
@@ -1,14 +1,4 @@
# Skills listed here are excluded from the top-level skills/ directory.
# One skill path per line, relative to the repo root (plugin/skills/skill-name).
# Lines starting with # are comments. Blank lines are ignored.
droid-control/skills/browser-use
droid-control/skills/capture
droid-control/skills/compose
droid-control/skills/desktop-use
droid-control/skills/droid-cli
droid-control/skills/droid-control
droid-control/skills/pty-capture
droid-control/skills/showcase
droid-control/skills/terminal-use
droid-control/skills/true-input
droid-control/skills/verify
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ Terminal, browser, and computer automation for Droids. Record demos, verify beha

**Commands:** `/demo`, `/verify`, `/qa-test`

**Skills:** `droid-control` (orchestrator), `terminal-use`, `true-input`, `browser-use`, `desktop-use`, `droid-cli`, `pty-capture`, `capture`, `compose`, `verify`, `showcase`
**Skills:** `droid-control`, which routes to its atoms: `terminal-use`, `true-input`, `browser-use`, `desktop-use`, `droid-cli`, `pty-capture`, `capture`, `compose`, `verify`, `showcase`

See [plugins/droid-control/README.md](plugins/droid-control/README.md) for details.

Expand Down
2 changes: 1 addition & 1 deletion plugins/droid-control/.factory-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"name": "droid-control",
"description": "Terminal, browser, and native desktop automation for testing, demos, QA, and computer-use tasks",
"version": "1.1.0"
"version": "1.2.0"
}
28 changes: 15 additions & 13 deletions plugins/droid-control/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ The plugin is designed to keep a droid focused while it operates real software:
- **Low context load:** load the Linux tuistory path without dragging in Windows KVM notes, macOS VM controls, browser automation, and Remotion internals.
- **Evidence-first workflows:** every command starts by making commitments, then ends by verifying the artifact against those commitments.
- **Parallel execution:** independent capture environments and render jobs can run in workers. Shared desktop input stays serialized; session names do not isolate focus.
- **Clear ownership:** commands decide *what* must be produced; atom skills decide *how* to execute their slice.
- **Clear ownership:** commands decide *what* must be produced; atoms decide *how* to execute their slice.
- **Platform specificity:** OS-specific mechanics live in platform subdocuments, not in global instructions.

## Commands are intent contracts
Expand All @@ -40,13 +40,15 @@ This is the first guardrail against agent drift. The droid does not start with "
| **Stage** | What does the workflow need? | capture, compose, verify |
| **Artifact** | Does compose need polish tools? | showcase presets, effects, keystroke overlays |

The routes compose without a cross-product explosion. Adding a new target means writing one target skill and one routing row; capture, compose, and verify can work with it immediately if the handoff shape is respected.
The routes compose without a cross-product explosion. Adding a new target means writing one target atom and one routing row; capture, compose, and verify can work with it immediately if the handoff shape is respected.

## Atom skills are runtime surfaces
## Atoms are runtime surfaces

Each atom skill is a self-contained surface the droid reads at a specific point in the workflow:
Atoms are plain files under `skills/droid-control/atoms/<atom>/ATOM.md`, not skills. Only `droid-control` appears in a session's skills list; a droid reads an atom only after routing selects it, by the absolute path the orchestrator gives it through `${DROID_PLUGIN_ROOT}`. Every installed droid-control therefore costs one skill entry of context, not eleven.

| Atom type | Skills | Responsibility |
Each atom is a self-contained surface the droid reads at a specific point in the workflow:

| Atom type | Atoms | Responsibility |
|---|---|---|
| Driver atoms | `terminal-use`, `true-input`, `browser-use`, `desktop-use` | How to drive a class of environment. `terminal-use` is the terminal entrypoint; it runs the tuistory backend and routes real-terminal proof to `true-input`. |
| Target atoms | `droid-cli`, `pty-capture` | Target-specific shortcuts, launch rules, and byte-capture patterns. |
Expand All @@ -57,7 +59,7 @@ The important property is not just smaller files. It is temporal relevance: the

## Waterfall by handoff, not framework

The workflow is a waterfall because each skill hands the next skill exactly what it needs:
The workflow is a waterfall because each atom hands the next atom exactly what it needs:

```text
command commitments
Expand Down Expand Up @@ -156,13 +158,13 @@ The key property is that the main composition is data-driven: the droid never wr
Platform-specific mechanics live below the atom that needs them:

```text
skills/true-input/platforms/linux.md
skills/true-input/platforms/windows.md
skills/true-input/platforms/macos.md
skills/pty-capture/platforms/linux.md
skills/pty-capture/platforms/windows.md
skills/pty-capture/platforms/macos.md
skills/desktop-use/SKILL.md
skills/droid-control/atoms/true-input/platforms/linux.md
skills/droid-control/atoms/true-input/platforms/windows.md
skills/droid-control/atoms/true-input/platforms/macos.md
skills/droid-control/atoms/pty-capture/platforms/linux.md
skills/droid-control/atoms/pty-capture/platforms/windows.md
skills/droid-control/atoms/pty-capture/platforms/macos.md
skills/droid-control/atoms/desktop-use/ATOM.md
```

A Linux droid reads Linux Wayland instructions. A Windows VM byte-capture task reads Windows KVM instructions. The system does not rely on the droid to skim irrelevant sections correctly.
Expand Down
4 changes: 2 additions & 2 deletions plugins/droid-control/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ Then open a Droid session and run `/demo`, `/verify`, or `/qa-test`.

For ordinary desktop work, ask directly: **“Using only cua, open Calculator and compute 17 × 23.”** Desktop-use runs the observe/act/verify loop without loading video-production stages.

The [desktop-use skill](skills/desktop-use/SKILL.md) includes setup and operating guidance. Install the `cua-driver` executable if missing; no separate Cua skill installation is needed. Installed driver versions and Wayland compositors may support different capabilities.
The [desktop-use atom](skills/droid-control/atoms/desktop-use/ATOM.md) includes setup and operating guidance. Install the `cua-driver` executable if missing; no separate Cua skill installation is needed. Installed driver versions and Wayland compositors may support different capabilities.

## Commands

Expand All @@ -72,7 +72,7 @@ Runs automated QA against terminal CLIs, web apps, or Electron apps. Accepts a U

1. **Commands** parse user intent into commitments.
2. **The orchestrator** routes by target, stage, and artifact needs.
3. **Atom skills** provide only the mechanics needed right now: drivers, target patterns, capture, compose, verify, and showcase polish.
3. **Atoms** provide only the mechanics needed right now: drivers, target patterns, capture, compose, verify, and showcase polish.
4. **Workers** handle independent capture/render jobs. The parent keeps short interactive desktop tasks, including observations, input, permission waits, and cleanup.
5. **Verify** checks the final evidence against the original commitments.

Expand Down
4 changes: 2 additions & 2 deletions plugins/droid-control/commands/demo.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,11 +59,11 @@ For simple PRs this is one sentence. For complex ones, sketch a brief table:
|-------|--------------|----------------|
| ... | ... | ... |

## Load Skills
## Load Atoms

Use the **droid-control** routing tables. Do all three lookups:

1. **Target route** -- find the row matching your target, load listed driver/target skills
1. **Target route** -- find the row matching your target, read listed driver/target atoms
2. **Stage route** -- load **capture** + **compose** + **verify** (demos always need all three)
3. **Artifact route** -- if showcase or keystroke overlay was committed, also load **showcase**

Expand Down
4 changes: 2 additions & 2 deletions plugins/droid-control/commands/qa-test.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,11 +32,11 @@ If showcase is committed, resolve the **preset** using the first matching rule:
| `minimal`, `inline`, `docs embed` | `minimal` |
| _(none of the above)_ | `macos` |

## Load Skills
## Load Atoms

Use the **droid-control** routing tables:

1. **Target route** -- find the row matching your target, load listed driver/target skills
1. **Target route** -- find the row matching your target, read listed driver/target atoms
2. **Stage route** -- load **capture** + **verify** always; load **compose** if video recording or showcase was committed
3. **Artifact route** -- if showcase committed, also load **showcase**

Expand Down
4 changes: 2 additions & 2 deletions plugins/droid-control/commands/verify.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,11 +44,11 @@ Determine the single specific behavior to observe. What would a skeptic need to
- Visual claim (rendering, layout, colors) → needs screenshot from a real compositor
- Functional claim (feature works, flow completes) → needs interaction + state verification

## Load Skills
## Load Atoms

Use the **droid-control** routing tables:

1. **Target route** -- find the row matching your target, load listed driver/target skills
1. **Target route** -- find the row matching your target, read listed driver/target atoms
2. **Stage route** -- load **capture** + **verify** always; load **compose** if video proof or showcase was committed
3. **Artifact route** -- if showcase committed, also load **showcase**

Expand Down
25 changes: 21 additions & 4 deletions plugins/droid-control/skills/droid-control/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Automate terminals, browsers, and desktop apps. Route by the user's requested me

1. **Real apps, real environments.** Non-deterministic behavior (LLM responses, network latency, variable output) is expected. Handle it with `wait` / `wait-idle`. Never substitute fixtures or mocked data.
2. **Recover from evidence.** After a failed or uncertain action, observe current state before retrying. Honor method constraints and permission boundaries; a refusal does not authorize another driver or broader target.
3. **Atoms include their references.** Load linked material on demand. Desktop-use does not require a separately installed cua skill.
3. **Atoms are files, not skills.** Read each routed atom with the Read tool at `${DROID_PLUGIN_ROOT}/skills/droid-control/atoms/<atom>/ATOM.md` (index under [Atoms](#atoms)); the Skill tool cannot load them. A bold atom name in this plugin's commands and atoms means that file. Atoms link their own references with paths relative to the atom file; read them on demand. Atom files are not expanded, so wherever one writes the DROID_PLUGIN_ROOT variable, substitute this plugin root: `${DROID_PLUGIN_ROOT}` (quote it in shell commands). Desktop-use does not require a separately installed cua skill.
4. **`tctl` owns recorded terminal sessions.** It wraps `asciinema rec` around the PTY; browser and desktop drivers own their separate lifecycles. Never call `tuistory launch` directly. Resolve `TCTL` to an absolute path only for terminal workflows or worker handoffs.
5. **Isolate every run.** Multiple droids may be filming simultaneously on the same machine. Session names and output paths share a global namespace (`/tmp/tctl-sessions/`). At the start of every workflow, generate a run ID (`RUN_ID=$(date +%s)-$$` or similar) and use it as a prefix for all session names and a scoped temp directory for all output files:
```bash
Expand All @@ -23,13 +23,30 @@ Automate terminals, browsers, and desktop apps. Route by the user's requested me
Never use bare session names like `-s demo`, `-s before`, `-s after` — they will collide with concurrent runs.
Separate names and paths do not isolate shared desktop focus or keyboard input. Keep one controller for a visible desktop.

## Atoms

Each atom lives at `${DROID_PLUGIN_ROOT}/skills/droid-control/atoms/<atom>/ATOM.md`.

| Atom | Scope |
|---|---|
| **terminal-use** | Terminal TUI driver: tuistory virtual PTY by default, true-input for real terminal proof |
| **true-input** | Real terminal emulator driver via a headless Wayland compositor or VM |
| **browser-use** | Web page and Electron app driver via agent-browser |
| **desktop-use** | Native GUI app driver via trycua cua-driver |
| **droid-cli** | Droid CLI target patterns, shortcuts, modes, and launch helpers |
| **pty-capture** | Ground-truth byte sequences from real terminal emulators |
| **capture** | Recording lifecycle for terminal and browser sessions |
| **compose** | Video assembly via Remotion: title cards, layout, transitions, effects, and showcase polish |
| **verify** | Deliverable verification against commitments |
| **showcase** | Visual polish: Remotion window chrome, animations, and branded backgrounds |

## Routing

Three independent lookups. Do all three, then load the union of skills they produce.
Three independent lookups. Do all three, then read the union of atoms they produce.

### 1. Target route — what are you driving?

| Target | Load these skills |
| Target | Read these atoms |
|---|---|
| User explicitly requests cua-only, native GUI input, or desktop control (including Electron) | **desktop-use**; method constraints override the defaults below |
| Droid CLI (`droid-dev`, `droid exec`) | **terminal-use** + **droid-cli** |
Expand All @@ -44,7 +61,7 @@ Three independent lookups. Do all three, then load the union of skills they prod

Every workflow passes through stages. Load the atoms for each stage you'll use.

| Stage | Skill | When to load |
| Stage | Atom | When to read |
|---|---|---|
| Capture | **capture** | Recording, scripted multi-step evidence, or a demo/QA deliverable; ordinary desktop operation uses the driver's observe/verify loop |
| Compose | **compose** | When the deliverable is a produced artifact (video, annotated screenshots, comparison image) |
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,3 @@
---
name: browser-use
description: Background knowledge for droid-control workflows -- not invoked directly. Browser-use driver mechanics for web page and Electron desktop app automation via agent-browser.
user-invocable: false
---

# Browser Use

The orchestrator routed you here. Execute the browser portion of its action
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,3 @@
---
name: capture
description: Background knowledge for droid-control workflows -- not invoked directly. Recording lifecycle for terminal and browser sessions.
user-invocable: false
---

# Capture

The orchestrator routed you here. This atom owns the full recording lifecycle: launch a target, execute an interaction script, collect raw outputs.
Expand All @@ -21,7 +15,7 @@ The command that invoked you should have provided:

## Recording lifecycle

For desktop-use, follow its [recording contract](../desktop-use/SKILL.md#recording); do not translate the terminal commands below into desktop commands. Routine desktop snapshots stay in the driver's observe/act/verify loop.
For desktop-use, follow its [recording contract](../desktop-use/ATOM.md#recording); do not translate the terminal commands below into desktop commands. Routine desktop snapshots stay in the driver's observe/act/verify loop.

### 1. Pre-flight

Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,3 @@
---
name: compose
description: Background knowledge for droid-control workflows -- not invoked directly. Video assembly via Remotion — title cards, layout, transitions, effects, and showcase polish.
user-invocable: false
---

# Compose

This atom owns the full video assembly pipeline. You receive raw outputs from the **capture** stage and produce a single polished artifact. Follow the pipeline below step by step.
Expand Down Expand Up @@ -286,7 +280,7 @@ Keep it short — aim for ≤ 15 lines per card, hold for 3–6 seconds.

### Transition styles

`transitionStyle` selects the title→content and content→outro crossfade presentation. Both transitions in one render share the same style. `flash` and `light-leak` derive their tint from the preset palette. Default `motion-blur` is always safe; preset-tier guidance lives in `showcase/SKILL.md`.
`transitionStyle` selects the title→content and content→outro crossfade presentation. Both transitions in one render share the same style. `flash` and `light-leak` derive their tint from the preset palette. Default `motion-blur` is always safe; preset-tier guidance lives in `../showcase/ATOM.md`.

| Style | Feel | Use when… |
|---|---|---|
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,3 @@
---
name: desktop-use
description: Background knowledge for droid-control workflows -- not invoked directly. Desktop-use driver mechanics for native GUI app automation via trycua cua-driver.
user-invocable: false
---

# Desktop Use

One controller operates an exact GUI target, observes each effect, and stops when the user's postcondition is proved.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,12 +1,6 @@
---
name: droid-cli
description: Background knowledge for droid-control workflows -- not invoked directly. Droid CLI target patterns, shortcuts, modes, and launch helpers.
user-invocable: false
---

# Droid CLI Target

The orchestrator routed you here. Layer these target-specific patterns on top of the driver skill you already loaded.
The orchestrator routed you here. Layer these target-specific patterns on top of the driver atom you already read.

Droid-specific shortcuts, modes, and launch patterns.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,3 @@
---
name: pty-capture
description: Background knowledge for droid-control workflows -- not invoked directly. Capture ground-truth byte sequences from real terminal emulators.
user-invocable: false
---

# PTY Byte Capture

The orchestrator routed you here. Use these mechanics to execute your plan.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,3 @@
---
name: showcase
description: Background knowledge for droid-control workflows -- not invoked directly. Visual polish for videos via Remotion-powered window chrome, animations, and branded backgrounds.
user-invocable: false
---

# Showcase Polish

This atom describes the visual polish system. It is invoked by the **compose** atom — you should not need to invoke it directly. Load it when you need to understand what the presets look like and how the cinematic layers work.
Expand Down Expand Up @@ -67,15 +61,15 @@ Palette is auto-selected based on preset. Factory/factory-hero use the warm pale

## Transition styles

`transitionStyle` selects the crossfade presentation. Schema lives in `compose/SKILL.md`; preset-tier matching:
`transitionStyle` selects the crossfade presentation. Schema lives in `../compose/ATOM.md`; preset-tier matching:

| Preset | Recommended (default first) | Avoid |
|---|---|---|
| `factory`, `factory-hero` | `motion-blur`, `light-leak`, `whip-pan`, `flash` | `glitch-lite` (clashes with warm tone) |
| `hero`, `presentation` | `motion-blur`, `whip-pan`, `flash` | `light-leak` (warm sweep clashes with cool palette) |
| `macos`, `minimal` | `motion-blur` | `glitch-lite`, `light-leak` (too much personality for utilitarian frames) |

`codeAnnotations` is preset-agnostic — palette and font stack are auto-derived. See `compose/SKILL.md` for schema and authoring rules.
`codeAnnotations` is preset-agnostic — palette and font stack are auto-derived. See `../compose/ATOM.md` for schema and authoring rules.

## Operational notes

Expand All @@ -86,13 +80,13 @@ Palette is auto-selected based on preset. Factory/factory-hero use the warm pale
- Missing clips in `public/`: render fails with "Could not read file." The render script stages clips into its own per-render directory; never run `npx remotion render` directly.
- Missing npm dependencies: run `cd ${REMOTION_DIR} && npm install` if rendering fails on first use.

**Debugging layout**: `render-showcase.sh --still <frame>` renders one frame through the same normalization and staging as a full render (see compose/SKILL.md Step 3).
**Debugging layout**: `render-showcase.sh --still <frame>` renders one frame through the same normalization and staging as a full render (see `../compose/ATOM.md` Step 3).

**Cleanup**: `render-showcase.sh` removes only the staged directory it created, on success, failure, or cancellation via Ctrl-C / process-group signal. A signal to the script's PID alone is deferred until the `npx remotion` child exits.

## Rendering

Use the render script from **compose** — see compose/SKILL.md Step 3 for full usage:
Use the render script from **compose** — see `../compose/ATOM.md` Step 3 for full usage:

```bash
RENDER=${DROID_PLUGIN_ROOT}/scripts/render-showcase.sh
Expand Down
Loading
Loading