Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 19 additions & 38 deletions .github/workflows/droid-control-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,70 +6,51 @@ on:
- 'plugins/droid-control/bin/tctl'
- 'plugins/droid-control/scripts/render-showcase.sh'
- 'plugins/droid-control/tests/**'
- 'plugins/droid-control/remotion/**'
- 'plugins/droid-control/fframes/**'
- '.github/workflows/droid-control-tests.yml'
push:
branches: [master]
paths:
- 'plugins/droid-control/bin/tctl'
- 'plugins/droid-control/scripts/render-showcase.sh'
- 'plugins/droid-control/tests/**'
- 'plugins/droid-control/remotion/**'
- 'plugins/droid-control/fframes/**'
- '.github/workflows/droid-control-tests.yml'

env:
PYTHONDONTWRITEBYTECODE: 1
CARGO_TERM_COLOR: always

jobs:
render-helper:
name: render-showcase.sh behavior
showcase:
name: Showcase composition, render helper, real renders
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: Install ffmpeg and agg
- name: Install ffmpeg, agg, and the fframes build dependencies
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get install -y --no-install-recommends ffmpeg
sudo apt-get install -y --no-install-recommends ffmpeg clang libclang-dev libx264-dev
curl -fsSL -o agg https://github.com/asciinema/agg/releases/download/v1.9.0/agg-x86_64-unknown-linux-gnu
echo "f111e315cd71056b116302342553dd765b7297579ed511f111d0cedb442aeda6 agg" | sha256sum -c -
sudo install -m 0755 agg /usr/local/bin/agg

- name: Run helper integration tests
run: python3 -m unittest discover -v -s plugins/droid-control/tests -p 'test_*.py'

remotion:
name: Remotion typecheck, duration tests, real render
runs-on: ubuntu-latest
defaults:
run:
working-directory: plugins/droid-control/remotion
steps:
- uses: actions/checkout@v4

- uses: actions/setup-node@v4
- uses: dtolnay/rust-toolchain@stable
with:
# node --test with built-in type stripping (22.18+); Remotion runtime itself needs >= 18
node-version: 22
cache: npm
cache-dependency-path: plugins/droid-control/remotion/package-lock.json
components: clippy, rustfmt

- run: npm ci

- run: npx tsc --noEmit -p .

- run: npm test
- uses: Swatinem/rust-cache@v2
with:
workspaces: plugins/droid-control/fframes

- name: Install ffmpeg and the Remotion browser
- name: Format, lint, and unit tests
working-directory: plugins/droid-control/fframes
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get install -y --no-install-recommends ffmpeg
npx remotion browser ensure
cargo fmt --check
cargo clippy --all-targets --release -- -D warnings
cargo test --release

- name: Render a synthetic clip through render-showcase.sh and check the encoded file
working-directory: .
env:
RENDER_SHOWCASE_E2E: 1
run: python3 -m unittest discover -v -s plugins/droid-control/tests -p 'test_*.py' -k RealRender
- name: Helper integration tests and real renders
run: python3 -m unittest discover -v -s plugins/droid-control/tests -p 'test_*.py'
2 changes: 0 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,2 @@
.factory/
node_modules/
# Per-render clip staging created by render-showcase.sh; only survives a SIGKILL
plugins/droid-control/remotion/public/render-*/
42 changes: 21 additions & 21 deletions plugins/droid-control/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Editable source: [`diagrams/architecture-routing.excalidraw`](diagrams/architect

The plugin is designed to keep a droid focused while it operates real software:

- **Low context load:** load the Linux tuistory path without dragging in Windows KVM notes, macOS VM controls, browser automation, and Remotion internals.
- **Low context load:** load the Linux tuistory path without dragging in Windows KVM notes, macOS VM controls, browser automation, and renderer internals.
- **Evidence-first workflows:** every command starts by making commitments, then ends by verifying the artifact against those commitments.
- **Parallel execution:** independent capture environments and render jobs can run in workers. Shared desktop input stays serialized; session names do not isolate focus.
- **Clear ownership:** commands decide *what* must be produced; atoms decide *how* to execute their slice.
Expand Down Expand Up @@ -91,7 +91,7 @@ The parent droid keeps judgment. Workers get exact commands.
| Interpret PR / claim / QA goal | Parent | Requires context and judgment. |
| Write the interaction script | Parent | Defines the proof story. |
| Capture baseline and candidate branches | Workers only for independent environments | A shared desktop must be captured serially. |
| Render Remotion video | Worker droid | Mechanical once props and clips are fixed. |
| Render showcase video | Worker droid | Mechanical once props and clips are fixed. |
| Verify commitments | Parent | Requires the original contract and evidence judgment. |

This boundary follows the stage handoffs. Capture workers need resolved `tctl` commands and worktree paths, not PR context. Render workers need a props JSON and clip paths, not a feature explanation.
Expand Down Expand Up @@ -126,32 +126,32 @@ Browser/Electron and native-desktop workflows intentionally do **not** go throug

## Video composition

The compose stage uses the Remotion project in `remotion/` as a single video engine. The droid writes a `Showcase` props JSON; `scripts/render-showcase.sh` handles the mechanical rendering pipeline:
The compose stage uses the [fframes](https://github.com/dmtrKovalenko/fframes) project in `fframes/` as a single video engine: a Rust crate whose `droid-showcase` binary renders the `Showcase` composition from SVG frames. The droid writes a `Showcase` props JSON; `scripts/render-showcase.sh` builds the binary on first use (cargo) and runs it. The binary owns the mechanical rendering pipeline:

1. Accept `.cast`, `.mp4`, and `.webm` clips only; normalize props and resolve fidelity (omitted: side-by-side `inspect`, single `standard`).
1. Accept `.cast`, `.mp4`, and `.webm` clips only; validate props and resolve defaults and fidelity (omitted: side-by-side `inspect`, single `standard`).
2. Convert `.cast` recordings through `agg` and `ffmpeg` at 1x, keeping the recording's timeline.
3. Stage clips as `clip-<index>` inside a directory created under Remotion `public/` for this render only.
4. Set `clipDuration` to the longest clip with `ffprobe`.
5. Render the `Showcase` composition as limited-range `yuv420p`/`bt709` H.264, or one frame with `--still`.
6. Remove that render's staged directory and conversion outputs on exit.
3. Stage clips as `clip-<index>` inside a work directory created for this render only.
4. Probe every clip with `ffprobe`; the longest one sets the content length.
5. Print the resolved plan (`showcase plan: {...}` on stderr), then render every frame on all cores and encode limited-range `yuv420p`/`bt709` H.264 through `ffmpeg`, or one PNG frame with `--still`.
6. Remove that render's work directory on exit, failure, or SIGINT/SIGTERM.

`remotion/src/lib/duration.ts` owns the timeline: the composition applies `speed` once to every clip, the clips run for `clipDuration / speed`, and the content sequence is padded by one crossfade on each side so the clips start after the title crossfade and the final frame is held through the outro crossfade. Total length is `4s title + clipDuration / speed + 3.5s outro`; a shorter clip holds its final frame. This keeps droids out of the common failure modes: two `recording.mp4` inputs overwriting each other in `public/`, concurrent renders deleting each other's clips, mismatched `clipDuration`, casts sped up twice, wrong `agg` theme, invalid pixel formats, and hand-written Remotion commands with missing encode flags.
`fframes/src/timing.rs` owns the timeline: the composition applies `speed` once to every clip, the clips run for `longest clip / speed`, and the content sequence is padded by one crossfade on each side so the clips start after the title crossfade and the final frame is held through the outro crossfade. Total length is `4s title + longest clip / speed + 3.5s outro`; a shorter clip holds its final frame. This keeps droids out of the common failure modes: two `recording.mp4` inputs overwriting each other, concurrent renders deleting each other's clips, a mismatched clip duration, casts sped up twice, wrong `agg` theme, invalid pixel formats, and hand-written encode commands with missing flags.

### Composition surface

The `Showcase` composition in `remotion/src/compositions/Showcase.tsx` is the only video entry point. Everything else lives in `remotion/src/components/` and is composed by props:
`fframes/src/showcase.rs` is the only video entry point (`Showcase`, an fframes `Video`). It composes the layer modules by props:

| Layer | Purpose | Controlled by |
|---|---|---|
| Background + FloatingParticles | Preset-driven warmth or coolness | `preset` |
| TitleCard / DroidOutro | Opening and closing cards (outro plays fanning rotor → crossfade → DROID wordmark) | `title`, `subtitle`, `speedNote` |
| Window chrome + layouts | `SingleLayout` or `SideBySideLayout` | `layout`, `labels`, `objectFit` |
| ZoomEffect / SpotlightOverlay / KeystrokeOverlay / SectionHeader | Timed in-scene overlays | `effects`, `keys`, `sections` |
| CodeAnnotationOverlay | Timed syntax-highlighted code cards | `codeAnnotations` |
| Transition presentation | Title→content and content→outro crossfade | `transitionStyle` (default `motion-blur`) |
| NoiseOverlay + ColorGradeOverlay + Watermark | Topmost polish pass | `fidelity`, `preset` |
| Layer | Module | Purpose | Controlled by |
|---|---|---|---|
| Background + particles | `scenery.rs` | Preset-driven warmth or coolness | `preset` |
| Title card / Droid outro | `title.rs`, `outro.rs` | Opening and closing cards (outro plays fanning rotor → crossfade → DROID wordmark) | `title`, `subtitle`, `speedNote` |
| Window chrome + layouts | `content.rs` | Single or side-by-side windows, panel entrances, zooms, spotlights | `layout`, `labels`, `windowTitle`, `objectFit`, `effects` |
| Callouts / section sweeps and headers / keystrokes | `overlays.rs` | Timed in-scene overlays | `effects`, `sections`, `keys` |
| Code cards | `overlays.rs`, `code.rs` | Timed syntax-highlighted code cards | `codeAnnotations` |
| Transition presentation | `transition.rs` | Title→content and content→outro crossfade | `transitionStyle` (default `motion-blur`) |
| Noise + colour grade + watermark | `scenery.rs` | Topmost polish pass | `fidelity`, `preset` |

The key property is that the main composition is data-driven: the droid never writes Remotion JSX. Adding a new overlay or transition style is a new component plus a schema field, not a new composition.
The key property is that the main composition is data-driven: the droid never writes composition code. Adding a new overlay or transition style is a new layer function plus a props field, not a new composition. Fonts (Geist, Geist Mono) are embedded in the binary, so renders look the same on every machine.

## Platform isolation

Expand Down Expand Up @@ -182,7 +182,7 @@ Use the same composition rules when adding capability:
| New user workflow | Add a command that parses arguments into commitments, then routes through existing atoms. |
| New target type | Add one target atom and one target-route row. |
| New capture backend | Add a driver atom or extend `tctl` only if it belongs behind the same terminal boundary. |
| New visual treatment | Add Remotion props/schema support and document compose/showcase behavior. |
| New visual treatment | Add props support and a layer in `fframes/src/`, then document compose/showcase behavior. |
| New platform mechanics | Add a `platforms/<os>.md` file under the relevant atom. |

If a change makes every droid read more global instructions, it is probably fighting the architecture. Prefer a new scoped surface over a larger shared surface.
Expand Down
10 changes: 5 additions & 5 deletions plugins/droid-control/NOTICES.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,10 @@ This plugin depends on several third-party tools and libraries. Executables are

## Video rendering

- **[Remotion](https://www.remotion.dev/)** -- React-based video renderer used by the compose/showcase pipeline. Remotion is free for individuals, small teams (<=3 employees), and non-profits. Larger companies require a [company license](https://www.remotion.pro/). See the [full license terms](https://github.com/remotion-dev/remotion/blob/main/LICENSE.md).
- **[React](https://react.dev/)** -- MIT License
- **[Zod](https://zod.dev/)** -- MIT License
- **[prism-react-renderer](https://github.com/FormidableLabs/prism-react-renderer)** -- MIT License. Powers the syntax highlighting in the `CodeAnnotationOverlay` component.
- **[fframes](https://github.com/dmtrKovalenko/fframes)** (MIT) -- Rust SVG-based video framework that renders the `fframes/` showcase composition. Its SVG stack, [svgr/usvgr](https://github.com/dmtrKovalenko/fframes), is MPL-2.0; its clip decoder statically links [FFmpeg](https://ffmpeg.org/) (LGPL-2.1+) through `ffmpeg-sys-fframes` (WTFPL).
- **[Geist and Geist Mono](https://github.com/vercel/geist-font)** -- SIL Open Font License 1.1, embedded in the renderer from `fframes/media/` (license text in `fframes/media/GEIST-OFL.txt`).
- **[syntect](https://github.com/trishume/syntect)** (MIT) and **[two-face](https://github.com/CosmicHorrorDev/two-face)** (MIT OR Apache-2.0) -- syntax highlighting for code annotations.
- **[clap](https://github.com/clap-rs/clap)**, **[serde](https://serde.rs/)**, **[signal-hook](https://github.com/vorner/signal-hook)**, **[tempfile](https://github.com/Stebalien/tempfile)** -- MIT OR Apache-2.0.

## Terminal automation

Expand All @@ -31,4 +31,4 @@ This plugin depends on several third-party tools and libraries. Executables are

## Design influences

- **[@hyperframes/shader-transitions](https://github.com/heygen-com/hyperframes/tree/main/packages/shader-transitions)** (Apache-2.0) -- the `transitionStyle` prop's naming and taxonomy (`whip-pan`, `light-leak`, `flash`, `glitch-lite`) was shaped by Hyperframes' shader-transitions catalog. Implementations in `ShowcaseTransition.tsx` are original Remotion-native CSS/SVG overlays, not GLSL ports.
- **[@hyperframes/shader-transitions](https://github.com/heygen-com/hyperframes/tree/main/packages/shader-transitions)** (Apache-2.0) -- the `transitionStyle` prop's naming and taxonomy (`whip-pan`, `light-leak`, `flash`, `glitch-lite`) was shaped by Hyperframes' shader-transitions catalog. Implementations in `fframes/src/transition.rs` are original SVG filter and overlay effects, not GLSL ports.
13 changes: 7 additions & 6 deletions plugins/droid-control/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,9 +39,9 @@ droid plugin marketplace add https://github.com/Factory-AI/factory-plugins
# Install the plugin
droid plugin install droid-control@factory-plugins --scope user

# Install Remotion dependencies (one-time, only needed for video rendering)
# Build the video renderer (one-time, only needed for video rendering; needs Rust, clang, libx264)
# Find the plugin install path with: droid plugin list --scope user
cd <plugin-path>/remotion && npm install
cargo build --release --manifest-path <plugin-path>/fframes/Cargo.toml
```

Or use the `/plugins` UI: Browse tab, select droid-control, install.
Expand Down Expand Up @@ -80,9 +80,9 @@ For the full rationale and runtime pipeline, see [`ARCHITECTURE.md`](ARCHITECTUR

## Video rendering

The compose stage uses [Remotion](https://www.remotion.dev/) for video compositing. Presets provide window chrome, spacing, palettes, backgrounds, particles, noise, color grading, configurable transitions (`motion-blur`, `flash`, `whip-pan`, `light-leak`, `glitch-lite`), zooms, spotlights, callout annotations, keystroke overlays, section headers, and syntax-highlighted code annotations.
The compose stage uses [fframes](https://github.com/dmtrKovalenko/fframes), a Rust SVG video framework, for video compositing. Presets provide window chrome, spacing, palettes, backgrounds, particles, noise, color grading, configurable transitions (`motion-blur`, `flash`, `whip-pan`, `light-leak`, `glitch-lite`), zooms, spotlights, callout annotations, keystroke overlays, section headers, and syntax-highlighted code annotations.

The `render-showcase.sh` helper owns the full pipeline: `.cast` conversion via `agg`, per-render clip staging, longest-clip duration, Remotion rendering (or a `--still` preview), and cleanup. Playback `speed` is applied once by the composition to every clip.
The `render-showcase.sh` helper owns the full pipeline: `.cast` conversion via `agg`, per-render clip staging, longest-clip duration, rendering on every core (or a `--still` preview), and cleanup. It builds the `droid-showcase` binary from `fframes/` on first use. Playback `speed` is applied once by the composition to every clip.

## Prerequisites

Expand All @@ -95,7 +95,7 @@ The `render-showcase.sh` helper owns the full pipeline: `.cast` conversion via `
| browser-use | All | `agent-browser` |
| desktop-use | All | `cua-driver` |
| compose | All | `ffmpeg`, `ffprobe`, `agg` |
| showcase | All | Node.js (>= 18), Chrome/Chromium |
| showcase | All | Rust (stable, via rustup), clang/libclang, libx264 |

```bash
npm install -g tuistory # virtual PTY driver
Expand All @@ -104,7 +104,8 @@ cargo install --git https://github.com/asciinema/agg # .cast -> .gif converter
sudo apt-get install -y ffmpeg # video processing
agent-browser install # browser automation (downloads Chromium)
curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.sh | bash # native desktop GUI automation
cd plugins/droid-control/remotion && npm install # Remotion video rendering
sudo apt-get install -y clang libclang-dev libx264-dev # renderer build deps (macOS: xcode-select --install; brew install x264)
cargo build --release --manifest-path plugins/droid-control/fframes/Cargo.toml # video renderer
```

Only install what you need, with approval. Terminal demos need tuistory, asciinema, agg, and ffmpeg. Web/Electron automation defaults to browser-use; an explicit cua-only/native-input request uses desktop-use instead. Native desktop automation needs cua-driver plus the graphical session and OS permissions reported by its preflight. Recording and rendering have additional dependencies; they are not required for ordinary desktop tasks.
Original file line number Diff line number Diff line change
Expand Up @@ -985,15 +985,15 @@
"textAlign": "center",
"verticalAlign": "middle",
"containerId": "compose",
"originalText": "Compose\nRemotion props\nrender-showcase",
"originalText": "Compose\nShowcase props\nrender-showcase",
"autoResize": true,
"lineHeight": 1.25,
"id": "t_compose",
"x": 338,
"y": 758,
"width": 194,
"height": 104,
"text": "Compose\nRemotion props\nrender-showcase"
"text": "Compose\nShowcase props\nrender-showcase"
},
{
"angle": 0,
Expand Down
Loading
Loading