MotionBridge

A generative-video motion-control compiler for Apple Silicon.

MotionBridge takes an existing video and compiles temporally synchronized control assets — masks, isolated subjects, motion, pose, depth and camera tracks — that let Seedance 2.5 recreate the original performance and camera move while the character, product, wardrobe or environment changes. Everything runs locally with local computer vision; no proprietary code or private APIs.

VIDEO → shot understanding → object identification → temporal tracking → motion
      → pose → depth → camera → synchronized control renders → Seedance 2.5 package

Built for an M4 Max MacBook Pro (Metal / MLX / PyTorch-MPS). The main workflow never needs CUDA or an NVIDIA GPU.

Demo

▶ Watch the full 58-second demo — a walkthrough of the real app (import, selection, tracking, corrections, analysis, references, export) followed by the controls it produced for the clip above, playing side by side.

Everything in it was generated by the commands in this README, on one 345-frame 1080p clip, on a MacBook Pro. The individual control videos are attached to the latest release if you want to inspect them frame by frame, and the prompt it generated is docs/example-seedance-prompt.txt.

What it produces

For a selected subject or object, frame-locked to the source at its native resolution:

Output	What it carries
subject_green.mp4 / object_green.mp4	the isolated subject on a key colour (green/blue/black)
subject_rgba.mov	ProRes 4444 with a real alpha channel
mask · edit_mask · protect_mask · silhouette	region controls, with expansion and feather
motion_debug.mp4 + motion_tracks.json	trajectory, velocity, scale, rotation, occlusion per frame
pose_overlay.mp4 + pose.json	body keypoints for the tracked subject only
depth.mp4 + 16-bit depth_frames/	depth normalized once per shot (no flicker)
camera_motion_preview.mp4 + camera_track.json	camera move, separated from subject motion
seedance_prompt.txt + manifest.json	the prompt, the @VideoN order, and Motion Lock weights

Every one of those videos is ffprobe-verified against the master: same fps, same frame count, same duration, same resolution. Zero frames of drift.

One frame of a 345-frame clip: pose (top-left), depth (top-right), motion with tracked IDs, scale and velocity (bottom-left), and the camera analysis (bottom-right).

Requirements

• Apple Silicon Mac (M-series), macOS 14+
• Homebrew, Python 3.11, uv, FFmpeg, Node + pnpm
• Blender (optional, for Phase 3 clay renders)
• ~2 GB of disk for the default models, downloaded on request

Quick start

git clone https://github.com/Kayariyan28/motionbridge.git
cd motionbridge
./install.sh      # verifies the toolchain, installs Python + UI dependencies (no models)
./dev.sh          # backend on :8765, UI on :5173, opens the browser

Then open Settings → Models and download what you need. Nothing downloads silently.

Model	Size	Role	License
SAM 3 · 8-bit (MLX)	1.0 GB	text / concept selection ("person", "man on left", "red car")	SAM License
SAM 2.1 Hiera-L (MPS)	0.9 GB	clicks, boxes, video tracking, corrections	Apache-2.0
ViTPose base (MPS)	0.34 GB	body keypoints — default pose model	Apache-2.0
Depth Anything V2 Small (MPS)	0.1 GB	depth	Apache-2.0
ViTPose+ base / huge	0.5 / 3.6 GB	higher-accuracy pose	Apache-2.0

CLI equivalent: backend/.venv/bin/motionbridge models download vitpose-base.

Workflow

1. Import an MP4/MOV/M4V. MotionBridge probes it, normalizes variable frame rate to a constant-rate ProRes working_master.mov, builds analysis proxies and a browser preview, and detects hard cuts. Every later asset takes its timing from that master. Optionally conform the master to a different frame rate here — Seedance 2.5 rejects references outside 24–60 fps, so 23.976 fps footage needs it (details).
2. Select the subject: type what to find (person, man on left, watch), click it, alt-click to exclude, or drag a box.
3. Track Through Shot — memory-based tracking with an optical-flow guard, inside the shot only, never across a cut.
4. Fix and propagate. Correct a bad frame with points, a box, paint or erase, then propagate forward or backward. Propagation stops at the next keyframe and as soon as it converges with the previous result, so one fix never re-tracks the whole shot.
5. Analyze — motion, pose, depth and camera, one model resident at a time.
6. References — auto-extract the best frames of the subject (front / three-quarter / side, sharp, unobstructed, de-duplicated) or upload your own character, wardrobe, product or environment images.
7. Motion Lock — set how much of the body, hands, head, object and camera motion to preserve, and how much of identity, wardrobe and environment to take from references.
8. Export for Seedance — builds MotionBridge_Project/ and validates every file.

Using the package with Seedance 2.5

This is what the whole tool exists for. Seedance 2.5 can generate a new video from reference videos and images — but only if what you feed it carries the motion, timing and camera move of the original. That is the package MotionBridge builds.

Sources: BytePlus ModelArk / Volcano Engine Ark docs for dreamina-seedance-2-5-260628 and doubao-seedance-2-5-260628, plus the Seedance 2.5 product page, read 2026-09-17. Limits change — check the docs before trusting the numbers below.

The rules that shape the package

Rule	Value
Reference assets per request	50 total: 30 images + 10 videos + 10 audio
Total reference video length	30 seconds combined — this is the binding limit
Length of each reference video	2–30 s (4–30 s when the task type is edit)
Frame rate of each reference video	24–60 fps
Size of each reference video	300–6000 px per side, aspect ratio 0.4–2.5
Reference video format	mp4 or mov, H.264/AVC or H.265/HEVC, ≤200 MB each
Reference image format	jpeg/png/webp/etc., ≤30 MB each
How assets are sent	video: URL or asset id (never Base64); images may be Base64. Whole request body ≤64 MB
Referring to an asset in the prompt	"the performer in Video 1" — type + number, never @Video1 and never a raw asset id
Numbering	an asset's position among assets of the same type, in upload order
Not supported on 2.5	camera_fixed and frames — those are Seedance 1.x parameters

The 30-second cap is the one that bites. Every MotionBridge control is exactly as long as your clip, so an 11.5-second clip only fits two reference videos. The generated prompt does this arithmetic for you and names the two to upload.

Will Seedance accept your clip?

The per-video limits apply to every control, because every control inherits the master's timing and size. Two of them catch people out:

• A clip under 2 seconds cannot be a reference video at all. Shoot or trim to at least 2 s (4 s if you are using an edit task).

• 23.976 fps is below the 24 fps floor. That is the standard NTSC film rate, so plenty of footage lands there. Conform it on import: pick a rate in Frame rate on the import screen, or pass --fps 24.

  motionbridge analyze take_03.mov --fps 24

  A small rate change (≤5%, e.g. 23.976 → 24) is a conform: every source frame is kept and only the timestamps move, exactly like a film conform, so nothing is invented and the duration shifts by the same 0.1%. A larger change (30 → 24) is a resample, which duplicates or drops frames to keep the duration — the app says which one it did. Either way the master becomes the new timing authority and every control inherits the conformed rate.

MotionBridge checks this twice, so you find out before you upload rather than after: at import (a warning under Source in the left panel) and at export, where clip_compatibility writes the verdict to the top of prompts/seedance_prompt.txt and to metadata/manifest.json under seedance.clip_compatibility:

CLIP COMPATIBILITY — Seedance would REJECT these videos as references:
- clip is 1.50s; a reference video must be at least 2s (reference task). Use a longer clip.
- clip is 23.976 fps; reference video must be 24-60 fps. Conform the master to 24 or 25 fps before exporting.
- Every control shares the master's timing and size, so this applies to all of them.

What to upload, by goal

Goal	Upload (in this order)	Plus images
Motion transfer / character replacement	source.mp4, then subject_green.mp4 (or clay_body.mp4 in Phase 3)	character references
Wardrobe swap	source.mp4, subject_green.mp4	wardrobe references
Environment replacement	source.mp4, depth.mp4	environment reference
Object swap	source.mp4, object_green.mp4	product references
Camera-motion preservation	source.mp4	— (describe the move; camera_track.json gives you the words)

Step by step

1. Run the export: Export for Seedance in the UI, or motionbridge export project.mb --mode motion-transfer.
2. Open prompts/seedance_prompt.txt. It lists the upload order and flags if you are over the 30-second budget.
3. In your Seedance 2.5 surface — Dreamina / CapCut, Jimeng, Doubao Pro, or the API — upload the videos in that order, then the reference images.
4. Paste the prompt. Adjust the Motion Lock wording if you want less of the original kept.
5. Generate. The performance, timing and camera move should survive; the look changes.

Honest limits

• Masks are not a Seedance input. Nothing in the documented API accepts a spatial mask. edit_mask / protect_mask are for your own compositing, and for tools that do take masks. In Seedance, restrict the edit in words ("change only the watch on her left wrist").
• Pose and motion overlays are diagnostics, not references. ByteDance explicitly warns against reference frames containing "trajectory lines, coordinate lines, camera cones, or similar visual interference" — which is exactly what pose_overlay.mp4 and motion_debug.mp4 look like. MotionBridge therefore leaves them out of the upload list. They remain genuinely useful for checking the tracking, and for other pipelines (ControlNet-style conditioning, Blender, compositing).
• Documented structural controls are clay/white-model and green-screen tracks. That is why clay_body.mp4 (Phase 3) is the highest-value control this tool can give you, and why subject_green.mp4 is the best available one today.
• Reference-video resolution: the model page says 480p/720p while a generic page says up to 4K. Assume 720p for reference videos until that is resolved; your controls are rendered at source resolution, so downscale before uploading if needed.
• Nobody has verified end-to-end results here. MotionBridge guarantees the inputs: correct content, correct timing, zero frame drift. What Seedance does with them is its own model's behaviour, and consumer UIs (Dreamina/CapCut) do not document their own per-upload limits.

Architecture

frontend/          React + TypeScript + Vite + Tailwind (dark editing UI)
backend/motionbridge/
  video/           ffprobe, CFR normalization, frame-exact FFmpeg I/O, proxy, shot detection
  providers/       interfaces + SAM 3 (MLX) · SAM 2.1 (MPS) · ViTPose · Depth Anything · classical
  tracking/        TrackEngine (flow guard, re-anchor, occlusion, keyframes), masks, motion analysis
  pose/ depth/ camera/   Phase 2 stages and their controls
  export/          mask derivation, control rendering, references, prompts, package, validate_sync
  pipeline/        job queue, event bus, single-model slot, perf monitor, service layer
  models/          Model Manager (explicit downloads, statuses, sizes, licenses)
  api/ cli.py      FastAPI + WebSocket, Typer CLI
tests/             synthetic ground-truth footage, tracking, camera, depth, API, sync tests

Design rules

1. Frame-lock is non-negotiable. One clock: working_master.mov. Every render is validated by validate_sync (fps, frame count, duration, resolution).
2. One model resident at a time, through ModelSlot, which unloads the previous model and frees MLX/MPS caches before loading the next.
3. No silent downloads. Providers load only what is already cached and raise ModelNotInstalled otherwise.
4. Optional features degrade, never crash. A failed stage is reported and the rest of the chain continues.
5. No faked results. If a control cannot be produced honestly, it is omitted and listed in manifest.json under missing.

Model choices (measured, not assumed)

• SAM 3 (MLX) handles text prompts. MotionBridge post-processes the detector itself, because mlx-vlm 0.7's Sam3Predictor swaps width/height for numpy input.
• SAM 2.1 (PyTorch-MPS) handles clicks and tracking: mlx-vlm 0.7's SAM 3 tracker head returns unusable masks (IoU ≈ 0 against synthetic ground truth). bfloat16 on MPS is 3.4 fps versus 2.4 fp32 with mask IoU 0.998 between them.
• ViTPose is the default pose model, not MediaPipe. MediaPipe Tasks 1.0.1 aborts the process on macOS 26 (Check failed: service_ Service is unavailable in its Metal helper) even with the CPU delegate — and SIGABRT cannot be caught, so it must not run in the backend. It stays available behind MOTIONBRIDGE_ENABLE_MEDIAPIPE=1. ViTPose is also a better fit: it is top-down, so MotionBridge feeds it the tracked mask's box and the pose provably belongs to the tracked subject.
• Depth Anything V2 per frame, with MotionBridge's own shot-wide alignment for temporal stability (see below).

Why depth doesn't flicker

Monocular depth models predict every frame on its own arbitrary scale, so playing raw output back pulses. MotionBridge fits a robust scale+shift (median/MAD) from each frame onto a slowly adapting per-shot reference, then normalizes the entire shot once. Genuine depth changes still track; frame-to-frame noise does not.

Frame-perfect synchronization

backend/.venv/bin/python -m motionbridge.export.validate_sync \
    source/working_master.mov controls/*.mp4 [--decode]

Compares FPS, frame count, duration and resolution; tolerance is 1 frame and the goal is 0.

CLI

motionbridge analyze video.mp4            # add --fps 24 to conform the master's rate
motionbridge track video.mp4 --text "person"          # or --point x,y / --box x1,y1,x2,y2
motionbridge analyze-stages project.mb --steps motion,pose,depth,camera
motionbridge export project.mb --mode motion-transfer  # renders controls + builds the package
motionbridge models list
motionbridge validate-sync working_master.mov controls/*.mp4

Tests

cd backend
.venv/bin/python -m pytest                     # fast, model-free
MB_RUN_SLOW=1 .venv/bin/python -m pytest       # + real SAM 3 / SAM 2.1 on synthetic and human footage

The suite generates its own footage with known ground truth — a moving white circle with a hard cut, and a textured background panning at a known rate behind a subject moving the other way — then asserts that masks follow the circle within 2.5 px, that the camera analyzer recovers the known pan while ignoring the subject, that depth alignment removes per-frame scale flicker, and that input and output frame counts, fps and duration match exactly.

Projects

Default location ~/Movies/MotionBridge/<name>_<id>/, override with MOTIONBRIDGE_HOME: source/ cache/ tracks/ geometry/ controls/ references/ exports/ project.json <name>.mb. Close and reopen any time — nothing is recomputed, because stages are keyed by content hash.

Roadmap

• Phase 1 — done: import, playback, timeline, selection, tracking, corrections, green screen, masks, sync validation.
• Phase 2 — done: motion tracks, pose, depth, camera analysis, reference manager, Motion Lock, Seedance prompt and package export.
• Phase 3: Blender clay mannequin driven by pose + depth, Blender camera from the camera track, white-model renders.
• Phase 4: Object Swap mode, occlusion masks, advanced reference controls.
• Phase 5: VGGT sparse reconstruction, optional SAM 3D providers, clean plate.

License

Apache-2.0 — see LICENSE and NOTICE.

MotionBridge does not redistribute model weights. Models are downloaded by you from their original sources and keep their own licenses; SAM weights are covered by Meta's SAM License, and some depth and geometry checkpoints are non-commercial. You are responsible for complying with the license of every model you download.