Virtual try-on in the browser: point your camera, pick a garment, see yourself wearing it.
🚧 Status: planning / early development. No working demo yet.
The project combines two modes:
- Live preview — real-time 2D overlay of the garment on the camera feed (MediaPipe pose + WebGL mesh warp), running entirely in the browser.
- Photorealistic try-on — one tap sends the best frame to a GPU pipeline built around an open-source diffusion VTON model, which returns a realistic photo of you wearing the garment.
The focus is on ML quality: rather than using a model "as is", the pipeline adds frame-quality checks, garment preparation, category-aware masking, best-of-N reranking and compositing, and every improvement is measured on a benchmark (ablation table coming soon).
| Layer | Tech |
|---|---|
| Frontend | React · TypeScript · Vite · Tailwind CSS · Framer Motion |
| In-browser CV | MediaPipe Tasks Vision (pose, segmentation) · WebGL |
| Backend | Python · FastAPI on Modal (serverless GPU) |
| ML | Diffusion VTON (CatVTON / IDM-VTON / FitDiT — chosen by benchmark) · DWPose · SCHP · DensePose · DINOv2 |
| Evaluation | SSIM · LPIPS · FID/KID (clean-fid) · garment fidelity (DINOv2) · ΔE color |
| Hosting | Vercel (frontend) · Modal (API + GPU) |
frontend/ React app: camera, live preview, UI
backend/ FastAPI API + ML try-on pipeline (deployed on Modal)
ml/ benchmark, experiment notebooks, fine-tuning
catalog/ garment catalog (catalog.json + images)
tools/ scripts for preparing garments
docs/ project plan and architecture
- Project plan (in Russian)
- Architecture (in Russian)
- Stage 0 — model benchmark & live-preview prototype
- Stage 1 — repo skeleton, end-to-end flow (upload → result)
- Stage 2 — ML quality pipeline + ablations
- Stage 3 — camera mode & UI
- Stage 4 — public demo
User photos are never stored: they exist only in memory while the try-on is being generated.
The source code of this repository is MIT-licensed. Third-party VTON models used by the pipeline have their own licenses (many are non-commercial, e.g. CC BY-NC-SA); this project is a non-commercial portfolio project. Model credits will be listed here once the baseline is chosen.