Skip to content

About

Virtual clothes try-on in the browser: live camera preview (MediaPipe + WebGL) + photorealistic diffusion VTON pipeline

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

1 Commit

Folders and files

Repository files navigation

Clothes Try-On

Virtual try-on in the browser: point your camera, pick a garment, see yourself wearing it.

🚧 Status: planning / early development. No working demo yet.

The project combines two modes:

  1. Live preview — real-time 2D overlay of the garment on the camera feed (MediaPipe pose + WebGL mesh warp), running entirely in the browser.
  2. Photorealistic try-on — one tap sends the best frame to a GPU pipeline built around an open-source diffusion VTON model, which returns a realistic photo of you wearing the garment.

The focus is on ML quality: rather than using a model "as is", the pipeline adds frame-quality checks, garment preparation, category-aware masking, best-of-N reranking and compositing, and every improvement is measured on a benchmark (ablation table coming soon).

Planned stack

Layer Tech
Frontend React · TypeScript · Vite · Tailwind CSS · Framer Motion
In-browser CV MediaPipe Tasks Vision (pose, segmentation) · WebGL
Backend Python · FastAPI on Modal (serverless GPU)
ML Diffusion VTON (CatVTON / IDM-VTON / FitDiT — chosen by benchmark) · DWPose · SCHP · DensePose · DINOv2
Evaluation SSIM · LPIPS · FID/KID (clean-fid) · garment fidelity (DINOv2) · ΔE color
Hosting Vercel (frontend) · Modal (API + GPU)

Repository layout

frontend/   React app: camera, live preview, UI
backend/    FastAPI API + ML try-on pipeline (deployed on Modal)
ml/         benchmark, experiment notebooks, fine-tuning
catalog/    garment catalog (catalog.json + images)
tools/      scripts for preparing garments
docs/       project plan and architecture

Documentation

Roadmap

  • Stage 0 — model benchmark & live-preview prototype
  • Stage 1 — repo skeleton, end-to-end flow (upload → result)
  • Stage 2 — ML quality pipeline + ablations
  • Stage 3 — camera mode & UI
  • Stage 4 — public demo

Privacy

User photos are never stored: they exist only in memory while the try-on is being generated.

Acknowledgements & licenses

The source code of this repository is MIT-licensed. Third-party VTON models used by the pipeline have their own licenses (many are non-commercial, e.g. CC BY-NC-SA); this project is a non-commercial portfolio project. Model credits will be listed here once the baseline is chosen.

About

Virtual clothes try-on in the browser: live camera preview (MediaPipe + WebGL) + photorealistic diffusion VTON pipeline

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors