CtrlVTON: Controllable Virtual Try-On via Visual-Instance-Prompt Segmentation

Seungyong Lee, Hyun Jun Jang, Sangoh Kim, Sungjoon Park

CtrlVTON recasts virtual try-on as an image-editing problem and adds pixel-level spatial control through segmentation masks, enabling fine-grained control over garment size, style, and placement — across single- and multi-garment scenarios.

News

• [Coming Soon] Visit the project page for more details and results.
• [2026-07-10] 📝 CtrlVTON is now on arXiv.
• [2026-07-10] 🤗 The VITON-HD-edit dataset is available on Hugging Face.

Overview

Virtual try-on (VTO) has made significant progress in realistically transferring garments onto a target person, yet most systems offer little control over how a garment should be worn — its size (loose or fitted), style (tucked or untucked, open or closed), and spatial placement on the body. CtrlVTON addresses this gap with two complementary contributions:

• VIP-SAM — We define and solve Visual-Instance-Prompt Segmentation: given a flatlay image of a garment, segment that specific instance in a photograph of a person wearing it. This is an instance-level task, distinct from the typically studied category-level visual-reference segmentation. VIP-SAM injects support features into the SAM backbone at early stages, enabling robust segmentation under same-category distractors, heavy occlusion, and non-rigid deformation between flatlay and on-body imagery.
• CtrlVTON — A controllable VTO framework that recasts try-on as an image editing problem (rather than inpainting) and adds segmentation masks as pixel-level control over garment layout. Built as a lightweight LoRA on top of an editing diffusion transformer, CtrlVTON unifies garment swapping, layering, selective switching, and multi-garment composition within one framework.

Results

Visual-Instance-Prompt Segmentation (VIP-SAM)

Existing visual-reference segmentation methods often include surrounding garments or the entire body, whereas VIP-SAM consistently isolates only the queried instance — even under same-category distractors and heavy occlusion.

Fine-grained Control via Mask Conditioning

Given the same garment and person reference, CtrlVTON produces diverse outputs by varying only the input mask — tucking, buttoning, sleeve length, layering, sizing, and more.

Mask-Controllable Try-On vs. Proprietary Editing Models

Compared against the strongest proprietary editing systems (Nano Banana Pro, GPT Image 1.5, Seedream 4.5, FLUX.2 [pro]), CtrlVTON follows hand-drawn control masks far more faithfully while matching them on garment fidelity. Each output is overlaid with per-image mask-adherence metrics $(IoU, d_{\text{Hu}}, d_H)$.

Dataset: VITON-HD-edit (HF dataset)

Training the editing model requires triplets $(p, p_{\text{ref}}, g_{\text{ref}})$ that no public dataset provides. We build a data pipeline that synthesizes the reference person $p_{\text{ref}}$ from a source pair $(p, g_{\text{ref}})$ and pairs it with garment-instance masks produced by VIP-SAM.

Applying this pipeline to the full VITON-HD test set (2,032 images) yields VITON-HD-edit, a public benchmark that supports three evaluations: image-editing VTO, instance-level visual-prompt segmentation, and spatial controllability.

What's in the release

VITON-HD-edit augments the official VITON-HD test set with three new fields contributed by our pipeline (highlighted in yellow below):

VITON-HD-edit sample.

Field	Source	Description
image-edit	VITON-HD-edit	Synthesized reference person $p_{\text{ref}}$ — same identity, pose, and background as image, but wearing a different garment.
image-edit_cloth-mask	VITON-HD-edit	VIP-SAM mask of the swapped-in garment in image-edit, i.e. $M_{p_{\text{ref}}}$.
gt_cloth_warped_mask	VITON-HD-edit	VIP-SAM mask of the reference garment on the target person, i.e. $M_p$.
image, cloth, cloth-mask, agnostic-v3.2, agnostic-mask, …	VITON-HD	Standard VITON-HD test-set fields.

📦 Download from 🤗 NXN-Labs/VITON-HD-edit.

Citation

@misc{lee2026ctrlvton,
      title={CtrlVTON: Controllable Virtual Try-On via Visual-Instance-Prompt Segmentation}, 
      author={Seungyong Lee and Hyun Jun Jang and Sangoh Kim and Sungjoon Park},
      year={2026},
      eprint={2607.09362},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.09362}, 
}

License

This project is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

For commercial use, please visit NXN Labs.

Acknowledgments

All human model and clothing images are either free-license images from the internet, drawn from public datasets (VITON-HD, DressCode, DressCode-MR, Garments2Look, OmniTry Bench), or generated. All images and brands are the property of their respective owners.