DVPSFormer: Efficient Online Depth-aware Video Panoptic Segmentation for Autonomous Driving

DVPSFormer: Efficient Online Depth-aware Video Panoptic Segmentation for Autonomous Driving
Yung-Hsu Yang, Luigi Piccinelli, Siyuan Li, Mattia Segu, Lei Ke, Martin Danelljan, Yuqian Fu, Zuria Bauer, Fisher Yu, Hermann Blum, Marc Pollefeys
arXiv 2026, Paper at arXiv 2607.26165

News and ToDo

• 31.07.2026: Release code and models.

Qualitative Results

Getting Started

We use Vis4D as the framework to implement DVPSFormer. Please check the document for more details.

Installation

We support Python 3.11+ and PyTorch 2.7.1+. Please install the correct PyTorch version according to your own hardware settings.

conda create -n dvpsformer python=3.11 -y

conda activate dvpsformer

# Install Vis4D
pip install torch==2.7.1 torchvision==0.22.1 --index-url https://download.pytorch.org/whl/cu126
pip install vis4d==1.0.0 --index-url https://download.pytorch.org/whl/cu126

# Install CUDA ops
pip install git+https://github.com/SysCV/vis4d_cuda_ops.git --no-build-isolation --no-cache-dir

# Install panoptic API
pip install git+https://github.com/cocodataset/panopticapi.git

# Install DVPSFormer
pip install -v -e .

Data Preparation

We follow PolyphonicFormer to prepare the datasets and pack them into HDF5 format.

Model Zoo

• Cityscapes-DVPS: https://huggingface.co/RoyYang0714/dvpsformer_r50_cityscapes-dvps
• SemKITTI-DVPS: https://huggingface.co/RoyYang0714/dvpsformer_r50_semkitti-dvps

Testing

# Cityscapes-DVPS
vis4d test --config dvpsformer/zoo/dvpsformer/dvpsformer_r50_cityscapes_dvps.py --gpus 8 --ckpt https://huggingface.co/RoyYang0714/dvpsformer_r50_cityscapes-dvps/resolve/main/dvpsformer_r50_cityscapes-dvps.pt

# SemKITTI-DVPS
vis4d test --config dvpsformer/zoo/dvpsformer/dvpsformer_r50_semkitti_dvps.py --gpus 1 --ckpt https://huggingface.co/RoyYang0714/dvpsformer_r50_semkitti-dvps/resolve/main/dvpsformer_r50_semkitti-dvps.pt

It is worth noting that since the SemKITTI-DVPS validation set is only one sequence, so it can only be evaluated using single GPU.

Training

We follow previous methods and have multi-stage training.

First download the Mapilary pre-trained weights here and save it to vis4d-workspace/checkpoints/mask2former_r50_mv.pkl.

Cityscapes

Download the Cityscapes data.

data
└── cityscapes
    ├── leftImg8bit
    └── gtFine

Train the PS model on Cityscapes to get ${YOUR_PS_CKPT_PATH}.

vis4d fit --config dvpsformer/zoo/mask2former/mask2former_r50_90k_cityscapes.py --gpus 8

Cityscapes-DVPS

Use the pre-train PS model ${YOUR_PS_CKPT_PATH} for DPS training and get ${YOUR_DPS_CKPT_PATH}.

vis4d fit --config dvpsformer/zoo/dvpsformer/mask2former_r50_depth_cityscapes_dvps.py --gpus 8 --ckpt ${YOUR_PS_CKPT_PATH}

Train the tacking module wtih ${YOUR_DPS_CKPT_PATH}.

vis4d fit --config dvpsformer/zoo/dvpsformer/dvpsformer_r50_cityscapes_dvps.py --gpus 8 --ckpt ${YOUR_DPS_CKPT_PATH}

SemKITTI-DVPS

Use the pre-train PS model ${YOUR_PS_CKPT_PATH} for DPS training and get ${YOUR_DPS_CKPT_PATH}.

vis4d fit --config dvpsformer/zoo/dvpsformer/mask2former_r50_depth_semkitti_dvps.py --gpus 8 --ckpt ${YOUR_PS_CKPT_PATH}

Train the tacking module with ${YOUR_DPS_CKPT_PATH}.

vis4d fit --config dvpsformer/zoo/dvpsformer/dvpsformer_r50_semkitti_dvps.py --gpus 8 --ckpt ${YOUR_DPS_CKPT_PATH}

Note that for SemKITTI-DVPS, we don't do validation for multi-gpu training.

Visualization

You will see the dumped segmenation masks, depth maps, and colored point cloud under the output folder.

# Cityscapes-DVPS
vis4d test --config dvpsformer/zoo/dvpsformer/dvpsformer_r50_cityscapes_dvps.py --gpus 8 --ckpt https://huggingface.co/RoyYang0714/dvpsformer_r50_cityscapes-dvps/resolve/main/dvpsformer_r50_cityscapes-dvps.pt --vis

# SemKITTI-DVPS
vis4d test --config dvpsformer/zoo/dvpsformer/dvpsformer_r50_semkitti_dvps.py --gpus 1 --ckpt https://huggingface.co/RoyYang0714/dvpsformer_r50_semkitti-dvps/resolve/main/dvpsformer_r50_semkitti-dvps.pt --vis

Citation

If you find our work useful in your research please consider citing our publications:

@misc{yang2026dvpsformerefficientonlinedepthaware,
      title={DVPSFormer: Efficient Online Depth-aware Video Panoptic Segmentation for Autonomous Driving}, 
      author={Yung-Hsu Yang and Luigi Piccinelli and Siyuan Li and Mattia Segu and Lei Ke and Martin Danelljan and Yuqian Fu and Zuria Bauer and Fisher Yu and Hermann Blum and Marc Pollefeys},
      year={2026},
      eprint={2607.26165},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.26165}, 
}