PVCPM

Scene Semantics-Guided Probabilistic Pedestrian Pre-Collision Motion Prediction

简体中文 | English

Abstract

Pre-collision motion prediction aims to forecast the emergency posture and subsequent motion evolution of pedestrians under imminent vehicle-collision risk. This task is highly passive and uncertain, and is tightly coupled with complex traffic scenarios. The unconstrained diverse predictions generated by existing human motion prediction methods may therefore be inconsistent with realistic collision scenarios. To address this issue, we construct PVCPM, a pedestrian pre-collision motion prediction dataset specifically tailored to traffic scenarios. Moreover, we analyze its motion characteristics using the discrete cosine transform (DCT), revealing pronounced high-frequency differences between PVCPM and conventional daily motion datasets. Furthermore, we propose a scene semantics-guided probabilistic motion prediction framework based on a two-stage Conditional Variational Autoencoder (CVAE). The proposed framework decouples motion generation from diverse sampling and incorporates scene semantic features extracted by a Vision-Language Model (VLM). These features are used to guide motion sequence modulation through Feature-wise Linear Modulation (FiLM). Our work considers both the unique dynamic characteristics of pedestrian pre-collision motion and realistic semantic traffic contexts, enabling diverse predictions that remain consistent with real-world traffic scenarios. Extensive experiments on PVCPM demonstrate that the proposed method achieves competitive performance compared with representative baseline and related state-of-the-art methods, validating its effectiveness for probabilistic pedestrian pre-collision motion prediction.

---

Environment Setup

Our code and environment are inherited from the DivSamp project. You can also follow that project to configure the environment. The environment uses python 3.12, PyTorch 2.10.0+cu128, and requirements.txt.

conda create -n PVCPM_env python=3.12
conda activate PVCPM_env

pip install torch==2.10.0 torchvision==0.25.0 torchaudio==2.10.0 --index-url https://download.pytorch.org/whl/cu128
pip install -r requirements.txt

Data Download

• Download the dataset and pretrained ckpt folders from the following link: PVCPM
• Download Qwen3.5-9B with:

huggingface-cli download --resume-download Qwen/Qwen3.5-9B --local-dir vlm_model/Qwen/Qwen3.5-9B

📂 Project Directory Structure (click to expand/collapse)

PVCPM
├── dataset
│   └── pvcp
│       ├── data_3d_pvcp_video_m35.npz
│       ├── pvcp_scene_splits_8_2.json
│       ├── data_3d_pvcp_video_m35_Qwen3.5-9B_ped_pose_ego_vehicle_hist10_d512.rfeat.audit.jsonl
│       ├── data_3d_pvcp_video_m35_Qwen3.5-9B_ped_pose_ego_vehicle_hist10_d512.rfeat.npz
│       ├── build_qwen_ped_pose_features_10.py
│       ├── frame
│       └── SMPLX_models
│           ├── SMPL_NEUTRAL.npz
│           └── SMPLX_NEUTRAL.npz
├── vlm_model
│   └── Qwen
│       └── Qwen3.5-9B
├── ckpt
│   ├── pvcp_video_t1
│   └── pvcp_video_t2
├── pvcp_video
│   ├── configs
│   ├── datas
│   ├── nets
│   ├── runs
│   └── inference.py
├── main.py
├── run_main.sh
├── run_main_t1.sh
├── run_main_t2.sh
├── run_inference_vis_pvcp_video.sh
├── requirement.txt
├── assets
└── README.md

Train + Eval

CUDA_VISIBLE_DEVICES=0 SEED=44 bash run_main.sh # Stage-1 + Stage-2 + evaluate

or

CUDA_VISIBLE_DEVICES=0 SEED=44 bash run_main_t1.sh # Stage-1

CUDA_VISIBLE_DEVICES=0 SEED=44 STAGE1_CKPT_NAME=pvcp_video_t1 bash run_main_t2.sh # Stage-2 + evaluate

or

python main.py --exp_name=pvcp_video_t1 --is_train=1 --seed=44  # Stage-1

python main.py --exp_name=pvcp_video_t2 --is_train=1 --seed=44 \
                --model_path_t1="ckpt/pvcp_video_t1/models/pvcp_video_t1_best.pth"  # Stage-2

python main.py --exp_name=pvcp_video_t2 --is_load=1 --seed=44 \
                --model_path="ckpt/pvcp_video_t2/models/pvcp_video_t2_best.pth"  # evaluate

Inference + Visualization

First, configure AMASS-skeleton-to-SMPL and place it in the same parent directory as PVCPM.

CUDA_VISIBLE_DEVICES=0 SORT_PREDICTIONS_BY_GT=1 SEED=44 bash run_inference_vis_pvcp_video.sh

📣 Our paper is currently under review...