ACE-Ego-Hand: Repurposing Video Diffusion Models for
Occlusion-Robust Egocentric 3D Hand Motion Recovery

Yufei Liu^1,4, Xixi Wang², Hao Li^3,4, Ganlong Zhao^3,4, Kaitong Cai⁴, Chengkai Jin^2,4, Chunxiao Liu⁴, Jianbo Liu⁴, Siyuan Huang⁴ †, Xingang Pan², Hongsheng Li^3,4 ✉

¹SJTU   ²NTU   ³CUHK   ⁴ACE Robotics    _{† Project leader   ✉ Corresponding author}

Bimanual 3D hand motion — MANO pose, shape and camera-space translation, per frame — from egocentric RGB video, through occlusion and out-of-sight gaps, with no external hand detector.

📦 Code: the code is hosted at github.com/ggxxii/ACE-Ego-Hand. It ships inference only; training code is not included yet (see the TODO). Every command below runs from a clone of that repository.

Requirements

git clone https://github.com/ggxxii/ACE-Ego-Hand.git
cd ACE-Ego-Hand

conda create -n ace-ego-hand python=3.10 && conda activate ace-ego-hand
pip install -r requirements.txt

The Wan backbone is loaded through VideoX-Fun. Clone it to third_party/:

git clone https://github.com/aigc-apps/VideoX-Fun third_party

MANO drives the translation decode. Get the official release (mano.is.tue.mpg.de, research licence) and convert it once — the official pickles embed chumpy, which no longer imports on modern numpy, so the converter rewrites them as plain-numpy pickles:

python scripts/convert_mano_pkls.py --src /path/to/mano_v1_2/models --dst data/mano/mano

Download checkpoints

Download Wan2.2-Fun-5B-Control from the VideoX-Fun release to ckpt/Wan2.2-Fun-5B-Control/ (needs config.json, diffusion_pytorch_model.safetensors, Wan2.2_VAE.pth, models_t5_umt5-xxl-enc-bf16.pth and google/umt5-xxl/), then precompute the fixed caption embedding once:

python scripts/precompute_caption.py

Ours come from 🤗 acerobotics2025/ACE-Ego-Hand and go in checkpoints/:

model	checkpoint	camera intrinsics
K-given	ace_ego_hand_k.pt	required	download
K-free	ace_ego_hand_kfree.pt	not needed	download

⚠️ Pinhole only. The two released checkpoints do not support native fisheye input. To run on fisheye footage, undistort it to a pinhole view first and pass the pinhole intrinsics.

pip install -U "huggingface_hub[cli]"
hf download acerobotics2025/ACE-Ego-Hand --local-dir checkpoints

Run on a video

# K-free: no camera parameters needed at all
python infer_video.py --video clip.mp4 \
    --opt options/ace_ego_hand_kfree.yml --ckpt checkpoints/ace_ego_hand_kfree.pt \
    --out results/clip

# K-given: pass the camera calibration
python infer_video.py --video clip.mp4 --camera cam.json \
    --opt options/ace_ego_hand_k.yml --ckpt checkpoints/ace_ego_hand_k.pt \
    --out results/clip

Frames wider than 832 px are downscaled to 832; narrower videos are encoded at their native size, never upscaled (--encode_w sets the cap).

cam.json looks like this — intrinsics are taken as constant over the clip, in the calibration's own pixel grid, and are rescaled automatically if the video resolution differs (--intrinsics fx,fy,cx,cy is an inline shorthand):

{"image_width": 1280, "image_height": 720,
 "frames": [{"intrinsics": {"fx": 736.6, "fy": 736.6, "cx": 640.0, "cy": 360.0}}]}

The predictions land in one pickle per clip. claim (F, 2) marks the frames where each hand is visible (exists_2d > 0.5); pose arrays are NaN elsewhere.

To see them, render an overlay (--mesh shades the MANO mesh through a built-in CPU rasterizer, so no GPU renderer is needed):

python scripts/viz_preds.py --pred_dir results/clip --video clip.mp4 \
    --out results/clip_viz --mesh

TODO

• arXiv preprint
• Project page
• Inference code
• Pretrained checkpoints — K-given and K-free
• Evaluation code
• Fisheye support
• Training code and data preparation scripts

Acknowledgement

Built on VideoX-Fun / Wan and smplx.

License

The code is MIT (see LICENSE). The checkpoints are CC BY-NC 4.0 — they derive from MANO and research-licensed datasets (ARCTIC, HOT3D, H2O, OakInk2, FreiHAND, RHD, Re:InterHand), so commercial use is not permitted. The Wan weights, VideoX-Fun and MANO remain under their own licences.

BibTeX

@misc{liu2026aceegohand,
    title={ACE-Ego-Hand: Repurposing Video Diffusion Models for Occlusion-Robust Egocentric 3D Hand Motion Recovery},
    author={Yufei Liu and Xixi Wang and Hao Li and Ganlong Zhao and Kaitong Cai and Chengkai Jin and Chunxiao Liu and Jianbo Liu and Siyuan Huang and Xingang Pan and Hongsheng Li},
    year={2026},
    eprint={2608.20308},
    archivePrefix={arXiv},
    primaryClass={cs.CV},
    url={https://arxiv.org/abs/2608.20308},
}