Fusion4D Linux Port

This is an unofficial Linux port of the Fusion4D paper implementation you can find here.

This builds the Linux library and produces a demo executable that allows you to do non-rigid reconstruction from RGBD sequences on Linux.

Results on VolumeDeform dataset

Hoodie	Upperbody	Umbrella

1. Install System Packages

• Install CUDA Toolkit (recommended v.12.6)

• Make sure nvcc is on PATH before configuring CMake.

• Install the compiler, CMake, CUDA toolkit, OpenGL/GLUT, SuiteSparse, Ceres, JSONCPP, BLAS/LAPACK, and METIS:

sudo apt update
sudo apt install -y \
  build-essential cmake git pkg-config ccache \
  libgl1-mesa-dev libglu1-mesa-dev freeglut3-dev \
  libsuitesparse-dev libceres-dev libjsoncpp-dev \
  libblas-dev liblapack-dev libmetis-dev \

• Install VxL from source

cd Dependencies
git clone --depth 1 git@github.com:vxl/vxl.git
cd vxl
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=../vxl-install
make -j
make install

You need OpenCV with CUDA support. For more details refer to my gist guide: https://gist.github.com/Hydran00/a5b8890a01127c222d40102e28e7fc7e

3. Configure

From the repository root:

cmake -S . -B build-linux \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CUDA_ARCHITECTURES=75 \ # or adjust for your GPU
  -DPEABODY_VOLUME_CUBES_DIM_MAX=50 # or adjust based on your VRAM

If your OpenCV or VXL installs are somewhere else, pass their paths when configuring:

-DOpenCV_DIR=/path/to/opencv/build
-DVXL_DIR=/path/to/vxl-install/share/vxl/cmake

Adjust CMAKE_CUDA_ARCHITECTURES for your GPU if needed. For example, use 86 for many RTX 30-series cards.

4. Build

cmake --build build-linux --target DeformableFusionDemo -j

If ccache cannot write to its default cache directory, use temporary paths:

cmake -E env CCACHE_DIR=/tmp/ccache CCACHE_TEMPDIR=/tmp \
  cmake --build build-linux --target DeformableFusionDemo -j

5. Run Upperbody Demo

• Download VolumeDeform Dataset from here and put it in Dataset/

From build-linux:

./DeformableFusionDemo \
  --input ../Dataset/upperbody/data \ # or another dataset path
  --depth-token depth \
  --rgb-token color \
  --output ../Dataset/outputs \
  --camera-count 1 \
  --gpu 0 \
  --config ../config/OfflineFolderFusion_upperbody.cfg \
  --fx 570.342 \
  --fy 570.342 \
  --cx 320.0 \
  --cy 240.0 \
  --bbox -300,300,-300,300,-300,300 \
  --viewer3d \
  --dump-surface-every 10 \
  --dump-surface-format ply \
  --preview

The application writes F2F outputs to ../Dataset/outputs/f2f and Fusion4D outputs to ../Dataset/outputs/fusion4d.

6. Multi-camera sequences

The demo can fuse synchronized RGB-D streams directly into one shared volume. Within each token-filtered stream (depth and color), camera files for each timestamp must be consecutive after natural filename sorting:

depth_00000000_cam0.png
depth_00000000_cam1.png
color_00000000_cam0.png
color_00000000_cam1.png
depth_00000001_cam0.png
depth_00000001_cam1.png
color_00000001_cam0.png
color_00000001_cam1.png

Depth images must be single-channel 16-bit PNGs containing millimetres. RGB images must correspond to the same timestamps and camera indices. The current mesh colouring assumes that each RGB image is registered pixel-to-pixel with the depth image from the same camera.

Set the number of cameras consistently in both the configuration file and the command line:

[DepthGeneration]
DepthCameraCount=2

Calibration uses the 3DTM-style JSON structure shown in config/multicamera_setup.json. Each element of inputCameras must contain a depth object with image resolution, intrinsics, distortion coefficients, and static extrinsics.

Internally, Fusion4D expects world-to-camera extrinsics:

p_camera = R * p_world + T

If the JSON instead stores optical-camera-to-world/robot-base poses:

p_world = R_cw * p_camera + t_cw

pass --calib-camera-to-world. The demo then applies R = R_cw^T and T = -R_cw^T * t_cw. The standard 3DTM loader expects translation values in millimetres; add --calib-translation-meters when the JSON translations are expressed in metres.

Example for two cameras, run from build-linux:

./DeformableFusionDemo \
  --input ../Dataset/f1/data \
  --depth-token depth \
  --rgb-token color \
  --camera-count 2 \
  --calib ../config/multicamera_setup.json \
  --calib-camera-to-world \
  --calib-translation-meters \
  --config ../config/multicamera.cfg \
  --bbox 0,210,-70,70,-20,60 \
  --output ../Dataset/outputs \
  --gpu 0 \
  --viewer3d \
  --dump-surface-every 10 \
  --dump-surface-format ply

The bounding box is expressed in centimetres in the world/robot-base frame. Choose it around the subject rather than using extremely large bounds. The maximum representable span along one axis is approximately:

PEABODY_VOLUME_CUBES_DIM_MAX * 8 * voxel_resolution_cm

For example, the default dimension 50 with vxl_res=0.5 covers about 200 cm per axis. Increase the voxel size or rebuild with a larger PEABODY_VOLUME_CUBES_DIM_MAX if the calibrated capture volume is larger; the latter also increases GPU memory usage.

7. Notes

• Use --viewer3d only on a machine/session with OpenGL display access.
• Use --preview only when OpenCV can open UI windows.
• The default PEABODY_VOLUME_CUBES_DIM_MAX=50 is sized for the upperbody demo and avoids the large fixed TSDF allocation that can cause CUDA out-of-memory errors.

References

• Original Fusion4D implementation
• Fusion4D paper VolumeDeform paper