Training time and computational budget for EmbodiedSplat

[akang-wang]

Hi Authors,

Thank you for this amazing work and for open-sourcing the project! The integration of online feed-forward semantics with 3D Gaussian Splatting for open-vocabulary scene understanding is highly impressive.

I am planning to reproduce your results and potentially extend the work. To help me plan our computational budget, could you kindly share some details regarding the training time? Specifically:

Total Training Time: Approximately how long does the entire training process take to converge?

Thank you again for your time and contribution to the community.

Best regards,
Akang Wang

[0nandon]

Thanks for your interest on our work!

As we mentioned in the implementation details of the main paper, we train the whole model with single NVIDIA RTX 6000 Ada (48GB) GPU with batch size 1. 1 batch size may already consume roughly 42GB GPU memory.

Since we train with a batch size of 1 on a single GPU, the overall training time is relatively long. The main reason we use a single GPU is due to MinkowskiEngine; based on my experience, enabling multi-GPU DDP training with MinkowskiEngine is quite tricky.

The detailed training time is below:

• Stage 1 (Single view): 3-4 days
• Stage 2 (Multi-views): 5-6 days

However, please note that training is only required when you use 3D CLIP features. The biggest contribution of this paper is how we bind the CLIP features to each Gaussian in memory-efficient way where we propose CLIP global codebook and online sparse coefficient fields. Those module is training-free approach adopted on top of pretrained feed-forward model.

Hence, EmbodiedSplat-fast is training-free built on top of pretrained FreeSplat++, since it only uses 2D CLIP features. And using 2D features alone already shows quite good performance. Combining 3D CLIP features lead further performance improvements in mIoU metric by following the spirit of OpenScene while it requires additional training.