I kept running into this error when training skin for frame 1
( running the guitar dataset with undistorted images.)
Output folder: /media/dual_output/guitar_test/ [06/11 16:59:10]
Reading camera 81/81 [06/11 16:59:10]
Loading Training Cameras [06/11 16:59:10]
Loading Test Cameras [06/11 16:59:13]
[16:59:13] loading motion from control point init.py:84 [06/11 16:59:13]
[16:59:14] Loading ply file : /media/dual_output/guitar_test/ckt/point_cloud_0.ply gaussian_model.py:290 [06/11 16:59:14]
number of gaussians: 215771 [06/11 16:59:14]
Training progress: 0%| | 0/15000 [00:00<?, ?it/s]Traceback (most recent call last):
File "/media/DualGS/train.py", line 383, in
training_skin(lp.extract(args), op.extract(args), pp.extract(args), lossp2, args.test_iterations, args.debug_from, is_start_frame, frame_idx, args=args)
File "/media/DualGS/train.py", line 235, in training_skin
loss.backward()
File "/home/anaconda3/envs/dualgs/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward
torch.autograd.backward(
File "/home/anaconda3/envs/dualgs/lib/python3.10/site-packages/torch/autograd/init.py", line 251, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: CUDA error: invalid configuration argument
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
Training progress: 0%| | 0/15000 [00:01<?, ?it/s]
Any possible cause or solution?
Any help will be very much appreciated!! thanks!!
I kept running into this error when training skin for frame 1
( running the guitar dataset with undistorted images.)
Output folder: /media/dual_output/guitar_test/ [06/11 16:59:10]
Reading camera 81/81 [06/11 16:59:10]
Loading Training Cameras [06/11 16:59:10]
Loading Test Cameras [06/11 16:59:13]
[16:59:13] loading motion from control point init.py:84 [06/11 16:59:13]
[16:59:14] Loading ply file : /media/dual_output/guitar_test/ckt/point_cloud_0.ply gaussian_model.py:290 [06/11 16:59:14]
number of gaussians: 215771 [06/11 16:59:14]
Training progress: 0%| | 0/15000 [00:00<?, ?it/s]Traceback (most recent call last):
File "/media/DualGS/train.py", line 383, in
training_skin(lp.extract(args), op.extract(args), pp.extract(args), lossp2, args.test_iterations, args.debug_from, is_start_frame, frame_idx, args=args)
File "/media/DualGS/train.py", line 235, in training_skin
loss.backward()
File "/home/anaconda3/envs/dualgs/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward
torch.autograd.backward(
File "/home/anaconda3/envs/dualgs/lib/python3.10/site-packages/torch/autograd/init.py", line 251, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: CUDA error: invalid configuration argument
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with
TORCH_USE_CUDA_DSAto enable device-side assertions.Training progress: 0%| | 0/15000 [00:01<?, ?it/s]
Any possible cause or solution?
Any help will be very much appreciated!! thanks!!