RuntimeError: CUDA error: device-side assert triggered

[tl123987]

lidar_aug_matrix[:3,:3] = common_utils.angle2matrix(torch.tensor(noise_rot)) @ lidar_aug_matrix[:3,:3]
../aten/src/ATen/native/cuda/Loss.cu:95: operator(): block: [0,0,0], thread: [0,0,0] Assertion target_val >= zero && target_val <= one failed.
../aten/src/ATen/native/cuda/Loss.cu:95: operator(): block: [0,0,0], thread: [29,0,0] Assertion target_val >= zero && target_val <= one failed.
epochs: 0%| | 0/100 [00:02<?, ?it/s]
Traceback (most recent call last):
File "/home/q/桌面/OpenPCDet/tools/train.py", line 238, in
main()
File "/home/q/桌面/OpenPCDet/tools/train.py", line 183, in main
train_model(
File "/home/q/桌面/OpenPCDet/tools/train_utils/train_utils.py", line 180, in train_model
accumulated_iter = train_one_epoch(
File "/home/q/桌面/OpenPCDet/tools/train_utils/train_utils.py", line 56, in train_one_epoch
loss, tb_dict, disp_dict = model_func(model, batch)
File "/home/q/桌面/OpenPCDet/tools/../pcdet/models/init.py", line 44, in model_func
ret_dict, tb_dict, disp_dict = model(batch_dict)
File "/home/q/anaconda3/envs/openpecd/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/q/anaconda3/envs/openpecd/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/home/q/桌面/OpenPCDet/tools/../pcdet/models/detectors/point_rcnn.py", line 14, in forward
loss, tb_dict, disp_dict = self.get_training_loss()
File "/home/q/桌面/OpenPCDet/tools/../pcdet/models/detectors/point_rcnn.py", line 27, in get_training_loss
loss_rcnn, tb_dict = self.roi_head.get_loss(tb_dict)
File "/home/q/桌面/OpenPCDet/tools/../pcdet/models/roi_heads/roi_head_template.py", line 223, in get_loss
rcnn_loss_cls, cls_tb_dict = self.get_box_cls_layer_loss(self.forward_ret_dict)
File "/home/q/桌面/OpenPCDet/tools/../pcdet/models/roi_heads/roi_head_template.py", line 217, in get_box_cls_layer_loss
tb_dict = {'rcnn_loss_cls': rcnn_loss_cls.item()}
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

data iswrong?

[tl123987]

[Petros626]

@tl123987 it seems not to be an OpenPCDet related issue, did you installed it after the guide? Try this CUDA_LAUNCH_BLOCKING=1 python your_script.py