torch.FatalError: function_attributes(): after cudaFuncGetAttributes: invalid device function

[HLH13297997663]

The following error occurred when I ran the train_mobile_mvd:
/workfs/lhhuang/anaconda3/envs/pytorch0.3/lib/python3.6/site-packages/torch/cuda/init.py:95: UserWarning:
Found GPU0 Tesla V100-SXM2-16GB which requires CUDA_VERSION >= 9000 for
optimal performance and fast startup time, but your PyTorch was compiled
with CUDA_VERSION 8000. Please install the correct PyTorch binary
using instructions from http://pytorch.org

warnings.warn(incorrect_binary_warn % (d, name, 9000, CUDA_VERSION))
^M 0%| | 0/9000 [00:00<?, ?it/s]['/workfs/lhhuang/segment/LightNet-master2/LightNet-master/scripts', '/workfs/lhhuang/segment/LightNet-master2/LightNet-master/scripts', '/dat01/lhhuang/models/research', '/dat01/lhhuang/models/research/slim', '/dat01/lhhuang/models/research/delf', '/workfs/lhhuang/anaconda3/envs/pytorch0.3/lib/python36.zip', '/workfs/lhhuang/anaconda3/envs/pytorch0.3/lib/python3.6', '/workfs/lhhuang/anaconda3/envs/pytorch0.3/lib/python3.6/lib-dynload', '/home/lhhuang/.local/lib/python3.6/site-packages', '/workfs/lhhuang/anaconda3/envs/pytorch0.3/lib/python3.6/site-packages', '/workfs/lhhuang/segment/LightNet-master2/LightNet-master/datasets', '/workfs/lhhuang/segment/LightNet-master2/LightNet-master']

+++++++++++++++++++++++++++++++++++++++++++++++++++++++

0. Setting up DataLoader...

+++++++++++++++++++++++++++++++++++++++++++++++++++++++

There are 66 labels in the config file
Found 18000 training images...
There are 66 labels in the config file
Found 2000 validation images...

+++++++++++++++++++++++++++++++++++++++++++++++++++++++

1. Setting up Model...
2. Model Training start...
   ^M 0%| | 1/9000 [00:04<12:23:13, 4.96s/it]^M> Epoch [1/256]: 0%| | 1/9000 [00:04<12:23:13, 4.96s/it]Traceback (most recent call last):
   File "train_mobile_mvd.py", line 348, in
   train(train_args, data_path, save_path)
   File "train_mobile_mvd.py", line 185, in train
   out_stg1, out_stg2, out_stg3 = model(images) # Here we have 3 output for 3 loss
   File "/workfs/lhhuang/anaconda3/envs/pytorch0.3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in call
   result = self.forward(*input, **kwargs)
   File "/workfs/lhhuang/segment/LightNet-master2/LightNet-master/models/mobilenetv2plus.py", line 169, in forward
   de_stg1 = self.aspp(stg8)[1]
   File "/workfs/lhhuang/anaconda3/envs/pytorch0.3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in call
   result = self.forward(*input, **kwargs)
   File "/workfs/lhhuang/anaconda3/envs/pytorch0.3/lib/python3.6/site-packages/torch/nn/modules/container.py", line 67, in forward
   input = module(input)
   File "/workfs/lhhuang/anaconda3/envs/pytorch0.3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in call
   result = self.forward(*input, **kwargs)
   File "/workfs/lhhuang/segment/LightNet-master2/LightNet-master/modules/misc.py", line 334, in forward
   x = self.in_norm(x)
   File "/workfs/lhhuang/anaconda3/envs/pytorch0.3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in call
   result = self.forward(*input, **kwargs)
   File "/workfs/lhhuang/segment/LightNet-master2/LightNet-master/modules/bn.py", line 96, in forward
   self.activation, self.slope)
   File "/workfs/lhhuang/segment/LightNet-master2/LightNet-master/modules/functions.py", line 97, in forward
   _act_forward(ctx, x)
   File "/workfs/lhhuang/segment/LightNet-master2/LightNet-master/modules/functions.py", line 41, in _act_forward
   _ext.leaky_relu_cuda(x, ctx.slope)
   File "/workfs/lhhuang/anaconda3/envs/pytorch0.3/lib/python3.6/site-packages/torch/utils/ffi/init.py", line 180, in safe_call
   result = torch._C._safe_call(*args, **kwargs)
   torch.FatalError: function_attributes(): after cudaFuncGetAttributes: invalid device function