RuntimeError: cuDNN error: CUDNN_STATUS_MAPPING_ERROR

[k1234567jkldhd]

Input the command:
python demo.py tracking,ddd --load_model ../models/nuScenes_3Dtracking.pth --dataset nuscenes --pre_hm --track_thresh 0.1 --demo ../videos/nuscenes_mini.mp4 --test_focal_length 633
result:
/home/h/anaconda3/envs/CenterTrack/lib/python3.6/site-packages/sklearn/utils/linear_assignment_.py:22: FutureWarning: The linear_assignment_ module is deprecated in 0.21 and will be removed from 0.23. Use scipy.optimize.linear_sum_assignment instead.
FutureWarning)
Running tracking
Using tracking threshold for out threshold! 0.1
Fix size testing.
training chunk_sizes: [32]
input h w: 448 800
heads {'hm': 10, 'reg': 2, 'wh': 2, 'tracking': 2, 'dep': 1, 'rot': 8, 'dim': 3, 'amodel_offset': 2}
weights {'hm': 1, 'reg': 1, 'wh': 0.1, 'tracking': 1, 'dep': 1, 'rot': 1, 'dim': 1, 'amodel_offset': 1}
head conv {'hm': [256], 'reg': [256], 'wh': [256], 'tracking': [256], 'dep': [256], 'rot': [256], 'dim': [256], 'amodel_offset': [256]}
Creating model...
Using node type: (<class 'model.networks.dla.DeformConv'>, <class 'model.networks.dla.DeformConv'>)
Warning: No ImageNet pretrain!!
loaded ../models/nuScenes_3Dtracking.pth, epoch 70
out_name nuscenes_mini.mp4
Initialize tracking!
Traceback (most recent call last):
File "demo.py", line 128, in
demo(opt)
File "demo.py", line 83, in demo
ret = detector.run(img)
File "/home/h/CenterTrack/src/lib/detector.py", line 119, in run
images, self.pre_images, pre_hms, pre_inds, return_time=True)
File "/home/h/CenterTrack/src/lib/detector.py", line 339, in process
output = self.model(images, pre_images, pre_hms)[-1]
File "/home/h/anaconda3/envs/CenterTrack/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "/home/h/CenterTrack/src/lib/model/networks/base_model.py", line 75, in forward
feats = self.imgpre2feats(x, pre_img, pre_hm)
File "/home/h/CenterTrack/src/lib/model/networks/dla.py", line 631, in imgpre2feats
x = self.base(x, pre_img, pre_hm)
File "/home/h/anaconda3/envs/CenterTrack/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "/home/h/CenterTrack/src/lib/model/networks/dla.py", line 307, in forward
x = self.base_layer(x)
File "/home/h/anaconda3/envs/CenterTrack/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "/home/h/anaconda3/envs/CenterTrack/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
input = module(input)
File "/home/h/anaconda3/envs/CenterTrack/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "/home/h/anaconda3/envs/CenterTrack/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 343, in forward
return self.conv2d_forward(input, self.weight)
File "/home/h/anaconda3/envs/CenterTrack/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 340, in conv2d_forward
self.padding, self.dilation, self.groups)
RuntimeError: cuDNN error: CUDNN_STATUS_MAPPING_ERROR

The code was tested on Ubuntu 20.04, with [Anaconda] Python 3.6, CUDA 10.0, and [PyTorch]v1.10.2.

[unbeliveyu]

Have you solved this problem

[bryanbocao]

@unbeliveyu Try using Docker:
https://github.com/bryanbocao/CenterTrack/blob/master/readme/Docker.md
#290

Run CenterTrack using Docker

DCNv2 Compilation Issue Fixed

docker pull bryanbocao/centertrack
docker run -d --ipc=host --shm-size=16384m -it -v /:/share --gpus all --network=bridge bryanbocao/centertrack /bin/bash
docker ps -a

Check <CONTAINER_ID>:

CONTAINER ID   IMAGE                    COMMAND                  CREATED          STATUS                       PORTS                NAMES
89bb79551ccb   bryanbocao/centertrack   "/usr/local/bin/nvid…"   49 seconds ago   Up 38 seconds                6006/tcp, 8888/tcp   competent_northcutt

docker exec -it <CONTAINE_ID> /bin/bash

docker exec -it 89bb79551ccb /bin/bash

Inside the container:

cd /root/CenterTrack/src/lib/model/networks/DCNv2
python3 setup.py build develop
cd /root/CenterTrack/src/
python3 demo.py tracking,ddd --load_model ../models/nuScenes_3Dtracking.pth --dataset nuscenes --pre_hm --track_thresh 0.1 --demo ../videos/nuscenes_mini.mp4 --test_focal_length 633

The DCNv2 is built from MatthewHowe_DCNv2. Thanks MatthewHowe for the contribution.
Then you should be able to see the results.

Go to /share to transfer data between the container and your root system.