onnxruntime-gpu requires NVIDIA's *.-devel packages?

[mkserge]

Hello,

It seems like ONNX runtime for GPU is requiring NVIDIA's -devel packages, instead of the -runtime. Can anyone confirm if that is true?

For example, using the pytorch/pytorch:1.7.1-cuda11.0-cudnn8-runtime from DockerHub as the base image to serve models with onnxruntime results in a failure on import because it seems like the *-runtime package does not actually contain libcudnn binary

    _libcudnn = CDLL("libcudnn.so.8", mode=RTLD_GLOBAL)
  File "/opt/conda/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libcudnn.so.8: cannot open shared object file: No such file or directory

The *-devel packages are quite a bit larger, but contain the nvcc, etc. I rather avoid using them, if possible.

Can anyone help me understand what exactly the issue is here? Does ONNX require nvcc when doing inference only?

Thank you.

[snnn]

Yes, you are right. A recent change #7110 requires the dev packages.

Before that, I think it doesn't. See: https://github.com/microsoft/onnxruntime/blob/master/dockerfiles/Dockerfile.cuda . Onnx runtime works well with the runtime only nvidia docker image.

[mkserge]

Understood, thank you.

[RandySheriffH]

@snnn, #7110 did not introduce that, cupti comes with cuda toolkit by default.
@mkserge: I think you just need to install cudnn, that's it.

[mkserge]

Thanks @RandySheriffH I'll give it a try and post an update here.