nvidia-container-runtime segfault

[xhejtman]

Hello,

I am getting randomly:

[4746547.221468] nvidia-containe[2847336]: segfault at 7f6ee8000020 ip 00007f6f24ca1cf2 sp 00007f6ef7ffed60 error 6 in libc.so.6[7f6f24c28000+195000] likely on CPU 127 (core 31, socket 1)

is is not related to a particular pod, I think this is result of some monitoring action (liveness/readiness probes). It is hard to debug as there is no core dump file.

Version is 1.14.3.

[elezar]

@xhejtman could you describe your environment?

Is this happening sporadically, or can it be reproduced?

[xhejtman]

Sporadically, but quite often. It may happen, that two probes run in parallel and sometimes it segfaults.

It is kubernetes 1.26.9 using RKE2. Gpu operator 23.6.1, but I tried today nvidia toolkit from the latest gpu-operator.

I can test or provide logs if needed.