Race condition while generating resolv-for-kubelet.conf

[mimiteto]

How to categorize this issue?

/area robustness
/kind bug
/platform openstack

What happened:
/etc/resolv-for-kubelet.conf was generated without any nameserver in it.
Observing the (redacted) logs:

2024-05-27T08:45:12.168190+0000 node-name systemd[1]: Starting update-resolv-conf.service - update /etc/resolv-for-kubelet.conf on start and after each change of /run/systemd/resolve/resolv.conf...
2024-05-27T08:45:12.192931+0000 node-name update-resolv-conf.sh[2138034]: updated /etc/resolv-for-kubelet.conf
2024-05-27T08:45:12.193163+0000 node-name systemd[1]: update-resolv-conf.service: Deactivated successfully.
2024-05-27T08:45:12.193349+0000 node-name systemd[1]: Finished update-resolv-conf.service - update /etc/resolv-for-kubelet.conf on start and after each change of /run/systemd/resolve/resolv.conf.

and when checked:

root@node-name:/# ls -la --full /run/systemd/resolve/resolv.conf
-rw-r--r-- 1 systemd-resolve systemd-resolve 833 2024-05-27 08:45:12.188413386 +0000 /run/systemd/resolve/resolv.conf

root@node-name:/# ls -la --full /etc/resolv*
-rw-r--r-- 1 root root 906 2024-05-27 08:45:12.168411571 +0000 /etc/resolv-for-kubelet.conf
lrwxrwxrwx 1 root root  32 2023-11-03 00:00:00.000000000 +0000 /etc/resolv.conf -> /run/systemd/resolve/resolv.conf

What you expected to happen:
/etc/resolv-for-kubelet.conf to always be generated with the latest data.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:
I see that we are handling this in https://github.com/gardener/gardener-extension-provider-openstack/blob/master/pkg/webhook/controlplane/ensurer.go#L390
We are generating a shell script that reacts on being called by a one shot service on path changes.
This works nicely on a file change but fails if called multiple times while systemd is processing the spawn of that process.
Issue stems from the way systemd will handle multiple calls from the same time if received for the state we are already in.

Environment:

• Gardener version (if relevant):
• Extension version: v1.39.2
• Kubernetes version (use kubectl version): v1.28.6
• Cloud provider or hardware configuration: Openstack
• Others: