FailedAttachVolume due to rpc error: code = Internal desc = Operation failed: InvalidResponse("error in response: status code '408 Request Timeout', text ''")

[kuldipshpe]

Describe the bug
Not able to attach volume, pod is showing below error:
FailedAttachVolume due to rpc error: code = Internal desc = Operation failed: InvalidResponse("error in response: status code '408 Request Timeout', text ''")

Events:
Type Reason Age From Message

---

Warning FailedAttachVolume 4m2s attachdetach-controller AttachVolume.Attach failed for volume "pvc-e775fe38-8dbe-407b-902c-9dbea3574643" : rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Warning FailedAttachVolume 2m57s (x6 over 5m9s) attachdetach-controller AttachVolume.Attach failed for volume "pvc-e775fe38-8dbe-407b-902c-9dbea3574643" : rpc error: code = Internal desc = Operation failed: InvalidResponse("error in response: status code '408 Request Timeout', text ''")
Warning FailedAttachVolume 55s (x3 over 4m51s) attachdetach-controller AttachVolume.Attach failed for volume "pvc-e775fe38-8dbe-407b-902c-9dbea3574643" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
core@sc-os-72-node1:~$
mayastor-2024-10-10--18-07-37-UTC.tar.gz

To Reproduce
Noticed this issue when reboot the node where a replica is running in 2 replica configured volume, there are other 98 volumes on the system with 2 replicas.

Expected behavior
Volume should be able to attach and pod going to Running state.

See attached mayastor dump system logs

Screenshots
If applicable, add screenshots to help explain your problem.

** OS info (please complete the following information):**

• Distro: [e.g. NixOS]
• Kernel version
• MayaStor revision or container image

Additional context
Add any other context about the problem here.

[tiagolobocastro]

We're seeing lots of this lately (CC @dsharma-dc):

ERROR io_engine::bdev::nvmx::handle:handle.rs:387] I/O completed with PI error

Could be due to the reboot of the node though.
Do you know what time you rebooted the node?

I see everything on csi-controller is timing out!

I wonder if the csi-controller rest client is not coping well after the node restarted.
If you try to create a new volume, does that succeed?

[kuldipshpe]

In attach logs rebooted the node at 10/15/2024, 12:48:40
For the pod noticed 408 request timeout and failed to attach volume.
Events:
Type Reason Age From Message

---

Warning FailedAttachVolume 90s (x5 over 99s) attachdetach-controller AttachVolume.Attach failed for volume "pvc-446b0f52-0971-467d-b8bc-3dc8eb4a9e97" : rpc error: code = Internal desc = Operation failed: InvalidResponse("error in response: status code '408 Request Timeout', text ''")
Warning FailedAttachVolume 85s (x5 over 95s) attachdetach-controller AttachVolume.Attach failed for volume "pvc-0e74c247-ea56-444d-81e8-dc47dac95183" : rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Warning FailedAttachVolume 82s (x6 over 100s) attachdetach-controller AttachVolume.Attach failed for volume "pvc-454bc89f-c37d-480e-a081-f89af71e027a" : rpc error: code = Internal desc = Operation failed: InvalidResponse("error in response: status code '408 Request Timeout', text ''")
Warning FailedAttachVolume 82s attachdetach-controller AttachVolume.Attach failed for volume "pvc-2fffe9b1-09fc-4d19-8a77-6e6caee7c052" : rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Warning FailedAttachVolume 81s attachdetach-controller AttachVolume.Attach failed for volume "pvc-446b0f52-0971-467d-b8bc-3dc8eb4a9e97" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
Warning FailedAttachVolume 76s attachdetach-controller AttachVolume.Attach failed for volume "pvc-0e74c247-ea56-444d-81e8-dc47dac95183" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
Warning FailedAttachVolume 65s (x6 over 100s) attachdetach-controller AttachVolume.Attach failed for volume "pvc-2fffe9b1-09fc-4d19-8a77-6e6caee7c052" : rpc error: code = Internal desc = Operation failed: InvalidResponse("error in response: status code '408 Request Timeout', text '’")

mayastor-2024-10-15--20-30-40-UTC-nodes-reboot-408-error.tar.gz