[Bug]: CI nextest failed by timeout

[lxl66566]

Description about the bug

There was a case that nextest failed by timeout found on github workflow. When I reruned the code without modifing any code, the failure disappeared.
It's a rare error that cannot be reproduced stably. And It can be either the code bug or a nextest bug.

Version

0.6.1 (Default)

Relevant log output

the failed CI: https://github.com/xline-kv/Xline/actions/runs/10003614333/job/27650819920?pr=905
the success(rerun) CI: https://github.com/xline-kv/Xline/actions/runs/10003614333?pr=905
The failed test is: curp::it server::shutdown_rpc_should_shutdown_the_cluster
There're a lot of RpcTransport(()) error

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

👋 Thanks for opening this issue!

Reply with the following command on its own line to get help or engage:

• /contributing-agreement : to print Contributing Agreements.
• /assignme : to assign this issue to you.

[lxl66566]

That might caused by the following:

• A put client opens a thread and sends 10 Put requests one by one.
• Another client proposed shutdown to cluster
• The putclient waits for a response, exceeding the time limit. now the cluster may have shut down successfully, and RPC closed.
• The put client retries, and gets an RpcTransport error, keep retrying. Each put request may retry 3 times, 10.5s.
• each 10 put request retry for at most 105s, exceed 30s test timeout.

This should be fixed when #918 merge.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 14 days.