one thread in the service cpu load reach the 99%

[suningzh]

Describe the bug

there are four threads in my service and i closed the work_stealing，When it has been running for two days
i found one thread in the service cpu load reach the 99%

Pingora info

Please include the following information about your environment:

Pingora version: 0.3.0
Rust version: cargo 1.80.0 (376290515 2024-07-16)
Operating system version: Debian 12.4

the perf top message:
25.82% pingora-demo [.] <tokio::future::poll_fn::PollFn as core::future::future::Future>::poll
17.01% pingora-demo [.] pingora_proxy::proxy_h1::$LT$impl$u20$pingora_proxy..HttpProxy$LT$SV$GT$$GT$::proxy_handle_downstream::$u7b$$u7b$closure$u7d$$u7d$::h68
9.24% libc-2.31.so [.] malloc
5.81% pingora-demo [.] pingora_error::Error::create
5.58% pingora-demo [.] <tokio::sync::batch_semaphore::Acquire as core::future::future::Future>::poll
5.25% pingora-demo [.] tokio::macros::support::thread_rng_n
4.17% libc-2.31.so [.] cfree
2.42% pingora-demo [.] core::ptr::drop_in_place<pingora_error::Error>
1.90% pingora-demo [.] core::ptr::drop_in_place$LT$$LP$pingora_core..protocols..http..server..Session..read_body_or_idle..$u7b$$u7b$closure$u7d$$u7d$$C$tokio..s
1.76% pingora-demo [.] core::ptr::drop_in_place$LT$pingora_proxy..proxy_h1..$LT$impl$u20$pingora_proxy..HttpProxy$LT$SV$GT$$GT$..proxy_handle_downstream..$u7b$$
1.46% pingora-demo [.] __rdl_dealloc
1.33% pingora-demo [.] __rdl_alloc
1.18% pingora-demo [.] <tokio::sync::batch_semaphore::Acquire as core::ops::drop::Drop>::drop
1.04% pingora-demo [.] tokio::sync::batch_semaphore::Semaphore::try_acquire
0.94% libc-2.31.so [.] 0x000000000008262f
0.83% pingora-demo [.] core::ptr::drop_in_place$LT$pingora_proxy..proxy_cache..ServeFromCache..next_http_task..$u7b$$u7b$closure$u7d$$u7d$$GT$::hbebc4c258c82f58
0.83% libc-2.31.so [.] 0x0000000000082606
0.72% pingora-demo [.] __rust_dealloc
0.67% libc-2.31.so [.] 0x00000000000825a9
0.67% libc-2.31.so [.] 0x00000000000825a0
0.63% libc-2.31.so [.] 0x00000000000825c3
0.62% libc-2.31.so [.] 0x00000000000825d0
0.61% libc-2.31.so [.] 0x00000000000825b1
0.61% libc-2.31.so [.] 0x0000000000082875
0.58% libc-2.31.so [.] 0x000000000008261a
0.56% libc-2.31.so [.] 0x00000000000826f2
0.56% libc-2.31.so [.] 0x00000000000825f0
0.54% libc-2.31.so [.] 0x0000000000082864
0.54% libc-2.31.so [.] 0x00000000000826d5
0.53% libc-2.31.so [.] 0x00000000000826ea

[drcaramelsyrup]

This is curious, do you have an example project demonstrating this?

[deep-gaurav]

Ok I don't know if this is same issue, or how to reproduce it. But in my project as well sometimes one thread locks up to 99% cpu usage. Even with no request incoming it never drops. I have to restart service to fix it.
I thought it's something logical error in my codebase and couldn't debug so didn't reported.