Abort/timeout long running requests

[PSeitz]

Currently long running requests may continue to consume a large amount resources, while the original request already timed-out in a http-layer above. We should add a configurable timeout to abort requests in quickwit

There are two different timeouts configured:

Tonic Timeout

tonic timeouts return errors like:

{
  "message": "internal error: `Timeout expired`"
}

This timeout is originating here:

let timeout_channel = Timeout::new(node.channel(), Duration::from_secs(30));
let search_client = create_search_client_from_channel(
    grpc_addr,
    timeout_channel,
    max_message_size,
);
Some(Change::Insert(grpc_addr, search_client))

Tower Timeout

There's another similar timeout, from tower

{
  "message": "internal error: `request timed out`"
{

The tower timeout is defined here:

/// Creates a channel from a socket address.
///
/// The function is marked as `async` because it requires an executor (`connect_lazy`).
pub async fn make_channel(socket_addr: SocketAddr) -> Channel {
    let uri = Uri::builder()
        .scheme("http")
        .authority(socket_addr.to_string())
        .path_and_query("/")
        .build()
        .expect("provided arguments should be valid");
    Endpoint::from(uri)
        .connect_timeout(Duration::from_secs(5))
        .timeout(Duration::from_secs(30))
        .connect_lazy()
}

The behavior of timeouts also differs currently on the dispatch type, local dispatch never times out.

Retries

The current behavior is to retry directly after a timeout (cluster_client.rs::leaf_search). This could add additional load on a overloaded node.

The first step would be to properly transform the error type to keep the semantics

[PSeitz]

Fixed by #5402