Long time run in server side causes client exception in client side

[cuauty]

I use the aiohttp as http server on LINUX and return the response. My code looks like:

async def handle_post(request):
loop = asyncio.get_running_loop()
handle_fn = functools.partial(call_service, await request.json())
my_result = await asyncio.wait_for(
loop.run_in_executor(None, handle_fn),
timeout=300.0,)
return web.Response(text=my_result, status=200)

if __name__ == "__main__":
app = web.Application()
app.add_routes([
web.post("/post", handle_post),
])
web.run_app(app)

My function "call_service" needs long time (almost 250s) to run so I put it into executor for parallel. However, my client, also use aiohttp session, enters the following error.

Traceback (most recent call last):
File "/home/chris/testspace/test_2_t.py", line 34, in
asyncio.run(main())
File "/home/chris/miniconda3/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/home/chris/miniconda3/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
File "/home/chris/testspace/test_2_t.py", line 30, in main
async with session.post(url, json=req, headers=headers, timeout=timeout) as resp:
File "/home/chris/miniconda3/lib/python3.10/site-packages/aiohttp/client.py", line 1353, in aenter
self._resp = await self._coro
File "/home/chris/miniconda3/lib/python3.10/site-packages/aiohttp/client.py", line 684, in _request
await resp.start(conn)
File "/home/chris/miniconda3/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 999, in start
message, payload = await protocol.read() # type: ignore[union-attr]
File "/home/chris/miniconda3/lib/python3.10/site-packages/aiohttp/streams.py", line 640, in read
await self._waiter
aiohttp.client_exceptions.ClientOSError: [Errno 104] Connection reset by peer.

If I use simple mock function as "call_service" and return quickly, everything is OK. But if the call_service works long, the above occurs again.

It looks the connection is closed for lack of message too long. Is there any setting I can use to avoid this issue? Thanks a lot.

[Dreamsorcerer]

I don't that's what aiohttp does, there must be some other issue with the code. Can you create a full reproducer, so we can just copy/paste and run it?

[cuauty]

@Dreamsorcerer

Thanks for your reply. I can reproduce the case by the following code.

Server side.

import asyncio
import aiohttp

async def handle_sleep_post(request):

    data = await request.json()
    sleep_duration=int(data["sleep_duration"])

    await asyncio.sleep(sleep_duration)

    return web.json_response(data)

if __name__ == "__main__":
    app = web.Application()
    app.add_routes(
        [
            web.post("/sleep", handle_sleep_post),
        ]
    )
    web.run_app(
        app,
        shutdown_timeout=900.0,
        keepalive_timeout=750.0,
        backlog=1000,
        reuse_address=True,
        reuse_port=True,
    )

The server receives POST and sleeps the assigned number of seconds by asyncio.sleep

client side. The "url" in the code is the server's address in the running time.

import aiohttp
import asyncio
import time

headers = {"Content-Type": "application/json"}
req = {"sleep_duration": 100}
timeout = aiohttp.ClientTimeout(total=300, connect=300, sock_connect=300, sock_read=300, ceil_threshold=300)
async def main():
    async with aiohttp.ClientSession() as session:
        start_point = time.perf_counter()
        async with session.post(url, json=req, headers=headers, timeout=timeout) as resp:
            print(resp.status)
            print(await resp.text())
        end_point = time.perf_counter()
        print(f"{end_point - start_point}")

asyncio.run(main())

The client sends the request, which includes "sleep_duration" to specify the number of seconds for sleep. If sleep_duration is greater than 90, the error occurs as follows.

Traceback (most recent call last):
  File "/home/chris/testspace/test_2_t.py", line 39, in <module>
    asyncio.run(main())
  File "/home/chris/miniconda3/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/home/chris/miniconda3/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/chris/testspace/test_2_t.py", line 33, in main
    async with session.post(url, json=req, headers=headers, timeout=timeout) as resp:
  File "/home/chris/miniconda3/lib/python3.10/site-packages/aiohttp/client.py", line 1353, in __aenter__
    self._resp = await self._coro
  File "/home/chris/miniconda3/lib/python3.10/site-packages/aiohttp/client.py", line 684, in _request
    await resp.start(conn)
  File "/home/chris/miniconda3/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 999, in start
    message, payload = await protocol.read()  # type: ignore[union-attr]
  File "/home/chris/miniconda3/lib/python3.10/site-packages/aiohttp/streams.py", line 640, in read
    await self._waiter
aiohttp.client_exceptions. ClientOSError: [Errno 104] Connection reset by peer

[cuauty]

More info for this case, hope supply more clues.

1. It can't reproduce the case if server and client runs in same machine.
2. When I use "curl" command to send the request, the error disappears even though sleep_duration is set to 200, which means server will sleep by "await asyncio.sleep" then return the response.

curl https://xx.xx.xx.xx/sleep \
     -H "Content-Type: application/json" \
     -d '{"sleep_duration": 200}'

That makes me wonder whether I need to set something in either session or post method. However I try to go through the document and can't find anything.

[Dreamsorcerer]

It can't reproduce the case if server and client runs in same machine.

That seems suspicious. That makes it sound like a problem with the network/firewall etc. that is killing the idle connection..

I'll try it out myself, probably tomorrow, and see if I can get the same result.

[cuauty]

@Dreamsorcerer Thanks for your reply.

I continue to dive down this issue, and find that the issue can disappear if I try to set socket options, e.g. (socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1), (socket.SOL_TCP, socket.TCP_KEEPIDLE, 20), and so on. That can explain why the curl program can work in this case, which maybe set these options by default.

So the question turns to, how to set these socket options into aiohttp session? Could you give me a hand for this? Thanks you again.

[Dreamsorcerer]

KEEPALIVE should already be set by default:

aiohttp/aiohttp/tcp_helpers.py

Lines 11 to 16 in d049f4d

	if hasattr(socket, "SO_KEEPALIVE"):
	def tcp_keepalive(transport: asyncio.Transport) -> None:
	sock = transport.get_extra_info("socket")
	if sock is not None:
	sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)

Can you try verifying that code is executing on your system, and whether it's just the KEEPIDLE that needs to be added?

[cuauty]

@Dreamsorcerer

I use the following socket options.

    (socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1),
    (socket.SOL_TCP, socket.TCP_KEEPIDLE, 20),
    (socket.SOL_TCP, socket.TCP_KEEPINTVL, 20),
    (socket.SOL_TCP, socket.TCP_KEEPCNT, 50),

Could you help take a look for these options in the code of aiohttp? Furthermore, is there any interface I can call to set them in my code? Thanks.

[Dreamsorcerer]

A quick search suggests that keepidle defaults to 7200, keepintvl to 75 and keepcnt to 9. These all seem reasonable and I'm not sure we'd want to change them.

Can you verify the previously mentioned code is actually running to set the KEEPALIVE (e.g. by editing aiohttp code and printing it out)? And if so, can you add the other options one-at-a-time to figure out which options are actually required for your usecase.

[cuauty]

@Dreamsorcerer

7200 keepidle is too long, actually in the most of environments the idle connection will be closed over 90 seconds. Even if we hope to relieve the load of servers, keepidle should be less than 90.

I have to turn to other solution for my case by using streaming. In streaming way, the server can return its first part of response in 90 seconds. Is there any links for streaming in aiohttp serverside? Thanks.

[Dreamsorcerer]

90 seconds would break websockets, SSE and other use cases. Usually, I think you'd use timeouts in nginx or similar to handle shorter keep alives at a WAF level, rather than a TCP level.

I don't really know what you're trying to achieve, but you can use a StreamResponse, websockets, SSE...
https://docs.aiohttp.org/en/stable/web_reference.html#aiohttp.web.StreamResponse

[Dreamsorcerer]

Even if we hope to relieve the load of servers, keepidle should be less than 90.

Also, this seems to be the opposite of your original request...
You said the request needs 250s and was failing due to timeouts, now you're explicitly wanting a shorter timeout...