You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Looking at the STDOUT I see that there's an issue when restarting the worker with max requests: It times out after it appears to have successfully killed the worker. Does anyone have an idea what could be causing the timeout? I see that kafka is timing out, but if that were the issue I'd expect not to see "Application shutdown complete." or "Finished server process"
[2024-04-25 14:28:44 +0000] [29227] [INFO] Shutting down
[2024-04-25 14:28:44 +0000] [29227] [INFO] Waiting for application shutdown.
{"asctime": "2024-04-25 14:28:46,080", "name": "...subprocesses.update_workload", "levelname": "INFO", "levelno": 20, "pathname": "/app/.../update_workload.py", "lineno": 23, "funcName": "update_workload", "threadName": "MainThread", "processName": "sub-process: update-workload-worker for PID: 29227", "message": "Shutdown triggered, exited update_workload gracefully."}
%5|1714055329.954|REQTMOUT|rdkafka#consumer-3| [thrd:GroupCoordinator]: GroupCoordinator/3: Timed out LeaveGroupRequest in flight (after 5004ms, timeout #0): possibly held back by preceeding blocking JoinGroupRequest with timeout in 99698ms
%4|1714055329.954|REQTMOUT|rdkafka#consumer-3| [thrd:GroupCoordinator]: GroupCoordinator/3: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
%3|1714055329.955|FAIL|rdkafka#consumer-3| [thrd:GroupCoordinator]: GroupCoordinator: 1 request(s) timed out: disconnect (after 2219257ms in state UP)
[2024-04-25 14:28:50 +0000] [29227] [INFO] Worker exiting (pid: 29227)
[2024-04-25 14:28:50 +0000] [29227] [INFO] Application shutdown complete.
[2024-04-25 14:28:50 +0000] [29227] [INFO] Finished server process [29227]
[2024-04-25 14:29:50 +0000] [69] [CRITICAL] WORKER TIMEOUT (pid:29227)
[2024-04-25 14:29:50 +0000] [69] [ERROR] Worker (pid:29227) was sent code 134!
[2024-04-25 14:29:50 +0000] [31401] [INFO] Booting worker with pid: 31401
The text was updated successfully, but these errors were encountered:
I have gunicorn managing a Flask app run using the following docker command
Looking at the STDOUT I see that there's an issue when restarting the worker with max requests: It times out after it appears to have successfully killed the worker. Does anyone have an idea what could be causing the timeout? I see that kafka is timing out, but if that were the issue I'd expect not to see "Application shutdown complete." or "Finished server process"
The text was updated successfully, but these errors were encountered: