RabbitMQ config & compute_all_grades_for_course ETA are unable to work together, causing unrecoverable error on celery

[cc3613]

Since RabbitMQ 3.8.16, the configuration consumer_timeout is introduced and defaults to 30 mins.

compute_all_grades_for_course ETA/countdown is set to an hour in code (why is it an hour anyway?)

celery would not ACK the task until countdown is over and task starts processing. RMQ waits for 30 minutes and not gets ACK, so it deems the consumer dead.

Here's official doc (it happens to : https://docs.celeryq.dev/en/stable/userguide/calling.html#eta-and-countdown

Here's the same issue describe on celery: celery/celery#6760

I cannot change the default consumer_timeout as I don't control the rmq. What's the easiest way to deal with this in edx code?

EDUCATOR-1088 for this change: https://github.com/openedx/edx-platform/pull/16165/files