Show rate limit issues in the UI

[rbren]

What problem or use case are you trying to solve?

I'm getting rate limited by Anthropic. But it just looks like the agent is kinda stuck while it cools down.

Describe the UX of the solution you'd like

I'd like the indicator to turn yellow, and show a relevant message about rate limits

Do you have thoughts on the technical implementation?

@tobitege has done a little preliminary work here. Basically I think we need to turn the status/badge from "agent status" to "system status"

Describe alternatives you've considered

Additional context

[tobitege]

Btw, during benches since yesterday, I received server error 502 with some html error message (lot of file edits back and forth in a short amount of time), but have a feeling that that is the error you've also experienced when getting limited?

[rbren]

Yeah exactly--I think it was due to file editing issues

[tobitege]

litellm completion calls can have a cooldown parameter with number of seconds for cooldown after hitting rate limits, i.e. it'll happen automatically without raising an exception.

[tobitege]

Just an example I found in my logs (added linebreaks for readability):
(429 is default code for rate limiting in litellm)

18:04:47 - openhands:ERROR: llm.py:128 - litellm.RateLimitError: RateLimitError: OpenAIException - Error code: 429 - 
{'error': {'message': 'No deployments available for selected model, Try again in 60 seconds. Passed model=claude-3-5-sonnet@20240620. pre-call-checks=False,
allowed_model_region=n/a, cooldown_list=[(\'75365eba-c184-48b9-8195-f845d4b812ab\', 
{\'Exception Received\': \'litellm.RateLimitError: BedrockException - {"message":"Too many requests, please wait before trying again.
You have sent too many requests.  Wait before trying again."}\', \'Status Code\': \'429\'}), 
(\'0fba6cb1-2b22-45a1-9ec4-f292d74213d4\', {\'Exception Received\': \'litellm.RateLimitError: litellm.RateLimitError: VertexAIException - 
{\\n  "error": {\\n    "code": 429,\\n    "message": "Online prediction request quota exceeded for anthropic-claude-3-5-sonnet.
Please try again later with backoff.",\\n    "status": "RESOURCE_EXHAUSTED"\\n  }\\n}\\n\', \'Status Code\': \'429\'})]', 
'type': 'None', 'param': 'None', 'code': '429'}}. Attempt #1 | You can customize these settings in the configuration.