Skip to content
This repository has been archived by the owner on Jul 25, 2024. It is now read-only.

Add number of messages in queues in system status page #992

Open
chaws opened this issue Jun 9, 2021 · 0 comments
Open

Add number of messages in queues in system status page #992

chaws opened this issue Jun 9, 2021 · 0 comments

Comments

@chaws
Copy link
Collaborator

chaws commented Jun 9, 2021

This will be useful for monitoring squad's health.

Here are the steps for doing so:

  1. RabbitMQ's host:
    • Open ports 15672 and 15671 only for squad-frontend
    • Enable management plugin sudo rabbitmq-plugins enable rabbitmq_management
  2. Squad-frontent
    • GET rabbitmq's host c = requests.get('http://<private-ip>:15672/api/queues/%2f/ci_fetch/', auth = HTTPBasicAuth('guest', 'guest'))
    • Get number of message (check if keys exist, they might not if queue is empty): c.json()['message_stats']['publish']

There's probably an endpoint to list all the queues, so we don't hard-code queues.

Then add a grafana dash grabbing that endpoint and create an alert if queues grows too much (probably due to worker outage)

  • ci_fetch shouldn't have more than 10k tasks
  • celery shouldn't have more than 300k tasks
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant