Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Specific endpoint for prometheus exporter health checking #435

Merged
merged 4 commits into from
Jan 14, 2024

Conversation

biancamip
Copy link
Contributor

@biancamip biancamip commented Jan 8, 2024

Problem:
A service that installs a prometheus metrics exporter will return all metrics data on any request, regardless of the endpoint requested. This presents an issue when doing health checks using a service networking solution such as Consul, which performs requests on short intervals and spreads the response status + body across all server nodes. Since the response body contains all metrics collected and will probably change from one check to the next, the issue is that this behavior will generate a high network traffic, resulting in delays, elevated costs, and so on.

Solution:
For health checking, the response could be as simple as a successful status code with a short body content. I've implemented this for the path "/health", while maintaining the previous behavior for all other cases.

If necessary, this solution could be adjusted to allow configuration of the health path on PrometheusBuilder struct initialization, and it could also be disabled in case the current behavior is desired, but I think this minor change would make significant difference for now.

@biancamip biancamip changed the title Health check http Specific endpoint for prometheus exporter health checking Jan 8, 2024
@tobz
Copy link
Member

tobz commented Jan 14, 2024

@biancamip Apologies on the delay reviewing this. 👋🏻

I've made two small little changes, but otherwise, I think this is fine to merge as soon as CI passes.

@tobz tobz merged commit 757dcac into metrics-rs:main Jan 14, 2024
12 checks passed
@tobz tobz added C-exporter Component: exporters such as Prometheus, TCP, etc. E-simple Effort: simple. T-enhancement Type: enhancement. S-awaiting-release Status: awaiting a release to be considered fixed/implemented. labels Jan 14, 2024
@tobz
Copy link
Member

tobz commented Feb 11, 2024

Forgot to update the changelog prior to release, but this has now been released as part of [email protected].

Thanks again for your contribution. ❤️

@tobz tobz removed the S-awaiting-release Status: awaiting a release to be considered fixed/implemented. label Feb 11, 2024
mnpw pushed a commit to mnpw/metrics that referenced this pull request Mar 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-exporter Component: exporters such as Prometheus, TCP, etc. E-simple Effort: simple. T-enhancement Type: enhancement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants