Too Many Requests: ingestion rate limit (50000) exceeded while adding 500 samples

[miladjahandideh]

Which Parameter should I increase?

Too Many Requests: ingestion rate limit (50000) exceeded while adding 500 samples and 0 metadata

my config:

mimir:
  config: |
    multitenancy_enabled: false

    limits:
      max_label_names_per_series: 200
      max_global_series_per_user: 1000000000
      ingestion_burst_size: 200000
      ingestion_rate: 150000
      max_global_series_per_metric: 200000000
  

Log of Prometheus Pod

ts=2022-06-11T07:20:37.778Z caller=dedupe.go:112 component=remote level=error remote_name=130066 url=http://XXX:8080/api/v1/push msg="non-recoverable error" count=500 exemplarCount=0 err="server returned HTTP status 429 Too Many Requests: ingestion rate limit (50000) exceeded while adding 500 samples and 0 metadata"

And Also Alert

Description: Prometheus monitoring/prometheus-kube-prometheus-stack-prometheus-0 failed to send 16.5% of the samples to 2f0a43:http://grafana-mimir-XXX:8080/api/v1/push

[pracucci]

You need to increase this per-tenant limit (see doc):

# Per-tenant ingestion rate limit in samples per second.
# CLI flag: -distributor.ingestion-rate-limit
[ingestion_rate: <float> | default = 10000]

Too Many Requests: ingestion rate limit (50000) exceeded while adding 500 samples and 0 metadata

The global ingestion_rate: 150000 rate limit is implemented as a local per-distributor rate limit, where each distributor limit is set to limit / number of distributors. Since the reported value is 50000, I assume you're running 3 distributors. I understand it's a bit confusing, and we'll fix it. Anyway, that's the limit to increase.

[miladjahandideh]

I was thinking about the relationship between adding 500 samples and ingestion rate limit (50000)! Now I released the problem. I've set ingestion_rate: 1500000 and the problem is solved. Thank you @pracucci.

[pracucci]

Good point. It may be confusing including in the error message the actual number of samples that we were trying to add, given the limit is order of magnitude bigger than that. We could probably just drop it.

[pracucci]

The PR #2104 should improve the error message related to this limit.