Error 500 - Failed to send batch; context deadline exceeded

[vyngur]

Hi everyone,

I'm currently testing Thanos Receive as a solution for some upcomfing environments where I need to get some metrics without running the full stack locally. I can see the blocks getting populated inside the Minio bucket, but I'm not really sure if everyting's ok considering I'm getting some errors in the logs.

I've added the necessary lines into the config file of the prometheus i'll be monitoring:

global:

[...]

remote_write:
- url: "http://monitoring-dev:10910/api/v1/receive"
  remote_timeout: 5s
  queue_config:
    batch_send_deadline: 5s
    capacity: 1000
    min_backoff: 50ms
    max_backoff: 500ms
    max_samples_per_send: 100
    min_shards: 24
    max_shards: 100

[...]

scrape_configs:
  - job_name: 'receive'
    static_configs:
      - targets: ['monitoring-dev:10909']

On the receiving end, this is how I have things running in docker-compose:

[...]

thanos-receive-0:
    image: quay.io/thanos/thanos:v0.22.0
    container_name: thanos-receive-0
    volumes:
      - data-0:/tsdb-0
      - ./data/bucket.yml:/etc/thanos/bucket.yml
      - ./data/local_hashring.json:/etc/thanos_hashring.json
    command:
      - "receive"
      - "--debug.name=receive"
      - "--log.level=debug"
      - "--tsdb.path=/tsdb-0"
      - "--grpc-address=0.0.0.0:10908"
      - "--http-address=0.0.0.0:10909"
      - "--remote-write.address=0.0.0.0:10910"
      - "--receive.replication-factor=1"
      - "--label=receive_replica=\"0\""
      - "--receive.local-endpoint=thanos-receive-0:10908"
      - "--objstore.config-file=/etc/thanos/bucket.yml"
      - "--receive.hashrings-file=/etc/thanos_hashring.json"
    ports:
      - "10908:10908"
      - "10909:10909"
      - "10910:10910"
    depends_on:
     - minio
    networks:
      - thanos

[...]

My temp hashring config:

[
    {
        "endpoints": [
            "thanos-receive-0:10908"
        ]
    }
]

prometheus.yml (receiver):

scrape_configs:

- job_name: thanos-receive
  scrape_interval: 5s
  static_configs:
  - targets:
    - "thanos-receive-0:10909"

Error

This is what I'm seeing in the sender's logs:

prometheus      | level=info ts=2021-08-02T16:42:32.686Z caller=main.go:796 msg="Server is ready to receive web requests."
prometheus      | ts=2021-08-02T16:42:38.034Z caller=dedupe.go:112 component=remote level=info remote_name=72a9a9 url=http://monitoring-dev:10910/api/v1/receive msg="Done replaying WAL" duration=5.360684787s
prometheus      | ts=2021-08-02T16:42:52.731Z caller=dedupe.go:112 component=remote level=warn remote_name=72a9a9 url=http://monitoring-dev:10910/api/v1/receive msg="Failed to send batch, retrying" err="Post \"http://monitoring-dev:10910/api/v1/receive\": context deadline exceeded"

And the corresponding Thanos receive logs:

thanos-receive-0     | level=error name=receive ts=2021-08-02T16:42:57.737941354Z caller=handler.go:366 component=receive component=receive-handler err="context deadline exceeded" msg="internal server error"
thanos-receive-0     | level=debug name=receive ts=2021-08-02T16:42:57.737337656Z caller=handler.go:351 component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded"

What am I missing? Where did I messed up in this setup?
Thanks in advance :)