[Docker Swarm mode] ingester failing healthcheck / failed pushing to ingester

[Aymen-MTE]

Did someone succeed in deploying on docker swarm cluster?

Every configuration file I've tried resulted in a network error before pushing to object storage (it works fine with the same config file using binary and 2-3 separate nodes).

Error log from container :

"
ts=2023-07-11T15:51:52.498902962Z caller=log.go:194 level=debug msg="Initiating push/pull sync with: mimir-2-6b87b28f 172.28.0.3:27956"
ts=2023-07-11T15:51:52.521445784Z caller=log.go:194 level=debug msg="Stream connection from=172.28.0.3:35390"
 ts=2023-07-11T15:51:56.378680111Z caller=logging.go:111 level=warn traceID=32dfbacab6cc39bb msg="POST /api/v1/push (500) 2.003137276s Response: \"failed pushing to ingester: rpc error: code = DeadlineExceeded desc = context deadline exceeded\\n\" ws: false; Content-Encoding: snappy; Content-Length: 21169; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.45.0; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: swarm-test; "
 ts=2023-07-11T15:51:57.364755512Z caller=logging.go:111 level=warn traceID=5b84438d4c1219a4 msg="POST /api/v1/push (500) 2.001635398s Response: \"failed pushing to ingester: rpc error: code = DeadlineExceeded desc = context deadline exceeded\\n\" ws: false; Content-Encoding: snappy; Content-Length: 10132; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.45.0; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: swarm-test; "
ts=2023-07-11T15:51:58.358634193Z caller=pool.go:193 level=warn msg="removing ingester failing healthcheck" addr=10.0.0.11:19095 reason="rpc error: code = DeadlineExceeded desc = context deadline exceeded"
 ts=2023-07-11T15:51:58.358696374Z caller=pool.go:193 level=warn msg="removing ingester failing healthcheck" addr=10.0.0.12:29095 reason="rpc error: code = DeadlineExceeded desc = context deadline exceeded"

Swarm docker-compose.yml file

version: "3.9"
services:
  mimir-1:
    image: grafana/mimir:latest
    hostname: mimir-1
    command: ["-config.file=/etc/mimir.yaml"]
    volumes:
      - ./config/mimir-1.yaml:/etc/mimir.yaml
      - mimir-data:/data
    ports:
      - 9041:8080
      - 17956:17956
      - 19095:19095
    networks:
      - mimir-net
    deploy:
      placement:
        constraints: [node.hostname == server01]

  mimir-2:
    image: grafana/mimir:latest
    hostname: mimir-2
    command: ["-config.file=/etc/mimir.yaml"]
    volumes:
      - ./config/mimir-2.yaml:/etc/mimir.yaml
      - mimir-data:/data
    ports:
      - 9042:8080
      - 27956:27956
      - 29095:29095
    networks:
      - mimir-net
    deploy:
      placement:
        constraints: [node.hostname == server02]
      
volumes:
  mimir-data:

networks:
  mimir-net:
    driver: overlay
    attachable: true

config/mimir-1.yaml

Temp configuration

target: all

common:
  storage:
    backend: s3
    s3:
      endpoint: mydomain.com:8080
      secret_access_key: myaccesskey
      access_key_id: mykeyid
      bucket_name: mimir-storage-test

blocks_storage:
  s3:
    bucket_name: mimir-blocks-test
  storage_prefix: blocks
  tsdb:
    dir: /data/ingester
    block_ranges_period:
        - 2m
    retention_period: 72h0m0s
  bucket_store:
      sync_interval: 2m
      ignore_blocks_within: 0s

querier:
    query_store_after: 0s
    shuffle_sharding_ingesters_enabled: false

compactor:
    cleanup_interval: 15m

alertmanager_storage:
  s3:
    bucket_name: mimir-alertmanager-swarm-test

ruler_storage:
  s3:
    bucket_name: mimir-ruler-swarm-test

memberlist:
  join_members: [server01_IP:17956, server02_IP:27956]
# 27956 in mimir-2.yaml
  bind_port: 17956
  advertise_port: 17956

server:
  log_level: debug
# 29095 in mimir-2.yaml
  grpc_listen_port: 19095
  grpc_listen_address: 0.0.0.0

About docker Network

$ docker network inspect devag_mimir-net
[
    {
        "Name": "devag_mimir-net",
        "Id": "7zevmtq4t4ubrojl14o7g05ps",
        "Created": "2023-07-11T16:15:50.377636913+02:00",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.1.0/24",
                    "Gateway": "10.0.1.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "192055e3655ad296cbaddb22dd2c349d684a4d49811b85b172210e2c31988449": {
                "Name": "devag_mimir-1.1.9qsqos94dzwxlpp6qchyywa2c",
                "EndpointID": "22857515aa89fe85a330d29e78667e9c7ba2f35337cb232f43961fa13da8023c",
                "MacAddress": "02:42:0a:00:01:07",
                "IPv4Address": "10.0.1.7/24",
                "IPv6Address": ""
            },
            "7b0a1f04479da54b17b8b48a84dbced78576b39156f778640ffa55830358a957": {
                "Name": "devag_mimir-2.1.ki36w3rhq880ylvhrllx0f8e7",
                "EndpointID": "2a94f365ea6623526f21559d49d26d2952107f151e0cb8d04c23afc3421ae2cc",
                "MacAddress": "02:42:0a:00:01:08",
                "IPv4Address": "10.0.1.8/24",
                "IPv6Address": ""
...
$ docker network inspect ingress | grep 10.0.0.11 -A1 
-B3
                "Name": "devag_mimir-1.1.9qsqos94dzwxlpp6qchyywa2c",
                "EndpointID": "b4d39cead1f27020208fd4fc500a3a44787bb2af851e06c6948d0c7f8c0b3954",
                "MacAddress": "02:42:0a:00:00:0b",
                "IPv4Address": "10.0.0.11/24",
                "IPv6Address": ""

$ docker network inspect ingress | grep 10.0.0.12 -A1 
-B3
                "Name": "devag_mimir-2.1.ki36w3rhq880ylvhrllx0f8e7",
                "EndpointID": "9abbb7617fe0cfbd69581f901830a168d00f49c1af9a7964955e1551d514fc14",
                "MacAddress": "02:42:0a:00:00:0c",
                "IPv4Address": "10.0.0.12/24",
                "IPv6Address": ""

Memberlist

[akuntsch]

I'm hitting the exact same issue running mimir in monolithic mode with 3 instances on AWS ECS.

Did you figure out what causes the issue?

[Aymen-MTE]

It's an issue with ingress network and the mimir container network configuration . It's looks like ingester can't synchronize with other containers/cluster-member before send data to object_storage because he can't resolve the dns name.
Same issue with Grafana Loki.
grafana/loki#6648

We've switched to solo mode deployment, found in cortex website documentation

server:
  http_listen_port: 9009
  log_level: debug
  # Configure the server to allow messages up to 100MB.
  grpc_server_max_recv_msg_size: 1048576000
  grpc_server_max_send_msg_size: 1048576000
  grpc_server_max_concurrent_streams: 10000

distributor:
  pool:
    health_check_ingesters: true

ingester_client:
  grpc_client_config:
    # Configure the client to allow messages up to 100MB.
    max_recv_msg_size: 1048576000
    max_send_msg_size: 1048576000
    grpc_compression: gzip

ingester:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1

common:
  storage:
    backend: s3
    s3:
      endpoint: "endpoint_url:endpoint_port"
      secret_access_key: "${SECRET_ACCESS_KEY}" 
      access_key_id: "${ACCESS_KEY_ID}" 
      bucket_name: bucket-storage

blocks_storage:
  s3:
    bucket_name: bucket-blocks
  storage_prefix: blocks
  tsdb:
    dir: /data/ingester
limits:
    ingestion_rate: 100000
    ingestion_burst_size: 2000000
    max_label_name_length: 2048
    max_label_value_length: 4096
    max_label_names_per_series: 50
    max_global_series_per_user: 4500000
state: present

[akuntsch]

Thanks for the fast reply. In my case it turned out to be a proxy issue.

Looks like the gRPC library in Mimir picks up the proxy from http_proxy / https_proxy environment variables automatically for gRPC connections

[Marvin0098]

@akuntsch What were the changes you did? For now I set the replica factor to 1, not the best solution but works for now.
I'm also on ECS.