GCS bucket name and storage prefix

[SunSparc]

(Originally posted to the Community Forums.)

          common:
            storage:
              backend: gcs
              gcs:
                bucket_name: my-unique-bucket-name
                service_account: | ${GCP_CREDENTIALS}

          blocks_storage:
            backend: gcs
            storage_prefix: blocks
          alertmanager_storage:
            backend: gcs
            storage_prefix: alertmanager
          ruler_storage:
            backend: gcs
            storage_prefix: ruler

The result is:

error validating config: invalid bucket config: ruler storage:
  GCS bucket name and storage prefix cannot be the same
  as the one used in blocks storage config

I am not sure why I am getting this error. The storage_prefix is obviously different in both the ruler_storage and the blocks_storage config sections.

I am unable to discover any clarification on this in the documentation. However, I do see that Mimir will fail if I use the same bucket name and storage prefix. The bucket name is the same for all, which is why I am specifying a storage_prefix. Do they all require a different bucket name? If that is the case, then the storage_prefix is useless.

[56quarters]

They shouldn't need different bucket names because that would indeed make the prefix useless. I'm not able to replicate this locally.
Can you provide more details about how Mimir is being run: Are you using the helm chart? Any CLI flags being passed that might override the configuration?

[SunSparc]

@56quarters, this is running in a Nomad cluster.

mimir --config.file=${NOMAD_TASK_DIR}/config.yaml -config.expand-env=true

This is the config.yaml in the current state:

target: all

multitenancy_enabled: false

server:
  http_listen_port: {{ env "NOMAD_PORT_http" }}
  grpc_listen_port: {{ env "NOMAD_PORT_grpc" }}
  log_level: info

#common:
#  storage:
#    backend: gcs
#    gcs:
#      bucket_name: my-unique-bucket-name
#      service_account: | ${GCP_CREDENTIALS}

blocks_storage:
  backend: gcs
    bucket_name: my-unique-bucket-name
  gcs:
    bucket_name: my-unique-bucket-name
    service_account: | ${GCP_CREDENTIALS}
  storage_prefix: blocks

ruler_storage:
  backend: filesystem
  filesystem:
    dir: /tmp/mimir/data/ruler
  #backend: gcs
  #  bucket_name: my-unique-bucket-name
  #storage_prefix: ruler

compactor:
  sharding_ring:
      kvstore:
          store: consul

distributor:
  ring:
      kvstore:
          store: consul

ingester:
  ring:
      kvstore:
          store: consul

ruler:
  ring:
      kvstore:
          store: consul

store_gateway:
  sharding_ring:
      kvstore:
          store: consul

I was able to get Mimir to actually start when I removed the common section and changed the ruler_storage.backend to filesystem. Although it still does not want to startup.

caller=ingester.go:279 level=warn msg="-blocks-storage.backend=filesystem is for development and testing only;
  you should switch to an external object store for production use or use a shared filesystem"
caller=modules.go:593 level=info msg="The ruler is not being started because you need to configure the ruler storage."
caller=modules.go:603 level=info msg="RulerStorage is nil.  Not starting the ruler."

[SunSparc]

I think the problem was that I had an extra bucket_name specified. Apparently when something is wrong in the config, Mimir silently takes anything it understood before that and then applies the defaults to the rest. It would have been helpful if I had seen an error about that earlier.

[pracucci]

I think the problem was that I had an extra bucket_name specified. Apparently when something is wrong in the config, Mimir silently takes anything it understood before that and then applies the defaults to the rest. It would have been helpful if I had seen an error about that earlier.

This not expected and, if confirmed, looks like a bug. Could you open an issue with the smallest YAML config possible to reproduce it, please?

[SunSparc]

Ok, I just tried to reproduce this and was warned immediately. So my conclusion was apparently incorrect. I had a lot of config changes happening as I was trying to get the system working. The problem must have been elsewhere. And now the system is working. Sorry about that. If I am able to identify a problem in the future I will certainly submit an issue.

[joshaltcode]

What was your final config ? Im having a similar issues with getting mimir to use gcp buckets .. I keep getting errors bucket access denied or the account does not have storage.objects.get even though I assigned the object admin role.

[SunSparc]

Here is my config.yaml file:

# "all" includes: compactor, distributor, ingester, querier, query-frontend, ruler, store-gateway
target: all

multitenancy_enabled: false

server:
  http_listen_address: 10.1.0.5
  http_listen_port: 45678
  grpc_listen_address: 10.1.0.6
  grpc_listen_port: 45679
  log_level: warn

common:
  storage:
    backend: gcs
    gcs:
      bucket_name: mimir-bucket-name
      #service_account: # when not specified, defaults to using the GOOGLE_APPLICATION_CREDENTIALS env var

blocks_storage:
  storage_prefix: blocks
  tsdb:
    dir: /local/mimir/tsdb

frontend_worker:
  frontend_address: 10.1.0.6:45679
  dns_lookup_duration: 60m

frontend: # configures the query-frontend
  instance_interface_names:
    - eth0

ruler_storage:
  storage_prefix: ruler

compactor:
  sharding_ring:
    kvstore:
      store: consul
    instance_interface_names:
      - eth0

distributor:
  ring:
    kvstore:
      store: consul
    instance_interface_names:
      - eth0

ingester:
  ring:
    kvstore:
      store: consul
    instance_interface_names:
      - eth0

ruler:
  ring:
    kvstore:
      store: consul
    instance_interface_names:
      - eth0

store_gateway:
  sharding_ring:
    kvstore:
      store: consul
    instance_interface_names:
      - eth0

alertmanager:
  sharding_ring:
    kvstore:
      store: consul
    instance_interface_names:
      - eth0

query_scheduler:
  ring:
    kvstore:
      store: consul
    instance_interface_names:
      - eth0

overrides_exporter:
  ring:
    enabled: false
    kvstore:
      store: consul
    instance_interface_names:
      - eth0

limits:

  ingestion_rate: 340000
  ingestion_burst_size: 3400000

  max_global_series_per_user: 9000000
  max_label_names_per_series: 60

[celesteking]

I don't understand - should storage_prefix be different for different components or not? Docs say it's "" by default, which means it's equal for all. And that's confusing.