why store UI shows Invalid date and empty Announced LabelSets

[rgaoistheone]

Hey guys ,
When I browsing the querier UI , it shows invalid date and empty Announced LabelSets

Please ignore the 10.0.144.129:10901 down alert, thats caused by re-create the store sts .

Any ideas?

[yeya24]

Hi @hairongGao, this relates to the internal implementation of the two stores: Sidecar for Prometheus store and the store gateway.

Sidecar has a shipper component that watches your TSDB directory and uploads blocks to your obj store. The min-time of your sidecar will be updated once your TSDB dir is not empty: the shipper will read the timestamp from that block and update the min-time info.

For the announced labelSets of Store gateway, it is empty because there is no block in that bucket. If you have some blocks there, the announced labelSets will be updated to the blocks' labels.

[rgaoistheone]

Thanks so much for explanations , I got another question about thanos-store-gateway integration with remote cluster.
We're trying to collect metrics from several clusters using store-gateway, not sure if it's the right way to do it but here are some of the details.

        args:
        - query
        - --log.level=debug
        - --query.replica-label=replica
        - --store=dnssrv+thanos-store-gateway:10901
        - --store=dns+store-zero.com:80

store-zero.com is a remote store gateway which configured ingress access with istio.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: store-vs
  namespace: istio-system
spec:
  hosts:
    - "store-zero.com"
  gateways:
  - monitoring-gateway
  http:
    - route:
      - destination:
          host: thanos-store-gateway
          port:
            number: 10901
---

But istio keep throwing errors like

2021-03-30T08:41:32.052491497Z [2021-03-30T08:41:27.685Z] "POST /thanos.Store/Info HTTP/2" 200 NR "-" "-" 0 0 0 - "10.0.149.80" "grpc-go/1.29.1" "a0aee11e-1ccf-9b45-9495-020b1df83a6a" "3.227.180.177:80" "-" - - 10.0.142.209:8080 10.0.149.80:49680 - -
2021-03-30T08:41:42.053201774Z [2021-03-30T08:41:32.684Z] "POST /thanos.Store/Info HTTP/2" 200 NR "-" "-" 0 0 0 - "10.0.149.80" "grpc-go/1.29.1" "982ed741-e40c-98f0-89af-69ce82ecb690" "3.227.180.177:80" "-" - - 10.0.142.209:8080 10.0.149.80:22526 - -
2021-03-30T08:41:42.053224950Z [2021-03-30T08:41:37.684Z] "POST /thanos.Store/Info HTTP/2" 200 NR "-" "-" 0 0 0 - "10.0.149.80" "grpc-go/1.29.1" "ed579a4f-6b3d-910c-bfa7-d654cdbf4e5c" "3.224.51.59:80" "-" - - 10.0.142.209:8080 10.0.149.80:42418 - -
2021-03-30T08:41:52.051381882Z [2021-03-30T08:41:42.685Z] "POST /thanos.Store/Info HTTP/2" 200 NR "-" "-" 0 0 0 - "10.0.149.80" "grpc-go/1.29.1" "cc88072f-365c-9e0c-aedb-42ab54c77f49" "3.227.180.177:80" "-" - - 10.0.142.209:8080 10.0.149.80:33381 - -
2021-03-30T08:41:52.051400995Z [2021-03-30T08:41:47.684Z] "POST /thanos.Store/Info HTTP/2" 200 NR "-" "-" 0 0 0 - "10.0.149.80" "grpc-go/1.29.1" "df2e4a89-ffe1-95c0-9754-e37b52581d42" "3.227.180.177:80" "-" - - 10.0.142.209:8080 10.0.149.80:33956 - -

Looks like NR refers to “no route” but I’m really running out solutions , any ideas ?
Ref:
https://discuss.istio.io/t/istio-grpc-gateway-throwing-no-route-errors/10141