[Receiver Hashring Observation][v0.22.0] Default soft tenancy is used always when it is first entry in hashrings.json

[gunjangarge]

Hi All,

While playing around with Thanos Receiver v0.22.0, I observed that if the default hashring is the first object in hashrings.json then the hard tenancy is ignored.

[ { "hashring": "default" }, { "hashring": "tenant-a", "tenants": ["tenant-a"] } ]

For example, in the above hashrings.json, all remote write requests would be sent to the default receiver even if it contains the correct header value i.e. THANOS-TENANT: "tenant-a".
(data would be written in "tenant-a" tsdb created on the default receiver pod)

If I swap the order as below, then all remote write requests, with THANOS-TENANT: "tenant-a" header, are handled by the tenant-a receiver (data would be written in tenant-a tsdb created on tenant-a receiver pod). All other remote write requests are handled by the default receiver (data would be written in "default-tenant" tsdb created on the default receiver pod)

[ { "hashring": "tenant-a", "tenants": ["tenant-a"] }, { "hashring": "default" } ]

Has anyone observed such behavior? Is this expected?

[GiedriusS]

@ianbillett maybe you have some insights?

[pepov]

It looks like this is the way it works, although the documentation doesn't mention this explicitly. The initial proposal docs are somewhat misleading as it states that

A remote write request can be initially received by any receiver node, however, will only be dispatched to receiver endpoints that correspond to that hard tenant.

And then elaborates further based on the example, which shows an ordering where the soft tenant comes last so in that case it will work.

I beleive it would be useful to mention somewhere that the hashring config is processed in order and the first matching hashring will be used eagerly for any given tenant.

Original proposal, where the hashring concept is elaborated:
https://thanos.io/tip/proposals-done/201812-thanos-remote-receive.md/

Code that process the hashrings in order:
https://github.com/thanos-io/thanos/blob/v0.22.0/pkg/receive/hashring.go#L105

[TaylorMutch]

I also ran into this a while back (see here) and indeed this is the behavior (and not well documented). Would definitely recommend adding clarification to the docs, or if it's not an issue, we could change the behavior to not care about order. Intuitively it doesn't make sense to me why order should matter if the soft tenancy is what will be defaulted if there isn't a hard tenant set.

[squat]

Order absolutely matters today though because multiple hashrings can declare support for a single tenant. The hashrings file is a list of hashrings with a list of tenants. For example, the following is a totally valid hashrings.json:

[
    {
        "hashring": "1",
        "tenants": [
            "foo"
        ],
        "endpoints": [
            "foo:10901"
        ]
    },
    {
        "hashring": "2",
        "tenants": [
            "foo",
            "bar"
        ],
        "endpoints": [
            "foo:10901",
            "bar:10901"
        ]
    }
]

If order did not matter, then which hashring would be the one selected for tenant foo? Hashring 1 or hashring 2?

[pepov]

In what scenario will then hashring 2 be used for tenant "foo"?

[squat]

It won't be used for foo but the point is that we can't make the code order-independent without changing configuration format because otherwise all predictability goes out the window