Redis: Implement double write client

[Dav1dde]

For lossless Redis migration we will need to make it possible for Relay to double write quotas into two separate clusters. It should be possible to configure a 'double write' Redis client, with a primary and secondary (or multiple secondaries), the client only reads from the primary but writes to all configured Redis instances.

Sketeched implementation:

Config:

redis:
    a_good_namespace_for_multi_write:
    - url: redis://redis1 # primary
    - url: redis://redis2

# for the multi config
redis:
    project_configs:
        a_good_namespace_for_multi_write:
        - url: redis://redis1 # primary
        - url: redis://redis2
    misc: # No multi write
       url: redis://redis1 

Implementation:

There is already a Client Wrapper in relay-redis, implement a MultiWriteOrBetterNamedRedisClient which dispatches writes to all of it's clients but only reads from the primary. Writes should happen in concurrently (joined).

Error Behaviour:

Log errors of secondary nodes but not fail and only use the primary response.

[Dav1dde]

Planning this for early next week (~23.9.), ideally we'd like to separate the Redis cluster in the week after (~30.9.).

[mwarkentin]

Following this issue, rc-ingest in saas is currently on a quite old version of redis which doesn't support ACLs / read-only users (that I can see).

Apparently this is a blocker for migration to memorystore / newer redis version. This will be helpful for https://getsentry.atlassian.net/browse/SRE-307.