You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, if you run riot in non-cluster mode, and configure it to point to a replica node URI, you will get MOVED responses and the export/replication will fail.
However, it would be useful to be able to export/replicate data directly from replicas so as to avoid putting additional load on the primary nodes. This would theoretically also allow greater throughput because we could run a separate parallel riot process for each shard in the cluster.
I could see this working in one of two ways:
a --read-only flag (or similar) that would tell riot to send a READONLY command before any scan/read operations; then, the source URI could point to a replica node without fear of MOVED responses. In this case, it would be up to the caller of riot to understand the cluster topology and pass in the correct replica URIs, decide whether to run in parallel or in sequence, etc.
abstract this behavior behind some additional flag(s) when running in -c mode, such that riot itself could handle any potential parallelization (perhaps with tuning, e.g. --replica-read-threads N). Conceptually, this would mean the caller wouldn't need to know the details of the cluster topology when invoking riot, but this is probably a much more complex change to implement.
The text was updated successfully, but these errors were encountered:
Looking through the output for file-export --help, I see the --read-from REPLICA option. This covers the ability to read from replicas to keep load off the primary nodes, but doesn't help with parallelization across shards.
Currently, if you run
riot
in non-cluster mode, and configure it to point to a replica node URI, you will getMOVED
responses and the export/replication will fail.However, it would be useful to be able to export/replicate data directly from replicas so as to avoid putting additional load on the primary nodes. This would theoretically also allow greater throughput because we could run a separate parallel
riot
process for each shard in the cluster.I could see this working in one of two ways:
--read-only
flag (or similar) that would tellriot
to send aREADONLY
command before any scan/read operations; then, the source URI could point to a replica node without fear ofMOVED
responses. In this case, it would be up to the caller ofriot
to understand the cluster topology and pass in the correct replica URIs, decide whether to run in parallel or in sequence, etc.-c
mode, such thatriot
itself could handle any potential parallelization (perhaps with tuning, e.g.--replica-read-threads N
). Conceptually, this would mean the caller wouldn't need to know the details of the cluster topology when invokingriot
, but this is probably a much more complex change to implement.The text was updated successfully, but these errors were encountered: