Support traffic replay

[djshow832]

Feature Request

Describe your feature request related problem

There are some cases when users want to capture the traffic on the production cluster and replay the traffic on a testing cluster:

• A new TiDB version may have compatibility breakers, such as the statements failing, running slower, or resulting in different query results.
• When the cluster runs unexpectedly, users want to capture the traffic so that they can investigate it later by replaying the traffic.
• Test the maximum throughput of a scaled-up or scaled-down cluster using the real workload instead of Sysbench or TPCC.

Some traffic replay tools are widely used, including tcpcopy, mysql-replay, and query-playback. Tcpcopy and mysql-replay capture data like tcpdump, while query-playback is based on slow logs. Although some of them are built for MySQL, deploying them on the proxy instance also works.

However, they have some limitations:

• Users need to learn the deployment and usage of these tools.
• Tcpcopy only captures new connections, which is unfriendly for persistent connections. Mysql-replay can capture existing connections but it loses session states such as prepared statements and session variables, which may make replay fail.
• The tools replay the traffic with one username and one current schema, which requires modification to the testing cluster.
• Tcpcopy and mysql-replay don't support TLS because they can't decode the encrypted data.
• Users need to verify the results and performance manually.

Describe the feature you'd like

Capturing traffic on one TiDB cluster and replaying the traffic on a new TiDB cluster to verify the SQL compatibility and performance of the new TiDB cluster.

Tasks

Design

• RFC: Traffic Capture and Replay #643

Traffic Capture

P1:

• Support capturing commands, connection ID, and start time #645
• Store traffic to files with rotation and compression #647
• Output the session states to traffic files #674
• Output disconnection to traffic files #676
• Output metadata to the output directory #672
• Filter sensitive commands #681

P2:

• Output execution duration to traffic files
• Output the query result information to the traffic files
• Order by the statements by start time
• Support writing huge commands
• Log a command without query result information if it runs too long
• Support more command types
• Memory control of traffic capture

P3:

• Output traffic files with encryption
• Output traffic files to a remote address

Traffic Replay

P1:

• Connect to TiDB with a username and password #652
• Manage connections in traffic replay #654
• Load and parse traffic files #657
• Replay traffic at a customized speed #659
• Maintain the mapping of prepared statement ID and text #678
• Add a connection ID allocator #687

P2:

• Streaming reading and sending large commands
• Support statement filter for replay

P3:

• Support more auth plugins
• Support replaying traffic as fast as possible
• Support reading from MySQL slow logs
• Support reading from tcpdump files

Comparison and Report

P1:

• Output failed statements and error messages into TiDB tables #661
• Output the capture and replay time of failed statements #683

P2:

• Write failed statements into tables and then update their counts
• Output slower statements and execution time
• Output mismatched statements and their row counts and result set sizes

P3:

• Compare metrics like CPU, memory, QPS, duration

User Interface

P1:

• Manage capture and replay jobs #665
• Support HTTP API to capture and replay traffic #667
• Support TiProxyCtl to capture and replay traffic #669
• Show the history of traffic jobs #685

P2:

• Support SQL to capture and replay traffic
• Add a --force arg to remove the traffic file or report forcibly if it exists
• Show the estimated disk usage of the traffic files

P3:

• Support TiDB Dashboard to capture and replay traffic
• Support TiDB Dashboard to show report