Observability enhancement

[ekexium]

There are some questions that are often asked but hard to answer:

• Error
  • Why does this "region is unavailable" error happen?
  • How does my stale read request fail?
• Latency
  • Why does my request becomes slower? (when we cannot see corresponding latency change in TiKV)
  • Detailed latency decomposition of slow query. Slow log is not good enough SlowQuery: details of PrewriteTime missing, cannot check the real cost of time. pingcap/tidb#45691
• QPS
  • Does the change in QPS result from retrying requests?
  • What are the sources of certain requests of interest? For example, do they come from DDL? Analyze? Other internal SQL? What are the concrete SQLs?

We need more metrics and logs to answer them.

There are 3 levels of request

• Original "request", a call of Scanner.getData , handleSingleBatch, kv snapshot get,batchGetSingleRegion,..
• Region request, a request sent to SendReqCtx
• Lower-level request, a request sent to sendRequest

Metrics

• metrics: durations of region requests, counters of region request retries #986
• metrics: durations of original read requests #989
• metrics: durations in sendRequest #992
• original write requests

Logs

Slow logs

Info and Warn Logs