Merge branch 'release-6.5' into optimize/quick-start

TOC-tidb-cloud.md

@@ -122,7 +122,7 @@
     - [Built-in Metrics](/tidb-cloud/built-in-monitoring.md)
     - [Built-in Alerting](/tidb-cloud/monitor-built-in-alerting.md)
     - [Cluster Events](/tidb-cloud/tidb-cloud-events.md)
-    - [Third-Party Metrics Integrations](/tidb-cloud/third-party-monitoring-integrations.md)
+    - [Third-Party Metrics Integrations (Beta)](/tidb-cloud/third-party-monitoring-integrations.md)
   - Tune Performance
     - [Overview](/tidb-cloud/tidb-cloud-tune-performance-overview.md)
     - Analyze Performance

br/backup-and-restore-storages.md

@@ -92,7 +92,7 @@ This section provides some URL examples by using `external` as the `host` parame
 **Back up snapshot data to Amazon S3**
 
 ```shell
-./br restore full -u "${PD_IP}:2379" \
+./br backup full -u "${PD_IP}:2379" \
 --storage "s3://external/backup-20220915?access-key=${access-key}&secret-access-key=${secret-access-key}"
 ```
 

command-line-flags-for-tidb-configuration.md

@@ -113,7 +113,7 @@ When you start the TiDB cluster, you can use command-line options or environment
 
 ## `--proxy-protocol-fallbackable`
 
-- Controls whether to enable PROXY protocol fallback mode. When this parameter is set to `true`, TiDB accepts PROXY client connections and client connections without any PROXY protocol header. By default, TiDB only accepts client connections with a PROXY protocol header.
+- Controls whether to enable PROXY protocol fallback mode. When this parameter is set to `true`, TiDB accepts client connections that belong to `--proxy-protocol-networks` without using the PROXY protocol specification or without sending a PROXY protocol header. By default, TiDB only accepts client connections that belong to `--proxy-protocol-networks` and send a PROXY protocol header.
 - Default value: `false`
 
 ## `--proxy-protocol-networks`

develop/dev-guide-aws-appflow-integration.md

@@ -244,5 +244,4 @@ test> SELECT * FROM sf_account;
 
 - If anything goes wrong, you can navigate to the [CloudWatch](https://console.aws.amazon.com/cloudwatch/home) page on the AWS Management Console to get logs.
 - The steps in this document are based on [Building custom connectors using the Amazon AppFlow Custom Connector SDK](https://aws.amazon.com/blogs/compute/building-custom-connectors-using-the-amazon-appflow-custom-connector-sdk/).
-- [TiDB Serverless](https://docs.pingcap.com/tidbcloud/select-cluster-tier#tidb-serverless-beta) is **NOT** a production environment.
 - To prevent excessive length, the examples in this document only show the `Insert` strategy, but `Update` and `Upsert` strategies are also tested and can be used.

develop/dev-guide-build-cluster-in-cloud.md

@@ -46,15 +46,15 @@ This document walks you through the quickest way to get started with TiDB Cloud.
 
 > **Note:**
 >
-> For [TiDB Serverless clusters](https://docs.pingcap.com/tidbcloud/select-cluster-tier#tidb-serverless-beta), when you connect to your cluster, you must include the prefix for your cluster in the user name and wrap the name with quotation marks. For more information, see [User name prefix](https://docs.pingcap.com/tidbcloud/select-cluster-tier#user-name-prefix).
+> For [TiDB Serverless clusters](https://docs.pingcap.com/tidbcloud/select-cluster-tier#tidb-serverless), when you connect to your cluster, you must include the prefix for your cluster in the user name and wrap the name with quotation marks. For more information, see [User name prefix](https://docs.pingcap.com/tidbcloud/select-cluster-tier#user-name-prefix).
 
 </CustomContent>
 
 <CustomContent platform="tidb-cloud">
 
 > **Note:**
 >
-> For [TiDB Serverless clusters](/tidb-cloud/select-cluster-tier.md#tidb-serverless-beta), when you connect to your cluster, you must include the prefix for your cluster in the user name and wrap the name with quotation marks. For more information, see [User name prefix](/tidb-cloud/select-cluster-tier.md#user-name-prefix).
+> For [TiDB Serverless clusters](/tidb-cloud/select-cluster-tier.md#tidb-serverless), when you connect to your cluster, you must include the prefix for your cluster in the user name and wrap the name with quotation marks. For more information, see [User name prefix](/tidb-cloud/select-cluster-tier.md#user-name-prefix).
 
 </CustomContent>
 

develop/dev-guide-third-party-support.md

@@ -251,8 +251,8 @@ If you encounter problems when connecting to TiDB using the tools listed in this
       <tr>
          <td rowspan="4">Python</td>
          <td><a href="https://pypi.org/project/Django/" target="_blank" referrerpolicy="no-referrer-when-downgrade">Django</a></td>
-         <td>v4.0.5</td>
-         <td>Compatible</td>
+         <td>v4.1</td>
+         <td>Full</td>
          <td><a href="https://github.com/pingcap/django-tidb" target="_blank" referrerpolicy="no-referrer-when-downgrade">django-tidb</a></td>
          <td>N/A</td>
       </tr>

hardware-and-software-requirements.md

@@ -37,14 +37,14 @@ In v6.5 LTS, TiDB provides multi-level support for different quality standards o
 
     |  Operating systems   |   Supported CPU architectures   |
     |   :---   |   :---   |
-    |   macOS Catalina or later   |  <ul><li>x86_64</li><li>ARM 64</li></ul>  |
-    |  Oracle Enterprise Linux 7.3 or a later 7.x version  |  x86_64           |
-    |   Ubuntu LTS 18.04 or later   |  x86_64           |
+    | macOS 12 (Monterey) or later |  <ul><li>x86_64</li><li>ARM 64</li></ul>  |
+    | Oracle Enterprise Linux 7.3 or a later 7.x version  |  x86_64           |
+    | Ubuntu LTS 18.04 or later   |  x86_64           |
     | CentOS 8 Stream | <ul><li>x86_64</li><li>ARM 64</li></ul> |
-    |  Debian 9 (Stretch) or later |  x86_64           |
-    |  Fedora 35 or later   |  x86_64           |
-    |  openSUSE Leap later than v15.3 (not including Tumbleweed) |  x86_64           |
-    |  SUSE Linux Enterprise Server 15  |  x86_64                        |
+    | Debian 9 (Stretch) or later |  x86_64           |
+    | Fedora 35 or later   |  x86_64           |
+    | openSUSE Leap later than v15.3 (not including Tumbleweed) |  x86_64           |
+    | SUSE Linux Enterprise Server 15  |  x86_64                        |
 
     > **Note:**
     >

information-schema/information-schema-slow-query.md

@@ -11,7 +11,7 @@ The `SLOW_QUERY` table provides the slow query information of the current node,
 
 > **Note:**
 >
-> The `SLOW_QUERY` table is unavailable for [TiDB Serverless clusters](/tidb-cloud/select-cluster-tier.md#tidb-serverless-beta).
+> The `SLOW_QUERY` table is unavailable for [TiDB Serverless clusters](/tidb-cloud/select-cluster-tier.md#tidb-serverless).
 
 </CustomContent>
 

sql-statements/sql-statement-flashback-to-timestamp.md

@@ -11,7 +11,7 @@ TiDB v6.4.0 introduces the `FLASHBACK CLUSTER TO TIMESTAMP` syntax. You can use
 
 > **Warning:**
 >
-> The `FLASHBACK CLUSTER TO TIMESTAMP` syntax is not applicable to [TiDB Serverless](/tidb-cloud/select-cluster-tier.md#tidb-serverless-beta) clusters. Do not execute this statement on TiDB Serverless clusters to avoid unexpected results.
+> The `FLASHBACK CLUSTER TO TIMESTAMP` syntax is not applicable to [TiDB Serverless](/tidb-cloud/select-cluster-tier.md#tidb-serverless) clusters. Do not execute this statement on TiDB Serverless clusters to avoid unexpected results.
 
 </CustomContent>
 

statement-summary-tables.md

@@ -19,7 +19,7 @@ Therefore, starting from v4.0.0-rc.1, TiDB provides system tables in `informatio
 
 > **Note:**
 >
-> The following tables are unavailable for [TiDB Serverless clusters](/tidb-cloud/select-cluster-tier.md#tidb-serverless-beta): `statements_summary`, `statements_summary_history`, `cluster_statements_summary`, and `cluster_statements_summary_history`.
+> The following tables are unavailable for [TiDB Serverless clusters](/tidb-cloud/select-cluster-tier.md#tidb-serverless): `statements_summary`, `statements_summary_history`, `cluster_statements_summary`, and `cluster_statements_summary_history`.
 
 </CustomContent>
 

system-variables.md

@@ -1116,7 +1116,7 @@ MPP is a distributed computing framework provided by the TiFlash engine, which a
 
 > **Note:**
 >
-> To improve the speed for index creation using this variable, make sure that your TiDB cluster is hosted on AWS and your TiDB node size is at least 8 vCPU. For [TiDB Serverless](/tidb-cloud/select-cluster-tier.md#tidb-serverless-beta) clusters, this feature is unavailable.
+> To improve the speed for index creation using this variable, make sure that your TiDB cluster is hosted on AWS and your TiDB node size is at least 8 vCPU. For [TiDB Serverless](/tidb-cloud/select-cluster-tier.md#tidb-serverless) clusters, this feature is unavailable.
 
 </CustomContent>
 

ticdc/ticdc-bidirectional-replication.md

@@ -38,16 +38,36 @@ After the configuration takes effect, the clusters can perform bi-directional re
 
 ## Execute DDL
 
-Bi-directional replication does not support replicating DDL statements.
-
-If you need to execute DDL statements, take the following steps:
-
-1. Pause the write operations in the tables that need to execute DDL in all clusters. If the DDL statement is adding a non-unique index, skip this step.
+After the bidirectional replication is enabled, TiCDC does not replicate any DDL statements. You need to execute DDL statements in the upstream and downstream clusters respectively.
+
+Note that some DDL statements might cause table structure changes or data change time sequence problems, which might lead to data inconsistency after the replication. Therefore, after enabling bidirectional replication, only the DDL statements in the following table can be executed without stopping the write operations of the application.
+
+| Event | Does it cause changefeed errors | Note |
+|---|---|---|
+| create database | Yes | After you manually execute the DDL statements in the upstream and downstream clusters, the errors can be automatically recovered. |
+| drop database | Yes | You need to manually restart the changefeed and specify `--overwrite-checkpoint-ts` as the `commitTs` of the DDL statement to recover the errors. |
+| create table | Yes | After you manually execute the DDL statements in the upstream and downstream clusters, the errors can be automatically recovered. |
+| drop table | Yes | You need to manually restart the changefeed and specify `--overwrite-checkpoint-ts` as the `commitTs` of the DDL statement to recover the errors. |
+| alter table comment | No |  |
+| rename index | No |  |
+| alter table index visibility | No |  |
+| add partition | Yes | After you manually execute the DDL statements in the upstream and downstream clusters, the errors can be automatically recovered. |
+| drop partition | No |  |
+| create view | No |  |
+| drop view | No |  |
+| alter column default value | No |  |
+| reorganize partition | Yes | After you manually execute the DDL statements in the upstream and downstream clusters, the errors can be automatically recovered. |
+| alter table ttl | No |  |
+| alter table remove ttl | No |  |
+| add **not unique** index | No |  |
+| drop **not unique** index | No |  |
+
+If you need to execute DDL statements that are not in the preceding table, take the following steps:
+
+1. Pause the write operations in the tables that need to execute DDL in all clusters.
 2. After the write operations of the corresponding tables in all clusters have been replicated to other clusters, manually execute all DDL statements in each TiDB cluster.
 3. After the DDL statements are executed, resume the write operations.
 
-Note that a DDL statement that adds non-unique index does not break bi-directional replication, so you do not need to pause the write operations in the corresponding table.
-
 ## Stop bi-directional replication
 
 After the application has stopped writing data, you can insert a special record into each cluster. By checking the two special records, you can make sure that data in two clusters are consistent.

ticdc/ticdc-changefeed-config.md

@@ -188,4 +188,21 @@ storage = ""
 # Specifies whether to store the redo log in a file.
 # The default value is false.
 use-file-backend = false
+
+# The following configuration items only take effect when the downstream is Kafka. Supported starting from v6.5.3.
+[sink.kafka-config]
+# The mechanism of Kafka SASL authentication. The default value is empty, indicating that SASL authentication is not used.
+sasl-mechanism = "OAUTHBEARER"
+# The client-id in the Kafka SASL OAUTHBEARER authentication. The default value is empty. This parameter is required when the OAUTHBEARER authentication is used.
+sasl-oauth-client-id = "producer-kafka"
+# The client-secret in the Kafka SASL OAUTHBEARER authentication. The default value is empty. This parameter is required when the OAUTHBEARER authentication is used.
+sasl-oauth-client-secret = "cHJvZHVjZXIta2Fma2E="
+# The token-url in the Kafka SASL OAUTHBEARER authentication to obtain the token. The default value is empty. This parameter is required when the OAUTHBEARER authentication is used.
+sasl-oauth-token-url = "http://127.0.0.1:4444/oauth2/token"
+# The scopes in the Kafka SASL OAUTHBEARER authentication. The default value is empty. This parameter is optional when the OAUTHBEARER authentication is used.
+sasl-oauth-scopes = ["producer.kafka", "consumer.kafka"]
+# The grant-type in the Kafka SASL OAUTHBEARER authentication. The default value is "client_credentials". This parameter is optional when the OAUTHBEARER authentication is used.
+sasl-oauth-grant-type = "client_credentials"
+# The audience in the Kafka SASL OAUTHBEARER authentication. The default value is empty. This parameter is optional when the OAUTHBEARER authentication is used.
+sasl-oauth-audience = "kafka"
 ```

ticdc/ticdc-faq.md

@@ -284,3 +284,33 @@ This feature is currently not supported, which might be supported in a future re
 ## Does TiCDC replication get stuck if the upstream has long-running uncommitted transactions?
 
 TiDB has a transaction timeout mechanism. When a transaction runs for a period longer than [`max-txn-ttl`](/tidb-configuration-file.md#max-txn-ttl), TiDB forcibly rolls it back. TiCDC waits for the transaction to be committed before proceeding with the replication, which causes replication delay.
+
+## Why can't I use the `cdc cli` command to operate a TiCDC cluster deployed by TiDB Operator?
+
+This is because the default port number of the TiCDC cluster deployed by TiDB Operator is `8301`, while the default port number of the `cdc cli` command to connect to the TiCDC server is `8300`. When using the `cdc cli` command to operate the TiCDC cluster deployed by TiDB Operator, you need to explicitly specify the `--server` parameter, as follows:
+
+```shell
+./cdc cli changefeed list --server "127.0.0.1:8301"
+[
+  {
+    "id": "4k-table",
+    "namespace": "default",
+    "summary": {
+      "state": "stopped",
+      "tso": 441832628003799353,
+      "checkpoint": "2023-05-30 22:41:57.910",
+      "error": null
+    }
+  },
+  {
+    "id": "big-table",
+    "namespace": "default",
+    "summary": {
+      "state": "normal",
+      "tso": 441872834546892882,
+      "checkpoint": "2023-06-01 17:18:13.700",
+      "error": null
+    }
+  }
+]
+```

ticdc/ticdc-filter.md

@@ -87,3 +87,40 @@ Description of configuration parameters:
 >
 > - When TiDB updates a value in the column of the clustered index, TiDB splits an `UPDATE` event into a `DELETE` event and an `INSERT` event. TiCDC does not identify such events as an `UPDATE` event and thus cannot correctly filter out such events.
 > - When you configure a SQL expression, make sure all tables that matches `matcher` contain all the columns specified in the SQL expression. Otherwise, the replication task cannot be created. In addition, if the table schema changes during the replication, which results in a table no longer containing a required column, the replication task fails and cannot be resumed automatically. In such a situation, you must manually modify the configuration and resume the task.
+
+## DDL allow list
+
+Currently, TiCDC uses an allow list to replicate DDL statements. Only the DDL statements in the allow list are replicated to the downstream. The DDL statements not in the allow list are not replicated to the downstream.
+
+The allow list of DDL statements supported by TiCDC is as follows:
+
+- create database
+- drop database
+- create table
+- drop table
+- add column
+- drop column
+- create index / add index
+- drop index
+- truncate table
+- modify column
+- rename table
+- alter column default value
+- alter table comment
+- rename index
+- add partition
+- drop partition
+- truncate partition
+- create view
+- drop view
+- alter table character set
+- alter database character set
+- recover table
+- add primary key
+- drop primary key
+- rebase auto id
+- alter table index visibility
+- exchange partition
+- reorganize partition
+- alter table ttl
+- alter table remove ttl

ticdc/ticdc-open-api.md

@@ -167,7 +167,7 @@ The configuration parameters of sink are as follows:
 
 ### Example
 
-The following request creates a replication task with an ID of `test5` and a `sink_uri` of `blackhome://`.
+The following request creates a replication task with an ID of `test5` and a `sink_uri` of `blackhole://`.
 
 {{< copyable "shell-regular" >}}
 

ticdc/ticdc-sink-to-cloud-storage.md

@@ -31,6 +31,7 @@ The output is as follows:
 Info: {"upstream_id":7171388873935111376,"namespace":"default","id":"simple-replication-task","sink_uri":"s3://logbucket/storage_test?protocol=canal-json","create_time":"2023-03-10T18:52:05.566016967+08:00","start_ts":437706850431664129,"engine":"unified","config":{"case_sensitive":true,"enable_old_value":true,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"],"event_filters":null},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","schema_registry":"","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false},"column_selectors":null,"transaction_atomicity":"none","encoder_concurrency":16,"terminator":"\r\n","date_separator":"none","enable_partition_separator":false},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"storage":""}},"state":"normal","creator_version":"v6.5.3-master-dirty"}
 ```
 
+- `--server`: The address of any TiCDC server in the TiCDC cluster.
 - `--changefeed-id`: The ID of the changefeed. The format must match the `^[a-zA-Z0-9]+(\-[a-zA-Z0-9]+)*$` regular expression. If this ID is not specified, TiCDC automatically generates a UUID (the version 4 format) as the ID.
 - `--sink-uri`: The downstream address of the changefeed. For details, see [Configure sink URI](#configure-sink-uri).
 - `--start-ts`: The starting TSO of the changefeed. TiCDC starts pulling data from this TSO. The default value is the current time.