Skip to content

Commit

Permalink
update doc
Browse files Browse the repository at this point in the history
  • Loading branch information
@mymeiyi
mymeiyi committed Nov 2, 2023
1 parent 05eb1d1 commit a116942
Show file tree
Hide file tree
Showing 2 changed files with 132 additions and 10 deletions.
71 changes: 66 additions & 5 deletions docs/en/docs/data-operate/import/import-way/group-commit-manual.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,8 @@ curl --location-trusted -u {user}:{passwd} -T data.csv -H "group_commit:true" -
# The retured label is start with 'group_commit', which is the label of the real load job
```

See [Stream Load](stream-load-manual.md) for more detailed syntax used by **Stream Load**.

### Http Stream

```sql
Expand Down Expand Up @@ -142,6 +144,8 @@ curl --location-trusted -u {user}:{passwd} -T data.csv -H "group_commit:true" -
# The retured label is start with 'group_commit', which is the label of the real load job
```

See [Stream Load](stream-load-manual.md) for more detailed syntax used by **Http Stream**.

### Use `PreparedStatement`

To reduce the CPU cost of SQL parsing and query planning, we provide the `PreparedStatement` in the FE. When using `PreparedStatement`, the SQL and its plan will be cached in the session level memory cache and will be reused later on, which reduces the CPU cost of FE. The following is an example of using PreparedStatement in JDBC:
Expand All @@ -155,13 +159,70 @@ url = jdbc:mysql://127.0.0.1:9030/db?useServerPrepStmts=true
2. Using `PreparedStatement`

```java
PreparedStatement statement = conn.prepareStatement("INSERT INTO dt VALUES (?, ?, ?)");
statement.setInt(1, 5);
statement.setString(2, "Tiger");
statement.setInt(3, 100);
int rows = statement.executeUpdate();
private static final String JDBC_DRIVER = "com.mysql.jdbc.Driver";
private static final String URL_PATTERN = "jdbc:mysql://%s:%d/%s?useServerPrepStmts=true";
private static final String HOST = "127.0.0.1";
private static final int PORT = 9087;
private static final String DB = "db";
private static final String TBL = "dt";
private static final String USER = "root";
private static final String PASSWD = "";
private static final int INSERT_BATCH_SIZE = 10;

private static void groupCommitInsert() throws Exception {
Class.forName(JDBC_DRIVER);
try (Connection conn = DriverManager.getConnection(String.format(URL_PATTERN, HOST, PORT, DB), USER, PASSWD)) {
// enable session variable 'enable_insert_group_commit'
try (Statement statement = conn.createStatement()) {
statement.execute("SET enable_insert_group_commit = true;");
}

String query = "insert into " + TBL + " values(?, ?, ?)";
try (PreparedStatement stmt = conn.prepareStatement(query)) {
for (int i = 0; i < INSERT_BATCH_SIZE; i++) {
stmt.setInt(1, i);
stmt.setString(2, "name" + i);
stmt.setInt(3, i + 10);
int result = stmt.executeUpdate();
System.out.println("rows: " + result);
}
}
} catch (Exception e) {
e.printStackTrace();
}
}

private static void groupCommitInsertBatch() throws Exception {
Class.forName(JDBC_DRIVER);
// rewriteBatchedStatements=true
try (Connection conn = DriverManager.getConnection(
String.format(URL_PATTERN + "&rewriteBatchedStatements=true", HOST, PORT, DB), USER, PASSWD)) {
// enable session variable 'enable_insert_group_commit'
try (Statement statement = conn.createStatement()) {
statement.execute("SET enable_insert_group_commit = true;");
}

String query = "insert into " + TBL + " values(?, ?, ?)";
try (PreparedStatement stmt = conn.prepareStatement(query)) {
for (int j = 0; j < 5; j++) {
// 10 rows per insert
for (int i = 0; i < INSERT_BATCH_SIZE; i++) {
stmt.setInt(1, i);
stmt.setString(2, "name" + i);
stmt.setInt(3, i + 10);
stmt.addBatch();
}
int[] result = stmt.executeBatch();
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
```

See [Synchronize Data Using Insert Method](../import-scenes/jdbc-load.md) for more details about **JDBC**.

## Relevant system configuration

### Session variable
Expand Down
71 changes: 66 additions & 5 deletions docs/zh-CN/docs/data-operate/import/import-way/group-commit-manual.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,8 @@ curl --location-trusted -u {user}:{passwd} -T data.csv -H "group_commit:true" -
# 返回的Label是group_commit开头的，是真正消费数据的导入关联的label
```

关于 Stream Load 使用的更多详细语法及最佳实践，请参阅 [Stream Load](stream-load-manual.md)。

### Http Stream

```sql
Expand Down Expand Up @@ -142,6 +144,8 @@ curl --location-trusted -u {user}:{passwd} -T data.csv -H "group_commit:true" -
# 返回的Label是group_commit开头的，是真正消费数据的导入关联的label
```

关于 Http Stream 使用的更多详细语法及最佳实践，请参阅 [Stream Load](stream-load-manual.md)。

### 使用`PreparedStatement`

为了减少 SQL 解析和生成规划的开销， 我们在 FE 端支持了 MySQL 协议的`PreparedStatement`特性。当使用`PreparedStatement`时，SQL 和其导入规划将被缓存到 Session 级别的内存缓存中，后续的导入直接使用缓存对象，降低了 FE 的 CPU 压力。下面是在 JDBC 中使用 PreparedStatement 的例子：
Expand All @@ -155,13 +159,70 @@ url = jdbc:mysql://127.0.0.1:9030/db?useServerPrepStmts=true
2. 使用 `PreparedStatement`

```java
PreparedStatement statement = conn.prepareStatement("INSERT INTO dt VALUES (?, ?, ?)");
statement.setInt(1, 5);
statement.setString(2, "Tiger");
statement.setInt(3, 100);
int rows = statement.executeUpdate();
private static final String JDBC_DRIVER = "com.mysql.jdbc.Driver";
private static final String URL_PATTERN = "jdbc:mysql://%s:%d/%s?useServerPrepStmts=true";
private static final String HOST = "127.0.0.1";
private static final int PORT = 9087;
private static final String DB = "db";
private static final String TBL = "dt";
private static final String USER = "root";
private static final String PASSWD = "";
private static final int INSERT_BATCH_SIZE = 10;

private static void groupCommitInsert() throws Exception {
Class.forName(JDBC_DRIVER);
try (Connection conn = DriverManager.getConnection(String.format(URL_PATTERN, HOST, PORT, DB), USER, PASSWD)) {
// enable session variable 'enable_insert_group_commit'
try (Statement statement = conn.createStatement()) {
statement.execute("SET enable_insert_group_commit = true;");
}

String query = "insert into " + TBL + " values(?, ?, ?)";
try (PreparedStatement stmt = conn.prepareStatement(query)) {
for (int i = 0; i < INSERT_BATCH_SIZE; i++) {
stmt.setInt(1, i);
stmt.setString(2, "name" + i);
stmt.setInt(3, i + 10);
int result = stmt.executeUpdate();
System.out.println("rows: " + result);
}
}
} catch (Exception e) {
e.printStackTrace();
}
}

private static void groupCommitInsertBatch() throws Exception {
Class.forName(JDBC_DRIVER);
// rewriteBatchedStatements=true
try (Connection conn = DriverManager.getConnection(
String.format(URL_PATTERN + "&rewriteBatchedStatements=true", HOST, PORT, DB), USER, PASSWD)) {
// enable session variable 'enable_insert_group_commit'
try (Statement statement = conn.createStatement()) {
statement.execute("SET enable_insert_group_commit = true;");
}

String query = "insert into " + TBL + " values(?, ?, ?)";
try (PreparedStatement stmt = conn.prepareStatement(query)) {
for (int j = 0; j < 5; j++) {
// 10 rows per insert
for (int i = 0; i < INSERT_BATCH_SIZE; i++) {
stmt.setInt(1, i);
stmt.setString(2, "name" + i);
stmt.setInt(3, i + 10);
stmt.addBatch();
}
int[] result = stmt.executeBatch();
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
```

关于**JDBC**的更多用法，参考[使用Insert方式同步数据](../import-scenes/jdbc-load.md)。

## 相关系统配置

### Session变量
Expand Down

0 comments on commit a116942

Please sign in to comment.