[Enhancement](Nereids) Nereids supports group_commit with insert #32523
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
TPC-H: Total hot run time: 38135 ms |
TPC-DS: Total hot run time: 180904 ms |
TPC-H: Total hot run time: 37418 ms |
TPC-DS: Total hot run time: 181604 ms |
|
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' |
|
run buildall |
TPC-H: Total hot run time: 38484 ms |
TPC-DS: Total hot run time: 186787 ms |
|
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' |
|
run buildall |
TPC-H: Total hot run time: 38355 ms |
TPC-DS: Total hot run time: 186015 ms |
|
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' |
|
run buildall |
TPC-H: Total hot run time: 38428 ms |
TPC-DS: Total hot run time: 185943 ms |
|
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' |
TPC-H: Total hot run time: 37736 ms |
TPC-DS: Total hot run time: 184702 ms |
|
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' |
|
PR approved by at least one committer and no changes requested. |
github-actions
bot
added
the
approved
mymeiyi
reviewed
Jun 4, 2024
| @Override | ||
| protected void onFail(Throwable t) { | ||
| errMsg = t.getMessage() == null ? "unknown reason" : t.getMessage(); | ||
| StringBuilder sb = new StringBuilder(); |
mymeiyi
reviewed
Jun 4, 2024
| @Override | ||
| protected void beforeExec() { | ||
| String queryId = DebugUtil.printId(ctx.queryId()); | ||
| LOG.info("start insert [{}] with query id {} and txn id {}", labelName, queryId, txnId); |
There was a problem hiding this comment.
the txnId is -1, no need to log it
mymeiyi
reviewed
Jun 4, 2024
| try { | ||
| handleGroupCommit(ctx, sink, olapSink, planner); | ||
| } catch (TException | RpcException | ExecutionException | InterruptedException e) { | ||
| LOG.error("errors when group commit insert. {}", e); |
mymeiyi
reviewed
Jun 4, 2024
| ctx.getState().setError(ErrorCode.ERR_UNKNOWN_ERROR, t.getMessage()); | ||
| // set insert result in connection context, | ||
| // so that user can use `show insert result` to get info of the last insert operation. | ||
| ctx.setOrUpdateInsertResult(txnId, labelName, database.getFullName(), table.getName(), |
There was a problem hiding this comment.
when execute error, should we set row num?
mymeiyi
reviewed
Jun 4, 2024
| // TODO: in legacy, there is a retry, we need to implement | ||
| if (code == TStatusCode.DATA_QUALITY_ERROR && !errorMsgsList.isEmpty() && errorMsgsList.get(0) | ||
| .contains("schema version not match")) { | ||
| LOG.info("group commit insert failed. query id: {}, backend id: {}, status: {}, " |
There was a problem hiding this comment.
log here and return success to client, will cause the data lost?
mymeiyi
reviewed
Jun 4, 2024
| @@ -80,7 +93,7 @@ suite("insert_group_commit_into_max_filter_ratio") { | |||
| logger.warn("insert result: " + result + ", expected_row_count: " + expected_row_count + ", sql: " + sql) | |||
| } | |||
| // assertEquals(result, expected_row_count) | |||
| assertTrue(serverInfo.contains("'status':'ABORTED'")) | |||
| assertTrue(serverInfo.contains("too many filtered rows")) | |||
There was a problem hiding this comment.
why change it, we can both check status and error msg
|
run buildall |
github-actions
bot
removed
the
approved
TPC-H: Total hot run time: 41131 ms |
TPC-DS: Total hot run time: 172654 ms |
ClickBench: Total hot run time: 30.68 s |
mymeiyi
approved these changes
Jun 6, 2024
github-actions
bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
dataroaring
pushed a commit
that referenced
this pull request
Jun 7, 2024
…atus (#35995) 1. Modify group commit p2 case to use nereids since #32523 support it. 2. We get many same logs when group commit is cancelled: ``` group_commit_mgr.cpp:182] cancel group commit, instance_id=xx, label=xx, status=[OK] ``` the problem is that `runtime_state->is_cancelled` is true, but `runtime_state->cancel_reason()` is `OK`
dataroaring
pushed a commit
that referenced
this pull request
Jun 7, 2024
) ## Proposed changes Nereids supports group_commit with insert Co-authored-by: abmdocrt <[email protected]> Co-authored-by: meiyi <[email protected]>
dataroaring
pushed a commit
that referenced
this pull request
Jun 7, 2024
…atus (#35995) 1. Modify group commit p2 case to use nereids since #32523 support it. 2. We get many same logs when group commit is cancelled: ``` group_commit_mgr.cpp:182] cancel group commit, instance_id=xx, label=xx, status=[OK] ``` the problem is that `runtime_state->is_cancelled` is true, but `runtime_state->cancel_reason()` is `OK`
mymeiyi
added a commit
to mymeiyi/doris
that referenced
this pull request
Jul 7, 2024
…atus (apache#35995) 1. Modify group commit p2 case to use nereids since apache#32523 support it. 2. We get many same logs when group commit is cancelled: ``` group_commit_mgr.cpp:182] cancel group commit, instance_id=xx, label=xx, status=[OK] ``` the problem is that `runtime_state->is_cancelled` is true, but `runtime_state->cancel_reason()` is `OK`
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
Nereids supports group_commit with insert
Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...