Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[shuffle](fix) Do not use copy assignment for TUniqueId #32969

Merged
merged 1 commit into from
Mar 29, 2024

Conversation

Gabriel39
Copy link
Contributor

@Gabriel39 Gabriel39 commented Mar 28, 2024

Proposed changes

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Gabriel39
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 38121 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1d7d88a11dfef50a44bb28d9f443ea9e94ca4295, data reload: false

------ Round 1 ----------------------------------
q1	17770	4844	4252	4252
q2	2652	163	156	156
q3	11205	1151	1222	1151
q4	10650	807	748	748
q5	7599	3029	2967	2967
q6	211	127	125	125
q7	1042	598	592	592
q8	9602	2051	2037	2037
q9	7133	6516	6396	6396
q10	8448	3531	3584	3531
q11	433	216	215	215
q12	374	199	191	191
q13	17812	2863	2850	2850
q14	240	202	203	202
q15	516	471	469	469
q16	493	366	363	363
q17	950	577	596	577
q18	7154	6652	6487	6487
q19	1574	1393	1479	1393
q20	550	279	252	252
q21	3598	2944	2871	2871
q22	339	296	303	296
Total cold run time: 110345 ms
Total hot run time: 38121 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4140	4084	4082	4082
q2	331	246	233	233
q3	3002	2853	2874	2853
q4	1852	1540	1533	1533
q5	5205	5212	5218	5212
q6	196	115	119	115
q7	2203	1841	1860	1841
q8	3189	3321	3291	3291
q9	8495	8441	8435	8435
q10	3853	3820	3757	3757
q11	563	449	443	443
q12	741	582	527	527
q13	16880	2821	2855	2821
q14	295	252	265	252
q15	497	456	455	455
q16	456	427	422	422
q17	1742	1497	1483	1483
q18	7308	7162	7198	7162
q19	1617	1562	1499	1499
q20	1901	1725	1699	1699
q21	4798	4624	4703	4624
q22	550	430	460	430
Total cold run time: 69814 ms
Total hot run time: 53169 ms

Status cancel_status = fragment_ctx->get_query_ctx()->exec_status();
_close_task(task, PipelineTaskState::CANCELED, cancel_status);
if (task->is_pending_finish()) {
// Only meet eos, should set task to PENDING_FINISH state
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should not wait finish.

@doris-robot
Copy link

TPC-DS: Total hot run time: 181207 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1d7d88a11dfef50a44bb28d9f443ea9e94ca4295, data reload: false

query1	933	356	343	343
query2	6525	2018	1897	1897
query3	6704	211	214	211
query4	31798	21573	21243	21243
query5	4347	399	393	393
query6	269	172	176	172
query7	4630	282	282	282
query8	225	173	169	169
query9	9083	2373	2332	2332
query10	559	242	257	242
query11	15424	14147	14177	14147
query12	131	99	92	92
query13	1630	424	417	417
query14	9843	7854	7846	7846
query15	246	189	182	182
query16	8234	259	262	259
query17	1976	551	553	551
query18	2080	276	267	267
query19	325	150	147	147
query20	90	91	84	84
query21	201	128	122	122
query22	5112	4940	4834	4834
query23	33555	32937	32621	32621
query24	11792	2883	2825	2825
query25	622	360	359	359
query26	1767	152	151	151
query27	3112	345	359	345
query28	7606	1898	1860	1860
query29	1000	595	595	595
query30	308	144	144	144
query31	969	730	755	730
query32	94	61	55	55
query33	776	254	251	251
query34	1120	485	486	485
query35	830	634	621	621
query36	1004	899	896	896
query37	267	67	65	65
query38	3567	3471	3456	3456
query39	1465	1447	1423	1423
query40	288	116	112	112
query41	50	49	49	49
query42	104	99	97	97
query43	496	468	453	453
query44	1193	737	741	737
query45	271	259	254	254
query46	1106	723	690	690
query47	1934	1852	1849	1849
query48	447	362	351	351
query49	1247	345	350	345
query50	763	368	365	365
query51	6930	6748	6787	6748
query52	107	92	87	87
query53	351	276	280	276
query54	316	253	246	246
query55	84	78	81	78
query56	243	232	230	230
query57	1220	1164	1160	1160
query58	234	211	206	206
query59	2801	2625	2549	2549
query60	277	242	264	242
query61	115	115	114	114
query62	649	409	403	403
query63	301	278	275	275
query64	6808	4097	4082	4082
query65	3106	3065	3040	3040
query66	1432	390	375	375
query67	15501	15042	14751	14751
query68	8809	546	526	526
query69	619	390	372	372
query70	1244	1161	1100	1100
query71	539	270	273	270
query72	6490	2516	2347	2347
query73	1258	313	320	313
query74	6991	6476	6375	6375
query75	3857	2215	2183	2183
query76	5389	868	887	868
query77	635	261	257	257
query78	11046	10090	10087	10087
query79	11734	529	523	523
query80	1722	370	373	370
query81	488	213	213	213
query82	239	80	87	80
query83	213	149	143	143
query84	289	80	76	76
query85	1110	314	307	307
query86	360	307	259	259
query87	3786	3514	3540	3514
query88	5200	2327	2291	2291
query89	499	369	368	368
query90	2073	170	171	170
query91	171	133	138	133
query92	69	45	47	45
query93	6314	497	477	477
query94	1326	174	176	174
query95	425	329	320	320
query96	613	263	274	263
query97	2655	2529	2481	2481
query98	226	205	205	205
query99	1064	819	839	819
Total cold run time: 316046 ms
Total hot run time: 181207 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 1d7d88a11dfef50a44bb28d9f443ea9e94ca4295 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.9 seconds inserted 10000000 Rows, about 719K ops/s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.62% (8834/24804)
Line Coverage: 27.32% (72396/264953)
Region Coverage: 26.56% (37554/141404)
Branch Coverage: 23.36% (19151/81980)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1d7d88a11dfef50a44bb28d9f443ea9e94ca4295_1d7d88a11dfef50a44bb28d9f443ea9e94ca4295/report/index.html

@Gabriel39 Gabriel39 changed the title [shuffle](fix) Do not release resources if rpc has not done [shuffle](fix) Do not use copy assignment for TUniqueId Mar 28, 2024
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@Gabriel39
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.62% (8836/24808)
Line Coverage: 27.32% (72394/264951)
Region Coverage: 26.55% (37549/141408)
Branch Coverage: 23.35% (19145/81980)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bae426d4657215a95446b10ed94a216b4d06d55d_bae426d4657215a95446b10ed94a216b4d06d55d/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 37598 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bae426d4657215a95446b10ed94a216b4d06d55d, data reload: false

------ Round 1 ----------------------------------
q1	17612	4172	4081	4081
q2	2108	156	152	152
q3	10593	1195	1211	1195
q4	10233	774	741	741
q5	7469	2936	2909	2909
q6	205	125	128	125
q7	1027	574	579	574
q8	9325	2016	2017	2016
q9	7312	6340	6347	6340
q10	8430	3447	3609	3447
q11	429	220	216	216
q12	464	203	193	193
q13	17795	2841	2867	2841
q14	256	201	204	201
q15	506	473	461	461
q16	494	375	371	371
q17	954	493	548	493
q18	7029	6383	6452	6383
q19	1799	1501	1404	1404
q20	557	253	242	242
q21	3506	2978	2916	2916
q22	345	306	297	297
Total cold run time: 108448 ms
Total hot run time: 37598 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4127	4031	4070	4031
q2	330	237	228	228
q3	2961	2794	2784	2784
q4	1851	1568	1571	1568
q5	5194	5204	5253	5204
q6	193	117	120	117
q7	2197	1853	1814	1814
q8	3178	3319	3309	3309
q9	8430	8451	8491	8451
q10	3767	3765	4650	3765
q11	574	462	447	447
q12	779	621	549	549
q13	17030	3096	3078	3078
q14	295	288	271	271
q15	532	508	486	486
q16	506	471	466	466
q17	1783	1535	1557	1535
q18	8136	7611	7566	7566
q19	5153	1535	1532	1532
q20	2009	1859	1733	1733
q21	4945	4820	4914	4820
q22	537	469	457	457
Total cold run time: 74507 ms
Total hot run time: 54211 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 182250 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bae426d4657215a95446b10ed94a216b4d06d55d, data reload: false

query1	915	351	349	349
query2	6252	2349	1898	1898
query3	6704	219	203	203
query4	31736	21355	21356	21355
query5	4293	406	398	398
query6	280	180	166	166
query7	4620	292	292	292
query8	233	167	175	167
query9	9054	2294	2284	2284
query10	426	262	267	262
query11	15521	14201	14222	14201
query12	131	83	88	83
query13	1643	432	426	426
query14	11349	7989	8039	7989
query15	248	178	185	178
query16	8264	265	265	265
query17	1989	581	558	558
query18	2104	304	285	285
query19	352	161	154	154
query20	97	86	84	84
query21	208	133	129	129
query22	4949	4829	4741	4741
query23	33786	32688	33068	32688
query24	10971	2995	2970	2970
query25	633	421	401	401
query26	1070	183	169	169
query27	3118	368	376	368
query28	7771	1970	1939	1939
query29	883	682	639	639
query30	311	163	159	159
query31	1014	792	780	780
query32	96	59	55	55
query33	767	257	251	251
query34	1308	509	527	509
query35	946	708	733	708
query36	1062	934	920	920
query37	127	66	63	63
query38	3694	3720	3735	3720
query39	1105	1050	1065	1050
query40	173	112	111	111
query41	51	51	48	48
query42	106	96	100	96
query43	498	464	459	459
query44	1151	757	762	757
query45	275	260	280	260
query46	1133	718	742	718
query47	2051	1942	1940	1940
query48	470	391	370	370
query49	934	341	374	341
query50	768	378	389	378
query51	7024	6821	6823	6821
query52	116	90	96	90
query53	352	282	278	278
query54	293	232	245	232
query55	86	78	83	78
query56	252	233	232	232
query57	1256	1200	1179	1179
query58	235	217	212	212
query59	2935	2672	2513	2513
query60	253	232	228	228
query61	92	88	89	88
query62	631	436	457	436
query63	301	276	272	272
query64	5112	3973	4043	3973
query65	3069	3048	3046	3046
query66	778	343	350	343
query67	15655	14774	15193	14774
query68	9184	530	537	530
query69	675	375	381	375
query70	1393	1154	1107	1107
query71	514	258	256	256
query72	6976	2502	2333	2333
query73	1607	322	323	322
query74	8263	6498	6455	6455
query75	3728	2250	2301	2250
query76	5829	926	878	878
query77	614	253	263	253
query78	11083	10237	10047	10047
query79	10631	545	525	525
query80	2062	386	388	386
query81	523	212	223	212
query82	476	89	83	83
query83	194	148	146	146
query84	287	80	80	80
query85	1027	307	303	303
query86	360	293	277	277
query87	3759	3498	3516	3498
query88	4600	2315	2292	2292
query89	488	363	375	363
query90	1956	173	178	173
query91	168	135	136	135
query92	57	47	47	47
query93	6924	496	486	486
query94	1154	179	175	175
query95	426	320	312	312
query96	613	272	276	272
query97	2678	2478	2476	2476
query98	243	226	215	215
query99	1096	859	870	859
Total cold run time: 315668 ms
Total hot run time: 182250 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit bae426d4657215a95446b10ed94a216b4d06d55d with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      33 seconds loaded 861443392 Bytes, about 24 MB/s
Insert into select:       14.0 seconds inserted 10000000 Rows, about 714K ops/s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 29, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@Gabriel39 Gabriel39 merged commit 0358501 into apache:master Mar 29, 2024
30 of 34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.1-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants