Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](merge-cloud) fix inconsistency between cloud mode and local mode for 2PC #33917

Merged
merged 1 commit into from
Apr 21, 2024
Merged

[fix](merge-cloud) fix inconsistency between cloud mode and local mode for 2PC #33917

merged 1 commit into from
Apr 21, 2024

Conversation

liaoxin01
Copy link
Contributor

@liaoxin01 liaoxin01 commented Apr 19, 2024 •
edited
Loading

Proposed changes

Issue Number: close #xxx

  1. fix test_stream_load case
  2. When the labels are the same, return the associated txn id
  3. Repeat commit and return error message for 2pc

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@liaoxin01
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

gavinchou
gavinchou requested changes Apr 19, 2024
View reviewed changes
gensrc/proto/cloud.proto Show resolved Hide resolved
cloud/src/meta-service/meta_service_txn.cpp Show resolved Hide resolved
cloud/src/meta-service/meta_service_txn.cpp
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pls add UT to cover all the newly added code: line/branch coverage 90+%

@doris-robot
Copy link

TPC-H: Total hot run time: 38684 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1c2d18cedae4e470cd6906efa7731c85bf7912fe, data reload: false

------ Round 1 ----------------------------------
q1	17817	4640	4277	4277
q2	2001	192	182	182
q3	10567	1136	1147	1136
q4	10235	821	697	697
q5	7495	2727	2677	2677
q6	221	134	138	134
q7	1007	619	581	581
q8	9229	2057	2043	2043
q9	7498	6603	6530	6530
q10	8496	3530	3527	3527
q11	462	241	237	237
q12	516	228	215	215
q13	17795	2983	2967	2967
q14	258	229	222	222
q15	516	485	486	485
q16	512	384	384	384
q17	974	671	685	671
q18	7343	6913	6877	6877
q19	5772	1538	1494	1494
q20	662	313	300	300
q21	3628	2738	2857	2738
q22	369	330	310	310
Total cold run time: 113373 ms
Total hot run time: 38684 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4313	4222	4273	4222
q2	372	260	268	260
q3	2970	2761	2743	2743
q4	1878	1604	1550	1550
q5	5283	5305	5296	5296
q6	212	124	126	124
q7	2227	1835	1923	1835
q8	3200	3340	3348	3340
q9	8568	8552	8661	8552
q10	4097	3957	4001	3957
q11	610	492	522	492
q12	783	635	638	635
q13	16355	3242	3157	3157
q14	323	289	292	289
q15	527	508	482	482
q16	511	465	448	448
q17	1841	1521	1502	1502
q18	8105	7830	7736	7736
q19	1700	1540	1490	1490
q20	1976	1897	1839	1839
q21	5172	4972	5033	4972
q22	573	479	499	479
Total cold run time: 71596 ms
Total hot run time: 55400 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186494 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1c2d18cedae4e470cd6906efa7731c85bf7912fe, data reload: false

query1	904	373	364	364
query2	6218	2565	2452	2452
query3	6675	203	195	195
query4	22781	21366	21409	21366
query5	4130	430	416	416
query6	266	189	179	179
query7	4589	294	298	294
query8	242	195	185	185
query9	8432	2358	2342	2342
query10	432	260	261	260
query11	14810	14348	14346	14346
query12	142	91	87	87
query13	1653	380	375	375
query14	9458	7697	8102	7697
query15	270	185	190	185
query16	8246	256	258	256
query17	1989	567	556	556
query18	2101	285	257	257
query19	326	150	150	150
query20	95	83	88	83
query21	196	128	128	128
query22	5041	4812	4762	4762
query23	33820	33406	33400	33400
query24	10795	3003	3037	3003
query25	627	391	383	383
query26	723	161	157	157
query27	2223	369	379	369
query28	6112	2071	2090	2071
query29	881	620	621	620
query30	308	183	188	183
query31	972	752	759	752
query32	96	54	56	54
query33	658	273	281	273
query34	911	500	514	500
query35	889	702	716	702
query36	1084	915	956	915
query37	118	70	75	70
query38	3490	3375	3308	3308
query39	1652	1604	1583	1583
query40	175	133	135	133
query41	49	44	42	42
query42	109	105	101	101
query43	615	576	550	550
query44	1126	754	780	754
query45	305	281	252	252
query46	1131	744	751	744
query47	2025	1920	1953	1920
query48	382	307	315	307
query49	823	411	435	411
query50	792	414	403	403
query51	6919	6903	6702	6702
query52	106	86	89	86
query53	341	282	280	280
query54	309	236	229	229
query55	80	74	77	74
query56	250	234	228	228
query57	1228	1115	1156	1115
query58	219	199	201	199
query59	3514	3286	3069	3069
query60	249	235	245	235
query61	109	104	104	104
query62	580	448	439	439
query63	312	282	281	281
query64	5035	4027	4037	4027
query65	3077	3031	3043	3031
query66	758	336	337	336
query67	15613	15164	14902	14902
query68	6649	535	564	535
query69	540	313	313	313
query70	1239	1218	1179	1179
query71	1452	1277	1280	1277
query72	6858	2739	2588	2588
query73	731	323	323	323
query74	6826	6345	6366	6345
query75	3683	2680	2618	2618
query76	4381	982	1035	982
query77	633	272	273	272
query78	10857	10279	10422	10279
query79	9904	515	524	515
query80	2055	465	453	453
query81	532	246	247	246
query82	350	96	101	96
query83	216	175	168	168
query84	269	90	85	85
query85	949	414	251	251
query86	352	302	292	292
query87	3565	3317	3254	3254
query88	4855	2330	2343	2330
query89	514	372	367	367
query90	2050	189	184	184
query91	126	97	95	95
query92	56	49	48	48
query93	6712	502	492	492
query94	1345	176	175	175
query95	402	291	294	291
query96	616	264	262	262
query97	3129	2889	2946	2889
query98	237	211	224	211
query99	1167	856	847	847
Total cold run time: 293836 ms
Total hot run time: 186494 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.17 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1c2d18cedae4e470cd6906efa7731c85bf7912fe, data reload: false

query1	0.04	0.04	0.03
query2	0.09	0.04	0.04
query3	0.23	0.05	0.05
query4	1.69	0.07	0.07
query5	0.49	0.49	0.50
query6	1.41	0.73	0.72
query7	0.02	0.01	0.02
query8	0.06	0.04	0.04
query9	0.56	0.48	0.49
query10	0.56	0.56	0.54
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.63	0.59	0.58
query14	0.75	0.78	0.77
query15	0.82	0.81	0.80
query16	0.35	0.35	0.36
query17	0.97	0.94	0.97
query18	0.21	0.25	0.23
query19	1.77	1.75	1.75
query20	0.01	0.01	0.02
query21	15.39	0.65	0.65
query22	4.14	6.94	1.80
query23	18.27	1.40	1.18
query24	1.93	0.25	0.21
query25	0.14	0.08	0.08
query26	0.26	0.15	0.15
query27	0.07	0.07	0.08
query28	13.25	1.02	0.98
query29	12.60	3.31	3.29
query30	0.26	0.06	0.05
query31	2.86	0.38	0.37
query32	3.28	0.48	0.46
query33	2.87	2.83	2.81
query34	17.17	4.37	4.47
query35	4.48	4.49	4.49
query36	0.65	0.50	0.49
query37	0.19	0.16	0.15
query38	0.15	0.15	0.15
query39	0.05	0.03	0.03
query40	0.18	0.14	0.14
query41	0.09	0.05	0.04
query42	0.05	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.33 s
Total hot run time: 30.17 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 1c2d18cedae4e470cd6906efa7731c85bf7912fe with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      33 seconds loaded 861443392 Bytes, about 24 MB/s
Insert into select:       13.3 seconds inserted 10000000 Rows, about 751K ops/s

@liaoxin01
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 38516 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d4506766bb18c8819712f3739441028b21cc7e0d, data reload: false

------ Round 1 ----------------------------------
q1	17599	4347	4330	4330
q2	2016	195	184	184
q3	10453	1208	1179	1179
q4	10184	766	754	754
q5	7482	2650	2652	2650
q6	226	132	138	132
q7	1001	600	579	579
q8	9238	2049	2032	2032
q9	7201	6547	6542	6542
q10	8611	3535	3520	3520
q11	459	232	229	229
q12	486	220	214	214
q13	17769	2938	2939	2938
q14	257	224	220	220
q15	521	474	484	474
q16	515	402	373	373
q17	968	646	686	646
q18	7460	6707	6767	6707
q19	8639	1477	1503	1477
q20	664	318	296	296
q21	3516	2730	2742	2730
q22	371	310	315	310
Total cold run time: 115636 ms
Total hot run time: 38516 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4363	4261	4303	4261
q2	375	270	276	270
q3	2957	2778	2784	2778
q4	1881	1536	1579	1536
q5	5290	5300	5275	5275
q6	215	126	128	126
q7	2226	1892	1915	1892
q8	3228	3359	3313	3313
q9	8558	8538	8683	8538
q10	4079	3884	3950	3884
q11	610	515	486	486
q12	817	631	637	631
q13	15963	3257	3187	3187
q14	329	299	278	278
q15	529	510	485	485
q16	470	436	445	436
q17	1851	1549	1525	1525
q18	8182	7992	7920	7920
q19	1673	1611	1575	1575
q20	2074	1856	1878	1856
q21	9814	4902	4911	4902
q22	568	492	472	472
Total cold run time: 76052 ms
Total hot run time: 55626 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186306 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d4506766bb18c8819712f3739441028b21cc7e0d, data reload: false

query1	887	375	361	361
query2	6310	2643	2344	2344
query3	6666	200	194	194
query4	23070	21236	21479	21236
query5	4100	396	401	396
query6	273	183	183	183
query7	4584	302	287	287
query8	244	185	185	185
query9	8638	2345	2301	2301
query10	417	266	250	250
query11	14786	14211	14413	14211
query12	129	87	83	83
query13	1635	354	356	354
query14	9286	8091	7876	7876
query15	252	180	180	180
query16	8104	252	251	251
query17	2025	559	533	533
query18	1915	274	258	258
query19	340	141	149	141
query20	88	84	82	82
query21	196	125	124	124
query22	4994	4802	4800	4800
query23	33848	33131	33144	33131
query24	11037	3100	2986	2986
query25	602	381	380	380
query26	707	159	154	154
query27	2460	362	373	362
query28	5999	2107	2089	2089
query29	877	644	618	618
query30	310	177	178	177
query31	1040	801	789	789
query32	91	56	52	52
query33	641	250	244	244
query34	915	492	510	492
query35	843	724	713	713
query36	1071	940	933	933
query37	109	72	75	72
query38	3486	3327	3303	3303
query39	1620	1610	1565	1565
query40	178	126	121	121
query41	53	43	47	43
query42	104	94	92	92
query43	591	544	551	544
query44	1099	740	738	738
query45	290	278	263	263
query46	1097	727	756	727
query47	2005	1956	1961	1956
query48	379	308	319	308
query49	848	426	432	426
query50	799	419	392	392
query51	6975	6966	6939	6939
query52	102	87	86	86
query53	351	275	269	269
query54	308	225	221	221
query55	75	67	69	67
query56	245	218	214	214
query57	1218	1129	1141	1129
query58	218	204	197	197
query59	3306	3148	3072	3072
query60	246	228	219	219
query61	87	85	88	85
query62	604	423	426	423
query63	300	279	273	273
query64	5012	3744	3934	3744
query65	3074	3036	3035	3035
query66	741	322	339	322
query67	15512	15497	14684	14684
query68	5388	537	528	528
query69	515	294	303	294
query70	1238	1172	1176	1172
query71	1427	1260	1263	1260
query72	6633	2616	2460	2460
query73	737	319	316	316
query74	6887	6389	6472	6389
query75	3590	2633	2598	2598
query76	4115	1065	1031	1031
query77	615	258	255	255
query78	10942	10473	10263	10263
query79	6823	524	518	518
query80	1347	444	435	435
query81	510	244	237	237
query82	246	97	96	96
query83	194	165	167	165
query84	260	85	88	85
query85	831	264	256	256
query86	345	318	321	318
query87	3538	3305	3282	3282
query88	4825	2322	2327	2322
query89	491	380	382	380
query90	2032	179	177	177
query91	118	94	96	94
query92	54	44	45	44
query93	5600	509	501	501
query94	1113	177	169	169
query95	1087	1086	1085	1085
query96	609	263	255	255
query97	3084	2974	2981	2974
query98	231	222	219	219
query99	1205	857	849	849
Total cold run time: 287042 ms
Total hot run time: 186306 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.24 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d4506766bb18c8819712f3739441028b21cc7e0d, data reload: false

query1	0.04	0.03	0.04
query2	0.07	0.03	0.04
query3	0.23	0.05	0.06
query4	1.68	0.08	0.06
query5	0.48	0.49	0.52
query6	1.46	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.56	0.50	0.51
query10	0.54	0.56	0.53
query11	0.16	0.12	0.11
query12	0.14	0.11	0.12
query13	0.61	0.59	0.58
query14	0.76	0.78	0.78
query15	0.84	0.81	0.80
query16	0.37	0.35	0.35
query17	0.96	0.95	0.95
query18	0.22	0.21	0.28
query19	1.82	1.68	1.67
query20	0.01	0.01	0.00
query21	15.41	0.64	0.63
query22	4.55	7.02	2.03
query23	18.29	1.42	1.25
query24	1.81	0.24	0.20
query25	0.14	0.08	0.08
query26	0.25	0.16	0.16
query27	0.08	0.08	0.08
query28	13.38	1.00	0.97
query29	12.96	3.24	3.22
query30	0.25	0.07	0.06
query31	2.87	0.38	0.37
query32	3.28	0.46	0.47
query33	2.82	2.78	2.83
query34	17.20	4.44	4.40
query35	4.53	4.47	4.51
query36	0.66	0.46	0.48
query37	0.18	0.16	0.15
query38	0.15	0.14	0.16
query39	0.04	0.04	0.04
query40	0.18	0.14	0.15
query41	0.09	0.05	0.05
query42	0.05	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.23 s
Total hot run time: 30.24 s

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Apr 20, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 20, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@dataroaring dataroaring merged commit 4d91640 into apache:master Apr 21, 2024
26 of 30 checks passed
dataroaring pushed a commit that referenced this pull request Apr 21, 2024
@liaoxin01 @dataroaring
[fix](merge-cloud) fix inconsistency between cloud mode and local mod…
127b877 
…e for 2PC (#33917)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.0-merged meta-change reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@liaoxin01 @doris-robot @gavinchou @dataroaring