Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](cloud) mysqlload works with cluster and refactor code #33027

Closed
wants to merge 1 commit into from

Conversation

dataroaring
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@dataroaring
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.58% (8839/24842)
Line Coverage: 27.31% (72450/265278)
Region Coverage: 26.51% (37499/141441)
Branch Coverage: 23.33% (19120/81972)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a0b4e8133862db11d9dac188757ab8e8b1e2d80b_a0b4e8133862db11d9dac188757ab8e8b1e2d80b/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 38284 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a0b4e8133862db11d9dac188757ab8e8b1e2d80b, data reload: false

------ Round 1 ----------------------------------
q1	17613	4300	4227	4227
q2	2078	195	192	192
q3	10425	1164	1176	1164
q4	10197	750	751	750
q5	7523	2728	2766	2728
q6	214	134	140	134
q7	1055	629	611	611
q8	9216	1992	1977	1977
q9	7352	6511	6464	6464
q10	8473	3430	3516	3430
q11	440	249	257	249
q12	501	217	215	215
q13	17770	3053	2995	2995
q14	270	232	227	227
q15	513	481	468	468
q16	507	396	387	387
q17	949	613	559	559
q18	7437	6918	6766	6766
q19	7205	1439	1382	1382
q20	669	309	318	309
q21	3604	2740	2829	2740
q22	370	310	323	310
Total cold run time: 114381 ms
Total hot run time: 38284 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4349	4234	4233	4233
q2	372	261	290	261
q3	3034	2767	2769	2767
q4	1910	1574	1536	1536
q5	5355	5268	5342	5268
q6	212	126	129	126
q7	2354	1970	1975	1970
q8	3279	3372	3336	3336
q9	8752	8860	8906	8860
q10	4182	3906	3990	3906
q11	627	514	481	481
q12	831	616	648	616
q13	17898	3261	3096	3096
q14	323	275	302	275
q15	539	502	510	502
q16	505	451	454	451
q17	1822	1485	1500	1485
q18	8226	7960	7819	7819
q19	1661	1549	1598	1549
q20	2163	1770	1775	1770
q21	10855	4950	4950	4950
q22	575	484	462	462
Total cold run time: 79824 ms
Total hot run time: 55719 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 182124 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a0b4e8133862db11d9dac188757ab8e8b1e2d80b, data reload: false

query1	874	1137	352	352
query2	7163	2068	1913	1913
query3	6667	219	210	210
query4	37125	21463	21352	21352
query5	4266	397	397	397
query6	245	178	172	172
query7	3992	300	295	295
query8	225	175	171	171
query9	5589	2290	2278	2278
query10	379	234	242	234
query11	14764	14266	14366	14266
query12	138	100	84	84
query13	967	366	356	356
query14	9364	6953	6952	6952
query15	199	181	173	173
query16	7241	271	275	271
query17	1426	590	569	569
query18	1500	284	281	281
query19	205	161	157	157
query20	94	90	89	89
query21	205	133	137	133
query22	4972	4882	4814	4814
query23	33309	32711	33130	32711
query24	10999	2983	2875	2875
query25	568	413	403	403
query26	737	166	156	156
query27	3033	367	374	367
query28	5992	1900	1890	1890
query29	912	659	649	649
query30	1099	169	1079	169
query31	942	764	758	758
query32	66	68	56	56
query33	443	258	283	258
query34	919	530	509	509
query35	854	720	745	720
query36	1037	935	939	935
query37	102	77	70	70
query38	3764	3651	3691	3651
query39	1703	1642	1603	1603
query40	186	137	130	130
query41	50	52	52	52
query42	107	104	96	96
query43	497	465	468	465
query44	1144	764	729	729
query45	301	270	285	270
query46	1125	732	697	697
query47	2052	1956	1941	1941
query48	407	319	311	311
query49	845	387	366	366
query50	798	425	418	418
query51	6790	6825	6775	6775
query52	104	93	94	93
query53	357	287	288	287
query54	253	236	228	228
query55	83	83	79	79
query56	252	225	236	225
query57	1260	1195	1167	1167
query58	226	204	207	204
query59	2941	2794	2547	2547
query60	246	243	227	227
query61	91	88	90	88
query62	590	443	435	435
query63	301	271	270	270
query64	4546	4124	4070	4070
query65	3046	3032	3033	3032
query66	737	339	335	335
query67	15479	14837	14814	14814
query68	9252	524	541	524
query69	615	312	302	302
query70	1285	1154	1106	1106
query71	522	264	275	264
query72	6882	2637	2485	2485
query73	880	324	324	324
query74	7142	6412	6372	6372
query75	3653	2388	2346	2346
query76	5892	1023	982	982
query77	578	268	245	245
query78	11021	10232	10126	10126
query79	9729	536	532	532
query80	1893	430	416	416
query81	535	217	215	215
query82	797	89	89	89
query83	214	173	169	169
query84	264	83	79	79
query85	910	279	274	274
query86	597	306	280	280
query87	3735	3503	3543	3503
query88	5316	2400	2385	2385
query89	508	381	380	380
query90	1847	176	171	171
query91	128	98	97	97
query92	62	48	45	45
query93	6650	504	505	504
query94	1127	184	182	182
query95	398	311	314	311
query96	611	264	265	264
query97	2713	2453	2447	2447
query98	235	216	220	216
query99	1102	846	839	839
Total cold run time: 305986 ms
Total hot run time: 182124 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.93 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a0b4e8133862db11d9dac188757ab8e8b1e2d80b, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.66	0.08	0.09
query5	0.50	0.48	0.50
query6	1.13	0.66	0.65
query7	0.01	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.50	0.49
query10	0.53	0.55	0.53
query11	0.15	0.10	0.11
query12	0.15	0.11	0.10
query13	0.58	0.58	0.59
query14	0.76	0.77	0.77
query15	0.84	0.81	0.81
query16	0.37	0.36	0.42
query17	0.97	0.95	1.02
query18	0.23	0.22	0.26
query19	1.80	1.66	1.69
query20	0.02	0.01	0.01
query21	15.53	0.64	0.63
query22	2.54	2.60	2.26
query23	17.06	1.02	0.87
query24	1.41	0.24	0.20
query25	0.10	0.08	0.07
query26	0.23	0.17	0.16
query27	0.09	0.08	0.08
query28	13.93	0.93	0.93
query29	12.50	3.16	3.16
query30	0.29	0.10	0.10
query31	2.79	0.37	0.38
query32	3.29	0.46	0.45
query33	2.79	2.82	2.85
query34	16.75	4.38	4.41
query35	4.50	4.50	4.44
query36	0.60	0.47	0.46
query37	0.19	0.15	0.15
query38	0.15	0.14	0.13
query39	0.05	0.04	0.04
query40	0.19	0.14	0.16
query41	0.09	0.05	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 105.82 s
Total hot run time: 29.93 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit a0b4e8133862db11d9dac188757ab8e8b1e2d80b with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.8 seconds inserted 10000000 Rows, about 724K ops/s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants