Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[branch-2.0](routine-load) self-adaption backoff timeout #32591

Merged
merged 4 commits into from
Mar 21, 2024
Merged

[branch-2.0](routine-load) self-adaption backoff timeout #32591

merged 4 commits into from
Mar 21, 2024

Conversation

sollhui
Copy link
Contributor

@sollhui sollhui commented Mar 21, 2024

Proposed changes

#32227

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@sollhui
Copy link
Contributor Author

sollhui commented Mar 21, 2024

run buildall

@github-actions github-actions bot added the area/load Issues or PRs related to all kinds of load label Mar 21, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Mar 21, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@sollhui sollhui force-pushed the backoff_timeout_2.0 branch 2 times, most recently from c24786c to 9b07bce Compare March 21, 2024 11:48
@sollhui
Copy link
Contributor Author

sollhui commented Mar 21, 2024

run buildall

@sollhui
Copy link
Contributor Author

sollhui commented Mar 21, 2024

run buildall

@sollhui
Copy link
Contributor Author

sollhui commented Mar 21, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 50278 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 75b189d68ff345643b03f86eae9d6fb74f3f6906, data reload: false

------ Round 1 ----------------------------------
q1	17535	4422	4372	4372
q2	2033	154	148	148
q3	10279	1924	1891	1891
q4	10110	1261	1337	1261
q5	8708	4029	3990	3990
q6	235	122	123	122
q7	2027	1604	1628	1604
q8	9256	2729	2721	2721
q9	11000	11265	10469	10469
q10	8694	3561	3555	3555
q11	416	240	236	236
q12	473	295	299	295
q13	18334	4003	4074	4003
q14	355	326	333	326
q15	513	458	451	451
q16	686	595	592	592
q17	1143	1011	984	984
q18	7226	6891	7040	6891
q19	1703	1576	1529	1529
q20	521	292	300	292
q21	4474	4146	4154	4146
q22	505	402	400	400
Total cold run time: 116226 ms
Total hot run time: 50278 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4320	4289	4287	4287
q2	322	219	225	219
q3	4174	4138	4138	4138
q4	2753	2769	2757	2757
q5	7304	7284	7259	7259
q6	234	118	117	117
q7	3296	2820	2874	2820
q8	4371	4513	4464	4464
q9	17030	17151	17003	17003
q10	4283	4304	4244	4244
q11	738	662	688	662
q12	1034	849	862	849
q13	6132	3747	3754	3747
q14	456	423	411	411
q15	494	472	457	457
q16	749	715	704	704
q17	3883	3865	3846	3846
q18	8800	8729	8821	8729
q19	1706	1707	1673	1673
q20	2436	2125	2110	2110
q21	8564	8532	8639	8532
q22	1061	949	972	949
Total cold run time: 84140 ms
Total hot run time: 79977 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 202490 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 75b189d68ff345643b03f86eae9d6fb74f3f6906, data reload: false

query1	927	389	384	384
query2	6523	2211	2121	2121
query3	6914	204	201	201
query4	20166	17892	17955	17892
query5	19717	6459	6472	6459
query6	298	238	216	216
query7	4158	298	300	298
query8	266	251	226	226
query9	3101	2674	2582	2582
query10	428	304	302	302
query11	11493	10789	10756	10756
query12	123	73	72	72
query13	5566	641	643	641
query14	17964	13382	13347	13347
query15	359	226	239	226
query16	6460	274	265	265
query17	1711	1465	884	884
query18	2284	402	409	402
query19	201	152	152	152
query20	77	76	76	76
query21	189	93	97	93
query22	5398	5057	5054	5054
query23	32707	32029	32017	32017
query24	6987	6450	6450	6450
query25	517	422	409	409
query26	524	156	153	153
query27	1904	287	292	287
query28	6168	2303	2255	2255
query29	2869	2765	2630	2630
query30	235	160	162	160
query31	893	727	735	727
query32	66	56	57	56
query33	386	250	255	250
query34	848	475	475	475
query35	1122	888	916	888
query36	1349	1219	1195	1195
query37	99	62	64	62
query38	3108	2912	2918	2912
query39	1380	1305	1314	1305
query40	199	94	95	94
query41	36	33	32	32
query42	80	89	87	87
query43	663	538	559	538
query44	1105	726	723	723
query45	240	228	232	228
query46	1227	970	986	970
query47	1708	1690	1649	1649
query48	991	684	657	657
query49	629	367	365	365
query50	883	599	598	598
query51	4781	4667	4731	4667
query52	86	85	76	76
query53	435	308	303	303
query54	2639	2469	2467	2467
query55	86	73	76	73
query56	224	208	204	204
query57	1132	1108	1112	1108
query58	203	200	212	200
query59	3552	3425	3258	3258
query60	213	197	195	195
query61	88	85	86	85
query62	785	448	468	448
query63	482	319	332	319
query64	2528	1498	1448	1448
query65	3611	3541	3548	3541
query66	791	376	358	358
query67	17476	16077	18502	16077
query68	6860	661	660	660
query69	567	354	342	342
query70	1720	1401	1392	1392
query71	401	312	313	312
query72	6514	3477	3448	3448
query73	726	326	326	326
query74	6292	5783	5740	5740
query75	4513	3712	3642	3642
query76	4412	1138	1218	1138
query77	549	249	262	249
query78	12493	11890	11897	11890
query79	8920	623	643	623
query80	1117	389	388	388
query81	497	226	232	226
query82	1733	97	93	93
query83	174	141	128	128
query84	250	71	67	67
query85	853	289	288	288
query86	337	286	286	286
query87	3244	3009	3037	3009
query88	5020	2333	2328	2328
query89	382	338	299	299
query90	1903	199	206	199
query91	159	127	127	127
query92	63	53	49	49
query93	5130	551	545	545
query94	736	208	212	208
query95	1110	1060	1046	1046
query96	650	329	330	329
query97	6491	6320	6422	6320
query98	193	173	167	167
query99	2805	1019	978	978
Total cold run time: 309172 ms
Total hot run time: 202490 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 75b189d68ff345643b03f86eae9d6fb74f3f6906 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       22.3 seconds inserted 10000000 Rows, about 448K ops/s

@liaoxin01 liaoxin01 merged commit b2239c6 into apache:branch-2.0 Mar 21, 2024
26 of 28 checks passed
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@sollhui
[branch-2.0](routine-load) self-adaption backoff timeout (apache#32591)
01ff835
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CalvinKirs CalvinKirs CalvinKirs approved these changes

@liaoxin01 liaoxin01 liaoxin01 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. area/load Issues or PRs related to all kinds of load reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@sollhui @doris-robot @CalvinKirs @liaoxin01