Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improve](routine load) delay schedule EOF tasks to avoid too many small transactions (#39975) #40498

Merged
merged 1 commit into from
Sep 8, 2024
Merged

[improve](routine load) delay schedule EOF tasks to avoid too many small transactions (#39975) #40498

merged 1 commit into from
Sep 8, 2024

Conversation

sollhui
Copy link
Contributor

@sollhui sollhui commented Sep 7, 2024

pick (#39975)

We encountered a scenario where a large number of small transactions were generated, resulting in an impact on query performance: Kafka's data comes in batches of very small data every very short time, which leads to tasks being frequently scheduled and ending very quickly, resulting in a large number of small transactions.

To solve this problem, we delay the scheduling of tasks that perceive EOF, which would not delay data consumption, for perceiving EOF indicates that the consumption speed is greater than the production speed.

@sollhui
[improve](routine load) delay schedule EOF tasks to avoid too many sm…
b665ddf 
…all transactions (apache#39975)

We encountered a scenario where a large number of small transactions
were generated, resulting in an impact on query performance:
Kafka's data comes in batches of very small data every very short time,
which leads to tasks being frequently scheduled and ending very quickly,
resulting in a large number of small transactions.

To solve this problem, we delay the scheduling of tasks that perceive
EOF, which would not delay data consumption, for perceiving EOF
indicates that the consumption speed is greater than the production
speed.
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@sollhui
Copy link
Contributor Author

sollhui commented Sep 7, 2024

run buildall

@github-actions github-actions bot added the area/load Issues or PRs related to all kinds of load label Sep 7, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 49085 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b665ddf2d219a8a30e1bef3ca1131f9f6104b934, data reload: false

------ Round 1 ----------------------------------
q1	17637	4322	4376	4322
q2	2107	154	140	140
q3	10422	1867	1926	1867
q4	10368	1261	1322	1261
q5	8510	3915	3925	3915
q6	235	123	123	123
q7	2014	1611	1598	1598
q8	9530	2721	2698	2698
q9	11576	10073	9924	9924
q10	8650	3518	3531	3518
q11	419	242	249	242
q12	470	302	292	292
q13	18341	3991	4001	3991
q14	354	327	320	320
q15	507	453	461	453
q16	535	469	455	455
q17	1132	946	901	901
q18	7301	6839	6769	6769
q19	1703	1559	1518	1518
q20	509	320	329	320
q21	4429	4199	4065	4065
q22	497	396	393	393
Total cold run time: 117246 ms
Total hot run time: 49085 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4343	4295	4283	4283
q2	320	228	223	223
q3	4193	4123	4163	4123
q4	2751	2729	2729	2729
q5	7163	7171	7158	7158
q6	232	120	118	118
q7	3267	2857	2859	2857
q8	4320	4438	4446	4438
q9	14341	13908	13897	13897
q10	4221	4229	4237	4229
q11	744	703	688	688
q12	1043	840	854	840
q13	6664	3740	3703	3703
q14	452	426	432	426
q15	510	471	452	452
q16	636	587	579	579
q17	3946	3803	3846	3803
q18	8848	8736	8749	8736
q19	1721	1665	1627	1627
q20	2407	2146	2087	2087
q21	8461	8452	8457	8452
q22	1058	967	974	967
Total cold run time: 81641 ms
Total hot run time: 76415 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 213186 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b665ddf2d219a8a30e1bef3ca1131f9f6104b934, data reload: false

query1	920	389	426	389
query2	6529	2344	2335	2335
query3	6920	203	206	203
query4	23414	21701	21939	21701
query5	19733	6555	6653	6555
query6	307	222	231	222
query7	4334	294	321	294
query8	261	273	247	247
query9	3067	2639	2581	2581
query10	416	324	300	300
query11	15818	15484	15507	15484
query12	133	78	76	76
query13	1038	477	448	448
query14	19717	13712	13372	13372
query15	372	218	227	218
query16	6870	282	256	256
query17	1959	937	928	928
query18	910	320	320	320
query19	216	158	151	151
query20	80	79	79	79
query21	193	102	103	102
query22	5223	5098	4971	4971
query23	34044	33624	33451	33451
query24	6688	6278	6294	6278
query25	540	443	433	433
query26	783	167	162	162
query27	2235	298	291	291
query28	6116	2261	2216	2216
query29	2901	2796	2641	2641
query30	244	169	169	169
query31	930	734	775	734
query32	70	56	61	56
query33	443	250	254	250
query34	871	468	474	468
query35	1137	913	934	913
query36	1300	1311	1023	1023
query37	93	61	57	57
query38	3062	2917	2915	2915
query39	1395	1342	1329	1329
query40	201	96	96	96
query41	40	37	36	36
query42	88	85	84	84
query43	565	591	533	533
query44	1163	709	719	709
query45	251	229	233	229
query46	1238	978	953	953
query47	1772	1666	1678	1666
query48	512	428	413	413
query49	611	386	372	372
query50	854	604	566	566
query51	4749	4724	4639	4639
query52	92	87	84	84
query53	232	182	178	178
query54	2679	2443	2458	2443
query55	85	86	83	83
query56	225	214	218	214
query57	1176	1046	1110	1046
query58	220	193	211	193
query59	3713	3399	3322	3322
query60	214	198	200	198
query61	100	96	98	96
query62	821	486	509	486
query63	206	178	169	169
query64	3007	1478	1474	1474
query65	3606	3540	3519	3519
query66	730	420	406	406
query67	18089	15818	15127	15127
query68	9491	655	638	638
query69	483	278	272	272
query70	1683	1487	1315	1315
query71	421	304	313	304
query72	6856	4862	4867	4862
query73	779	317	308	308
query74	6339	5891	5811	5811
query75	4658	3743	3633	3633
query76	4792	1118	1193	1118
query77	654	263	248	248
query78	12825	11894	11972	11894
query79	5411	633	633	633
query80	1075	390	400	390
query81	499	239	243	239
query82	1522	96	95	95
query83	171	140	130	130
query84	267	73	69	69
query85	955	323	327	323
query86	348	315	294	294
query87	3166	2989	3043	2989
query88	4780	2291	2313	2291
query89	361	314	278	278
query90	2068	207	208	207
query91	176	136	124	124
query92	59	52	51	51
query93	4869	587	592	587
query94	880	207	202	202
query95	2074	2060	2075	2060
query96	649	324	315	315
query97	6515	6422	6411	6411
query98	221	220	202	202
query99	2984	966	941	941
Total cold run time: 316552 ms
Total hot run time: 213186 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.03 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b665ddf2d219a8a30e1bef3ca1131f9f6104b934, data reload: false

query1	0.02	0.03	0.03
query2	0.07	0.03	0.03
query3	0.25	0.05	0.05
query4	1.79	0.07	0.07
query5	0.54	0.51	0.52
query6	1.23	0.60	0.63
query7	0.02	0.00	0.01
query8	0.04	0.03	0.02
query9	0.53	0.49	0.48
query10	0.53	0.53	0.54
query11	0.12	0.09	0.09
query12	0.13	0.09	0.09
query13	0.62	0.63	0.61
query14	0.78	0.79	0.80
query15	0.79	0.76	0.76
query16	0.37	0.38	0.38
query17	0.97	1.02	0.99
query18	0.22	0.25	0.24
query19	1.90	1.86	1.85
query20	0.01	0.00	0.01
query21	15.50	0.53	0.56
query22	2.01	2.27	1.77
query23	17.51	1.00	0.96
query24	5.75	1.04	2.00
query25	0.34	0.09	0.06
query26	0.78	0.15	0.15
query27	0.03	0.04	0.03
query28	6.49	0.74	0.73
query29	12.61	2.29	2.28
query30	0.54	0.52	0.51
query31	2.81	0.39	0.37
query32	3.38	0.49	0.49
query33	3.05	3.06	3.07
query34	15.27	4.80	4.82
query35	4.89	4.86	4.89
query36	1.06	1.02	1.02
query37	0.06	0.04	0.05
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.15
query41	0.07	0.02	0.01
query42	0.02	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 103.34 s
Total hot run time: 31.03 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit b665ddf2d219a8a30e1bef3ca1131f9f6104b934 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.7 seconds inserted 10000000 Rows, about 460K ops/s

@dataroaring dataroaring merged commit 5b23ab7 into apache:branch-2.0 Sep 8, 2024
22 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
area/load Issues or PRs related to all kinds of load
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@sollhui @doris-robot @dataroaring