Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[opt](index compaction)Use RAM dir to create tmp index_writer #41371

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

[opt](index compaction)Use RAM dir to create tmp index_writer #41371

wants to merge 1 commit into from

Conversation

qidaye
Copy link
Contributor

@qidaye qidaye commented Sep 26, 2024

Proposed changes

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@qidaye qidaye force-pushed the opt_index_compaction_tmp_idx_writer branch from 0776c52 to f0abe3a Compare September 26, 2024 13:07
@qidaye
Copy link
Contributor Author

qidaye commented Sep 26, 2024

run buildall

airborne12
airborne12 approved these changes Sep 26, 2024
View reviewed changes
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 26, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.31% (9634/25823)
Line Coverage: 28.72% (79731/277656)
Region Coverage: 28.13% (41208/146506)
Branch Coverage: 24.75% (20983/84766)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f0abe3a65952a7d68dc92c38959e04ad3915c68a_f0abe3a65952a7d68dc92c38959e04ad3915c68a/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 40665 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f0abe3a65952a7d68dc92c38959e04ad3915c68a, data reload: false

------ Round 1 ----------------------------------
q1	17563	7389	7291	7291
q2	2049	280	277	277
q3	12164	1055	1123	1055
q4	10581	754	805	754
q5	7746	2878	2796	2796
q6	236	149	147	147
q7	953	608	605	605
q8	9350	1889	1921	1889
q9	6473	6456	6403	6403
q10	7023	2286	2350	2286
q11	441	247	252	247
q12	430	220	213	213
q13	17760	2969	2995	2969
q14	242	215	217	215
q15	563	522	514	514
q16	649	581	601	581
q17	964	595	528	528
q18	7169	6731	6697	6697
q19	1396	1069	946	946
q20	493	218	202	202
q21	3929	3136	3056	3056
q22	1078	1001	994	994
Total cold run time: 109252 ms
Total hot run time: 40665 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7299	7195	7254	7195
q2	328	228	228	228
q3	2935	2872	2938	2872
q4	2008	1866	1811	1811
q5	5741	5735	5728	5728
q6	231	144	143	143
q7	2246	1848	1796	1796
q8	3352	3560	3516	3516
q9	8931	8912	8850	8850
q10	3581	3525	3536	3525
q11	572	480	490	480
q12	856	607	604	604
q13	9036	3196	3150	3150
q14	303	285	273	273
q15	563	539	536	536
q16	678	637	643	637
q17	1830	1598	1629	1598
q18	8200	7702	7603	7603
q19	1731	1490	1537	1490
q20	2094	1873	1886	1873
q21	5565	5407	5424	5407
q22	1116	1055	1058	1055
Total cold run time: 69196 ms
Total hot run time: 60370 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191297 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f0abe3a65952a7d68dc92c38959e04ad3915c68a, data reload: false

query1	952	390	385	385
query2	6241	2122	2042	2042
query3	8683	191	201	191
query4	33736	23481	23789	23481
query5	3927	463	456	456
query6	273	159	180	159
query7	4186	307	295	295
query8	273	223	216	216
query9	9233	2701	2688	2688
query10	484	270	284	270
query11	17857	15219	15222	15219
query12	146	96	97	96
query13	1519	439	412	412
query14	9776	6674	7493	6674
query15	263	183	175	175
query16	8069	439	471	439
query17	1753	616	597	597
query18	2208	315	318	315
query19	388	160	150	150
query20	119	113	116	113
query21	213	104	110	104
query22	4606	4465	4475	4465
query23	34954	34107	34024	34024
query24	11103	2840	2880	2840
query25	606	404	407	404
query26	965	165	161	161
query27	2159	305	305	305
query28	7365	2441	2420	2420
query29	814	438	436	436
query30	295	153	153	153
query31	1024	773	812	773
query32	101	55	55	55
query33	768	306	300	300
query34	896	488	502	488
query35	911	729	740	729
query36	1089	946	956	946
query37	149	94	88	88
query38	4035	3836	3990	3836
query39	1475	1441	1409	1409
query40	211	96	97	96
query41	52	47	49	47
query42	114	97	94	94
query43	539	491	496	491
query44	1183	799	809	799
query45	196	165	166	165
query46	1159	722	709	709
query47	1928	1821	1832	1821
query48	473	377	385	377
query49	936	412	418	412
query50	809	430	415	415
query51	7455	6898	6897	6897
query52	100	91	87	87
query53	258	181	184	181
query54	1223	478	468	468
query55	83	75	78	75
query56	290	263	271	263
query57	1218	1116	1100	1100
query58	239	250	260	250
query59	3142	2990	3036	2990
query60	308	274	267	267
query61	106	105	104	104
query62	862	660	682	660
query63	218	187	186	186
query64	4012	666	621	621
query65	3288	3139	3192	3139
query66	833	306	301	301
query67	15944	15613	15494	15494
query68	4660	560	556	556
query69	526	306	303	303
query70	1110	1061	1126	1061
query71	353	267	282	267
query72	7471	4133	4013	4013
query73	772	344	350	344
query74	10218	9051	9045	9045
query75	3444	2714	2665	2665
query76	3079	876	907	876
query77	623	290	304	290
query78	10475	9511	9469	9469
query79	2181	599	597	597
query80	2833	443	456	443
query81	586	238	238	238
query82	693	144	132	132
query83	314	138	131	131
query84	282	83	86	83
query85	1817	293	288	288
query86	458	302	281	281
query87	4444	4378	4331	4331
query88	3461	2442	2378	2378
query89	403	293	295	293
query90	2252	189	185	185
query91	189	145	145	145
query92	68	47	48	47
query93	2274	563	540	540
query94	1305	305	303	303
query95	357	254	261	254
query96	644	291	287	287
query97	3227	3109	3146	3109
query98	224	200	203	200
query99	1538	1340	1325	1325
Total cold run time: 303074 ms
Total hot run time: 191297 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.01 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f0abe3a65952a7d68dc92c38959e04ad3915c68a, data reload: false

query1	0.05	0.05	0.05
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.65	0.10	0.10
query5	0.51	0.52	0.52
query6	1.13	0.73	0.72
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.55	0.49	0.51
query10	0.55	0.53	0.54
query11	0.13	0.11	0.10
query12	0.13	0.11	0.12
query13	0.62	0.60	0.60
query14	2.71	2.74	2.82
query15	0.89	0.82	0.82
query16	0.37	0.38	0.40
query17	1.02	1.06	1.05
query18	0.20	0.20	0.20
query19	1.94	1.80	2.04
query20	0.01	0.01	0.01
query21	15.34	0.59	0.58
query22	2.65	2.75	1.72
query23	17.32	0.85	0.87
query24	2.50	0.46	0.38
query25	0.19	0.09	0.08
query26	0.46	0.14	0.14
query27	0.05	0.04	0.05
query28	12.04	1.08	1.07
query29	12.55	3.21	3.24
query30	0.25	0.07	0.06
query31	2.87	0.37	0.37
query32	3.30	0.47	0.48
query33	2.99	3.05	3.07
query34	17.12	4.41	4.47
query35	4.50	4.49	4.47
query36	0.65	0.47	0.48
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.11	0.12
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.04
Total cold run time: 108.06 s
Total hot run time: 32.01 s

@qidaye qidaye force-pushed the opt_index_compaction_tmp_idx_writer branch from f0abe3a to 40a6c53 Compare October 8, 2024 02:44
@qidaye
Copy link
Contributor Author

qidaye commented Oct 8, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40812 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 40a6c5358994ce9346ed4ae98888ca95691c6a21, data reload: false

------ Round 1 ----------------------------------
q1	17894	7443	7261	7261
q2	2026	315	276	276
q3	12015	1038	1143	1038
q4	10571	744	755	744
q5	7768	2919	2847	2847
q6	234	149	146	146
q7	1042	622	600	600
q8	9358	1936	1971	1936
q9	6574	6506	6429	6429
q10	7003	2297	2304	2297
q11	445	243	252	243
q12	421	212	215	212
q13	17809	2974	2979	2974
q14	260	224	212	212
q15	571	531	524	524
q16	649	599	596	596
q17	981	571	596	571
q18	7240	6658	6791	6658
q19	1334	1018	881	881
q20	485	205	195	195
q21	4005	3174	3209	3174
q22	1098	1006	998	998
Total cold run time: 109783 ms
Total hot run time: 40812 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7309	7808	7233	7233
q2	333	231	233	231
q3	3065	2898	2951	2898
q4	2155	1843	1796	1796
q5	5758	5768	5771	5768
q6	237	147	141	141
q7	2274	1841	1851	1841
q8	3398	3534	3458	3458
q9	8930	8943	8953	8943
q10	3628	3573	3575	3573
q11	595	494	494	494
q12	868	642	600	600
q13	8605	3169	3195	3169
q14	331	293	278	278
q15	572	536	528	528
q16	691	641	625	625
q17	1850	1667	1618	1618
q18	8314	7748	7597	7597
q19	1733	1317	1529	1317
q20	2113	1928	1900	1900
q21	5597	5516	5204	5204
q22	1139	1064	1077	1064
Total cold run time: 69495 ms
Total hot run time: 60276 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191980 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 40a6c5358994ce9346ed4ae98888ca95691c6a21, data reload: false

query1	925	389	398	389
query2	6244	2081	2016	2016
query3	8694	193	204	193
query4	34074	23542	23333	23333
query5	3302	475	461	461
query6	261	185	164	164
query7	4189	306	307	306
query8	268	208	212	208
query9	9286	2654	2662	2654
query10	466	291	272	272
query11	17804	15304	15202	15202
query12	168	103	103	103
query13	1562	487	442	442
query14	9091	7233	7176	7176
query15	252	164	176	164
query16	7871	455	457	455
query17	1636	609	591	591
query18	1956	338	314	314
query19	241	162	157	157
query20	119	115	117	115
query21	205	108	115	108
query22	4911	4429	4632	4429
query23	35103	33921	34203	33921
query24	11163	2889	2931	2889
query25	622	400	404	400
query26	1225	164	165	164
query27	2530	306	302	302
query28	7628	2427	2431	2427
query29	846	454	446	446
query30	270	156	158	156
query31	1049	833	798	798
query32	103	54	55	54
query33	764	300	304	300
query34	935	525	500	500
query35	878	744	749	744
query36	1121	973	956	956
query37	157	92	93	92
query38	4014	4045	3956	3956
query39	1491	1443	1427	1427
query40	212	99	97	97
query41	48	45	46	45
query42	115	101	95	95
query43	544	505	488	488
query44	1728	836	818	818
query45	204	167	167	167
query46	1169	727	724	724
query47	1973	1873	1853	1853
query48	454	340	350	340
query49	946	416	399	399
query50	811	421	421	421
query51	7301	6917	6848	6848
query52	100	88	87	87
query53	256	182	189	182
query54	1122	488	481	481
query55	80	76	76	76
query56	281	262	262	262
query57	1336	1176	1166	1166
query58	223	223	238	223
query59	3262	3079	3263	3079
query60	284	272	260	260
query61	108	99	109	99
query62	873	711	652	652
query63	216	196	188	188
query64	4045	650	625	625
query65	3308	3181	3250	3181
query66	960	325	307	307
query67	17280	16288	15702	15702
query68	4781	573	562	562
query69	1439	296	312	296
query70	1199	1128	1053	1053
query71	417	272	267	267
query72	7935	3993	3981	3981
query73	764	344	344	344
query74	10129	8983	8888	8888
query75	4392	2692	2701	2692
query76	3820	941	946	941
query77	736	299	295	295
query78	10619	9518	9466	9466
query79	1816	598	595	595
query80	1028	453	442	442
query81	550	242	242	242
query82	662	135	144	135
query83	363	140	131	131
query84	283	79	75	75
query85	1381	288	290	288
query86	397	284	308	284
query87	4392	4273	4371	4273
query88	3499	2445	2444	2444
query89	401	281	297	281
query90	1899	185	184	184
query91	142	109	109	109
query92	65	47	49	47
query93	1892	576	559	559
query94	954	294	285	285
query95	349	255	249	249
query96	618	282	289	282
query97	3305	3203	3139	3139
query98	213	206	199	199
query99	1649	1304	1303	1303
Total cold run time: 304752 ms
Total hot run time: 191980 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.1 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 40a6c5358994ce9346ed4ae98888ca95691c6a21, data reload: false

query1	0.05	0.05	0.04
query2	0.07	0.03	0.02
query3	0.23	0.06	0.06
query4	1.65	0.10	0.10
query5	0.54	0.50	0.52
query6	1.14	0.73	0.73
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.55	0.50	0.48
query10	0.56	0.55	0.53
query11	0.14	0.11	0.11
query12	0.13	0.11	0.11
query13	0.61	0.60	0.60
query14	2.71	2.69	2.72
query15	0.93	0.82	0.82
query16	0.38	0.38	0.41
query17	1.04	1.06	0.97
query18	0.20	0.19	0.20
query19	1.94	1.80	1.94
query20	0.01	0.00	0.01
query21	15.35	0.59	0.58
query22	2.61	2.28	2.20
query23	17.19	0.87	0.89
query24	2.84	1.11	1.39
query25	0.24	0.17	0.13
query26	0.51	0.14	0.13
query27	0.04	0.03	0.04
query28	10.25	1.09	1.07
query29	12.57	3.23	3.23
query30	0.24	0.05	0.06
query31	2.88	0.38	0.38
query32	3.28	0.46	0.46
query33	2.96	2.99	3.02
query34	16.86	4.46	4.44
query35	4.47	4.49	4.51
query36	0.64	0.48	0.48
query37	0.10	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.17	0.12	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 106.36 s
Total hot run time: 33.1 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.28% (9631/25832)
Line Coverage: 28.68% (79894/278580)
Region Coverage: 28.11% (41296/146929)
Branch Coverage: 24.73% (21042/85086)
Coverage Report: http://coverage.selectdb-in.cc/coverage/40a6c5358994ce9346ed4ae98888ca95691c6a21_40a6c5358994ce9346ed4ae98888ca95691c6a21/report/index.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xiaokang xiaokang xiaokang left review comments

@airborne12 airborne12 airborne12 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.x dev/2.1.x dev/3.0.x reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@qidaye @doris-robot @xiaokang @airborne12