Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[enhance](mtmv)Optimize the speed of obtaining the last update time of Hive #40169

Merged
merged 4 commits into from
Sep 3, 2024
Merged

[enhance](mtmv)Optimize the speed of obtaining the last update time of Hive #40169

merged 4 commits into from
Sep 3, 2024

Conversation

zddr
Copy link
Contributor

@zddr zddr commented Aug 30, 2024

Previously, to obtain the last update time of a hive table, it was necessary to obtain the last update time of all partitions under the table, which required generating a large map.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zddr
Copy link
Contributor Author

zddr commented Aug 30, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37843 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c7780492aa2d2969fc1dc2179f321e5ef9f8f8bd, data reload: false

------ Round 1 ----------------------------------
q1	17602	4448	4302	4302
q2	2011	183	178	178
q3	11760	985	1141	985
q4	10511	763	747	747
q5	7761	2786	2787	2786
q6	228	144	141	141
q7	956	620	611	611
q8	9326	2042	2057	2042
q9	7275	6511	6542	6511
q10	7006	2179	2129	2129
q11	463	242	232	232
q12	402	228	229	228
q13	17760	3026	3022	3022
q14	265	239	224	224
q15	517	497	474	474
q16	569	504	531	504
q17	976	642	698	642
q18	7334	6980	6760	6760
q19	1395	1013	1019	1013
q20	672	337	342	337
q21	3879	3117	2969	2969
q22	1108	1006	1024	1006
Total cold run time: 109776 ms
Total hot run time: 37843 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4363	4299	4267	4267
q2	382	268	262	262
q3	2891	2620	2730	2620
q4	1913	1627	1660	1627
q5	5582	5668	5670	5668
q6	235	139	146	139
q7	2244	1809	1859	1809
q8	3319	3426	3412	3412
q9	8871	8875	8819	8819
q10	3557	3358	3377	3358
q11	595	497	517	497
q12	822	671	649	649
q13	14842	3310	3200	3200
q14	323	302	288	288
q15	539	477	493	477
q16	642	564	596	564
q17	1839	1531	1515	1515
q18	8036	7810	7964	7810
q19	1748	1598	1542	1542
q20	2158	1907	1933	1907
q21	5792	5631	5488	5488
q22	1170	1040	1060	1040
Total cold run time: 71863 ms
Total hot run time: 56958 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193662 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c7780492aa2d2969fc1dc2179f321e5ef9f8f8bd, data reload: false

query1	1247	895	853	853
query2	6293	1991	1895	1895
query3	10593	3991	3933	3933
query4	59668	25342	23271	23271
query5	5516	517	515	515
query6	399	163	164	163
query7	5766	302	308	302
query8	294	217	245	217
query9	8966	2518	2508	2508
query10	517	285	283	283
query11	18203	15056	15529	15056
query12	156	105	101	101
query13	1544	415	385	385
query14	11352	7803	7576	7576
query15	246	170	188	170
query16	7504	510	487	487
query17	1123	595	613	595
query18	2037	312	302	302
query19	289	168	168	168
query20	129	121	122	121
query21	214	112	110	110
query22	4704	4597	4463	4463
query23	34307	33535	33362	33362
query24	5910	2871	2847	2847
query25	534	384	383	383
query26	689	161	157	157
query27	1798	285	287	285
query28	3874	2096	2085	2085
query29	684	421	434	421
query30	243	161	152	152
query31	939	747	777	747
query32	83	54	59	54
query33	498	296	283	283
query34	887	494	494	494
query35	834	704	724	704
query36	1066	936	936	936
query37	154	102	89	89
query38	3980	3894	3867	3867
query39	1479	1377	1519	1377
query40	195	122	117	117
query41	46	48	49	48
query42	119	95	96	95
query43	525	481	481	481
query44	1133	747	747	747
query45	194	170	163	163
query46	1082	748	771	748
query47	1880	1771	1808	1771
query48	364	297	310	297
query49	756	441	458	441
query50	828	462	427	427
query51	7294	7103	7027	7027
query52	97	90	92	90
query53	256	180	184	180
query54	565	458	446	446
query55	78	77	80	77
query56	270	255	262	255
query57	1175	1105	1066	1066
query58	228	226	224	224
query59	3081	2787	2806	2787
query60	300	274	279	274
query61	107	97	100	97
query62	744	661	653	653
query63	217	186	188	186
query64	2847	695	653	653
query65	3174	3150	3160	3150
query66	609	343	337	337
query67	15393	15019	15001	15001
query68	2956	587	585	585
query69	408	278	282	278
query70	1206	1128	1136	1128
query71	358	275	276	275
query72	6101	4105	3993	3993
query73	774	337	339	337
query74	9147	8874	8846	8846
query75	3387	2677	2696	2677
query76	1486	1027	1038	1027
query77	614	331	316	316
query78	11030	9371	9299	9299
query79	1060	552	555	552
query80	700	531	506	506
query81	529	236	232	232
query82	252	153	151	151
query83	175	149	157	149
query84	261	80	79	79
query85	678	293	287	287
query86	314	285	308	285
query87	4301	4261	4276	4261
query88	3017	2340	2348	2340
query89	368	289	284	284
query90	1808	201	197	197
query91	124	98	100	98
query92	64	52	52	52
query93	1056	539	537	537
query94	609	306	304	304
query95	354	262	266	262
query96	598	279	275	275
query97	3221	3166	3076	3076
query98	219	201	271	201
query99	1538	1260	1218	1218
Total cold run time: 306989 ms
Total hot run time: 193662 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.17 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c7780492aa2d2969fc1dc2179f321e5ef9f8f8bd, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.04
query4	1.69	0.08	0.08
query5	0.49	0.50	0.48
query6	1.14	0.74	0.74
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.56	0.48	0.49
query10	0.54	0.53	0.54
query11	0.15	0.12	0.12
query12	0.15	0.13	0.13
query13	0.60	0.59	0.59
query14	2.02	2.09	2.05
query15	0.91	0.84	0.85
query16	0.36	0.37	0.39
query17	1.10	1.06	1.00
query18	0.22	0.20	0.21
query19	1.89	1.80	1.72
query20	0.01	0.01	0.01
query21	15.39	0.67	0.65
query22	4.24	7.39	1.82
query23	18.28	1.44	1.37
query24	2.06	0.23	0.23
query25	0.15	0.08	0.08
query26	0.26	0.19	0.17
query27	0.08	0.08	0.07
query28	13.25	1.05	1.03
query29	12.57	3.41	3.44
query30	0.24	0.06	0.05
query31	2.87	0.40	0.41
query32	3.23	0.48	0.48
query33	3.02	3.03	3.01
query34	16.97	4.41	4.43
query35	4.46	4.40	4.45
query36	0.67	0.48	0.47
query37	0.19	0.16	0.16
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.14	0.12	0.12
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 110.72 s
Total hot run time: 32.17 s

@zddr
Copy link
Contributor Author

zddr commented Aug 30, 2024

run buildall

@zddr
Copy link
Contributor Author

zddr commented Aug 30, 2024

run buildall

@zddr
Copy link
Contributor Author

zddr commented Aug 30, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38271 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 93752ec0acb44e6c45ca1e5e8be9715f4fb8832f, data reload: false

------ Round 1 ----------------------------------
q1	17951	4600	4449	4449
q2	2208	182	186	182
q3	10480	1173	1096	1096
q4	10138	861	698	698
q5	7718	2904	2803	2803
q6	232	137	145	137
q7	978	622	609	609
q8	9333	2097	2100	2097
q9	7165	6580	6559	6559
q10	7002	2273	2214	2214
q11	449	241	257	241
q12	406	225	222	222
q13	17765	3023	3036	3023
q14	292	236	224	224
q15	531	477	480	477
q16	580	508	521	508
q17	981	742	699	699
q18	7511	6932	6799	6799
q19	1390	1022	987	987
q20	675	339	331	331
q21	3902	2923	2983	2923
q22	1114	993	998	993
Total cold run time: 108801 ms
Total hot run time: 38271 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4358	4328	4292	4292
q2	384	274	276	274
q3	2941	2700	2707	2700
q4	1938	1681	1643	1643
q5	5442	5416	5403	5403
q6	225	133	132	132
q7	2117	1727	1734	1727
q8	3217	3378	3379	3378
q9	8430	8419	8425	8419
q10	3458	3172	3206	3172
q11	598	495	509	495
q12	795	627	620	620
q13	12262	3050	3049	3049
q14	317	282	264	264
q15	524	476	492	476
q16	591	542	550	542
q17	1803	1484	1498	1484
q18	7917	7573	7523	7523
q19	1690	1570	1583	1570
q20	2078	1841	1812	1812
q21	5490	5156	5185	5156
q22	1090	1046	1027	1027
Total cold run time: 67665 ms
Total hot run time: 55158 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 188223 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 93752ec0acb44e6c45ca1e5e8be9715f4fb8832f, data reload: false

query1	922	373	365	365
query2	6466	2002	1887	1887
query3	6650	207	235	207
query4	32975	23512	23314	23314
query5	4151	511	496	496
query6	258	162	163	162
query7	4585	287	288	287
query8	265	234	217	217
query9	8541	2490	2488	2488
query10	443	262	275	262
query11	17682	15185	15023	15023
query12	155	102	106	102
query13	1651	365	379	365
query14	10135	7421	7498	7421
query15	289	172	181	172
query16	7390	485	475	475
query17	1654	599	531	531
query18	1962	289	285	285
query19	331	154	145	145
query20	114	113	111	111
query21	211	103	102	102
query22	4552	4122	4117	4117
query23	33979	33348	34010	33348
query24	11072	2895	2795	2795
query25	629	421	387	387
query26	1153	165	160	160
query27	2604	281	272	272
query28	7427	2124	2092	2092
query29	810	421	407	407
query30	308	157	150	150
query31	1078	754	779	754
query32	96	53	59	53
query33	751	288	287	287
query34	965	463	491	463
query35	829	728	709	709
query36	1102	890	948	890
query37	159	93	90	90
query38	4051	3862	3926	3862
query39	1438	1408	1411	1408
query40	260	118	114	114
query41	46	45	45	45
query42	111	95	95	95
query43	509	463	473	463
query44	1205	747	729	729
query45	198	167	173	167
query46	1093	731	753	731
query47	1895	1813	1770	1770
query48	376	307	291	291
query49	975	413	413	413
query50	807	417	404	404
query51	7248	7105	6990	6990
query52	103	88	89	88
query53	253	221	176	176
query54	940	450	444	444
query55	78	73	77	73
query56	273	261	247	247
query57	1178	1098	1081	1081
query58	248	227	233	227
query59	3068	2995	2733	2733
query60	295	266	271	266
query61	106	104	100	100
query62	807	662	667	662
query63	216	182	179	179
query64	4428	725	645	645
query65	3192	3153	3147	3147
query66	1011	336	337	336
query67	15485	15426	15316	15316
query68	3242	577	567	567
query69	391	267	280	267
query70	1187	1165	1135	1135
query71	338	276	271	271
query72	6324	4122	4008	4008
query73	754	331	332	331
query74	9157	8849	8994	8849
query75	3384	2683	2704	2683
query76	1993	979	964	964
query77	436	305	303	303
query78	9563	9118	8909	8909
query79	1042	548	530	530
query80	731	503	511	503
query81	454	235	233	233
query82	254	148	147	147
query83	172	149	160	149
query84	236	79	76	76
query85	682	291	273	273
query86	322	275	310	275
query87	4514	4317	4409	4317
query88	2956	2323	2418	2323
query89	381	297	281	281
query90	1794	203	193	193
query91	122	102	101	101
query92	63	53	51	51
query93	1027	530	527	527
query94	688	303	303	303
query95	352	262	263	262
query96	591	272	271	271
query97	3188	3075	3063	3063
query98	219	260	204	204
query99	1471	1272	1261	1261
Total cold run time: 284265 ms
Total hot run time: 188223 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.2 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 93752ec0acb44e6c45ca1e5e8be9715f4fb8832f, data reload: false

query1	0.05	0.05	0.04
query2	0.08	0.05	0.04
query3	0.23	0.04	0.05
query4	1.68	0.08	0.07
query5	0.51	0.50	0.51
query6	1.14	0.73	0.75
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.47	0.49
query10	0.55	0.55	0.54
query11	0.16	0.12	0.12
query12	0.16	0.12	0.13
query13	0.62	0.60	0.58
query14	2.10	2.06	2.04
query15	0.89	0.83	0.82
query16	0.37	0.38	0.37
query17	0.99	1.03	1.06
query18	0.22	0.21	0.21
query19	1.94	1.85	1.71
query20	0.01	0.02	0.01
query21	15.40	0.68	0.66
query22	4.43	7.37	1.98
query23	18.24	1.44	1.38
query24	2.07	0.24	0.23
query25	0.15	0.09	0.08
query26	0.28	0.18	0.18
query27	0.08	0.08	0.08
query28	13.32	1.02	0.99
query29	12.60	3.35	3.33
query30	0.25	0.06	0.06
query31	2.86	0.41	0.39
query32	3.25	0.50	0.47
query33	2.98	2.96	2.98
query34	17.18	4.39	4.40
query35	4.46	4.47	4.52
query36	0.65	0.47	0.49
query37	0.18	0.15	0.15
query38	0.15	0.16	0.15
query39	0.04	0.04	0.04
query40	0.15	0.12	0.13
query41	0.10	0.05	0.05
query42	0.06	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 111.25 s
Total hot run time: 32.2 s

morningman
morningman approved these changes Aug 30, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 30, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@zddr
Copy link
Contributor Author

zddr commented Sep 2, 2024

run cloud_p0

@morrySnow morrySnow merged commit 5ea6e4b into apache:master Sep 3, 2024
26 of 28 checks passed
zddr added a commit to zddr/incubator-doris that referenced this pull request Sep 5, 2024
@zddr
[enhance](mtmv)Optimize the speed of obtaining the last update time o…
78931ce 
…f Hive (apache#40169)

Previously, to obtain the last update time of a hive table, it was
necessary to obtain the last update time of all partitions under the
table, which required generating a large map.
@zddr zddr mentioned this pull request Sep 5, 2024
Merged
yiguolei pushed a commit that referenced this pull request Sep 9, 2024
@zddr
[enhance](mtmv)Optimize the speed of obtaining the last update time o… (
4084ff2 
#40421)

…f Hive (#40169)

pick: #40169
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
@zddr @dataroaring
[enhance](mtmv)Optimize the speed of obtaining the last update time o…
6c216dc 
…f Hive (#40169)

Previously, to obtain the last update time of a hive table, it was
necessary to obtain the last update time of all partitions under the
table, which required generating a large map.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morningman morningman morningman approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.7-merged dev/3.0.3-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@zddr @doris-robot @morningman @morrySnow @yiguolei