Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](Nereids) fix date function rewrite on datetimev1 column bug #32569 #32719

Merged
merged 4 commits into from
Mar 25, 2024
Merged

[fix](Nereids) fix date function rewrite on datetimev1 column bug #32569 #32719

merged 4 commits into from
Mar 25, 2024

Conversation

ziyanTOP
Copy link
Contributor

Proposed changes

cherry picked from #32569

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@ziyanTOP
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49660 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a86844ee36782df308bdabc017f92adb9ce264ff, data reload: false

------ Round 1 ----------------------------------
q1	17757	4369	4364	4364
q2	2043	148	136	136
q3	10372	1899	1971	1899
q4	10309	1220	1313	1220
q5	8475	4142	3969	3969
q6	224	122	122	122
q7	2019	1573	1591	1573
q8	9502	2697	2702	2697
q9	13496	10532	10220	10220
q10	8576	3513	3503	3503
q11	406	239	243	239
q12	475	303	297	297
q13	18333	4000	4010	4000
q14	346	323	327	323
q15	505	457	444	444
q16	673	598	596	596
q17	1127	992	942	942
q18	7279	6882	6983	6882
q19	1696	1598	1477	1477
q20	506	285	315	285
q21	4454	4132	4084	4084
q22	483	399	388	388
Total cold run time: 119056 ms
Total hot run time: 49660 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4297	4299	4287	4287
q2	316	220	215	215
q3	4162	4170	4105	4105
q4	2777	2737	2734	2734
q5	7251	7234	7285	7234
q6	240	120	117	117
q7	3261	2856	2884	2856
q8	4419	4424	4485	4424
q9	17072	16915	17050	16915
q10	4203	4273	4254	4254
q11	753	680	686	680
q12	1044	827	866	827
q13	6959	3772	3748	3748
q14	445	422	423	422
q15	502	446	454	446
q16	761	705	693	693
q17	3830	3917	3868	3868
q18	8866	8734	8879	8734
q19	1729	1665	1625	1625
q20	2359	2181	2136	2136
q21	8508	8612	8617	8612
q22	1056	962	978	962
Total cold run time: 84810 ms
Total hot run time: 79894 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 200523 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a86844ee36782df308bdabc017f92adb9ce264ff, data reload: false

query1	916	385	384	384
query2	6515	2344	1977	1977
query3	6917	203	203	203
query4	23994	18721	18498	18498
query5	20902	6670	6464	6464
query6	415	211	244	211
query7	5160	298	295	295
query8	290	253	261	253
query9	3051	2618	2557	2557
query10	418	281	298	281
query11	11380	10568	10707	10568
query12	119	73	72	72
query13	5578	657	631	631
query14	18997	13404	13211	13211
query15	357	229	239	229
query16	6405	278	269	269
query17	1419	1408	843	843
query18	2304	406	393	393
query19	197	144	144	144
query20	81	74	76	74
query21	183	90	97	90
query22	5212	5047	4985	4985
query23	32654	31868	31766	31766
query24	6755	6545	6443	6443
query25	533	431	408	408
query26	455	160	153	153
query27	1589	289	287	287
query28	6057	2225	2197	2197
query29	2842	2758	2838	2758
query30	245	159	162	159
query31	905	729	717	717
query32	66	58	59	58
query33	389	235	252	235
query34	839	451	474	451
query35	1116	952	937	937
query36	1307	1116	1205	1116
query37	90	60	58	58
query38	3025	2930	2896	2896
query39	1387	1309	1314	1309
query40	210	92	93	92
query41	37	32	31	31
query42	85	80	84	80
query43	685	704	553	553
query44	1131	733	716	716
query45	239	226	222	222
query46	1222	976	962	962
query47	1972	1752	1754	1752
query48	987	662	664	662
query49	609	380	359	359
query50	862	613	584	584
query51	4836	4679	4641	4641
query52	97	77	76	76
query53	443	316	314	314
query54	2640	2412	2473	2412
query55	80	70	79	70
query56	218	210	207	207
query57	1143	1067	1066	1066
query58	209	200	213	200
query59	3557	3148	3323	3148
query60	199	189	195	189
query61	86	86	82	82
query62	819	496	509	496
query63	473	328	330	328
query64	2160	1479	1341	1341
query65	3662	3575	3553	3553
query66	759	374	359	359
query67	15512	15411	14831	14831
query68	10698	661	681	661
query69	569	336	322	322
query70	2191	1517	1599	1517
query71	430	304	306	304
query72	6465	3472	3489	3472
query73	735	325	321	321
query74	6346	5869	5759	5759
query75	5432	3748	3691	3691
query76	6889	1181	1232	1181
query77	1178	254	246	246
query78	13122	11816	11424	11424
query79	7584	624	654	624
query80	912	396	413	396
query81	459	234	227	227
query82	876	102	92	92
query83	159	139	131	131
query84	253	69	68	68
query85	855	290	284	284
query86	331	286	310	286
query87	3222	3018	3014	3014
query88	4366	2345	2367	2345
query89	459	308	285	285
query90	1926	201	220	201
query91	153	118	115	115
query92	53	54	51	51
query93	5947	586	618	586
query94	708	209	205	205
query95	1106	1078	1064	1064
query96	647	331	332	331
query97	6404	6364	6415	6364
query98	186	179	176	176
query99	3071	874	986	874
Total cold run time: 319728 ms
Total hot run time: 200523 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit a86844ee36782df308bdabc017f92adb9ce264ff with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       22.3 seconds inserted 10000000 Rows, about 448K ops/s

@924060929
Copy link
Contributor

run buildall

@924060929 924060929 changed the title [fix] (Nereids) fix date function rewrite on datetimev1 column bug #32569 [fix](Nereids) fix date function rewrite on datetimev1 column bug #32569 Mar 25, 2024
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 25, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 50475 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a86844ee36782df308bdabc017f92adb9ce264ff, data reload: false

------ Round 1 ----------------------------------
q1	17696	4442	4372	4372
q2	2079	158	142	142
q3	10336	1909	1969	1909
q4	10120	1250	1340	1250
q5	8548	4033	4052	4033
q6	231	124	121	121
q7	2042	1615	1612	1612
q8	9295	2769	2741	2741
q9	10921	10640	10744	10640
q10	8686	3496	3561	3496
q11	412	229	242	229
q12	466	292	296	292
q13	18360	3968	4054	3968
q14	351	322	329	322
q15	501	458	451	451
q16	706	594	584	584
q17	1147	982	984	982
q18	7243	6899	6913	6899
q19	1721	1653	1610	1610
q20	531	313	304	304
q21	4529	4125	4146	4125
q22	493	393	398	393
Total cold run time: 116414 ms
Total hot run time: 50475 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4340	4358	4312	4312
q2	315	222	216	216
q3	4160	4166	4166	4166
q4	2767	2743	2744	2743
q5	7309	7278	7340	7278
q6	235	117	116	116
q7	3260	2875	2887	2875
q8	4383	4515	4507	4507
q9	17063	17186	17002	17002
q10	4269	4332	4280	4280
q11	729	711	676	676
q12	1014	870	862	862
q13	6964	3747	3733	3733
q14	446	426	415	415
q15	495	452	448	448
q16	768	705	690	690
q17	3850	3925	3800	3800
q18	8886	8720	8881	8720
q19	1734	1716	1695	1695
q20	2406	2197	2091	2091
q21	8590	8587	8615	8587
q22	1057	960	964	960
Total cold run time: 85040 ms
Total hot run time: 80172 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 201424 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a86844ee36782df308bdabc017f92adb9ce264ff, data reload: false

query1	944	398	395	395
query2	6519	2287	2141	2141
query3	6914	203	205	203
query4	20001	17884	17924	17884
query5	19720	6519	6500	6500
query6	272	217	226	217
query7	4154	295	304	295
query8	255	241	237	237
query9	3151	2692	2610	2610
query10	422	285	299	285
query11	11238	10754	10807	10754
query12	119	82	71	71
query13	5573	678	635	635
query14	17524	13355	13537	13355
query15	363	239	240	239
query16	6476	269	264	264
query17	1712	1458	844	844
query18	2348	409	405	405
query19	199	145	149	145
query20	75	77	77	77
query21	187	101	94	94
query22	5244	4929	5026	4929
query23	32709	31935	32148	31935
query24	7695	6510	6408	6408
query25	531	432	422	422
query26	1163	162	161	161
query27	2211	295	292	292
query28	6104	2266	2211	2211
query29	2959	2667	2718	2667
query30	239	169	163	163
query31	925	712	736	712
query32	68	62	58	58
query33	417	253	247	247
query34	856	467	474	467
query35	1109	884	940	884
query36	1372	1183	1186	1183
query37	87	64	61	61
query38	3061	2909	2949	2909
query39	1367	1322	1317	1317
query40	305	96	92	92
query41	34	32	32	32
query42	88	84	85	84
query43	654	612	574	574
query44	1130	734	737	734
query45	241	226	228	226
query46	1235	995	951	951
query47	1734	1714	1756	1714
query48	995	687	669	669
query49	640	365	348	348
query50	868	586	597	586
query51	4755	4757	4666	4666
query52	88	78	78	78
query53	452	314	316	314
query54	2651	2483	2477	2477
query55	85	79	74	74
query56	229	212	201	201
query57	1152	1131	1037	1037
query58	211	199	210	199
query59	3503	3255	3213	3213
query60	213	195	192	192
query61	85	83	82	82
query62	806	510	531	510
query63	473	333	330	330
query64	2789	1540	1457	1457
query65	3594	3603	3565	3565
query66	789	370	366	366
query67	16608	15527	15377	15377
query68	9619	683	685	683
query69	588	342	368	342
query70	1718	1317	1402	1317
query71	407	299	309	299
query72	6249	3449	3438	3438
query73	733	322	329	322
query74	6315	6092	5931	5931
query75	5293	3689	3712	3689
query76	6002	1192	1092	1092
query77	1003	252	254	252
query78	12516	11954	11547	11547
query79	8421	657	639	639
query80	1418	393	390	390
query81	489	233	229	229
query82	1594	96	96	96
query83	169	138	129	129
query84	258	69	66	66
query85	837	288	278	278
query86	330	292	289	289
query87	3282	3000	3056	3000
query88	4832	2352	2358	2352
query89	383	284	283	283
query90	1970	217	207	207
query91	156	116	127	116
query92	59	54	53	53
query93	6444	597	560	560
query94	699	207	210	207
query95	1119	1074	1054	1054
query96	640	327	325	325
query97	6471	6329	6402	6329
query98	188	185	166	166
query99	2886	973	920	920
Total cold run time: 315778 ms
Total hot run time: 201424 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit a86844ee36782df308bdabc017f92adb9ce264ff with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       22.5 seconds inserted 10000000 Rows, about 444K ops/s

@924060929 924060929 merged commit f0839fb into apache:branch-2.0 Mar 25, 2024
27 of 30 checks passed
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@ziyanTOP
[fix](Nereids) fix date function rewrite on datetimev1 column bug apa…
f979af9 
…che#32569 (apache#32719)

cherry picked from apache#32569
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@924060929 924060929 924060929 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. area/nereids kind/test reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ziyanTOP @doris-robot @924060929