[feat](nereids) support Iceberg time travel syntax #34681

zhangbutao · 2024-05-11T01:18:47Z

Proposed changes

#15418 added Iceberg time travel in legacy parser but not added this syntax Neredis. If we enable nereids and disable fallback to original palnner, time travel won't be available.

This PR added time travel syntas in Neredis.

BTW, we already have nereids time travel regression-test in https://github.com/apache/doris/blob/master/regression-test/suites/external_table_p2/iceberg/test_external_catalog_icebergv2_nereids.groovy, this regression-test will always fail without this PR.

doris/regression-test/suites/external_table_p2/iceberg/test_external_catalog_icebergv2_nereids.groovy

Lines 70 to 75 in 88530bf

    
           // test for 'FOR TIME AS OF' and 'FOR VERSION AS OF' 
        
           def q04 = { 
        
               qt_q15 """ select count(*) from ${hms_catalog_name}.tpch_1000_icebergv2.customer_small FOR TIME AS OF '2022-12-22 02:29:30' """ 
        
               qt_q16 """ select count(*) from ${hms_catalog_name}.tpch_1000_icebergv2.customer_small FOR VERSION AS OF 6113938156088124425 """ 
        
               qt_q17 """ select count(*) from ${iceberg_catalog_name}.tpch_1000_icebergv2.customer_small FOR TIME AS OF '2022-12-22 02:29:30' """ 
        
               qt_q18 """ select count(*) from ${iceberg_catalog_name}.tpch_1000_icebergv2.customer_small FOR VERSION AS OF 6113938156088124425 """

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-05-11T01:18:53Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

zhangbutao · 2024-05-11T01:19:36Z

run buildall

zhangbutao · 2024-05-11T01:25:43Z

@morningman @morrySnow kindly request your review. Thx.

fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundRelation.java

fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4

zhangbutao · 2024-05-11T14:43:12Z

run buildall

doris-robot · 2024-05-11T15:01:18Z

TPC-H: Total hot run time: 39892 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fa83cce7bede1c1f53806f37981ee6ab5f4bd6b7, data reload: false

------ Round 1 ----------------------------------
q1	17764	4608	4366	4366
q2	2618	188	193	188
q3	10985	1200	1167	1167
q4	10630	807	746	746
q5	7638	2661	2704	2661
q6	225	134	132	132
q7	944	528	515	515
q8	9204	2063	2021	2021
q9	8583	6516	6516	6516
q10	8883	3776	3659	3659
q11	442	239	239	239
q12	462	216	213	213
q13	17754	2932	2941	2932
q14	253	223	220	220
q15	503	477	471	471
q16	517	384	380	380
q17	958	733	817	733
q18	8020	7543	7459	7459
q19	4600	1517	1518	1517
q20	641	338	311	311
q21	5083	3168	3267	3168
q22	348	278	279	278
Total cold run time: 117055 ms
Total hot run time: 39892 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4371	4242	4243	4242
q2	375	271	261	261
q3	2976	2798	2694	2694
q4	1871	1589	1541	1541
q5	5285	5254	5252	5252
q6	216	124	125	124
q7	1736	1377	1345	1345
q8	3168	3315	3310	3310
q9	8262	8344	8325	8325
q10	3834	3620	3638	3620
q11	582	474	486	474
q12	772	593	600	593
q13	17639	2987	2997	2987
q14	287	265	267	265
q15	527	468	463	463
q16	459	400	405	400
q17	1757	1475	1469	1469
q18	7546	7566	7406	7406
q19	1638	1567	1546	1546
q20	1945	1752	1785	1752
q21	4833	4933	4830	4830
q22	563	501	494	494
Total cold run time: 70642 ms
Total hot run time: 53393 ms

doris-robot · 2024-05-11T15:12:55Z

TPC-DS: Total hot run time: 186552 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fa83cce7bede1c1f53806f37981ee6ab5f4bd6b7, data reload: false

query1	918	370	341	341
query2	6445	2382	2385	2382
query3	6655	207	210	207
query4	23063	21131	21075	21075
query5	4155	421	423	421
query6	267	173	170	170
query7	4592	290	290	290
query8	258	189	187	187
query9	8738	2404	2392	2392
query10	444	250	257	250
query11	14772	14110	14330	14110
query12	133	94	91	91
query13	1649	371	366	366
query14	10595	8380	7599	7599
query15	262	171	172	171
query16	8174	257	263	257
query17	1867	563	542	542
query18	2101	273	280	273
query19	328	147	147	147
query20	90	89	88	88
query21	212	127	124	124
query22	5064	4883	4805	4805
query23	34218	33801	33493	33493
query24	11594	3005	2945	2945
query25	635	380	358	358
query26	1736	153	152	152
query27	2815	317	322	317
query28	7264	2053	2030	2030
query29	1004	603	591	591
query30	291	149	150	149
query31	950	745	728	728
query32	92	52	55	52
query33	751	249	246	246
query34	1022	485	493	485
query35	813	683	672	672
query36	1066	913	912	912
query37	175	65	65	65
query38	2881	2737	2731	2731
query39	1631	1569	1538	1538
query40	282	125	124	124
query41	41	40	39	39
query42	102	97	100	97
query43	584	515	542	515
query44	1201	714	739	714
query45	260	250	259	250
query46	1071	725	704	704
query47	1964	1902	1896	1896
query48	371	297	291	291
query49	1184	393	396	393
query50	785	386	392	386
query51	6895	6765	6696	6696
query52	104	83	90	83
query53	348	276	274	274
query54	943	431	429	429
query55	73	71	72	71
query56	238	223	217	217
query57	1244	1124	1139	1124
query58	224	194	197	194
query59	3378	3158	3130	3130
query60	263	238	238	238
query61	90	88	86	86
query62	668	476	473	473
query63	309	279	277	277
query64	9766	7366	7325	7325
query65	3146	3076	3076	3076
query66	1382	349	344	344
query67	15627	14958	15130	14958
query68	4906	538	533	533
query69	525	300	306	300
query70	1154	1097	1118	1097
query71	427	267	264	264
query72	7551	2591	2348	2348
query73	724	322	324	322
query74	6656	6142	5958	5958
query75	3630	2601	2602	2601
query76	3610	986	985	985
query77	639	276	263	263
query78	10574	10090	10076	10076
query79	1349	519	508	508
query80	1062	432	436	432
query81	493	220	220	220
query82	653	95	94	94
query83	193	163	167	163
query84	263	93	89	89
query85	1511	264	258	258
query86	403	289	305	289
query87	3334	3128	3119	3119
query88	3452	2424	2410	2410
query89	474	389	383	383
query90	2009	188	190	188
query91	121	98	97	97
query92	58	48	48	48
query93	1285	510	504	504
query94	1120	179	188	179
query95	384	304	309	304
query96	584	268	271	268
query97	3168	2993	3014	2993
query98	245	220	214	214
query99	1302	915	889	889
Total cold run time: 286122 ms
Total hot run time: 186552 ms

zhangbutao · 2024-05-12T04:29:19Z

run buildall

fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundRelation.java

zhangbutao · 2024-05-14T03:09:54Z

run buildall

github-actions · 2024-05-14T03:28:26Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-05-14T03:28:28Z

PR approved by anyone and no changes requested.

doris-robot · 2024-05-14T03:30:58Z

TPC-H: Total hot run time: 40423 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f18eb36bd4f1121530cf1b910b13b85645a36674, data reload: false

------ Round 1 ----------------------------------
q1	17675	4338	4184	4184
q2	2016	196	190	190
q3	10575	1292	1218	1218
q4	10547	760	777	760
q5	7523	2668	2632	2632
q6	211	132	132	132
q7	982	585	594	585
q8	9634	2091	2065	2065
q9	9207	6646	6597	6597
q10	9601	3717	3855	3717
q11	444	241	241	241
q12	447	214	216	214
q13	17778	2946	2956	2946
q14	269	207	230	207
q15	514	466	477	466
q16	522	386	384	384
q17	968	663	730	663
q18	8108	7325	7419	7325
q19	4221	1541	1494	1494
q20	652	305	302	302
q21	5318	3824	3867	3824
q22	349	277	290	277
Total cold run time: 117561 ms
Total hot run time: 40423 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4303	4200	4195	4195
q2	375	276	273	273
q3	2988	2761	2716	2716
q4	1823	1521	1594	1521
q5	5238	5251	5257	5251
q6	210	125	123	123
q7	2228	1867	1893	1867
q8	3158	3280	3328	3280
q9	8295	8285	8293	8285
q10	3819	3669	3650	3650
q11	580	499	494	494
q12	768	573	588	573
q13	16386	2960	2976	2960
q14	305	276	274	274
q15	516	478	471	471
q16	451	415	421	415
q17	1757	1467	1477	1467
q18	7710	7609	7331	7331
q19	1645	1593	1526	1526
q20	1953	1778	1761	1761
q21	4787	4766	4774	4766
q22	551	485	475	475
Total cold run time: 69846 ms
Total hot run time: 53674 ms

doris-robot · 2024-05-14T03:42:37Z

TPC-DS: Total hot run time: 187388 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f18eb36bd4f1121530cf1b910b13b85645a36674, data reload: false

query1	915	365	345	345
query2	6454	2395	2333	2333
query3	6638	208	208	208
query4	23075	21386	21203	21203
query5	4146	421	422	421
query6	256	172	177	172
query7	4579	296	283	283
query8	243	191	189	189
query9	8480	2491	2450	2450
query10	459	242	251	242
query11	14704	14285	14240	14240
query12	137	91	89	89
query13	1634	382	379	379
query14	10510	8194	7641	7641
query15	228	167	170	167
query16	8080	267	260	260
query17	1854	580	539	539
query18	2059	267	269	267
query19	214	155	151	151
query20	109	87	89	87
query21	202	130	134	130
query22	5027	4828	4887	4828
query23	34147	33526	33475	33475
query24	11904	2942	2943	2942
query25	624	389	362	362
query26	1695	155	153	153
query27	2987	322	336	322
query28	7441	2072	2061	2061
query29	982	626	611	611
query30	285	150	153	150
query31	975	766	742	742
query32	92	52	58	52
query33	751	249	238	238
query34	1052	517	499	499
query35	796	669	681	669
query36	1087	896	920	896
query37	133	67	67	67
query38	2890	2781	2752	2752
query39	1653	1556	1557	1556
query40	276	126	125	125
query41	41	39	39	39
query42	106	97	97	97
query43	571	549	572	549
query44	1218	733	737	733
query45	261	258	259	258
query46	1085	716	727	716
query47	1959	1893	1938	1893
query48	370	307	305	305
query49	1128	392	394	392
query50	777	392	399	392
query51	6830	6796	6854	6796
query52	107	86	92	86
query53	350	287	284	284
query54	965	431	439	431
query55	80	77	81	77
query56	254	232	265	232
query57	1268	1196	1162	1162
query58	219	203	191	191
query59	3511	3218	3616	3218
query60	263	230	239	230
query61	92	87	85	85
query62	665	471	461	461
query63	309	295	286	286
query64	9793	7396	7356	7356
query65	3154	3096	3087	3087
query66	1383	333	347	333
query67	15474	14923	14864	14864
query68	4555	535	532	532
query69	482	299	299	299
query70	1156	1138	1076	1076
query71	405	270	260	260
query72	7401	2675	2353	2353
query73	698	326	329	326
query74	6544	6200	6154	6154
query75	3344	2613	2622	2613
query76	2611	991	979	979
query77	402	269	264	264
query78	10570	10224	10001	10001
query79	2472	517	519	517
query80	1041	482	426	426
query81	508	223	222	222
query82	743	91	94	91
query83	247	164	173	164
query84	234	92	87	87
query85	1589	271	268	268
query86	521	304	310	304
query87	3312	3110	3108	3108
query88	4323	2447	2455	2447
query89	471	379	377	377
query90	1959	190	190	190
query91	127	97	98	97
query92	65	48	51	48
query93	1710	511	495	495
query94	1192	186	185	185
query95	415	303	307	303
query96	585	275	277	275
query97	3158	2970	2963	2963
query98	224	220	212	212
query99	1139	903	918	903
Total cold run time: 286209 ms
Total hot run time: 187388 ms

doris-robot · 2024-05-16T03:14:55Z

TPC-DS: Total hot run time: 186741 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f18eb36bd4f1121530cf1b910b13b85645a36674, data reload: false

query1	911	388	374	374
query2	6321	2456	2329	2329
query3	6654	218	216	216
query4	26214	21774	21931	21774
query5	3789	426	429	426
query6	280	187	171	171
query7	4542	297	296	296
query8	239	186	192	186
query9	8596	2405	2463	2405
query10	426	246	261	246
query11	15191	14946	14840	14840
query12	119	94	85	85
query13	1569	356	372	356
query14	10273	6770	7665	6770
query15	236	168	166	166
query16	8115	255	255	255
query17	1706	549	526	526
query18	2085	265	267	265
query19	194	151	142	142
query20	88	81	80	80
query21	186	130	129	129
query22	5099	4946	4918	4918
query23	33926	33701	33356	33356
query24	10680	2841	2898	2841
query25	597	360	355	355
query26	1169	152	147	147
query27	3006	310	322	310
query28	7136	2006	1995	1995
query29	853	635	607	607
query30	262	171	169	169
query31	987	739	748	739
query32	94	49	57	49
query33	734	245	238	238
query34	1069	490	472	472
query35	774	666	651	651
query36	1045	917	922	917
query37	138	70	68	68
query38	2903	2764	2835	2764
query39	1595	1567	1558	1558
query40	192	125	119	119
query41	47	42	41	41
query42	100	96	93	93
query43	548	551	527	527
query44	1186	725	732	725
query45	266	248	258	248
query46	1068	722	700	700
query47	1981	1889	1879	1879
query48	365	305	292	292
query49	895	386	381	381
query50	749	376	385	376
query51	6857	6789	6807	6789
query52	98	94	92	92
query53	346	280	278	278
query54	841	416	420	416
query55	75	72	72	72
query56	236	215	221	215
query57	1209	1178	1122	1122
query58	214	194	196	194
query59	3368	3181	2952	2952
query60	279	230	226	226
query61	86	85	88	85
query62	637	472	454	454
query63	311	280	283	280
query64	8802	7369	7396	7369
query65	3096	3076	3055	3055
query66	804	325	334	325
query67	15816	15012	14865	14865
query68	4531	517	520	517
query69	477	300	299	299
query70	1134	1085	1091	1085
query71	423	266	261	261
query72	7095	2591	2324	2324
query73	719	317	319	317
query74	6611	6195	6221	6195
query75	3415	2641	2607	2607
query76	2702	955	975	955
query77	365	261	263	261
query78	10568	10201	10017	10017
query79	2521	515	508	508
query80	1052	425	417	417
query81	579	240	241	240
query82	943	94	95	94
query83	260	164	163	163
query84	232	87	84	84
query85	1179	264	255	255
query86	473	318	295	295
query87	3352	3118	3201	3118
query88	4189	2300	2317	2300
query89	467	384	390	384
query90	1938	193	185	185
query91	123	97	107	97
query92	61	48	48	48
query93	1810	484	484	484
query94	1145	173	172	172
query95	387	293	315	293
query96	586	260	268	260
query97	3165	3066	3014	3014
query98	237	227	215	215
query99	1177	916	934	916
Total cold run time: 284189 ms
Total hot run time: 186741 ms

doris-robot · 2024-05-16T03:20:25Z

ClickBench: Total hot run time: 30.16 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f18eb36bd4f1121530cf1b910b13b85645a36674, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.24	0.04	0.05
query4	1.66	0.07	0.06
query5	0.49	0.49	0.52
query6	1.12	0.73	0.72
query7	0.01	0.02	0.01
query8	0.06	0.04	0.04
query9	0.53	0.48	0.48
query10	0.54	0.52	0.53
query11	0.16	0.12	0.11
query12	0.15	0.12	0.11
query13	0.59	0.59	0.60
query14	0.77	0.77	0.79
query15	0.82	0.80	0.80
query16	0.35	0.37	0.36
query17	0.97	0.93	0.92
query18	0.22	0.22	0.26
query19	1.79	1.69	1.70
query20	0.02	0.01	0.01
query21	15.73	0.65	0.64
query22	4.23	7.67	1.73
query23	18.31	1.45	1.29
query24	1.29	0.45	0.21
query25	0.14	0.09	0.08
query26	0.27	0.17	0.18
query27	0.08	0.08	0.08
query28	13.31	1.01	1.00
query29	13.14	3.38	3.33
query30	0.24	0.06	0.06
query31	2.86	0.39	0.37
query32	3.28	0.47	0.47
query33	2.81	2.80	2.81
query34	17.04	4.39	4.47
query35	4.48	4.48	4.47
query36	0.71	0.49	0.48
query37	0.17	0.15	0.15
query38	0.15	0.14	0.14
query39	0.04	0.04	0.03
query40	0.16	0.14	0.14
query41	0.09	0.05	0.04
query42	0.06	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.24 s
Total hot run time: 30.16 s

zhangbutao · 2024-05-16T16:31:25Z

@morrySnow Gentle ping. Can we merge this change? Or Is there anything else that needs to be done?

morrySnow · 2024-05-17T03:35:16Z

@morrySnow Gentle ping. Can we merge this change? Or Is there anything else that needs to be done?

we need 2 approve to merge pr. let's find another reviewer to review this PR

morningman · 2024-05-17T04:19:41Z

@wuwenchi PTAL

fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java

wuwenchi · 2024-05-17T06:53:33Z

fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalTable.java

@@ -169,6 +170,7 @@ public class HMSExternalTable extends ExternalTable implements MTMVRelatedTableI
    // for hudi incremental read
    private TableScanParams scanParams = null;
    private IncrementalRelation incrementalRelation = null;
+    private TableSnapshot tableSnapshot = null;


Can we get snapshot version information directly from IcebergScanNode?

This naturally supports all types of iceberg tables.

Specify that the snapshot belongs to the query, and its valid range is related to the query, so it is not appropriate to put this information in the HMSExternalTable.

Thanks for you suggesstion!

I refine this PR in 2108139.
Now we can support all types of iceberg tables, and i set the snapshot in IcebergScanNode through method PhysicalPlanTranslator::visitPhysicalFileScan .

Please help to check if this change is good for you.

fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4

zhangbutao · 2024-05-20T10:37:58Z

run buildall

zhangbutao · 2024-05-22T01:51:31Z

run buildall

doris-robot · 2024-05-22T03:14:14Z

TPC-H: Total hot run time: 42135 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 761c29f4fd17b4c6f3d87d65c447d6cfefa67621, data reload: false

------ Round 1 ----------------------------------
q1	17595	4906	4260	4260
q2	2017	193	197	193
q3	10447	1187	1260	1187
q4	10224	913	866	866
q5	7499	2705	2741	2705
q6	218	136	134	134
q7	1062	598	588	588
q8	9227	2163	2135	2135
q9	9112	6784	6661	6661
q10	9948	3846	3941	3846
q11	438	245	263	245
q12	431	228	217	217
q13	17486	3213	3259	3213
q14	256	215	209	209
q15	535	465	486	465
q16	522	411	394	394
q17	1002	689	761	689
q18	8504	7983	7888	7888
q19	4204	1542	1577	1542
q20	650	319	333	319
q21	5309	4097	4141	4097
q22	369	282	298	282
Total cold run time: 117055 ms
Total hot run time: 42135 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4543	4397	4435	4397
q2	378	276	268	268
q3	3168	2933	2755	2755
q4	1927	1607	1658	1607
q5	5529	5501	5572	5501
q6	221	124	125	124
q7	2354	1956	1964	1956
q8	3268	3443	3421	3421
q9	8705	8698	8620	8620
q10	3946	3863	3852	3852
q11	612	502	495	495
q12	818	632	616	616
q13	17417	3181	3086	3086
q14	285	272	257	257
q15	509	475	463	463
q16	470	409	413	409
q17	1778	1490	1457	1457
q18	7678	7648	7395	7395
q19	1692	1532	1605	1532
q20	2055	1814	1776	1776
q21	10752	4985	4855	4855
q22	598	503	490	490
Total cold run time: 78703 ms
Total hot run time: 55332 ms

doris-robot · 2024-05-22T03:25:24Z

TPC-DS: Total hot run time: 181302 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 761c29f4fd17b4c6f3d87d65c447d6cfefa67621, data reload: false

query1	910	397	374	374
query2	6440	2521	2392	2392
query3	6650	211	203	203
query4	22935	21234	21131	21131
query5	4107	422	426	422
query6	259	172	172	172
query7	4586	300	295	295
query8	235	191	188	188
query9	8479	2390	2380	2380
query10	452	251	258	251
query11	14855	14201	14072	14072
query12	135	94	88	88
query13	1635	390	359	359
query14	10454	8648	7741	7741
query15	248	172	165	165
query16	8071	271	251	251
query17	1718	541	542	541
query18	2100	281	265	265
query19	189	144	150	144
query20	95	86	86	86
query21	187	129	126	126
query22	5062	4847	4765	4765
query23	34198	33565	33268	33268
query24	6699	2779	2858	2779
query25	541	375	359	359
query26	700	154	156	154
query27	1887	311	330	311
query28	3619	2061	2048	2048
query29	879	610	595	595
query30	256	182	173	173
query31	978	775	727	727
query32	91	52	53	52
query33	485	239	246	239
query34	871	479	492	479
query35	750	678	659	659
query36	1062	908	899	899
query37	109	70	73	70
query38	2906	2773	2775	2773
query39	1622	1531	1618	1531
query40	211	130	133	130
query41	46	44	45	44
query42	104	96	96	96
query43	558	551	550	550
query44	1082	728	753	728
query45	263	266	237	237
query46	1060	730	712	712
query47	1973	1862	1872	1862
query48	391	304	288	288
query49	786	393	406	393
query50	773	391	387	387
query51	6835	6691	6753	6691
query52	105	91	91	91
query53	346	293	285	285
query54	526	429	430	429
query55	75	73	73	73
query56	238	219	224	219
query57	1230	1143	1147	1143
query58	220	199	199	199
query59	3523	3190	3225	3190
query60	255	237	239	237
query61	90	129	88	88
query62	614	457	467	457
query63	308	277	285	277
query64	8453	2245	1743	1743
query65	3204	3082	3147	3082
query66	789	333	340	333
query67	15485	15202	15279	15202
query68	4566	523	535	523
query69	481	313	312	312
query70	1161	1133	1166	1133
query71	390	282	274	274
query72	7887	2605	2350	2350
query73	713	316	316	316
query74	6704	6190	6215	6190
query75	3318	2630	2604	2604
query76	2328	938	1007	938
query77	396	268	268	268
query78	10567	10172	10107	10107
query79	1670	508	532	508
query80	966	433	444	433
query81	526	249	243	243
query82	961	102	101	101
query83	254	171	172	171
query84	256	89	94	89
query85	1049	334	262	262
query86	465	309	297	297
query87	3283	3065	3080	3065
query88	3061	2324	2333	2324
query89	475	379	388	379
query90	1976	201	193	193
query91	126	97	100	97
query92	58	50	48	48
query93	1470	505	495	495
query94	1094	189	184	184
query95	400	306	304	304
query96	577	263	269	263
query97	3178	3023	3054	3023
query98	240	229	217	217
query99	1230	898	937	898
Total cold run time: 268824 ms
Total hot run time: 181302 ms

doris-robot · 2024-05-22T03:30:55Z

ClickBench: Total hot run time: 30.89 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 761c29f4fd17b4c6f3d87d65c447d6cfefa67621, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.24	0.06	0.05
query4	1.66	0.08	0.08
query5	0.50	0.50	0.51
query6	1.12	0.72	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.55	0.49	0.49
query10	0.54	0.55	0.53
query11	0.15	0.12	0.11
query12	0.16	0.12	0.11
query13	0.60	0.60	0.60
query14	0.78	0.78	0.77
query15	0.85	0.82	0.81
query16	0.38	0.36	0.37
query17	1.01	0.99	0.99
query18	0.21	0.26	0.21
query19	1.80	1.67	1.68
query20	0.02	0.01	0.01
query21	15.73	0.65	0.64
query22	4.50	6.32	2.25
query23	18.27	1.43	1.35
query24	1.77	0.23	0.26
query25	0.15	0.08	0.08
query26	0.26	0.17	0.17
query27	0.08	0.08	0.08
query28	13.33	1.03	0.99
query29	13.76	3.27	3.22
query30	0.24	0.07	0.06
query31	2.83	0.41	0.39
query32	3.28	0.46	0.47
query33	2.91	2.95	2.95
query34	17.24	4.35	4.39
query35	4.47	4.50	4.63
query36	0.66	0.49	0.45
query37	0.17	0.17	0.15
query38	0.15	0.15	0.14
query39	0.04	0.04	0.04
query40	0.16	0.15	0.14
query41	0.09	0.05	0.04
query42	0.05	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 110.94 s
Total hot run time: 30.89 s

wuwenchi · 2024-05-23T03:42:55Z

LGTM 👍

github-actions · 2024-05-23T04:27:26Z

PR approved by at least one committer and no changes requested.

#15418 added Iceberg time travel in legacy parser but not added this syntax Neredis. If we enable nereids and disable fallback to original palnner, time travel won't be available. This PR added time travel syntas in Neredis. BTW, we already have nereids time travel regression-test in https://github.com/apache/doris/blob/master/regression-test/suites/external_table_p2/iceberg/test_external_catalog_icebergv2_nereids.groovy this regression-test will always fail without this PR. https://github.com/apache/doris/blob/88530bf9437e60190f198252ab82f53fa53d4c10/regression-test/suites/external_table_p2/iceberg/test_external_catalog_icebergv2_nereids.groovy#L70-L75

apache#15418 added Iceberg time travel in legacy parser but not added this syntax Neredis. If we enable nereids and disable fallback to original palnner, time travel won't be available. This PR added time travel syntas in Neredis. BTW, we already have nereids time travel regression-test in https://github.com/apache/doris/blob/master/regression-test/suites/external_table_p2/iceberg/test_external_catalog_icebergv2_nereids.groovy this regression-test will always fail without this PR. https://github.com/apache/doris/blob/88530bf9437e60190f198252ab82f53fa53d4c10/regression-test/suites/external_table_p2/iceberg/test_external_catalog_icebergv2_nereids.groovy#L70-L75

backport: #34681 Co-authored-by: Butao Zhang <[email protected]>

…36047) Related: #34681 #34681 supports the time function, but supports the type `HMSExternalTable`. This PR add support for `IcebergExternalTable` type.

morrySnow reviewed May 11, 2024

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundRelation.java Outdated Show resolved Hide resolved

fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4 Outdated Show resolved Hide resolved

zhangbutao force-pushed the nereids_timetravel_syntaz branch from 87196e8 to fa83cce Compare May 11, 2024 14:35

zhangbutao force-pushed the nereids_timetravel_syntaz branch from fa83cce to 3401f61 Compare May 12, 2024 04:23

zhangbutao requested a review from morrySnow May 13, 2024 08:24

morrySnow reviewed May 14, 2024

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundRelation.java Outdated Show resolved Hide resolved

zhangbutao force-pushed the nereids_timetravel_syntaz branch from 3401f61 to f18eb36 Compare May 14, 2024 02:51

morrySnow previously approved these changes May 14, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label May 14, 2024

github-actions bot added the reviewed label May 14, 2024

morrySnow requested a review from morningman May 17, 2024 03:35

wuwenchi reviewed May 17, 2024

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java Outdated Show resolved Hide resolved

wuwenchi reviewed May 17, 2024

View reviewed changes

morrySnow self-requested a review May 17, 2024 08:09

zhangbutao force-pushed the nereids_timetravel_syntaz branch from f18eb36 to 2108139 Compare May 20, 2024 10:29

github-actions bot removed the approved Indicates a PR has been approved by one committer. label May 20, 2024

zhangbutao requested a review from wuwenchi May 20, 2024 10:36

zhangbutao force-pushed the nereids_timetravel_syntaz branch from 2108139 to 1907600 Compare May 22, 2024 01:37

zhangbutao added 3 commits May 22, 2024 09:47

[feat](nereids) support Iceberg time travel syntax

2b03c27

optimize antlr expr

675fccd

Get_snapshot_from_IcebergScanNode

761c29f

zhangbutao force-pushed the nereids_timetravel_syntaz branch from 1907600 to 761c29f Compare May 22, 2024 01:47

wuwenchi approved these changes May 23, 2024

View reviewed changes

morrySnow approved these changes May 23, 2024

View reviewed changes

morrySnow added the dev/2.1.x label May 23, 2024

github-actions bot added the approved Indicates a PR has been approved by one committer. label May 23, 2024

morrySnow merged commit 5bbc149 into apache:master May 23, 2024
27 of 28 checks passed

AshinGau mentioned this pull request Jun 3, 2024

[feat](nereids) support Iceberg time travel syntax #35812

Merged

morningman pushed a commit that referenced this pull request Jun 3, 2024

[feat](nereids) support Iceberg time travel syntax (#35812)

940e26f

backport: #34681 Co-authored-by: Butao Zhang <[email protected]>

morningman added dev/2.1.4-merged and removed dev/2.1.x labels Jun 3, 2024

wuwenchi mentioned this pull request Jun 7, 2024

[bugfix](iceberg)Fixed missing type of iceberg table for timetravel #36047

Merged

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feat](nereids) support Iceberg time travel syntax #34681

[feat](nereids) support Iceberg time travel syntax #34681

zhangbutao commented May 11, 2024 •

edited

Loading

doris-robot commented May 11, 2024

zhangbutao commented May 11, 2024

zhangbutao commented May 11, 2024

zhangbutao commented May 11, 2024

doris-robot commented May 11, 2024

doris-robot commented May 11, 2024

zhangbutao commented May 12, 2024

zhangbutao commented May 14, 2024

github-actions bot commented May 14, 2024

github-actions bot commented May 14, 2024

doris-robot commented May 14, 2024

doris-robot commented May 14, 2024

doris-robot commented May 16, 2024

doris-robot commented May 16, 2024

zhangbutao commented May 16, 2024

morrySnow commented May 17, 2024

morningman commented May 17, 2024

wuwenchi May 17, 2024

zhangbutao May 20, 2024

zhangbutao commented May 20, 2024

zhangbutao commented May 22, 2024

doris-robot commented May 22, 2024

doris-robot commented May 22, 2024

doris-robot commented May 22, 2024

wuwenchi commented May 23, 2024

github-actions bot commented May 23, 2024

	// test for 'FOR TIME AS OF' and 'FOR VERSION AS OF'
	def q04 = {
	qt_q15 """ select count(*) from ${hms_catalog_name}.tpch_1000_icebergv2.customer_small FOR TIME AS OF '2022-12-22 02:29:30' """
	qt_q16 """ select count(*) from ${hms_catalog_name}.tpch_1000_icebergv2.customer_small FOR VERSION AS OF 6113938156088124425 """
	qt_q17 """ select count(*) from ${iceberg_catalog_name}.tpch_1000_icebergv2.customer_small FOR TIME AS OF '2022-12-22 02:29:30' """
	qt_q18 """ select count(*) from ${iceberg_catalog_name}.tpch_1000_icebergv2.customer_small FOR VERSION AS OF 6113938156088124425 """

[feat](nereids) support Iceberg time travel syntax #34681

[feat](nereids) support Iceberg time travel syntax #34681

Conversation

zhangbutao commented May 11, 2024 • edited Loading

Proposed changes

Further comments

doris-robot commented May 11, 2024

zhangbutao commented May 11, 2024

zhangbutao commented May 11, 2024

zhangbutao commented May 11, 2024

doris-robot commented May 11, 2024

doris-robot commented May 11, 2024

zhangbutao commented May 12, 2024

zhangbutao commented May 14, 2024

github-actions bot commented May 14, 2024

github-actions bot commented May 14, 2024

doris-robot commented May 14, 2024

doris-robot commented May 14, 2024

doris-robot commented May 16, 2024

doris-robot commented May 16, 2024

zhangbutao commented May 16, 2024

morrySnow commented May 17, 2024

morningman commented May 17, 2024

wuwenchi May 17, 2024

Choose a reason for hiding this comment

zhangbutao May 20, 2024

Choose a reason for hiding this comment

zhangbutao commented May 20, 2024

zhangbutao commented May 22, 2024

doris-robot commented May 22, 2024

doris-robot commented May 22, 2024

doris-robot commented May 22, 2024

wuwenchi commented May 23, 2024

github-actions bot commented May 23, 2024

zhangbutao commented May 11, 2024 •

edited

Loading