Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix](partial update) Fix __DORIS_SEQUENCE_COL__ is not set for newly inserted rows in partial update #40272

Merged

Conversation

bobhan1
Copy link
Contributor

@bobhan1 bobhan1 commented Sep 2, 2024

Proposed changes

1. Fix __DORIS_SEQUENCE_COL__ is not set for newly inserted rows in partial update

before:

MySQL root@127.1:d1> CREATE TABLE IF NOT EXISTS t3 (
                  ->   `k` BIGINT NOT NULL,
                  ->   `c1` int,
                  ->   `c2` datetime default current_timestamp,
                  ->  ) UNIQUE KEY(`k`)
                  ->  DISTRIBUTED BY HASH(`k`) BUCKETS 1
                  ->  PROPERTIES (
                  ->  "replication_allocation" = "tag.location.default: 1",
                  ->  "enable_unique_key_merge_on_write" = "true",
                  ->  "function_column.sequence_col" = "c2"
                  ->  );
Query OK, 0 rows affected
MySQL root@127.1:d1> set enable_insert_strict=false;
MySQL root@127.1:d1> set enable_unique_key_partial_update=true;
MySQL root@127.1:d1> insert into t3(k,c1) values(99,99);
MySQL root@127.1:d1> set show_hidden_columns=true;
MySQL root@127.1:d1> select * from t3;
+----+----+---------------------+-----------------------+-----------------------+------------------------+
| k  | c1 | c2                  | __DORIS_DELETE_SIGN__ | __DORIS_VERSION_COL__ | __DORIS_SEQUENCE_COL__ |
+----+----+---------------------+-----------------------+-----------------------+------------------------+
| 99 | 99 | 2024-09-02 11:03:09 | 0                     | 2                     | <null>                 |
+----+----+---------------------+-----------------------+-----------------------+------------------------+

after:

MySQL root@127.1:d1> CREATE TABLE IF NOT EXISTS t3 (
                  ->   `k` BIGINT NOT NULL,
                  ->   `c1` int,
                  ->   `c2` datetime default current_timestamp,
                  ->  ) UNIQUE KEY(`k`)
                  ->  DISTRIBUTED BY HASH(`k`) BUCKETS 1
                  ->  PROPERTIES (
                  ->  "replication_allocation" = "tag.location.default: 1",
                  ->  "enable_unique_key_merge_on_write" = "true",
                  ->  "function_column.sequence_col" = "c2"
                  ->  );
Query OK, 0 rows affected
MySQL root@127.1:d1> set enable_insert_strict=false;
MySQL root@127.1:d1> set enable_unique_key_partial_update=true;
MySQL root@127.1:d1> insert into t3(k,c1) values(1,10);
MySQL root@127.1:d1> set show_hidden_columns=true;
MySQL root@127.1:d1> select * from t3;
+---+----+---------------------+-----------------------+-----------------------+------------------------+
| k | c1 | c2                  | __DORIS_DELETE_SIGN__ | __DORIS_VERSION_COL__ | __DORIS_SEQUENCE_COL__ |
+---+----+---------------------+-----------------------+-----------------------+------------------------+
| 1 | 10 | 2024-09-02 16:49:50 | 0                     | 2                     | 2024-09-02 16:49:50    |
+---+----+---------------------+-----------------------+-----------------------+------------------------+

2. Fix current_timestamp() precision loss for newly inserted rows in partial update

before:

MySQL root@127.1:d1> CREATE TABLE IF NOT EXISTS t3 (
                  ->   `k` BIGINT NOT NULL,
                  ->   `c1` int,
                  ->   `c2` datetime(6) default current_timestamp(6),
                  ->  ) UNIQUE KEY(`k`)
                  ->  DISTRIBUTED BY HASH(`k`) BUCKETS 1
                  ->  PROPERTIES (
                  ->  "replication_allocation" = "tag.location.default: 1",
                  ->  "enable_unique_key_merge_on_write" = "true",
                  ->  "function_column.sequence_col" = "c2"
                  ->  );
Query OK, 0 rows affected
MySQL root@127.1:d1> set enable_unique_key_partial_update=true;
MySQL root@127.1:d1> set enable_insert_strict=false;
MySQL root@127.1:d1> insert into t3(k,c1) values(3,10);
MySQL root@127.1:d1> set show_hidden_columns=true;
MySQL root@127.1:d1> select * from t3;
+---+----+----------------------------+-----------------------+-----------------------+----------------------------+
| k | c1 | c2                         | __DORIS_DELETE_SIGN__ | __DORIS_VERSION_COL__ | __DORIS_SEQUENCE_COL__     |
+---+----+----------------------------+-----------------------+-----------------------+----------------------------+
| 3 | 10 | 2024-09-02 19:04:55        | 0                     | 2                     | <null>                     |
+---+----+----------------------------+-----------------------+-----------------------+----------------------------+

after:

MySQL root@127.1:d1> CREATE TABLE IF NOT EXISTS t3 (
                  ->   `k` BIGINT NOT NULL,
                  ->   `c1` int,
                  ->   `c2` datetime(6) default current_timestamp(6),
                  ->  ) UNIQUE KEY(`k`)
                  ->  DISTRIBUTED BY HASH(`k`) BUCKETS 1
                  ->  PROPERTIES (
                  ->  "replication_allocation" = "tag.location.default: 1",
                  ->  "enable_unique_key_merge_on_write" = "true",
                  ->  "function_column.sequence_col" = "c2"
                  ->  );
Query OK, 0 rows affected
MySQL root@127.1:d1> set enable_unique_key_partial_update=true;
MySQL root@127.1:d1> set enable_insert_strict=false;
MySQL root@127.1:d1> insert into t3(k,c1) values(3,10);
MySQL root@127.1:d1> set show_hidden_columns=true;
MySQL root@127.1:d1> select * from t3;
+---+----+----------------------------+-----------------------+-----------------------+----------------------------+
| k | c1 | c2                         | __DORIS_DELETE_SIGN__ | __DORIS_VERSION_COL__ | __DORIS_SEQUENCE_COL__     |
+---+----+----------------------------+-----------------------+-----------------------+----------------------------+
| 3 | 10 | 2024-09-02 19:04:55.464438 | 0                     | 2                     | 2024-09-02 19:04:55.464438 |
+---+----+----------------------------+-----------------------+-----------------------+----------------------------+

branch-2.1-pick: #40964
branch-2.0-pick: #40966

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@bobhan1 bobhan1 force-pushed the fix-partial-update-not-set-hidden-seq-col branch from 5c95c0b to 4d42e7a Compare September 2, 2024 11:16
@bobhan1
Copy link
Contributor Author

bobhan1 commented Sep 2, 2024

run buildall

@bobhan1 bobhan1 force-pushed the fix-partial-update-not-set-hidden-seq-col branch from 4d42e7a to e68de4d Compare September 2, 2024 11:21
Copy link
Contributor

github-actions bot commented Sep 2, 2024

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
Copy link
Contributor

github-actions bot commented Sep 2, 2024

clang-tidy review says "All clean, LGTM! 👍"

@bobhan1 bobhan1 force-pushed the fix-partial-update-not-set-hidden-seq-col branch from e68de4d to bc37aa8 Compare September 2, 2024 11:26
@bobhan1
Copy link
Contributor Author

bobhan1 commented Sep 2, 2024

run buildall

Copy link
Contributor

github-actions bot commented Sep 2, 2024

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
Copy link
Contributor

github-actions bot commented Sep 2, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 38332 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bc37aa87c8ddd62e1548c57696355b49d3773332, data reload: false

------ Round 1 ----------------------------------
q1	18061	4457	4338	4338
q2	2028	190	181	181
q3	11134	1006	1054	1006
q4	10290	697	715	697
q5	7759	2849	2783	2783
q6	229	139	138	138
q7	966	618	604	604
q8	9325	2082	2120	2082
q9	7251	6580	6530	6530
q10	7019	2253	2262	2253
q11	457	245	239	239
q12	395	223	225	223
q13	17775	3110	3089	3089
q14	275	235	236	235
q15	523	490	496	490
q16	594	525	508	508
q17	999	655	678	655
q18	7661	6892	6811	6811
q19	1416	1059	1022	1022
q20	695	329	330	329
q21	4122	3203	3125	3125
q22	1097	1022	994	994
Total cold run time: 110071 ms
Total hot run time: 38332 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4388	4364	4333	4333
q2	392	280	269	269
q3	2925	2675	2661	2661
q4	1958	1663	1666	1663
q5	5676	5714	5771	5714
q6	239	135	139	135
q7	2242	1826	1803	1803
q8	3348	3435	3447	3435
q9	8829	8809	8851	8809
q10	3590	3442	3351	3351
q11	615	512	505	505
q12	856	659	669	659
q13	13913	3227	3207	3207
q14	311	294	299	294
q15	541	512	486	486
q16	607	585	598	585
q17	1819	1542	1509	1509
q18	8132	7962	7798	7798
q19	1776	1506	1549	1506
q20	2140	1903	1933	1903
q21	5746	5622	5576	5576
q22	1126	1028	1029	1028
Total cold run time: 71169 ms
Total hot run time: 57229 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192200 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bc37aa87c8ddd62e1548c57696355b49d3773332, data reload: false

query1	1270	904	868	868
query2	6288	1977	1867	1867
query3	10605	4065	4023	4023
query4	59603	25360	23161	23161
query5	5250	497	498	497
query6	408	163	158	158
query7	5792	293	293	293
query8	288	209	234	209
query9	8506	2499	2491	2491
query10	475	277	271	271
query11	15798	15078	15058	15058
query12	154	107	107	107
query13	1503	397	382	382
query14	11221	7371	6917	6917
query15	235	181	171	171
query16	7632	456	436	436
query17	1148	578	564	564
query18	1902	286	288	286
query19	279	153	151	151
query20	120	121	111	111
query21	205	102	106	102
query22	4557	4546	4387	4387
query23	34495	33275	33206	33206
query24	5955	2853	2887	2853
query25	555	409	408	408
query26	686	183	155	155
query27	1743	277	275	275
query28	3862	2063	2049	2049
query29	670	397	403	397
query30	240	152	156	152
query31	948	758	746	746
query32	83	54	54	54
query33	481	286	280	280
query34	882	484	485	484
query35	810	731	708	708
query36	1076	922	962	922
query37	140	89	88	88
query38	4019	3838	3882	3838
query39	1474	1376	1396	1376
query40	213	111	113	111
query41	47	45	45	45
query42	115	96	97	96
query43	505	467	459	459
query44	1096	743	743	743
query45	194	161	162	161
query46	1093	744	721	721
query47	1914	1795	1823	1795
query48	371	295	303	295
query49	773	441	472	441
query50	809	419	404	404
query51	7120	6835	6800	6800
query52	101	88	89	88
query53	256	183	184	183
query54	577	482	469	469
query55	77	76	80	76
query56	301	287	260	260
query57	1234	1068	1093	1068
query58	231	240	242	240
query59	2977	2766	2740	2740
query60	311	286	279	279
query61	124	120	120	120
query62	755	673	652	652
query63	215	191	188	188
query64	2889	755	743	743
query65	3233	3154	3172	3154
query66	697	361	359	359
query67	15543	15267	15201	15201
query68	3222	592	589	589
query69	408	298	287	287
query70	1165	1142	1155	1142
query71	370	286	295	286
query72	6422	4101	4162	4101
query73	748	326	330	326
query74	9176	8847	8796	8796
query75	3340	2724	2725	2724
query76	1506	1023	1033	1023
query77	538	327	328	327
query78	9786	9018	9013	9013
query79	2283	535	535	535
query80	1011	500	504	500
query81	571	238	231	231
query82	252	144	138	138
query83	183	146	144	144
query84	261	76	76	76
query85	929	286	306	286
query86	446	291	301	291
query87	4535	4257	4257	4257
query88	4175	2300	2283	2283
query89	393	288	287	287
query90	1846	184	185	184
query91	122	99	101	99
query92	64	49	48	48
query93	2374	553	556	553
query94	880	292	307	292
query95	347	253	255	253
query96	587	265	260	260
query97	3213	3036	3085	3036
query98	214	211	196	196
query99	1564	1290	1245	1245
Total cold run time: 308060 ms
Total hot run time: 192200 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.06 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bc37aa87c8ddd62e1548c57696355b49d3773332, data reload: false

query1	0.05	0.05	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.08	0.07
query5	0.53	0.51	0.49
query6	1.12	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.56	0.49	0.49
query10	0.56	0.55	0.54
query11	0.16	0.12	0.11
query12	0.15	0.13	0.12
query13	0.61	0.59	0.58
query14	2.12	2.09	2.11
query15	0.85	0.81	0.83
query16	0.36	0.37	0.35
query17	1.04	1.01	1.05
query18	0.22	0.20	0.20
query19	1.94	1.85	1.87
query20	0.01	0.01	0.01
query21	15.40	0.66	0.67
query22	3.78	8.07	1.76
query23	18.28	1.42	1.32
query24	2.11	0.22	0.21
query25	0.15	0.08	0.08
query26	0.25	0.18	0.18
query27	0.07	0.08	0.08
query28	13.28	1.02	1.00
query29	12.61	3.37	3.40
query30	0.24	0.05	0.05
query31	2.90	0.41	0.39
query32	3.23	0.48	0.48
query33	2.97	2.97	2.98
query34	17.17	4.41	4.42
query35	4.43	4.43	4.41
query36	0.65	0.46	0.49
query37	0.19	0.17	0.16
query38	0.15	0.14	0.15
query39	0.04	0.04	0.04
query40	0.16	0.12	0.13
query41	0.09	0.06	0.05
query42	0.06	0.05	0.06
query43	0.05	0.04	0.04
Total cold run time: 110.6 s
Total hot run time: 32.06 s

@bobhan1 bobhan1 force-pushed the fix-partial-update-not-set-hidden-seq-col branch from bc37aa8 to 2ca8923 Compare September 2, 2024 16:16
Copy link
Contributor

github-actions bot commented Sep 2, 2024

clang-tidy review says "All clean, LGTM! 👍"

@bobhan1
Copy link
Contributor Author

bobhan1 commented Sep 2, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37998 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2ca8923af2c7e5fafcc38a255006406f1fd3cff9, data reload: false

------ Round 1 ----------------------------------
q1	18749	6990	4356	4356
q2	2015	186	192	186
q3	11665	939	1171	939
q4	10457	735	801	735
q5	7747	2901	2877	2877
q6	230	142	149	142
q7	986	610	602	602
q8	9327	2077	2098	2077
q9	7281	6531	6500	6500
q10	6994	2266	2222	2222
q11	451	239	240	239
q12	393	218	216	216
q13	17756	3099	3069	3069
q14	272	244	233	233
q15	527	495	479	479
q16	571	503	514	503
q17	975	682	717	682
q18	7544	6730	6884	6730
q19	1399	1017	1013	1013
q20	685	330	337	330
q21	3932	2977	2844	2844
q22	1102	1024	1033	1024
Total cold run time: 111058 ms
Total hot run time: 37998 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4392	4285	4264	4264
q2	388	272	268	268
q3	2935	2693	2676	2676
q4	1936	1664	1750	1664
q5	5757	5651	5704	5651
q6	226	132	133	132
q7	2245	1780	1802	1780
q8	3315	3508	3471	3471
q9	8882	8841	8943	8841
q10	3663	3423	3313	3313
q11	605	498	528	498
q12	873	681	734	681
q13	14543	3161	3281	3161
q14	327	297	283	283
q15	535	499	478	478
q16	637	614	556	556
q17	1862	1558	1523	1523
q18	8242	7805	7811	7805
q19	1720	1462	1555	1462
q20	2159	1892	1910	1892
q21	5769	5520	5363	5363
q22	1130	1034	1065	1034
Total cold run time: 72141 ms
Total hot run time: 56796 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192237 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2ca8923af2c7e5fafcc38a255006406f1fd3cff9, data reload: false

query1	1249	900	877	877
query2	6356	1941	1890	1890
query3	10645	3848	3928	3848
query4	58724	25041	23142	23142
query5	5534	500	498	498
query6	435	156	158	156
query7	5762	288	285	285
query8	292	210	219	210
query9	8797	2482	2472	2472
query10	508	285	268	268
query11	17943	15092	15215	15092
query12	158	106	107	106
query13	1551	411	384	384
query14	11139	7661	7631	7631
query15	218	165	191	165
query16	7135	460	431	431
query17	1186	576	560	560
query18	1860	292	302	292
query19	285	145	142	142
query20	126	112	112	112
query21	215	106	104	104
query22	4465	4516	4400	4400
query23	34214	34404	33126	33126
query24	5972	2966	2855	2855
query25	478	390	380	380
query26	668	151	149	149
query27	1767	273	274	273
query28	3843	2027	2029	2027
query29	634	399	402	399
query30	236	157	163	157
query31	949	741	782	741
query32	86	51	58	51
query33	420	282	279	279
query34	872	466	475	466
query35	841	744	721	721
query36	1091	943	959	943
query37	147	90	95	90
query38	3970	3952	3935	3935
query39	1453	1415	1413	1413
query40	197	117	116	116
query41	55	48	46	46
query42	122	100	97	97
query43	519	490	464	464
query44	1070	740	742	740
query45	196	166	170	166
query46	1095	778	734	734
query47	1920	1777	1796	1777
query48	365	310	298	298
query49	772	466	426	426
query50	822	409	409	409
query51	7177	6866	6841	6841
query52	98	84	86	84
query53	245	175	179	175
query54	573	444	446	444
query55	74	73	74	73
query56	270	248	259	248
query57	1175	1092	1053	1053
query58	216	223	222	222
query59	3029	2843	2751	2751
query60	289	261	287	261
query61	102	97	99	97
query62	754	656	640	640
query63	242	185	180	180
query64	2796	716	659	659
query65	3223	3161	3148	3148
query66	675	327	383	327
query67	15363	15259	15416	15259
query68	2930	581	582	581
query69	402	278	289	278
query70	1117	1123	1119	1119
query71	347	311	269	269
query72	5787	3958	4101	3958
query73	742	328	330	328
query74	9169	8745	8862	8745
query75	3393	2683	2659	2659
query76	1659	1035	974	974
query77	534	324	310	310
query78	9808	9055	10526	9055
query79	1034	523	527	523
query80	683	494	508	494
query81	460	235	233	233
query82	243	135	146	135
query83	166	148	155	148
query84	259	72	77	72
query85	686	282	276	276
query86	308	290	285	285
query87	4445	4350	4298	4298
query88	3052	2293	2276	2276
query89	378	289	284	284
query90	1960	183	186	183
query91	121	175	100	100
query92	55	48	49	48
query93	1022	530	530	530
query94	693	288	300	288
query95	343	244	254	244
query96	580	257	259	257
query97	3179	3052	3076	3052
query98	214	200	198	198
query99	1500	1232	1255	1232
Total cold run time: 302927 ms
Total hot run time: 192237 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.21 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2ca8923af2c7e5fafcc38a255006406f1fd3cff9, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.09	0.09
query5	0.50	0.51	0.48
query6	1.13	0.73	0.73
query7	0.02	0.02	0.01
query8	0.06	0.04	0.04
query9	0.55	0.50	0.49
query10	0.54	0.54	0.54
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.61	0.60	0.59
query14	2.04	2.12	2.11
query15	0.85	0.83	0.83
query16	0.37	0.39	0.39
query17	1.02	1.04	1.01
query18	0.21	0.20	0.20
query19	1.98	1.78	1.75
query20	0.02	0.01	0.01
query21	15.39	0.66	0.65
query22	4.28	5.65	2.99
query23	18.24	1.37	1.28
query24	2.10	0.23	0.22
query25	0.17	0.07	0.08
query26	0.28	0.18	0.18
query27	0.08	0.08	0.07
query28	13.24	1.02	1.00
query29	12.64	3.36	3.39
query30	0.24	0.05	0.06
query31	2.88	0.41	0.39
query32	3.24	0.49	0.48
query33	3.00	3.02	2.98
query34	17.09	4.42	4.38
query35	4.48	4.45	4.42
query36	0.65	0.47	0.50
query37	0.18	0.16	0.15
query38	0.16	0.15	0.15
query39	0.05	0.03	0.04
query40	0.16	0.13	0.12
query41	0.10	0.05	0.05
query42	0.05	0.04	0.05
query43	0.04	0.04	0.05
Total cold run time: 110.96 s
Total hot run time: 33.21 s

Copy link
Contributor

github-actions bot commented Sep 3, 2024

clang-tidy review says "All clean, LGTM! 👍"

@bobhan1
Copy link
Contributor Author

bobhan1 commented Sep 3, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38168 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a091bc750324d3ef7325a2694210158dbe8947e7, data reload: false

------ Round 1 ----------------------------------
q1	18160	6834	4419	4419
q2	2650	181	177	177
q3	11489	1117	1122	1117
q4	11387	786	753	753
q5	7786	2924	2869	2869
q6	233	141	140	140
q7	962	631	604	604
q8	9325	2064	2056	2056
q9	7160	6523	6531	6523
q10	6973	2240	2151	2151
q11	488	241	247	241
q12	394	221	220	220
q13	17799	3097	3039	3039
q14	279	235	239	235
q15	522	499	506	499
q16	568	509	517	509
q17	977	693	704	693
q18	7449	6856	6898	6856
q19	1387	1074	1082	1074
q20	677	335	330	330
q21	3907	3081	2640	2640
q22	1124	1037	1023	1023
Total cold run time: 111696 ms
Total hot run time: 38168 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4409	4306	4277	4277
q2	380	275	264	264
q3	2921	2670	2681	2670
q4	1902	1678	1632	1632
q5	5385	5394	5380	5380
q6	222	130	132	130
q7	2067	1717	1744	1717
q8	3172	3314	3345	3314
q9	8412	8435	8400	8400
q10	3444	3192	3222	3192
q11	593	504	495	495
q12	790	616	612	612
q13	10197	3088	3072	3072
q14	311	278	268	268
q15	528	476	484	476
q16	601	565	568	565
q17	1795	1495	1469	1469
q18	7816	7376	7446	7376
q19	1650	1461	1605	1461
q20	2046	1822	1847	1822
q21	5554	5255	5161	5161
q22	1105	1018	1048	1018
Total cold run time: 65300 ms
Total hot run time: 54771 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187520 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a091bc750324d3ef7325a2694210158dbe8947e7, data reload: false

query1	921	369	367	367
query2	6473	1933	1888	1888
query3	6649	222	226	222
query4	34164	23353	23239	23239
query5	4133	507	491	491
query6	242	168	173	168
query7	4576	293	304	293
query8	248	208	201	201
query9	8426	2479	2469	2469
query10	447	260	265	260
query11	17793	14903	15218	14903
query12	148	100	98	98
query13	1635	372	375	372
query14	9727	6976	7246	6976
query15	254	170	184	170
query16	8069	444	455	444
query17	1597	587	567	567
query18	2096	297	295	295
query19	288	166	141	141
query20	113	106	108	106
query21	207	105	99	99
query22	4425	4026	4124	4026
query23	34069	33384	33915	33384
query24	11491	2886	2859	2859
query25	675	376	380	376
query26	1448	157	152	152
query27	2877	283	275	275
query28	7658	2041	2025	2025
query29	918	411	403	403
query30	309	161	152	152
query31	985	771	790	771
query32	100	56	57	56
query33	764	286	290	286
query34	933	484	494	484
query35	850	730	732	730
query36	1092	941	956	941
query37	173	94	94	94
query38	3926	3783	3814	3783
query39	1420	1427	1413	1413
query40	270	116	116	116
query41	48	46	47	46
query42	114	101	98	98
query43	510	470	465	465
query44	1268	753	739	739
query45	203	167	171	167
query46	1113	719	723	719
query47	1876	1745	1770	1745
query48	374	292	299	292
query49	1136	425	426	425
query50	815	407	411	407
query51	7023	6864	6835	6835
query52	100	88	89	88
query53	252	184	188	184
query54	975	486	450	450
query55	78	78	78	78
query56	286	264	268	264
query57	1180	1054	1053	1053
query58	223	239	242	239
query59	3008	2762	2745	2745
query60	304	272	266	266
query61	107	102	103	102
query62	787	638	629	629
query63	223	188	185	185
query64	5268	756	671	671
query65	3205	3158	3154	3154
query66	1227	348	341	341
query67	15549	15189	15171	15171
query68	3542	578	565	565
query69	392	286	271	271
query70	1126	1172	1165	1165
query71	356	284	289	284
query72	6307	4080	3983	3983
query73	750	331	333	331
query74	9115	8810	8883	8810
query75	3446	2671	2675	2671
query76	1995	996	1056	996
query77	452	333	330	330
query78	9689	9128	9170	9128
query79	1060	546	534	534
query80	693	529	519	519
query81	499	237	236	236
query82	248	142	148	142
query83	173	163	161	161
query84	228	80	78	78
query85	797	293	280	280
query86	326	293	282	282
query87	4364	4310	4322	4310
query88	3110	2296	2296	2296
query89	387	298	282	282
query90	1864	196	193	193
query91	128	105	109	105
query92	57	52	49	49
query93	1032	558	550	550
query94	663	286	289	286
query95	346	252	248	248
query96	579	261	265	261
query97	3208	3068	3069	3068
query98	208	201	197	197
query99	1457	1256	1285	1256
Total cold run time: 288438 ms
Total hot run time: 187520 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.67 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a091bc750324d3ef7325a2694210158dbe8947e7, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.06	0.06
query4	1.66	0.08	0.08
query5	0.51	0.50	0.49
query6	1.13	0.73	0.72
query7	0.01	0.01	0.01
query8	0.05	0.04	0.04
query9	0.53	0.48	0.48
query10	0.56	0.55	0.55
query11	0.15	0.11	0.12
query12	0.16	0.13	0.12
query13	0.62	0.60	0.59
query14	2.07	2.07	2.12
query15	0.90	0.83	0.81
query16	0.37	0.38	0.39
query17	1.02	0.97	1.03
query18	0.20	0.20	0.20
query19	1.95	1.83	1.74
query20	0.01	0.01	0.02
query21	15.39	0.67	0.66
query22	4.30	7.27	1.44
query23	18.22	1.36	1.29
query24	2.14	0.22	0.21
query25	0.14	0.08	0.08
query26	0.26	0.18	0.17
query27	0.07	0.07	0.08
query28	13.26	1.02	1.01
query29	12.60	3.41	3.40
query30	0.24	0.05	0.05
query31	2.88	0.42	0.40
query32	3.22	0.49	0.49
query33	2.97	3.00	3.01
query34	17.11	4.42	4.39
query35	4.51	4.43	4.41
query36	0.66	0.47	0.47
query37	0.18	0.16	0.16
query38	0.16	0.16	0.15
query39	0.05	0.04	0.04
query40	0.17	0.13	0.13
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 110.99 s
Total hot run time: 31.67 s

@bobhan1 bobhan1 force-pushed the fix-partial-update-not-set-hidden-seq-col branch from a091bc7 to afa077a Compare September 3, 2024 05:16
@bobhan1
Copy link
Contributor Author

bobhan1 commented Sep 3, 2024

run buildall

Copy link
Contributor

github-actions bot commented Sep 3, 2024

clang-tidy review says "All clean, LGTM! 👍"

@bobhan1
Copy link
Contributor Author

bobhan1 commented Sep 3, 2024

run buildall

Copy link
Contributor

github-actions bot commented Sep 3, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 38326 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f6d535be4be0ec683be95ba81bee79653dc63d47, data reload: false

------ Round 1 ----------------------------------
q1	17623	4564	4348	4348
q2	2031	192	175	175
q3	11733	991	1148	991
q4	10513	766	678	678
q5	7773	2893	2845	2845
q6	231	140	138	138
q7	981	622	596	596
q8	9339	2109	2114	2109
q9	7289	6568	6632	6568
q10	6988	2259	2204	2204
q11	482	248	238	238
q12	395	219	219	219
q13	17767	3073	3081	3073
q14	289	226	231	226
q15	517	486	490	486
q16	595	513	510	510
q17	1007	614	744	614
q18	7451	6857	7005	6857
q19	1383	1080	1104	1080
q20	677	338	332	332
q21	3948	3136	3045	3045
q22	1112	1046	994	994
Total cold run time: 110124 ms
Total hot run time: 38326 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4399	4350	4359	4350
q2	382	269	265	265
q3	2851	2690	2652	2652
q4	1932	1684	1625	1625
q5	5705	5706	5772	5706
q6	236	145	138	138
q7	2272	1841	1821	1821
q8	3295	3441	3468	3441
q9	8916	8927	8839	8839
q10	3586	3380	3389	3380
q11	621	529	503	503
q12	840	676	660	660
q13	13451	3152	3268	3152
q14	320	286	290	286
q15	536	500	476	476
q16	614	598	601	598
q17	1852	1552	1545	1545
q18	8352	7770	7983	7770
q19	1751	1745	1527	1527
q20	2211	1922	1917	1917
q21	5831	5487	5395	5395
q22	1137	1059	1054	1054
Total cold run time: 71090 ms
Total hot run time: 57100 ms

@bobhan1 bobhan1 force-pushed the fix-partial-update-not-set-hidden-seq-col branch from 80ca2a8 to a0bada5 Compare September 18, 2024 06:36
@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Sep 18, 2024
Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 18, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@bobhan1
Copy link
Contributor Author

bobhan1 commented Sep 18, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41884 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a0bada52fd9e9468b93b697084707f689a561d5a, data reload: false

------ Round 1 ----------------------------------
q1	18013	7514	7463	7463
q2	2517	172	199	172
q3	10879	1145	1198	1145
q4	10211	720	712	712
q5	7743	3157	3074	3074
q6	234	145	147	145
q7	1018	613	593	593
q8	9433	2053	2065	2053
q9	6945	6422	6412	6412
q10	7004	2327	2262	2262
q11	432	246	252	246
q12	415	208	211	208
q13	17793	2984	2982	2982
q14	239	208	213	208
q15	565	532	523	523
q16	671	628	630	628
q17	991	798	833	798
q18	7389	6798	6753	6753
q19	1392	1037	1130	1037
q20	613	285	286	285
q21	3984	3177	3248	3177
q22	1124	1008	1013	1008
Total cold run time: 109605 ms
Total hot run time: 41884 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7267	7201	7220	7201
q2	329	228	231	228
q3	2861	2807	2773	2773
q4	1923	1683	1624	1624
q5	5412	5425	5390	5390
q6	229	137	138	137
q7	2069	1707	1688	1688
q8	3176	3313	3343	3313
q9	8395	8417	8371	8371
q10	3368	3354	3339	3339
q11	571	473	465	465
q12	752	557	560	557
q13	7649	3009	3021	3009
q14	285	264	259	259
q15	559	515	512	512
q16	717	657	661	657
q17	1763	1549	1529	1529
q18	7794	7504	7268	7268
q19	1677	1639	1337	1337
q20	2045	1825	1832	1825
q21	5304	5021	5041	5021
q22	1113	1013	1014	1013
Total cold run time: 65258 ms
Total hot run time: 57516 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193940 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a0bada52fd9e9468b93b697084707f689a561d5a, data reload: false

query1	948	369	365	365
query2	6510	2079	2037	2037
query3	6717	215	223	215
query4	34092	23376	23319	23319
query5	4386	469	466	466
query6	271	173	188	173
query7	4623	296	294	294
query8	304	222	224	222
query9	9515	2642	2636	2636
query10	451	274	285	274
query11	18136	15243	15273	15243
query12	155	98	96	96
query13	1627	412	390	390
query14	10690	7247	6979	6979
query15	270	176	179	176
query16	8028	476	462	462
query17	1643	575	558	558
query18	2159	305	305	305
query19	368	148	146	146
query20	123	109	110	109
query21	218	101	102	101
query22	4565	4087	4066	4066
query23	34628	33722	33701	33701
query24	11181	2864	2865	2864
query25	651	409	412	409
query26	1429	160	163	160
query27	2828	285	284	284
query28	8358	2454	2429	2429
query29	899	420	431	420
query30	317	153	154	153
query31	1034	783	797	783
query32	100	55	55	55
query33	799	297	295	295
query34	948	494	502	494
query35	876	734	723	723
query36	1049	940	936	936
query37	159	90	86	86
query38	4150	3969	3830	3830
query39	1461	1406	1407	1406
query40	276	100	98	98
query41	52	48	51	48
query42	123	97	97	97
query43	529	498	495	495
query44	1274	842	792	792
query45	195	164	168	164
query46	1132	754	764	754
query47	1851	1826	1821	1821
query48	442	356	359	356
query49	1161	409	414	409
query50	819	410	411	410
query51	7041	6933	6978	6933
query52	101	86	89	86
query53	254	186	184	184
query54	1291	472	470	470
query55	77	82	80	80
query56	288	279	273	273
query57	1182	1104	1041	1041
query58	275	229	238	229
query59	3329	3025	2934	2934
query60	304	263	270	263
query61	106	100	108	100
query62	860	654	661	654
query63	220	186	186	186
query64	5296	645	673	645
query65	3286	3199	3178	3178
query66	1434	316	309	309
query67	15801	15593	15414	15414
query68	3083	858	840	840
query69	451	361	342	342
query70	1199	1206	1213	1206
query71	341	342	336	336
query72	5932	3426	3331	3331
query73	586	583	585	583
query74	9434	8952	8968	8952
query75	3082	2869	2898	2869
query76	1938	874	873	873
query77	390	367	369	367
query78	9346	9298	9233	9233
query79	889	894	875	875
query80	600	568	560	560
query81	455	249	252	249
query82	232	239	226	226
query83	165	175	165	165
query84	249	108	97	97
query85	686	379	354	354
query86	314	312	327	312
query87	4346	4336	4360	4336
query88	4436	4089	4049	4049
query89	373	360	361	360
query90	1442	315	314	314
query91	164	167	167	167
query92	75	72	73	72
query93	907	901	900	900
query94	559	371	365	365
query95	428	424	411	411
query96	484	491	486	486
query97	3137	3118	3136	3118
query98	226	230	223	223
query99	1401	1286	1300	1286
Total cold run time: 293015 ms
Total hot run time: 193940 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.07 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a0bada52fd9e9468b93b697084707f689a561d5a, data reload: false

query1	0.05	0.05	0.04
query2	0.07	0.03	0.02
query3	0.23	0.06	0.06
query4	1.64	0.10	0.10
query5	0.52	0.52	0.51
query6	1.14	0.73	0.71
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.56	0.51	0.50
query10	0.55	0.57	0.55
query11	0.14	0.11	0.10
query12	0.14	0.11	0.11
query13	0.60	0.59	0.58
query14	3.00	2.96	2.94
query15	0.90	0.81	0.82
query16	0.39	0.39	0.38
query17	1.02	1.06	0.97
query18	0.22	0.20	0.21
query19	1.95	1.86	1.99
query20	0.01	0.00	0.01
query21	15.35	0.59	0.58
query22	2.64	3.19	1.94
query23	17.02	0.86	0.71
query24	2.86	0.46	2.06
query25	0.24	0.05	0.06
query26	0.53	0.14	0.13
query27	0.03	0.05	0.03
query28	10.47	1.08	1.07
query29	12.52	3.23	3.17
query30	0.24	0.06	0.06
query31	2.89	0.38	0.37
query32	3.29	0.46	0.47
query33	2.95	3.02	3.02
query34	16.52	4.37	4.39
query35	4.46	4.44	4.38
query36	0.66	0.48	0.48
query37	0.08	0.05	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.14
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.33 s
Total hot run time: 32.07 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.33% (9584/25674)
Line Coverage: 28.72% (79230/275889)
Region Coverage: 28.20% (41029/145518)
Branch Coverage: 24.82% (20909/84252)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a0bada52fd9e9468b93b697084707f689a561d5a_a0bada52fd9e9468b93b697084707f689a561d5a/report/index.html

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit dda76f3 into apache:master Sep 18, 2024
24 of 29 checks passed
xiaokang pushed a commit that referenced this pull request Sep 22, 2024
yiguolei pushed a commit that referenced this pull request Sep 26, 2024
…is not set for newly inserted rows in partial update #40272" (#40964)

picks #40272
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
…ly inserted rows in partial update (#40272)

## Proposed changes

### 1. Fix `__DORIS_SEQUENCE_COL__` is not set for newly inserted rows
in partial update
before:
```sql
MySQL [email protected]:d1> CREATE TABLE IF NOT EXISTS t3 (
                  ->   `k` BIGINT NOT NULL,
                  ->   `c1` int,
                  ->   `c2` datetime default current_timestamp,
                  ->  ) UNIQUE KEY(`k`)
                  ->  DISTRIBUTED BY HASH(`k`) BUCKETS 1
                  ->  PROPERTIES (
                  ->  "replication_allocation" = "tag.location.default: 1",
                  ->  "enable_unique_key_merge_on_write" = "true",
                  ->  "function_column.sequence_col" = "c2"
                  ->  );
Query OK, 0 rows affected
MySQL [email protected]:d1> set enable_insert_strict=false;
MySQL [email protected]:d1> set enable_unique_key_partial_update=true;
MySQL [email protected]:d1> insert into t3(k,c1) values(99,99);
MySQL [email protected]:d1> set show_hidden_columns=true;
MySQL [email protected]:d1> select * from t3;
+----+----+---------------------+-----------------------+-----------------------+------------------------+
| k  | c1 | c2                  | __DORIS_DELETE_SIGN__ | __DORIS_VERSION_COL__ | __DORIS_SEQUENCE_COL__ |
+----+----+---------------------+-----------------------+-----------------------+------------------------+
| 99 | 99 | 2024-09-02 11:03:09 | 0                     | 2                     | <null>                 |
+----+----+---------------------+-----------------------+-----------------------+------------------------+
```
after:
```sql
MySQL [email protected]:d1> CREATE TABLE IF NOT EXISTS t3 (
                  ->   `k` BIGINT NOT NULL,
                  ->   `c1` int,
                  ->   `c2` datetime default current_timestamp,
                  ->  ) UNIQUE KEY(`k`)
                  ->  DISTRIBUTED BY HASH(`k`) BUCKETS 1
                  ->  PROPERTIES (
                  ->  "replication_allocation" = "tag.location.default: 1",
                  ->  "enable_unique_key_merge_on_write" = "true",
                  ->  "function_column.sequence_col" = "c2"
                  ->  );
Query OK, 0 rows affected
MySQL [email protected]:d1> set enable_insert_strict=false;
MySQL [email protected]:d1> set enable_unique_key_partial_update=true;
MySQL [email protected]:d1> insert into t3(k,c1) values(1,10);
MySQL [email protected]:d1> set show_hidden_columns=true;
MySQL [email protected]:d1> select * from t3;
+---+----+---------------------+-----------------------+-----------------------+------------------------+
| k | c1 | c2                  | __DORIS_DELETE_SIGN__ | __DORIS_VERSION_COL__ | __DORIS_SEQUENCE_COL__ |
+---+----+---------------------+-----------------------+-----------------------+------------------------+
| 1 | 10 | 2024-09-02 16:49:50 | 0                     | 2                     | 2024-09-02 16:49:50    |
+---+----+---------------------+-----------------------+-----------------------+------------------------+
```
### 2. Fix `current_timestamp()` precision loss for newly inserted rows
in partial update
before:
```sql
MySQL [email protected]:d1> CREATE TABLE IF NOT EXISTS t3 (
                  ->   `k` BIGINT NOT NULL,
                  ->   `c1` int,
                  ->   `c2` datetime(6) default current_timestamp(6),
                  ->  ) UNIQUE KEY(`k`)
                  ->  DISTRIBUTED BY HASH(`k`) BUCKETS 1
                  ->  PROPERTIES (
                  ->  "replication_allocation" = "tag.location.default: 1",
                  ->  "enable_unique_key_merge_on_write" = "true",
                  ->  "function_column.sequence_col" = "c2"
                  ->  );
Query OK, 0 rows affected
MySQL [email protected]:d1> set enable_unique_key_partial_update=true;
MySQL [email protected]:d1> set enable_insert_strict=false;
MySQL [email protected]:d1> insert into t3(k,c1) values(3,10);
MySQL [email protected]:d1> set show_hidden_columns=true;
MySQL [email protected]:d1> select * from t3;
+---+----+----------------------------+-----------------------+-----------------------+----------------------------+
| k | c1 | c2                         | __DORIS_DELETE_SIGN__ | __DORIS_VERSION_COL__ | __DORIS_SEQUENCE_COL__     |
+---+----+----------------------------+-----------------------+-----------------------+----------------------------+
| 3 | 10 | 2024-09-02 19:04:55        | 0                     | 2                     | <null>                     |
+---+----+----------------------------+-----------------------+-----------------------+----------------------------+
```
after:
```sql
MySQL [email protected]:d1> CREATE TABLE IF NOT EXISTS t3 (
                  ->   `k` BIGINT NOT NULL,
                  ->   `c1` int,
                  ->   `c2` datetime(6) default current_timestamp(6),
                  ->  ) UNIQUE KEY(`k`)
                  ->  DISTRIBUTED BY HASH(`k`) BUCKETS 1
                  ->  PROPERTIES (
                  ->  "replication_allocation" = "tag.location.default: 1",
                  ->  "enable_unique_key_merge_on_write" = "true",
                  ->  "function_column.sequence_col" = "c2"
                  ->  );
Query OK, 0 rows affected
MySQL [email protected]:d1> set enable_unique_key_partial_update=true;
MySQL [email protected]:d1> set enable_insert_strict=false;
MySQL [email protected]:d1> insert into t3(k,c1) values(3,10);
MySQL [email protected]:d1> set show_hidden_columns=true;
MySQL [email protected]:d1> select * from t3;
+---+----+----------------------------+-----------------------+-----------------------+----------------------------+
| k | c1 | c2                         | __DORIS_DELETE_SIGN__ | __DORIS_VERSION_COL__ | __DORIS_SEQUENCE_COL__     |
+---+----+----------------------------+-----------------------+-----------------------+----------------------------+
| 3 | 10 | 2024-09-02 19:04:55.464438 | 0                     | 2                     | 2024-09-02 19:04:55.464438 |
+---+----+----------------------------+-----------------------+-----------------------+----------------------------+
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.15-merged dev/2.1.7-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants