[fix](compaction) fix duplicate key in agg/mor table caused by ordered data compaction (#38224)

[luwei16]

pick master #38224

[doris-robot]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

[luwei16]

run buildall

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[doris-robot]

TPC-H: Total hot run time: 49438 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 726d807fceb156fb20431da642c9c63504cb8d4e, data reload: false

------ Round 1 ----------------------------------
q1	18119	4354	4401	4354
q2	2031	152	148	148
q3	10275	1899	1951	1899
q4	10375	1212	1291	1212
q5	8407	3902	3888	3888
q6	234	125	121	121
q7	2047	1598	1584	1584
q8	9514	2732	2687	2687
q9	11900	10245	10197	10197
q10	8656	3504	3481	3481
q11	406	243	245	243
q12	474	305	300	300
q13	18359	3933	3994	3933
q14	347	335	324	324
q15	521	462	474	462
q16	683	577	591	577
q17	1131	933	964	933
q18	7132	6923	6766	6766
q19	1796	1662	1579	1579
q20	535	298	305	298
q21	4444	4085	4016	4016
q22	525	451	436	436
Total cold run time: 117911 ms
Total hot run time: 49438 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4319	4277	4281	4277
q2	327	227	222	222
q3	4181	4114	4128	4114
q4	2724	2729	2745	2729
q5	7157	7069	7121	7069
q6	237	119	116	116
q7	3237	2797	2825	2797
q8	4377	4548	4471	4471
q9	16916	16959	16886	16886
q10	4228	4289	4268	4268
q11	728	684	660	660
q12	1025	863	840	840
q13	7834	3774	3748	3748
q14	460	433	432	432
q15	505	468	453	453
q16	733	689	670	670
q17	3711	3893	3948	3893
q18	9254	9187	9146	9146
q19	1748	1731	1658	1658
q20	2382	2130	2137	2130
q21	8410	8517	8431	8431
q22	1039	961	963	961
Total cold run time: 85532 ms
Total hot run time: 79971 ms

[doris-robot]

TeamCity be ut coverage result:
Function Coverage: 37.75% (8104/21466)
Line Coverage: 29.39% (66367/225803)
Region Coverage: 28.91% (34249/118473)
Branch Coverage: 24.80% (17602/70978)
Coverage Report: http://coverage.selectdb-in.cc/coverage/726d807fceb156fb20431da642c9c63504cb8d4e_726d807fceb156fb20431da642c9c63504cb8d4e/report/index.html

[doris-robot]

TPC-DS: Total hot run time: 203163 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 726d807fceb156fb20431da642c9c63504cb8d4e, data reload: false

query1	950	422	376	376
query2	6565	2851	2675	2675
query3	6918	208	204	204
query4	20015	18000	17863	17863
query5	19746	6469	6563	6469
query6	387	217	238	217
query7	4847	308	315	308
query8	401	426	391	391
query9	3064	2640	2586	2586
query10	435	307	312	307
query11	11310	10629	10702	10629
query12	129	85	73	73
query13	5611	678	692	678
query14	18756	13320	13392	13320
query15	360	246	253	246
query16	6442	273	258	258
query17	1467	1466	874	874
query18	2264	426	402	402
query19	211	152	158	152
query20	77	81	79	79
query21	199	95	103	95
query22	5060	5015	5142	5015
query23	32589	31925	31710	31710
query24	6794	6544	6504	6504
query25	514	437	431	431
query26	531	161	160	160
query27	1847	295	294	294
query28	6218	2343	2309	2309
query29	2796	2703	2713	2703
query30	241	167	167	167
query31	911	728	726	726
query32	66	64	60	60
query33	405	253	253	253
query34	847	471	472	471
query35	1130	956	912	912
query36	1366	1329	1072	1072
query37	86	63	63	63
query38	3043	2920	2918	2918
query39	1361	1317	1306	1306
query40	208	96	98	96
query41	48	44	44	44
query42	84	83	83	83
query43	675	788	719	719
query44	1126	704	704	704
query45	250	233	235	233
query46	1214	955	976	955
query47	2048	1901	1715	1715
query48	1007	694	702	694
query49	619	359	371	359
query50	859	634	611	611
query51	4743	4645	4690	4645
query52	92	87	96	87
query53	446	326	324	324
query54	2648	2486	2451	2451
query55	95	91	83	83
query56	237	214	212	212
query57	1182	1143	1010	1010
query58	205	211	216	211
query59	4104	4104	4166	4104
query60	232	210	194	194
query61	96	93	98	93
query62	836	443	485	443
query63	500	339	351	339
query64	2590	1554	1393	1393
query65	3626	3529	3559	3529
query66	792	391	388	388
query67	15746	15429	16706	15429
query68	9190	650	655	650
query69	574	346	339	339
query70	1717	1401	1335	1335
query71	423	319	313	313
query72	6575	3476	3470	3470
query73	748	324	314	314
query74	6295	5853	5794	5794
query75	5321	3661	3620	3620
query76	5565	1174	1226	1174
query77	963	250	251	250
query78	12664	11835	17069	11835
query79	5878	631	617	617
query80	916	414	413	413
query81	488	240	234	234
query82	305	105	101	101
query83	178	132	129	129
query84	256	72	71	71
query85	830	332	331	331
query86	334	336	296	296
query87	3262	3030	3023	3023
query88	4023	2295	2285	2285
query89	373	295	283	283
query90	1912	217	212	212
query91	182	153	137	137
query92	64	56	56	56
query93	995	588	561	561
query94	677	204	206	204
query95	1109	1056	1034	1034
query96	632	319	319	319
query97	6443	6339	6352	6339
query98	193	173	171	171
query99	2905	829	864	829
Total cold run time: 304257 ms
Total hot run time: 203163 ms

[doris-robot]

ClickBench: Total hot run time: 31.08 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 726d807fceb156fb20431da642c9c63504cb8d4e, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.03	0.02
query3	0.24	0.05	0.05
query4	1.77	0.08	0.07
query5	0.53	0.53	0.51
query6	1.22	0.62	0.61
query7	0.01	0.01	0.01
query8	0.03	0.02	0.02
query9	0.54	0.50	0.48
query10	0.52	0.54	0.54
query11	0.12	0.09	0.09
query12	0.11	0.09	0.10
query13	0.62	0.63	0.60
query14	0.77	0.79	0.78
query15	0.78	0.75	0.76
query16	0.35	0.35	0.37
query17	0.98	0.99	1.00
query18	0.25	0.26	0.23
query19	1.85	1.81	1.84
query20	0.02	0.01	0.01
query21	15.51	0.56	0.56
query22	2.10	2.14	1.70
query23	17.47	0.97	1.15
query24	6.88	1.17	1.51
query25	0.37	0.13	0.06
query26	0.67	0.17	0.16
query27	0.05	0.04	0.03
query28	5.79	0.74	0.74
query29	12.63	2.34	2.32
query30	0.62	0.53	0.53
query31	2.81	0.39	0.37
query32	3.37	0.50	0.49
query33	3.10	3.10	3.08
query34	15.25	4.77	4.79
query35	4.83	4.85	4.82
query36	1.04	1.01	1.02
query37	0.06	0.05	0.05
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.17	0.15	0.14
query41	0.06	0.01	0.01
query42	0.02	0.02	0.02
query43	0.02	0.02	0.02
Total cold run time: 103.67 s
Total hot run time: 31.08 s

[doris-robot]

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 726d807fceb156fb20431da642c9c63504cb8d4e with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.0 seconds inserted 10000000 Rows, about 476K ops/s