[feature](merge-cloud) Add tablet rebalance run in cloud

[deardeng]

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

[doris-robot]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

[gavinchou]

LGTM

[github-actions]

PR approved by anyone and no changes requested.

[dataroaring]

run buildall

[dataroaring]

LGTM

[github-actions]

PR approved by at least one committer and no changes requested.

[doris-robot]

TPC-H: Total hot run time: 38667 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d7a152f6ead40f7701c9e25ed9342cc3770626d7, data reload: false

------ Round 1 ----------------------------------
q1	17604	4152	4104	4104
q2	2094	195	193	193
q3	10463	1231	1393	1231
q4	10206	839	990	839
q5	7456	2946	2890	2890
q6	218	132	130	130
q7	1091	627	612	612
q8	9392	2008	2017	2008
q9	6538	6177	6125	6125
q10	8457	3500	3510	3500
q11	419	247	239	239
q12	387	226	213	213
q13	17788	2917	2903	2903
q14	280	241	244	241
q15	533	495	473	473
q16	502	390	383	383
q17	947	911	911	911
q18	7137	6457	6549	6457
q19	1564	1523	1519	1519
q20	593	316	319	316
q21	3529	3087	3114	3087
q22	363	293	309	293
Total cold run time: 107561 ms
Total hot run time: 38667 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4078	4049	4050	4049
q2	335	228	229	228
q3	2941	2920	2938	2920
q4	1858	1859	1801	1801
q5	5197	5173	5204	5173
q6	212	123	122	122
q7	2231	1803	1818	1803
q8	3195	3270	3240	3240
q9	8480	8437	8423	8423
q10	3745	3931	4020	3931
q11	564	484	466	466
q12	754	625	582	582
q13	16775	3068	3067	3067
q14	309	275	267	267
q15	543	490	517	490
q16	519	478	447	447
q17	1784	1767	1759	1759
q18	8332	7904	7682	7682
q19	1744	1703	1676	1676
q20	2015	1870	1773	1773
q21	5160	5038	4969	4969
q22	503	434	455	434
Total cold run time: 71274 ms
Total hot run time: 55302 ms

[doris-robot]

TPC-DS: Total hot run time: 182474 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d7a152f6ead40f7701c9e25ed9342cc3770626d7, data reload: false

query1	1220	1120	372	372
query2	6385	2038	1854	1854
query3	6692	232	218	218
query4	25442	21624	22245	21624
query5	4156	436	424	424
query6	283	190	225	190
query7	4678	336	322	322
query8	245	187	188	187
query9	9032	2210	2242	2210
query10	522	254	304	254
query11	15424	14956	14524	14524
query12	149	98	101	98
query13	1648	384	388	384
query14	8995	7217	7056	7056
query15	215	180	185	180
query16	7775	288	282	282
query17	1330	610	601	601
query18	1968	293	298	293
query19	212	165	163	163
query20	99	94	95	94
query21	203	133	143	133
query22	5057	4822	4845	4822
query23	33389	32693	32879	32693
query24	8607	3316	3201	3201
query25	637	428	446	428
query26	727	182	179	179
query27	2807	391	386	386
query28	5014	1918	1874	1874
query29	1008	629	629	629
query30	299	164	163	163
query31	1020	780	764	764
query32	101	69	68	68
query33	483	268	262	262
query34	907	523	536	523
query35	859	740	734	734
query36	1005	858	883	858
query37	119	90	82	82
query38	3708	3543	3574	3543
query39	1662	1612	1619	1612
query40	210	141	148	141
query41	49	47	47	47
query42	126	111	114	111
query43	455	408	407	407
query44	1079	724	742	724
query45	293	296	269	269
query46	1097	826	788	788
query47	2025	1910	1922	1910
query48	394	322	329	322
query49	868	389	373	373
query50	829	407	407	407
query51	6816	6779	6892	6779
query52	117	102	101	101
query53	378	305	310	305
query54	264	241	234	234
query55	92	93	89	89
query56	240	238	245	238
query57	1282	1188	1190	1188
query58	256	231	228	228
query59	2610	2425	2379	2379
query60	266	232	241	232
query61	97	113	93	93
query62	625	459	474	459
query63	313	289	283	283
query64	4450	3383	3316	3316
query65	3057	3040	3023	3023
query66	756	336	333	333
query67	15478	14779	14906	14779
query68	9618	568	594	568
query69	582	349	322	322
query70	1392	1088	1105	1088
query71	512	274	291	274
query72	6707	2602	2458	2458
query73	1533	323	331	323
query74	6634	6316	6257	6257
query75	3614	2302	2316	2302
query76	5397	1110	1196	1110
query77	578	254	254	254
query78	10917	10252	10117	10117
query79	8780	535	541	535
query80	1355	447	437	437
query81	493	228	225	225
query82	262	111	100	100
query83	214	166	167	166
query84	269	92	90	90
query85	868	294	294	294
query86	347	296	284	284
query87	3684	3535	3535	3535
query88	3878	2333	2320	2320
query89	552	380	372	372
query90	1937	181	180	180
query91	143	109	108	108
query92	59	53	54	53
query93	6369	540	519	519
query94	1197	198	196	196
query95	440	342	333	333
query96	621	271	270	270
query97	2681	2486	2486	2486
query98	230	216	244	216
query99	1207	847	840	840
Total cold run time: 291674 ms
Total hot run time: 182474 ms

[doris-robot]

ClickBench: Total hot run time: 30.63 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d7a152f6ead40f7701c9e25ed9342cc3770626d7, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.05	0.04
query3	0.24	0.05	0.05
query4	1.68	0.06	0.07
query5	0.48	0.48	0.49
query6	1.15	0.66	0.65
query7	0.02	0.01	0.01
query8	0.06	0.05	0.04
query9	0.55	0.52	0.52
query10	0.58	0.56	0.57
query11	0.15	0.12	0.11
query12	0.14	0.11	0.11
query13	0.60	0.60	0.60
query14	0.76	0.80	0.78
query15	0.86	0.83	0.83
query16	0.36	0.36	0.36
query17	0.99	1.01	0.97
query18	0.26	0.24	0.24
query19	1.81	1.74	1.77
query20	0.01	0.01	0.01
query21	15.52	0.73	0.73
query22	3.12	5.12	2.18
query23	17.61	1.33	1.08
query24	1.43	0.22	0.36
query25	0.13	0.09	0.09
query26	0.29	0.18	0.17
query27	0.09	0.09	0.08
query28	13.52	0.96	0.96
query29	12.55	3.37	3.39
query30	0.27	0.10	0.09
query31	2.79	0.40	0.39
query32	3.27	0.48	0.47
query33	2.85	2.79	2.85
query34	15.54	4.34	4.34
query35	4.36	4.36	4.36
query36	0.67	0.47	0.48
query37	0.20	0.18	0.16
query38	0.19	0.16	0.17
query39	0.05	0.04	0.04
query40	0.19	0.15	0.15
query41	0.09	0.05	0.05
query42	0.06	0.05	0.06
query43	0.05	0.04	0.05
Total cold run time: 105.65 s
Total hot run time: 30.63 s

[doris-robot]

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit d7a152f6ead40f7701c9e25ed9342cc3770626d7 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       16.8 seconds inserted 10000000 Rows, about 595K ops/s

[deardeng]

run fe_ut

[deardeng]

run buildall

[doris-robot]

TPC-H: Total hot run time: 38925 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d7a152f6ead40f7701c9e25ed9342cc3770626d7, data reload: false

------ Round 1 ----------------------------------
q1	17653	4226	4076	4076
q2	2096	200	184	184
q3	10468	1229	1420	1229
q4	10198	895	1014	895
q5	7463	2973	2950	2950
q6	215	132	136	132
q7	1109	641	624	624
q8	9416	2093	2055	2055
q9	6702	6220	6184	6184
q10	8492	3539	3501	3501
q11	425	259	240	240
q12	410	222	208	208
q13	17774	2916	2928	2916
q14	277	247	246	246
q15	524	492	477	477
q16	503	389	378	378
q17	977	947	907	907
q18	7446	6649	6489	6489
q19	1625	1540	1546	1540
q20	606	315	313	313
q21	3546	3163	3071	3071
q22	374	312	310	310
Total cold run time: 108299 ms
Total hot run time: 38925 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4111	4081	4052	4052
q2	336	238	224	224
q3	2955	2967	2949	2949
q4	1940	1879	1857	1857
q5	5282	5271	5258	5258
q6	210	126	125	125
q7	2296	1825	1839	1825
q8	3258	3285	3288	3285
q9	8511	8529	8524	8524
q10	3804	4231	4025	4025
q11	576	484	485	484
q12	805	650	603	603
q13	16132	3142	3173	3142
q14	343	278	264	264
q15	522	500	482	482
q16	483	437	462	437
q17	1762	1792	1739	1739
q18	8341	7850	7766	7766
q19	1680	1647	1650	1647
q20	2030	1834	1819	1819
q21	5284	5069	5038	5038
q22	518	465	450	450
Total cold run time: 71179 ms
Total hot run time: 55995 ms

[doris-robot]

TPC-DS: Total hot run time: 182669 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d7a152f6ead40f7701c9e25ed9342cc3770626d7, data reload: false

query1	1222	1132	1124	1124
query2	6190	1916	1891	1891
query3	6667	211	202	202
query4	25114	21566	21767	21566
query5	4150	391	405	391
query6	291	194	196	194
query7	4604	305	301	301
query8	239	186	177	177
query9	8466	2225	2241	2225
query10	444	244	263	244
query11	14984	14590	14502	14502
query12	141	93	91	91
query13	1633	373	384	373
query14	8465	6906	6854	6854
query15	210	181	180	180
query16	6677	278	284	278
query17	957	613	597	597
query18	1847	291	289	289
query19	224	178	183	178
query20	98	98	91	91
query21	202	139	133	133
query22	4958	4846	4807	4807
query23	33516	32767	32584	32584
query24	11815	3301	3187	3187
query25	652	431	470	431
query26	772	176	174	174
query27	3238	389	386	386
query28	5629	1888	1906	1888
query29	1030	642	641	641
query30	306	170	174	170
query31	1020	767	743	743
query32	107	64	62	62
query33	641	270	276	270
query34	912	519	541	519
query35	866	745	732	732
query36	1007	872	860	860
query37	119	82	81	81
query38	3760	3550	3604	3550
query39	1610	1601	1580	1580
query40	189	156	148	148
query41	49	47	47	47
query42	112	112	106	106
query43	445	420	402	402
query44	1086	722	730	722
query45	313	280	285	280
query46	1088	851	809	809
query47	1942	1864	1906	1864
query48	397	326	336	326
query49	808	397	400	397
query50	877	437	425	425
query51	6876	6732	6777	6732
query52	120	97	96	96
query53	367	298	295	295
query54	296	231	229	229
query55	89	79	82	79
query56	239	226	224	224
query57	1180	1112	1133	1112
query58	244	223	221	221
query59	2607	2371	2452	2371
query60	263	226	243	226
query61	98	91	94	91
query62	668	463	468	463
query63	313	286	284	284
query64	4658	3099	3138	3099
query65	3028	2998	3002	2998
query66	884	338	331	331
query67	15415	15008	15255	15008
query68	8957	566	573	566
query69	581	327	332	327
query70	1369	1100	1123	1100
query71	496	283	274	274
query72	6522	2618	2460	2460
query73	1564	328	328	328
query74	6691	6427	6383	6383
query75	3602	2284	2317	2284
query76	5236	1126	1180	1126
query77	588	244	248	244
query78	10800	10150	10163	10150
query79	9751	539	547	539
query80	1694	443	434	434
query81	505	233	218	218
query82	588	107	101	101
query83	205	165	162	162
query84	271	86	91	86
query85	1052	304	299	299
query86	356	271	283	271
query87	3734	3554	3544	3544
query88	3735	2325	2297	2297
query89	552	384	379	379
query90	1958	184	185	184
query91	139	107	105	105
query92	58	50	50	50
query93	6450	519	520	519
query94	1187	202	197	197
query95	446	329	338	329
query96	604	270	271	270
query97	2695	2458	2476	2458
query98	237	215	216	215
query99	1255	849	839	839
Total cold run time: 293312 ms
Total hot run time: 182669 ms

[doris-robot]

ClickBench: Total hot run time: 30 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d7a152f6ead40f7701c9e25ed9342cc3770626d7, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.05	0.04
query3	0.23	0.05	0.04
query4	1.68	0.06	0.07
query5	0.49	0.48	0.50
query6	1.14	0.65	0.65
query7	0.02	0.01	0.01
query8	0.06	0.05	0.04
query9	0.57	0.52	0.51
query10	0.56	0.56	0.57
query11	0.14	0.11	0.11
query12	0.14	0.12	0.11
query13	0.59	0.59	0.59
query14	0.78	0.78	0.79
query15	0.86	0.85	0.82
query16	0.36	0.36	0.37
query17	0.98	1.00	0.94
query18	0.24	0.25	0.25
query19	1.82	1.68	1.72
query20	0.02	0.01	0.01
query21	15.53	0.72	0.69
query22	2.90	6.01	1.65
query23	17.49	1.27	1.15
query24	1.46	0.21	0.22
query25	0.13	0.09	0.10
query26	0.25	0.18	0.20
query27	0.08	0.08	0.08
query28	13.85	0.95	0.92
query29	12.59	3.37	3.46
query30	0.28	0.09	0.07
query31	2.82	0.38	0.37
query32	3.29	0.48	0.46
query33	2.85	2.86	2.86
query34	15.54	4.32	4.35
query35	4.39	4.40	4.39
query36	0.67	0.48	0.47
query37	0.20	0.18	0.17
query38	0.18	0.15	0.17
query39	0.05	0.04	0.05
query40	0.18	0.16	0.18
query41	0.09	0.05	0.05
query42	0.06	0.04	0.05
query43	0.05	0.05	0.04
Total cold run time: 105.72 s
Total hot run time: 30 s

[doris-robot]

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit d7a152f6ead40f7701c9e25ed9342cc3770626d7 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       16.1 seconds inserted 10000000 Rows, about 621K ops/s