Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feat](binlog) Reduce AlterJobV2/TruncateTable binlog size (#30505) #30567

Merged
merged 1 commit into from
Jan 30, 2024
Merged

[feat](binlog) Reduce AlterJobV2/TruncateTable binlog size (#30505) #30567

merged 1 commit into from
Jan 30, 2024

Conversation

JackDrogon
Copy link
Contributor

Proposed changes

Reduce AlterJobV2/TruncateTable binlog size (#30505)

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@JackDrogon
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 50469 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e197981b5f6f0da2de146fdb408da0b7cb198f18, data reload: false

------ Round 1 ----------------------------------
q1	17540	4451	4299	4299
q2	2038	161	145	145
q3	10428	1892	1892	1892
q4	10268	1262	1326	1262
q5	8427	3946	4000	3946
q6	229	122	123	122
q7	2027	1613	1601	1601
q8	9291	2717	2705	2705
q9	11017	10974	10901	10901
q10	8643	3486	3494	3486
q11	408	254	237	237
q12	465	296	307	296
q13	18594	3996	4026	3996
q14	353	338	330	330
q15	503	465	449	449
q16	702	591	617	591
q17	1126	983	994	983
q18	7316	6969	6941	6941
q19	1682	1549	1528	1528
q20	517	312	311	311
q21	4531	4123	4051	4051
q22	488	405	397	397
Total cold run time: 116593 ms
Total hot run time: 50469 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4324	4304	4278	4278
q2	322	225	223	223
q3	4145	4153	4119	4119
q4	2762	2746	2731	2731
q5	7282	7184	7172	7172
q6	235	122	118	118
q7	3265	2881	2892	2881
q8	4393	4503	4493	4493
q9	17136	17019	16872	16872
q10	4232	4292	4240	4240
q11	778	682	688	682
q12	1028	845	869	845
q13	6492	3745	3751	3745
q14	451	426	428	426
q15	495	452	455	452
q16	752	702	685	685
q17	3829	3870	3874	3870
q18	8925	8764	8737	8737
q19	1717	1725	1672	1672
q20	2400	2148	2155	2148
q21	8642	8450	8511	8450
q22	1023	919	916	916
Total cold run time: 84628 ms
Total hot run time: 79755 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 242387 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e197981b5f6f0da2de146fdb408da0b7cb198f18, data reload: false

query1	923	391	378	378
query2	6514	2150	2071	2071
query3	6917	203	199	199
query4	20094	17942	17766	17766
query5	19732	6248	6237	6237
query6	274	212	213	212
query7	4135	295	300	295
query8	266	256	266	256
query9	3094	2693	2620	2620
query10	406	302	289	289
query11	11287	10719	10686	10686
query12	113	77	72	72
query13	5644	644	627	627
query14	17621	13684	13407	13407
query15	367	238	234	234
query16	6441	263	258	258
query17	3245	880	864	864
query18	2268	403	401	401
query19	200	152	149	149
query20	77	79	76	76
query21	183	98	93	93
query22	5305	5150	5116	5116
query23	32393	32058	32014	32014
query24	7025	6500	6573	6500
query25	529	433	426	426
query26	527	163	154	154
query27	1862	298	293	293
query28	6112	2213	2182	2182
query29	2940	2746	2805	2746
query30	236	166	158	158
query31	890	741	746	741
query32	71	58	57	57
query33	397	243	243	243
query34	833	469	476	469
query35	1118	922	920	920
query36	1485	1366	1421	1366
query37	86	60	60	60
query38	3098	2967	2927	2927
query39	1373	1306	1323	1306
query40	200	100	92	92
query41	34	32	32	32
query42	94	80	81	80
query43	578	659	573	573
query44	1127	708	721	708
query45	238	232	232	232
query46	1241	973	983	973
query47	1841	1637	1665	1637
query48	960	682	676	676
query49	601	375	381	375
query50	857	586	624	586
query51	5618	5532	5539	5532
query52	94	89	88	88
query53	446	343	314	314
query54	2404	2230	2260	2230
query55	82	76	80	76
query56	197	203	200	200
query57	1232	1215	1167	1167
query58	206	183	216	183
query59	3683	3354	3391	3354
query60	214	184	199	184
query61	91	80	82	80
query62	810	509	511	509
query63	470	337	334	334
query64	2089	1345	1301	1301
query65	3680	3602	3590	3590
query66	802	372	369	369
query67	16506	15406	15034	15034
query68	8597	655	631	631
query69	575	354	345	345
query70	2063	2231	1685	1685
query71	380	316	320	316
query72	4392	3317	3354	3317
query73	720	337	319	319
query74	6227	5942	5922	5922
query75	4868	3575	3444	3444
query76	5416	1175	1185	1175
query77	828	260	249	249
query78	31933	53307	51977	51977
query79	15466	644	644	644
query80	4334	390	383	383
query81	572	230	233	230
query82	1558	97	98	97
query83	331	131	134	131
query84	257	68	67	67
query85	1702	280	279	279
query86	441	378	383	378
query87	3271	3022	3007	3007
query88	7029	2315	2296	2296
query89	436	287	292	287
query90	2508	213	214	213
query91	154	120	123	120
query92	65	57	52	52
query93	5889	567	537	537
query94	1667	205	206	205
query95	1126	1083	1096	1083
query96	644	331	330	330
query97	6468	6348	6252	6252
query98	194	171	174	171
query99	3691	882	868	868
Total cold run time: 346278 ms
Total hot run time: 242387 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.49 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e197981b5f6f0da2de146fdb408da0b7cb198f18, data reload: false

query1	0.03	0.02	0.02
query2	0.06	0.02	0.02
query3	0.24	0.05	0.04
query4	1.82	0.07	0.06
query5	0.53	0.51	0.52
query6	1.25	0.62	0.62
query7	0.01	0.01	0.01
query8	0.03	0.02	0.02
query9	0.52	0.48	0.49
query10	0.53	0.54	0.55
query11	0.11	0.09	0.08
query12	0.12	0.09	0.09
query13	0.61	0.61	0.61
query14	0.78	0.77	0.81
query15	0.77	0.77	0.75
query16	0.36	0.40	0.37
query17	1.00	1.01	1.01
query18	0.24	0.24	0.23
query19	1.92	1.87	1.85
query20	0.01	0.01	0.01
query21	15.46	0.55	0.55
query22	2.12	2.15	1.53
query23	17.30	0.92	0.79
query24	6.90	2.18	0.87
query25	1.49	0.14	0.11
query26	0.35	0.14	0.14
query27	0.12	0.11	0.11
query28	6.13	0.74	0.73
query29	12.62	2.43	2.27
query30	0.56	0.49	0.51
query31	2.83	0.38	0.39
query32	3.40	0.48	0.49
query33	3.10	3.10	3.08
query34	15.25	4.80	4.79
query35	4.87	4.87	4.83
query36	1.07	1.01	1.00
query37	0.06	0.05	0.05
query38	0.03	0.02	0.02
query39	0.02	0.02	0.01
query40	0.15	0.14	0.14
query41	0.07	0.01	0.02
query42	0.02	0.01	0.02
query43	0.02	0.01	0.02
Total cold run time: 104.88 s
Total hot run time: 30.49 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit e197981b5f6f0da2de146fdb408da0b7cb198f18 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s

dataroaring
dataroaring approved these changes Jan 30, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 30, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit 50a10c8 into apache:branch-2.0 Jan 30, 2024
26 of 29 checks passed
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@JackDrogon
[improvement](binlog) Reduce AlterJobV2/TruncateTable binlog size (ap…
44b9fb3 
…ache#30505) (apache#30567)

Signed-off-by: Jack Drogon <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@JackDrogon @doris-robot @dataroaring