Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fix](Jdbc-Hive) fix the order of partition keys #33963

Merged
merged 1 commit into from
Apr 22, 2024
Merged

[Fix](Jdbc-Hive) fix the order of partition keys #33963

merged 1 commit into from
Apr 22, 2024

Conversation

BePPPower
Copy link
Contributor

@BePPPower BePPPower commented Apr 22, 2024 •
edited
Loading

Proposed changes

Issue Number: close #xxx

The partition key information recorded in PARTITION_KEYS table is sorted according to the INTEGER_IDX field, so we need to add an 'order by' clause to ensure that the obtained partition names are ordered.

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@BePPPower
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38855 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5371e2d7a821df4c126cf9b56244a6d2de5d5ba2, data reload: false

------ Round 1 ----------------------------------
q1	6509	4561	4242	4242
q2	508	189	187	187
q3	1417	1083	1126	1083
q4	1332	771	832	771
q5	2617	2726	2685	2685
q6	215	131	138	131
q7	1023	621	598	598
q8	1989	2210	2090	2090
q9	6704	6613	6521	6521
q10	3698	3537	3560	3537
q11	417	241	231	231
q12	387	232	215	215
q13	16901	2934	2990	2934
q14	275	229	232	229
q15	514	486	488	486
q16	509	399	377	377
q17	951	662	687	662
q18	7512	6991	6906	6906
q19	1587	1530	1526	1526
q20	795	329	322	322
q21	11203	2807	2922	2807
q22	378	315	325	315
Total cold run time: 67441 ms
Total hot run time: 38855 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4281	4265	4185	4185
q2	372	261	257	257
q3	3029	2710	2735	2710
q4	1840	1564	1573	1564
q5	5319	5332	5342	5332
q6	207	122	120	120
q7	2235	1874	1874	1874
q8	3218	3346	3335	3335
q9	8568	8606	8594	8594
q10	3919	3727	3741	3727
q11	576	483	474	474
q12	761	603	604	603
q13	3485	2968	2936	2936
q14	298	280	263	263
q15	512	486	465	465
q16	472	422	415	415
q17	1737	1447	1472	1447
q18	7616	7735	7963	7735
q19	1711	1598	1596	1596
q20	2043	1891	1924	1891
q21	5185	4994	4877	4877
q22	584	470	499	470
Total cold run time: 57968 ms
Total hot run time: 54870 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186711 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5371e2d7a821df4c126cf9b56244a6d2de5d5ba2, data reload: false

query1	916	389	377	377
query2	6482	2550	2377	2377
query3	6643	200	202	200
query4	23183	21526	21364	21364
query5	4218	408	431	408
query6	268	170	172	170
query7	4591	287	285	285
query8	239	173	184	173
query9	8477	2334	2338	2334
query10	567	258	245	245
query11	14801	14299	14224	14224
query12	134	91	88	88
query13	1635	363	367	363
query14	9346	7951	7184	7184
query15	244	184	188	184
query16	8009	261	250	250
query17	1940	572	545	545
query18	2061	304	282	282
query19	193	151	149	149
query20	93	85	88	85
query21	206	131	125	125
query22	5075	4872	4905	4872
query23	33826	34132	34178	34132
query24	10882	3042	2933	2933
query25	593	390	390	390
query26	932	156	157	156
query27	2339	369	371	369
query28	6381	2039	2048	2039
query29	865	646	614	614
query30	310	185	192	185
query31	1084	764	735	735
query32	90	57	54	54
query33	751	243	232	232
query34	1234	499	514	499
query35	841	713	750	713
query36	1125	938	928	928
query37	109	75	69	69
query38	3509	3463	3305	3305
query39	1644	1616	1624	1616
query40	188	130	133	130
query41	49	47	46	46
query42	108	102	100	100
query43	610	564	562	562
query44	1247	753	755	753
query45	280	273	275	273
query46	1118	765	739	739
query47	2058	1976	1950	1950
query48	396	307	309	307
query49	851	397	382	382
query50	778	378	395	378
query51	7025	6908	6744	6744
query52	102	93	91	91
query53	350	282	287	282
query54	310	249	240	240
query55	80	75	75	75
query56	248	251	233	233
query57	1315	1206	1187	1187
query58	221	203	199	199
query59	3629	3294	3442	3294
query60	254	231	226	226
query61	90	86	82	82
query62	620	436	440	436
query63	300	280	285	280
query64	4795	4102	3823	3823
query65	3105	3048	3025	3025
query66	737	322	318	318
query67	15365	15120	14966	14966
query68	6315	528	545	528
query69	532	308	299	299
query70	1273	1205	1167	1167
query71	1466	1263	1269	1263
query72	6749	2754	2557	2557
query73	735	320	320	320
query74	6817	6479	6355	6355
query75	3688	2631	2663	2631
query76	4125	988	946	946
query77	576	272	270	270
query78	11341	10378	10207	10207
query79	8491	529	524	524
query80	1925	463	454	454
query81	534	253	257	253
query82	1632	96	98	96
query83	334	171	168	168
query84	269	87	87	87
query85	1459	317	313	313
query86	457	309	302	302
query87	3511	3255	3299	3255
query88	5193	2330	2339	2330
query89	558	383	379	379
query90	1916	182	183	182
query91	141	106	111	106
query92	58	50	49	49
query93	6525	503	501	501
query94	1103	188	184	184
query95	403	303	298	298
query96	600	261	273	261
query97	3137	2993	2963	2963
query98	230	225	224	224
query99	1241	870	865	865
Total cold run time: 295399 ms
Total hot run time: 186711 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5371e2d7a821df4c126cf9b56244a6d2de5d5ba2, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.07	0.07
query5	0.48	0.47	0.50
query6	1.47	0.72	0.71
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.55	0.49	0.50
query10	0.55	0.57	0.54
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.61	0.58	0.58
query14	0.76	0.78	0.77
query15	0.82	0.79	0.80
query16	0.37	0.37	0.35
query17	1.02	0.98	1.04
query18	0.20	0.26	0.23
query19	1.77	1.82	1.67
query20	0.02	0.01	0.01
query21	15.79	0.64	0.65
query22	4.57	7.61	1.78
query23	18.27	1.32	1.27
query24	2.05	0.22	0.20
query25	0.14	0.08	0.09
query26	0.25	0.15	0.17
query27	0.08	0.08	0.07
query28	13.32	0.98	0.97
query29	12.58	3.28	3.24
query30	0.25	0.07	0.05
query31	2.94	0.37	0.37
query32	3.21	0.45	0.46
query33	2.77	2.81	2.78
query34	17.04	4.42	4.42
query35	4.47	4.55	4.44
query36	0.65	0.46	0.47
query37	0.19	0.16	0.15
query38	0.16	0.14	0.15
query39	0.05	0.03	0.03
query40	0.16	0.14	0.13
query41	0.10	0.05	0.05
query42	0.06	0.04	0.05
query43	0.04	0.03	0.03
Total cold run time: 110.16 s
Total hot run time: 30 s

@xiaokang xiaokang added dev/2.0.x usercase Important user case type label labels Apr 22, 2024
morningman
morningman approved these changes Apr 22, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, please add comment

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 22, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@morningman morningman merged commit d84b35c into apache:master Apr 22, 2024
27 of 30 checks passed
yiguolei pushed a commit that referenced this pull request Apr 23, 2024
@BePPPower @Doris-Extras
[Fix](Jdbc-Hive) fix the order of partition keys (#33963)
6027236 
The partition key information recorded in PARTITION_KEYS table is sorted according to the INTEGER_IDX field, so we need to add an 'order by' clause to ensure that the obtained partition names are ordered.
yiguolei pushed a commit that referenced this pull request Apr 24, 2024
@BePPPower @Doris-Extras
[Fix](Jdbc-Hive) fix the order of partition keys (#33963)
a11ae2c 
The partition key information recorded in PARTITION_KEYS table is sorted according to the INTEGER_IDX field, so we need to add an 'order by' clause to ensure that the obtained partition names are ordered.
dataroaring pushed a commit that referenced this pull request Apr 24, 2024
@BePPPower @dataroaring
[Fix](Jdbc-Hive) fix the order of partition keys (#33963)
62db031 
The partition key information recorded in PARTITION_KEYS table is sorted according to the INTEGER_IDX field, so we need to add an 'order by' clause to ensure that the obtained partition names are ordered.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morningman morningman morningman approved these changes

@wuwenchi wuwenchi wuwenchi approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.3-merged reviewed usercase Important user case type label
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@BePPPower @doris-robot @morningman @wuwenchi @xiaokang