Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](statistics)Use utf-8 charset for internal query result. (#39989) #40025

Merged
merged 1 commit into from
Aug 28, 2024
Merged

[fix](statistics)Use utf-8 charset for internal query result. (#39989) #40025

merged 1 commit into from
Aug 28, 2024

Conversation

Jibing-Li
Copy link
Contributor

backport: #39989

@Jibing-Li
[fix](statistics)Use utf-8 charset for internal query result. (apache…
cd3d980 
…#39989)

Use utf-8 charset for internal query result.
User's environment may set default java file encode to some other
character set, which may cause internal query get Garbled Characters.
Use utf-8 to decode string bytes from BE.
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Jibing-Li Jibing-Li marked this pull request as ready for review August 28, 2024 06:06
@Jibing-Li
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 50063 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cd3d980fe1c55b7f38ff7cdb64247fb4e7e2024b, data reload: false

------ Round 1 ----------------------------------
q1	18068	4458	4368	4368
q2	2055	185	148	148
q3	10584	1917	1912	1912
q4	10355	1240	1328	1240
q5	8467	3906	3923	3906
q6	231	123	123	123
q7	2116	1644	1640	1640
q8	9699	2747	2742	2742
q9	10697	11021	10445	10445
q10	8968	3632	3553	3553
q11	422	236	251	236
q12	471	303	307	303
q13	19797	4080	4101	4080
q14	352	332	346	332
q15	523	459	456	456
q16	699	578	569	569
q17	1119	976	970	970
q18	7196	6890	6764	6764
q19	1667	1563	1484	1484
q20	517	312	282	282
q21	4447	4124	4130	4124
q22	472	411	386	386
Total cold run time: 118922 ms
Total hot run time: 50063 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4313	4289	4270	4270
q2	321	232	223	223
q3	4161	4111	4136	4111
q4	2721	2736	2723	2723
q5	7111	7155	7001	7001
q6	237	118	121	118
q7	3270	2831	2865	2831
q8	4386	4473	4473	4473
q9	16795	16823	16669	16669
q10	4237	4242	4223	4223
q11	769	674	702	674
q12	1023	853	827	827
q13	6718	3719	3701	3701
q14	454	438	419	419
q15	500	457	459	457
q16	742	689	672	672
q17	3843	3905	3804	3804
q18	8743	8732	8724	8724
q19	1726	1689	1675	1675
q20	2412	2089	2110	2089
q21	8445	8467	8408	8408
q22	1020	936	926	926
Total cold run time: 83947 ms
Total hot run time: 79018 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 201456 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cd3d980fe1c55b7f38ff7cdb64247fb4e7e2024b, data reload: false

query1	933	390	406	390
query2	6554	2229	2161	2161
query3	6920	209	207	207
query4	20024	18021	17927	17927
query5	19729	6490	6520	6490
query6	281	235	223	223
query7	4157	303	306	303
query8	241	276	248	248
query9	3137	2708	2644	2644
query10	527	299	289	289
query11	11243	10510	10753	10510
query12	123	76	78	76
query13	5589	657	660	657
query14	17885	13007	12942	12942
query15	360	214	235	214
query16	6449	283	261	261
query17	1720	1460	884	884
query18	2281	415	406	406
query19	213	152	141	141
query20	80	80	77	77
query21	188	94	102	94
query22	5073	5008	4960	4960
query23	32678	31707	32019	31707
query24	6934	6496	6468	6468
query25	529	431	421	421
query26	523	159	161	159
query27	1886	298	291	291
query28	6143	2290	2255	2255
query29	2816	2923	2747	2747
query30	248	165	166	165
query31	924	752	741	741
query32	68	58	55	55
query33	404	266	262	262
query34	825	473	474	473
query35	1115	918	907	907
query36	1236	1143	1266	1143
query37	93	58	60	58
query38	3055	2937	2896	2896
query39	1390	1315	1313	1313
query40	206	96	96	96
query41	39	38	36	36
query42	87	83	80	80
query43	662	584	549	549
query44	1089	709	719	709
query45	239	228	229	228
query46	1220	970	993	970
query47	1989	1763	1656	1656
query48	975	684	658	658
query49	608	370	367	367
query50	856	575	621	575
query51	4768	4595	4657	4595
query52	92	72	80	72
query53	436	323	318	318
query54	2659	2500	2500	2500
query55	93	71	80	71
query56	240	228	211	211
query57	1129	1108	1126	1108
query58	225	211	193	193
query59	3587	3143	3404	3143
query60	211	216	221	216
query61	95	91	91	91
query62	827	496	557	496
query63	482	341	350	341
query64	2590	1552	1491	1491
query65	3596	3555	3570	3555
query66	812	375	375	375
query67	15454	16772	16214	16214
query68	8645	626	637	626
query69	573	337	367	337
query70	1599	1359	1289	1289
query71	401	306	319	306
query72	6523	3421	3479	3421
query73	726	311	311	311
query74	6252	5818	5856	5818
query75	4739	3676	3718	3676
query76	4806	1171	1225	1171
query77	670	251	255	251
query78	12496	11589	11644	11589
query79	6941	627	623	623
query80	1678	399	398	398
query81	508	239	238	238
query82	1518	96	95	95
query83	178	133	131	131
query84	256	72	70	70
query85	1271	310	314	310
query86	351	304	289	289
query87	3233	3003	3010	3003
query88	5028	2277	2304	2277
query89	339	294	294	294
query90	1751	204	211	204
query91	168	120	126	120
query92	60	56	55	55
query93	2600	561	560	560
query94	871	207	206	206
query95	1114	1074	1051	1051
query96	619	323	321	321
query97	6422	6303	6308	6303
query98	182	181	171	171
query99	2963	930	863	863
Total cold run time: 305321 ms
Total hot run time: 201456 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.71 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cd3d980fe1c55b7f38ff7cdb64247fb4e7e2024b, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.03	0.02
query3	0.24	0.04	0.05
query4	1.79	0.06	0.07
query5	0.54	0.52	0.52
query6	1.23	0.61	0.60
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.52	0.48	0.47
query10	0.53	0.53	0.52
query11	0.12	0.08	0.09
query12	0.11	0.09	0.09
query13	0.63	0.61	0.62
query14	0.77	0.79	0.78
query15	0.78	0.77	0.76
query16	0.38	0.38	0.37
query17	1.01	1.00	1.02
query18	0.24	0.24	0.21
query19	1.93	1.85	1.90
query20	0.02	0.01	0.01
query21	15.44	0.54	0.52
query22	2.11	2.08	1.56
query23	17.41	1.13	0.81
query24	4.31	1.10	1.21
query25	0.37	0.12	0.05
query26	0.53	0.16	0.15
query27	0.03	0.05	0.03
query28	8.48	0.80	0.74
query29	12.63	2.39	2.38
query30	0.57	0.53	0.59
query31	2.80	0.40	0.38
query32	3.40	0.49	0.50
query33	3.09	3.05	3.06
query34	15.23	4.76	4.79
query35	4.88	4.85	4.84
query36	1.04	1.03	1.02
query37	0.06	0.04	0.04
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.14
query41	0.07	0.02	0.01
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 103.7 s
Total hot run time: 30.71 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit cd3d980fe1c55b7f38ff7cdb64247fb4e7e2024b with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.8 seconds inserted 10000000 Rows, about 458K ops/s

@Jibing-Li Jibing-Li merged commit ad998ef into apache:branch-2.0 Aug 28, 2024
23 of 25 checks passed
@Jibing-Li Jibing-Li deleted the utf82.0 branch August 28, 2024 07:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Jibing-Li @doris-robot