Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](stream_load)streamload with double quotes #39634

Merged
merged 5 commits into from
Sep 23, 2024

Conversation

amorynan
Copy link
Contributor

Proposed changes

backport: #27942

Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@dataroaring
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 50687 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 731f72f9600cf49b6aa5eea0a790afb26fb954a0, data reload: false

------ Round 1 ----------------------------------
q1	17860	6101	4440	4440
q2	2162	201	190	190
q3	10455	2005	1967	1967
q4	10408	1302	1367	1302
q5	8460	3957	3973	3957
q6	273	145	145	145
q7	2093	1612	1641	1612
q8	9487	2825	2778	2778
q9	10702	10373	10346	10346
q10	8641	3513	3525	3513
q11	454	269	290	269
q12	499	333	331	331
q13	18382	3981	4034	3981
q14	385	346	366	346
q15	558	507	501	501
q16	704	610	608	608
q17	1174	978	979	978
q18	7430	6985	7007	6985
q19	1769	1609	1598	1598
q20	596	320	323	320
q21	4528	4203	4089	4089
q22	546	438	431	431
Total cold run time: 117566 ms
Total hot run time: 50687 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4382	4379	4303	4303
q2	365	269	265	265
q3	4210	4156	4185	4156
q4	2782	2787	2779	2779
q5	7241	7181	7209	7181
q6	256	141	148	141
q7	3311	2887	2952	2887
q8	4510	4493	4560	4493
q9	16923	16928	16936	16928
q10	4253	4299	4295	4295
q11	765	750	703	703
q12	1052	902	868	868
q13	7548	3782	3752	3752
q14	470	491	463	463
q15	545	515	500	500
q16	770	706	701	701
q17	3845	3899	3878	3878
q18	8761	8814	8834	8814
q19	1765	1810	1717	1717
q20	2428	2196	2141	2141
q21	8475	8522	8580	8522
q22	1063	984	949	949
Total cold run time: 85720 ms
Total hot run time: 80436 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 209128 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 731f72f9600cf49b6aa5eea0a790afb26fb954a0, data reload: false

query1	978	415	411	411
query2	6776	2509	2235	2235
query3	6940	231	229	229
query4	21000	18324	18228	18228
query5	19947	6634	6820	6634
query6	330	258	271	258
query7	4186	332	333	332
query8	490	454	455	454
query9	3238	2807	2739	2739
query10	487	374	364	364
query11	11387	10863	10816	10816
query12	152	123	110	110
query13	5669	727	725	725
query14	18227	13755	13681	13681
query15	387	257	252	252
query16	6515	326	303	303
query17	1562	1505	938	938
query18	2337	480	477	477
query19	245	183	190	183
query20	116	112	117	112
query21	242	151	150	150
query22	5207	5101	5018	5018
query23	32755	32150	31783	31783
query24	7084	6584	6566	6566
query25	574	488	471	471
query26	561	201	190	190
query27	1823	331	334	331
query28	6081	2454	2387	2387
query29	3117	2802	2855	2802
query30	281	208	206	206
query31	1013	851	831	831
query32	101	98	94	94
query33	474	336	342	336
query34	918	536	536	536
query35	1172	994	984	984
query36	1385	1178	1238	1178
query37	117	91	87	87
query38	3103	2956	2933	2933
query39	1482	1445	1429	1429
query40	251	150	149	149
query41	156	159	154	154
query42	112	118	114	114
query43	740	737	697	697
query44	1201	799	783	783
query45	281	268	266	266
query46	1282	997	1039	997
query47	1886	1851	1928	1851
query48	1032	728	721	721
query49	846	576	584	576
query50	947	665	657	657
query51	4823	4807	4757	4757
query52	122	109	109	109
query53	501	378	385	378
query54	2717	2533	2508	2508
query55	108	102	103	102
query56	324	289	290	289
query57	1253	1324	1219	1219
query58	326	305	317	305
query59	3508	3266	3219	3219
query60	308	281	280	280
query61	158	159	156	156
query62	913	518	518	518
query63	545	401	401	401
query64	2662	1664	1595	1595
query65	3676	3625	3605	3605
query66	1229	829	806	806
query67	15819	15176	15323	15176
query68	8649	720	713	713
query69	636	423	417	417
query70	1641	1412	1327	1327
query71	463	355	384	355
query72	6631	3600	3559	3559
query73	778	377	367	367
query74	6296	5918	5872	5872
query75	5206	3858	3872	3858
query76	5278	1218	1256	1218
query77	919	423	414	414
query78	12518	11942	12744	11942
query79	11681	675	690	675
query80	1572	546	553	546
query81	533	284	282	282
query82	779	129	129	129
query83	248	216	218	216
query84	286	98	101	98
query85	1018	412	413	412
query86	371	326	365	326
query87	3303	3070	3032	3032
query88	5100	2572	2534	2534
query89	410	338	325	325
query90	1999	248	260	248
query91	197	163	169	163
query92	92	86	88	86
query93	4088	569	600	569
query94	804	246	247	246
query95	1186	1124	1126	1124
query96	665	347	342	342
query97	6586	6451	6456	6451
query98	224	215	209	209
query99	3087	892	959	892
Total cold run time: 319429 ms
Total hot run time: 209128 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.21 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 731f72f9600cf49b6aa5eea0a790afb26fb954a0, data reload: false

query1	0.02	0.03	0.03
query2	0.07	0.03	0.02
query3	0.26	0.05	0.05
query4	1.79	0.07	0.07
query5	0.54	0.52	0.52
query6	1.27	0.63	0.67
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.54	0.49	0.50
query10	0.56	0.53	0.54
query11	0.14	0.09	0.09
query12	0.13	0.10	0.11
query13	0.62	0.63	0.61
query14	0.80	0.79	0.80
query15	0.78	0.77	0.77
query16	0.37	0.37	0.36
query17	1.02	1.02	1.04
query18	0.22	0.25	0.24
query19	1.95	1.85	1.88
query20	0.01	0.01	0.01
query21	15.48	0.56	0.57
query22	2.04	2.29	1.46
query23	17.01	1.23	0.97
query24	6.63	0.99	1.12
query25	0.40	0.11	0.06
query26	0.63	0.16	0.15
query27	0.05	0.04	0.04
query28	6.41	0.76	0.73
query29	12.65	2.39	2.32
query30	0.76	0.71	0.72
query31	2.83	0.40	0.38
query32	3.36	0.52	0.50
query33	3.08	3.07	3.06
query34	15.26	4.80	4.82
query35	4.85	4.86	4.86
query36	1.08	1.03	1.02
query37	0.07	0.06	0.07
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.19	0.16	0.18
query41	0.08	0.03	0.04
query42	0.04	0.03	0.04
query43	0.04	0.03	0.04
Total cold run time: 104.18 s
Total hot run time: 31.21 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 731f72f9600cf49b6aa5eea0a790afb26fb954a0 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

@amorynan amorynan changed the title [fix] fix streamload with double quotes [fix](stream_load)streamload with double quotes Aug 23, 2024
@amorynan
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 49468 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bea07de00ba0fc794f5e54c42b54b48b28b4f75c, data reload: false

------ Round 1 ----------------------------------
q1	17743	4476	4383	4383
q2	2037	163	147	147
q3	10415	1916	1947	1916
q4	10320	1239	1313	1239
q5	8422	3907	3796	3796
q6	227	124	149	124
q7	2037	1616	1605	1605
q8	9283	2712	2694	2694
q9	10610	10368	10068	10068
q10	8622	3494	3556	3494
q11	420	244	255	244
q12	458	293	305	293
q13	18329	3947	3978	3947
q14	361	335	328	328
q15	504	468	454	454
q16	687	568	569	568
q17	1120	952	954	952
q18	7303	6946	6966	6946
q19	1693	1545	1494	1494
q20	549	299	295	295
q21	4447	4085	4092	4085
q22	492	396	400	396
Total cold run time: 116079 ms
Total hot run time: 49468 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4379	4330	4303	4303
q2	327	224	225	224
q3	4184	4125	4150	4125
q4	2766	2758	2732	2732
q5	7156	7047	7104	7047
q6	243	120	117	117
q7	3258	2821	2878	2821
q8	4353	4448	4490	4448
q9	16801	16770	16817	16770
q10	4278	4266	4242	4242
q11	775	717	670	670
q12	1020	874	870	870
q13	7487	3749	3739	3739
q14	452	427	422	422
q15	489	462	456	456
q16	733	714	697	697
q17	3832	3845	3874	3845
q18	8817	8875	8733	8733
q19	1701	1692	1678	1678
q20	2405	2155	2101	2101
q21	8522	8546	8466	8466
q22	1028	958	915	915
Total cold run time: 85006 ms
Total hot run time: 79421 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 201597 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bea07de00ba0fc794f5e54c42b54b48b28b4f75c, data reload: false

query1	940	389	379	379
query2	6587	2138	1988	1988
query3	6919	207	201	201
query4	21020	18206	17972	17972
query5	19729	6493	6548	6493
query6	300	218	235	218
query7	4169	312	329	312
query8	249	266	248	248
query9	3181	2718	2665	2665
query10	444	289	301	289
query11	11393	10667	10742	10667
query12	119	79	71	71
query13	5593	667	663	663
query14	18153	13326	13729	13326
query15	365	226	234	226
query16	6461	284	262	262
query17	1704	1452	874	874
query18	2316	417	412	412
query19	206	152	155	152
query20	80	79	86	79
query21	192	97	94	94
query22	5152	4970	4960	4960
query23	32655	32128	31917	31917
query24	6952	6518	6546	6518
query25	538	429	429	429
query26	524	163	163	163
query27	1877	290	292	290
query28	6080	2301	2287	2287
query29	2875	2757	2587	2587
query30	242	172	165	165
query31	911	710	750	710
query32	70	58	65	58
query33	404	261	254	254
query34	842	487	484	484
query35	1119	923	914	914
query36	1279	1328	1083	1083
query37	91	59	59	59
query38	3113	2894	3000	2894
query39	1358	1305	1318	1305
query40	204	100	94	94
query41	39	38	36	36
query42	85	87	79	79
query43	783	564	614	564
query44	1149	717	717	717
query45	243	230	231	230
query46	1215	948	982	948
query47	1935	1748	1730	1730
query48	988	689	675	675
query49	626	361	399	361
query50	860	613	617	613
query51	4745	4654	4664	4654
query52	90	88	81	81
query53	464	322	326	322
query54	2660	2487	2451	2451
query55	87	86	90	86
query56	243	204	217	204
query57	1221	1090	1064	1064
query58	213	206	192	192
query59	3425	3524	3362	3362
query60	209	191	199	191
query61	93	97	93	93
query62	827	497	524	497
query63	482	339	337	337
query64	2555	1523	1380	1380
query65	3618	3564	3589	3564
query66	827	379	389	379
query67	15621	15501	15144	15144
query68	8340	648	648	648
query69	559	352	333	333
query70	1751	1459	1498	1459
query71	399	302	319	302
query72	6489	3458	3461	3458
query73	733	317	325	317
query74	6312	5870	5867	5867
query75	4627	3808	3686	3686
query76	4679	1131	1161	1131
query77	585	255	252	252
query78	12785	12328	11723	11723
query79	12239	653	670	653
query80	836	405	409	405
query81	506	235	238	235
query82	938	102	95	95
query83	167	135	130	130
query84	255	72	71	71
query85	847	319	322	319
query86	326	315	296	296
query87	3266	2995	3000	2995
query88	5318	2295	2291	2291
query89	424	288	297	288
query90	1964	210	208	208
query91	158	123	121	121
query92	60	53	52	52
query93	5050	556	601	556
query94	748	205	209	205
query95	1128	1064	1064	1064
query96	640	327	320	320
query97	6425	6400	6374	6374
query98	195	176	176	176
query99	3015	851	880	851
Total cold run time: 313473 ms
Total hot run time: 201597 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.38 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bea07de00ba0fc794f5e54c42b54b48b28b4f75c, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.03	0.02
query3	0.25	0.05	0.05
query4	1.80	0.07	0.07
query5	0.53	0.53	0.52
query6	1.28	0.63	0.61
query7	0.02	0.01	0.01
query8	0.03	0.02	0.02
query9	0.53	0.49	0.48
query10	0.54	0.54	0.52
query11	0.12	0.09	0.09
query12	0.12	0.09	0.09
query13	0.63	0.61	0.62
query14	0.80	0.77	0.80
query15	0.80	0.77	0.76
query16	0.37	0.37	0.39
query17	1.02	1.00	1.04
query18	0.23	0.26	0.22
query19	1.94	1.83	1.84
query20	0.01	0.00	0.01
query21	15.47	0.55	0.56
query22	1.96	1.78	1.66
query23	17.28	1.01	0.82
query24	5.27	0.75	1.13
query25	0.35	0.13	0.05
query26	0.58	0.14	0.14
query27	0.05	0.03	0.05
query28	8.76	0.73	0.73
query29	12.68	2.23	2.38
query30	0.60	0.53	0.52
query31	2.82	0.40	0.37
query32	3.37	0.51	0.50
query33	3.07	3.10	3.08
query34	15.25	4.79	4.79
query35	4.87	4.82	4.82
query36	1.06	1.02	1.02
query37	0.06	0.05	0.05
query38	0.03	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.14	0.14
query41	0.07	0.02	0.01
query42	0.02	0.01	0.02
query43	0.02	0.02	0.02
Total cold run time: 104.93 s
Total hot run time: 30.38 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit bea07de00ba0fc794f5e54c42b54b48b28b4f75c with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.1 seconds inserted 10000000 Rows, about 473K ops/s

@xiaokang
Copy link
Contributor

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@amorynan
Copy link
Contributor Author

run p0

@amorynan
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.80% (8142/21542)
Line Coverage: 29.54% (66977/226743)
Region Coverage: 29.02% (34554/119050)
Branch Coverage: 24.94% (17806/71386)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8fafb480d14aa56d77425fa63da6c13e6c702c63_8fafb480d14aa56d77425fa63da6c13e6c702c63/report/index.html

@xiaokang xiaokang merged commit d3f9f9b into apache:branch-2.0 Sep 23, 2024
20 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants