Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](routine load) fix enclose and escape can not set in routine load job (#38402) #39184

Merged
merged 1 commit into from
Aug 11, 2024

Conversation

sollhui
Copy link
Contributor

@sollhui sollhui commented Aug 10, 2024

pick (#38402)

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@sollhui
Copy link
Contributor Author

sollhui commented Aug 10, 2024

run buildall

@github-actions github-actions bot added area/load Issues or PRs related to all kinds of load area/planner Issues or PRs related to the query planner kind/test labels Aug 10, 2024
@sollhui
Copy link
Contributor Author

sollhui commented Aug 10, 2024

run buildall

@sollhui
Copy link
Contributor Author

sollhui commented Aug 10, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49798 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e95997175cb90f0ad51d42dffa6da229e8335b3f, data reload: false

------ Round 1 ----------------------------------
q1	17855	4414	4309	4309
q2	2072	154	146	146
q3	10278	1867	1900	1867
q4	10365	1254	1335	1254
q5	8494	3887	3898	3887
q6	232	123	124	123
q7	2069	1612	1638	1612
q8	9489	2735	2714	2714
q9	13519	10368	10213	10213
q10	8672	3517	3536	3517
q11	419	238	245	238
q12	475	312	305	305
q13	18331	3994	4049	3994
q14	347	323	335	323
q15	510	470	454	454
q16	666	562	562	562
q17	1129	936	944	936
q18	7152	6950	6835	6835
q19	1788	1669	1612	1612
q20	521	302	312	302
q21	4420	4152	4149	4149
q22	531	446	446	446
Total cold run time: 119334 ms
Total hot run time: 49798 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4352	4313	4301	4301
q2	323	225	228	225
q3	4221	4162	4159	4159
q4	2759	2773	2772	2772
q5	7230	7127	7091	7091
q6	238	117	123	117
q7	3274	2924	2892	2892
q8	4425	4552	4581	4552
q9	16921	16902	16939	16902
q10	4254	4300	4308	4300
q11	779	702	690	690
q12	1051	853	870	853
q13	7660	3850	3859	3850
q14	479	451	452	451
q15	502	458	465	458
q16	759	734	721	721
q17	3984	3949	3761	3761
q18	9101	8883	9026	8883
q19	1774	1687	1699	1687
q20	2406	2142	2120	2120
q21	8602	8579	8482	8482
q22	1082	999	951	951
Total cold run time: 86176 ms
Total hot run time: 80218 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 204739 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e95997175cb90f0ad51d42dffa6da229e8335b3f, data reload: false

query1	931	388	411	388
query2	6543	2816	2683	2683
query3	6918	208	194	194
query4	20266	18155	17863	17863
query5	19734	6472	6541	6472
query6	334	227	235	227
query7	4452	303	307	303
query8	430	407	413	407
query9	3136	2708	2651	2651
query10	408	329	297	297
query11	11308	10741	10692	10692
query12	127	80	75	75
query13	5603	713	689	689
query14	18882	13399	13271	13271
query15	363	239	251	239
query16	6433	286	262	262
query17	1368	1591	881	881
query18	2264	413	406	406
query19	206	154	146	146
query20	74	78	78	78
query21	188	99	94	94
query22	5281	5061	5176	5061
query23	32576	31932	32101	31932
query24	6804	6518	6491	6491
query25	538	427	430	427
query26	526	161	160	160
query27	1816	299	292	292
query28	6163	2400	2365	2365
query29	3082	2795	2668	2668
query30	251	175	171	171
query31	900	727	740	727
query32	71	63	59	59
query33	399	245	260	245
query34	852	471	487	471
query35	1146	946	890	890
query36	1354	1329	1391	1329
query37	92	60	67	60
query38	3066	2929	2901	2901
query39	1377	1346	1315	1315
query40	214	87	91	87
query41	46	44	44	44
query42	78	88	92	88
query43	730	705	749	705
query44	1116	738	727	727
query45	242	235	238	235
query46	1220	941	1016	941
query47	1985	1723	1892	1723
query48	1021	724	730	724
query49	626	378	379	378
query50	866	630	617	617
query51	4715	4686	4746	4686
query52	96	76	85	76
query53	442	330	320	320
query54	2646	2468	2438	2438
query55	90	89	88	88
query56	240	217	214	214
query57	1131	1077	1105	1077
query58	230	209	196	196
query59	3883	4141	3931	3931
query60	214	207	201	201
query61	104	93	97	93
query62	771	480	444	444
query63	481	341	347	341
query64	2508	1535	1485	1485
query65	3639	3547	3541	3541
query66	773	381	379	379
query67	16487	16494	17392	16494
query68	6382	658	633	633
query69	556	330	354	330
query70	1530	1568	1348	1348
query71	413	308	325	308
query72	6528	3488	3502	3488
query73	729	343	316	316
query74	6322	5867	5845	5845
query75	4685	3749	3702	3702
query76	4153	1147	1194	1147
query77	542	250	255	250
query78	12459	11524	11552	11524
query79	7358	633	635	633
query80	2549	398	410	398
query81	528	241	231	231
query82	1584	97	99	97
query83	214	132	134	132
query84	260	73	72	72
query85	1437	325	326	325
query86	367	286	304	286
query87	3209	3017	3029	3017
query88	5159	2325	2322	2322
query89	346	292	287	287
query90	1689	213	216	213
query91	173	138	139	138
query92	63	56	54	54
query93	2886	622	586	586
query94	912	207	212	207
query95	1128	1056	1046	1046
query96	647	331	314	314
query97	6485	6442	6352	6352
query98	191	171	170	170
query99	2946	813	815	813
Total cold run time: 307286 ms
Total hot run time: 204739 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.04 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e95997175cb90f0ad51d42dffa6da229e8335b3f, data reload: false

query1	0.02	0.02	0.03
query2	0.06	0.02	0.02
query3	0.25	0.05	0.05
query4	1.80	0.05	0.06
query5	0.54	0.53	0.52
query6	1.25	0.62	0.61
query7	0.02	0.01	0.02
query8	0.03	0.02	0.02
query9	0.53	0.49	0.49
query10	0.55	0.54	0.52
query11	0.12	0.08	0.08
query12	0.12	0.09	0.09
query13	0.63	0.61	0.62
query14	0.77	0.80	0.77
query15	0.78	0.76	0.76
query16	0.36	0.38	0.38
query17	1.02	1.02	1.01
query18	0.21	0.26	0.25
query19	1.93	1.75	1.83
query20	0.01	0.01	0.01
query21	15.92	0.56	0.54
query22	1.93	2.14	1.44
query23	16.80	1.13	0.88
query24	4.00	3.46	1.69
query25	0.39	0.10	0.05
query26	0.72	0.16	0.16
query27	0.03	0.04	0.04
query28	6.08	0.71	0.75
query29	12.63	2.22	2.16
query30	0.51	0.52	0.52
query31	2.81	0.41	0.37
query32	3.37	0.50	0.50
query33	3.06	3.07	3.07
query34	15.25	4.79	4.79
query35	4.87	4.84	4.84
query36	1.05	1.01	1.02
query37	0.06	0.05	0.04
query38	0.04	0.03	0.02
query39	0.02	0.01	0.02
query40	0.16	0.15	0.14
query41	0.07	0.02	0.01
query42	0.02	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 100.81 s
Total hot run time: 31.04 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit e95997175cb90f0ad51d42dffa6da229e8335b3f with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s

@sollhui
Copy link
Contributor Author

sollhui commented Aug 10, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 50709 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a2c4976da520efc2f0da768f1b9be5ff6df526b3, data reload: false

------ Round 1 ----------------------------------
q1	17706	4459	4451	4451
q2	2093	159	150	150
q3	10426	1898	1930	1898
q4	10278	1236	1325	1236
q5	8611	3962	3955	3955
q6	240	129	125	125
q7	2063	1602	1619	1602
q8	9293	2755	2727	2727
q9	10937	10780	10477	10477
q10	8723	3595	3568	3568
q11	420	255	258	255
q12	496	318	317	317
q13	18367	4089	4151	4089
q14	367	339	341	339
q15	540	475	483	475
q16	707	609	602	602
q17	1137	996	964	964
q18	7423	7135	7066	7066
q19	1776	1647	1638	1638
q20	521	295	313	295
q21	4439	4146	4056	4056
q22	525	424	442	424
Total cold run time: 117088 ms
Total hot run time: 50709 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4395	4302	4357	4302
q2	320	227	221	221
q3	4173	4138	4206	4138
q4	2789	2750	2759	2750
q5	7219	7213	7187	7187
q6	242	124	124	124
q7	3258	2830	2827	2827
q8	4384	4483	4516	4483
q9	17475	17170	17093	17093
q10	4319	4316	4255	4255
q11	775	706	714	706
q12	1037	878	855	855
q13	6896	3778	3745	3745
q14	451	418	438	418
q15	503	455	466	455
q16	753	699	675	675
q17	3891	3876	3820	3820
q18	8882	8852	8920	8852
q19	1727	1739	1639	1639
q20	2390	2143	2153	2143
q21	8646	8487	8567	8487
q22	1066	968	954	954
Total cold run time: 85591 ms
Total hot run time: 80129 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 204708 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a2c4976da520efc2f0da768f1b9be5ff6df526b3, data reload: false

query1	931	389	421	389
query2	6569	2916	2662	2662
query3	6917	210	200	200
query4	21106	17915	18124	17915
query5	19730	6534	6489	6489
query6	285	222	226	222
query7	4158	300	308	300
query8	398	381	433	381
query9	3171	2732	2646	2646
query10	404	307	285	285
query11	11290	10726	10817	10726
query12	121	80	73	73
query13	5610	712	707	707
query14	17858	13282	13167	13167
query15	359	235	250	235
query16	6469	282	264	264
query17	1708	1450	867	867
query18	2339	424	417	417
query19	211	150	151	150
query20	80	80	77	77
query21	191	100	98	98
query22	5226	5106	5023	5023
query23	32435	32045	31864	31864
query24	7005	6488	6462	6462
query25	533	437	431	431
query26	537	161	160	160
query27	1887	296	294	294
query28	6145	2409	2368	2368
query29	2983	2811	2704	2704
query30	251	171	174	171
query31	894	750	775	750
query32	68	63	64	63
query33	402	268	271	268
query34	843	487	472	472
query35	1132	966	957	957
query36	1319	1003	1107	1003
query37	88	62	64	62
query38	3082	2976	2960	2960
query39	1386	1319	1321	1319
query40	205	97	94	94
query41	46	43	52	43
query42	89	98	79	79
query43	679	743	802	743
query44	1107	719	729	719
query45	249	234	238	234
query46	1232	946	961	946
query47	1929	1843	1847	1843
query48	1001	703	713	703
query49	624	384	379	379
query50	857	616	614	614
query51	4800	4666	4662	4662
query52	93	88	88	88
query53	457	323	320	320
query54	2664	2455	2456	2455
query55	90	86	89	86
query56	237	211	218	211
query57	1247	1126	1066	1066
query58	207	208	197	197
query59	3872	3870	4051	3870
query60	228	193	214	193
query61	102	96	100	96
query62	923	537	472	472
query63	483	345	337	337
query64	2288	1552	1537	1537
query65	3612	3572	3576	3572
query66	799	380	375	375
query67	15617	15940	18373	15940
query68	10570	697	650	650
query69	583	361	346	346
query70	2029	1307	1344	1307
query71	423	300	310	300
query72	6653	3540	3510	3510
query73	732	321	319	319
query74	6348	5929	5900	5900
query75	5601	3784	3754	3754
query76	6994	1151	1181	1151
query77	1182	259	252	252
query78	13000	11998	11858	11858
query79	7958	648	659	648
query80	985	410	408	408
query81	461	232	236	232
query82	806	97	106	97
query83	169	132	138	132
query84	259	73	73	73
query85	910	341	335	335
query86	345	292	292	292
query87	3242	3071	3050	3050
query88	4356	2299	2325	2299
query89	474	295	314	295
query90	1923	211	214	211
query91	172	140	138	138
query92	55	52	51	51
query93	6576	626	598	598
query94	705	207	210	207
query95	1119	1068	1050	1050
query96	642	331	323	323
query97	6528	6406	6493	6406
query98	185	182	174	174
query99	3061	867	887	867
Total cold run time: 316904 ms
Total hot run time: 204708 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.11 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a2c4976da520efc2f0da768f1b9be5ff6df526b3, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.03	0.02
query3	0.24	0.05	0.04
query4	1.78	0.07	0.08
query5	0.55	0.53	0.52
query6	1.23	0.67	0.63
query7	0.02	0.01	0.02
query8	0.03	0.03	0.02
query9	0.53	0.50	0.47
query10	0.53	0.54	0.53
query11	0.12	0.09	0.09
query12	0.12	0.09	0.09
query13	0.63	0.62	0.61
query14	0.80	0.79	0.78
query15	0.76	0.76	0.75
query16	0.39	0.40	0.37
query17	1.02	0.98	1.03
query18	0.21	0.26	0.26
query19	1.92	1.86	1.82
query20	0.02	0.01	0.01
query21	15.44	0.56	0.57
query22	2.05	2.72	1.79
query23	17.18	0.86	0.86
query24	6.71	1.39	1.11
query25	0.35	0.10	0.05
query26	0.78	0.15	0.15
query27	0.04	0.03	0.04
query28	6.21	0.77	0.74
query29	12.62	2.40	2.30
query30	0.61	0.53	0.53
query31	2.81	0.38	0.38
query32	3.39	0.50	0.49
query33	3.07	3.10	3.09
query34	15.25	4.83	4.82
query35	4.92	4.86	4.86
query36	1.08	1.01	1.01
query37	0.06	0.05	0.04
query38	0.03	0.02	0.02
query39	0.03	0.02	0.01
query40	0.16	0.14	0.14
query41	0.06	0.01	0.01
query42	0.02	0.01	0.02
query43	0.03	0.02	0.02
Total cold run time: 103.89 s
Total hot run time: 31.11 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit a2c4976da520efc2f0da768f1b9be5ff6df526b3 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.6 seconds inserted 10000000 Rows, about 462K ops/s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 11, 2024
Copy link
Contributor

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit f2a82f6 into apache:branch-2.0 Aug 11, 2024
23 of 26 checks passed
GoGoWen pushed a commit to GoGoWen/incubator-doris that referenced this pull request Aug 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. area/load Issues or PRs related to all kinds of load area/planner Issues or PRs related to the query planner kind/test reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants