Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[opt](inverted index) the "unicode" tokenizer can be configured to select stop words #33982 #34376

Merged
merged 1 commit into from
May 3, 2024
Merged

[opt](inverted index) the "unicode" tokenizer can be configured to select stop words #33982 #34376

merged 1 commit into from
May 3, 2024

Conversation

zzzxl1993
Copy link
Contributor

@zzzxl1993 zzzxl1993 commented May 1, 2024 •
edited
Loading

…sable stop words

Proposed changes

#33982

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zzzxl1993
Copy link
Contributor Author

run buildall

@github-actions github-actions bot added area/planner Issues or PRs related to the query planner kind/test labels May 1, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented May 1, 2024

clang-tidy review says "All clean, LGTM! 👍"

@zzzxl1993
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented May 2, 2024

clang-tidy review says "All clean, LGTM! 👍"

@xiaokang xiaokang changed the title [opt](inverted index) the "unicode" tokenizer can be configured to di… [opt](inverted index) the "unicode" tokenizer can be configured to select stop words May 2, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 49753 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 58bb321b747d1fcbeea9f645961080c76e148e86, data reload: false

------ Round 1 ----------------------------------
q1	17694	4501	4325	4325
q2	2031	152	141	141
q3	10470	2000	1917	1917
q4	10234	1269	1332	1269
q5	8484	4207	3943	3943
q6	232	126	125	125
q7	2061	1604	1595	1595
q8	9336	2737	2750	2737
q9	10718	10268	10325	10268
q10	8597	3545	3513	3513
q11	417	238	240	238
q12	464	296	300	296
q13	18329	3977	4097	3977
q14	358	321	316	316
q15	520	458	454	454
q16	684	571	584	571
q17	1128	949	974	949
q18	7310	6762	6821	6762
q19	1707	1575	1556	1556
q20	552	309	306	306
q21	4468	4143	4106	4106
q22	493	389	402	389
Total cold run time: 116287 ms
Total hot run time: 49753 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4344	4381	4265	4265
q2	320	230	217	217
q3	4192	4193	4170	4170
q4	2927	2752	2742	2742
q5	7223	7158	7109	7109
q6	236	119	118	118
q7	3232	2861	2830	2830
q8	4360	4491	4463	4463
q9	16875	16855	16819	16819
q10	4282	4283	4269	4269
q11	744	713	690	690
q12	1012	849	843	843
q13	6731	3785	3767	3767
q14	463	426	424	424
q15	503	448	457	448
q16	740	701	698	698
q17	3809	3760	3821	3760
q18	8795	8793	8849	8793
q19	1719	1731	1671	1671
q20	2389	2166	2134	2134
q21	8531	8547	8457	8457
q22	1034	950	935	935
Total cold run time: 84461 ms
Total hot run time: 79622 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.81% (8078/21363)
Line Coverage: 29.45% (65935/223871)
Region Coverage: 28.92% (33949/117376)
Branch Coverage: 24.78% (17420/70300)
Coverage Report: http://coverage.selectdb-in.cc/coverage/58bb321b747d1fcbeea9f645961080c76e148e86_58bb321b747d1fcbeea9f645961080c76e148e86/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 202717 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 58bb321b747d1fcbeea9f645961080c76e148e86, data reload: false

query1	915	394	380	380
query2	6541	2780	2866	2780
query3	6913	205	204	204
query4	20333	17932	18020	17932
query5	19760	6523	6514	6514
query6	280	215	224	215
query7	4149	292	298	292
query8	239	251	230	230
query9	3132	2677	2575	2575
query10	419	283	297	283
query11	11359	10656	10620	10620
query12	125	76	71	71
query13	5574	693	674	674
query14	17882	13201	13469	13201
query15	358	208	217	208
query16	6453	291	258	258
query17	1714	1437	864	864
query18	2329	404	412	404
query19	205	155	142	142
query20	77	75	75	75
query21	190	95	93	93
query22	5138	5049	5153	5049
query23	32592	31797	32025	31797
query24	6897	6508	6510	6508
query25	512	431	414	414
query26	500	161	153	153
query27	1798	292	293	292
query28	6205	2374	2337	2337
query29	3019	2952	2766	2766
query30	243	160	165	160
query31	906	715	740	715
query32	68	58	57	57
query33	391	245	252	245
query34	862	455	494	455
query35	1118	915	938	915
query36	1210	1235	1264	1235
query37	91	66	57	57
query38	3090	2916	2934	2916
query39	1373	1329	1313	1313
query40	198	97	89	89
query41	41	35	34	34
query42	84	85	84	84
query43	823	858	710	710
query44	1121	728	735	728
query45	243	225	222	222
query46	1215	971	988	971
query47	1865	1751	1787	1751
query48	1028	718	704	704
query49	619	355	363	355
query50	869	610	580	580
query51	4798	4634	4654	4634
query52	91	84	81	81
query53	439	314	316	314
query54	2645	2466	2464	2464
query55	97	78	85	78
query56	237	211	211	211
query57	1231	1134	1006	1006
query58	219	185	185	185
query59	3916	3877	4222	3877
query60	201	192	194	192
query61	87	82	91	82
query62	859	453	470	453
query63	462	327	332	327
query64	2347	1602	1381	1381
query65	3604	3538	3558	3538
query66	749	354	386	354
query67	15491	15035	16091	15035
query68	8004	684	660	660
query69	553	330	342	330
query70	1552	1512	1526	1512
query71	398	298	295	295
query72	6486	3446	3428	3428
query73	732	322	318	318
query74	6325	5960	5863	5863
query75	4692	3781	3614	3614
query76	4532	1140	1199	1140
query77	544	251	247	247
query78	12621	11752	12705	11752
query79	11725	656	625	625
query80	1890	396	390	390
query81	505	232	227	227
query82	1474	95	98	95
query83	159	134	133	133
query84	264	70	69	69
query85	1120	295	299	295
query86	341	306	290	290
query87	3229	3027	3037	3027
query88	5121	2342	2330	2330
query89	472	291	282	282
query90	1790	201	214	201
query91	178	144	134	134
query92	57	51	51	51
query93	6051	556	616	556
query94	706	201	207	201
query95	1096	1072	1046	1046
query96	654	332	324	324
query97	6518	6422	6477	6422
query98	185	169	162	162
query99	2736	859	985	859
Total cold run time: 313249 ms
Total hot run time: 202717 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.76 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 58bb321b747d1fcbeea9f645961080c76e148e86, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.24	0.05	0.04
query4	1.80	0.07	0.07
query5	0.53	0.53	0.53
query6	1.24	0.62	0.67
query7	0.01	0.01	0.01
query8	0.03	0.02	0.02
query9	0.53	0.47	0.49
query10	0.54	0.52	0.53
query11	0.12	0.08	0.09
query12	0.11	0.09	0.10
query13	0.62	0.61	0.61
query14	0.76	0.79	0.80
query15	0.78	0.77	0.76
query16	0.38	0.36	0.36
query17	1.00	1.00	1.02
query18	0.23	0.28	0.25
query19	1.84	1.88	1.84
query20	0.02	0.01	0.01
query21	15.45	0.56	0.54
query22	1.96	2.99	1.50
query23	16.54	0.93	0.93
query24	6.03	1.11	2.22
query25	0.38	0.07	0.06
query26	0.78	0.15	0.14
query27	0.05	0.03	0.03
query28	6.12	0.76	0.72
query29	12.64	2.24	2.28
query30	0.66	0.56	0.55
query31	2.82	0.37	0.38
query32	3.39	0.50	0.51
query33	3.15	3.08	3.07
query34	15.27	4.80	4.84
query35	4.85	4.82	4.87
query36	1.06	1.01	1.01
query37	0.06	0.04	0.05
query38	0.04	0.02	0.02
query39	0.01	0.01	0.02
query40	0.16	0.14	0.14
query41	0.06	0.01	0.02
query42	0.02	0.02	0.02
query43	0.02	0.02	0.02
Total cold run time: 102.39 s
Total hot run time: 30.76 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 58bb321b747d1fcbeea9f645961080c76e148e86 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.2 seconds inserted 10000000 Rows, about 471K ops/s

@xiaokang
Copy link
Contributor

xiaokang commented May 2, 2024

run p0 10

@zzzxl1993
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented May 2, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 49545 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit aa0e49de391fc9fc35bc602f3dde5049b7060890, data reload: false

------ Round 1 ----------------------------------
q1	17664	4395	4367	4367
q2	2028	152	148	148
q3	10455	1863	1890	1863
q4	10351	1233	1332	1233
q5	8352	3876	3914	3876
q6	230	121	122	121
q7	2017	1589	1585	1585
q8	9290	2920	2711	2711
q9	10495	10382	10239	10239
q10	8641	3539	3519	3519
q11	408	234	237	234
q12	466	307	307	307
q13	18373	3946	4027	3946
q14	358	336	315	315
q15	513	459	463	459
q16	673	571	565	565
q17	1124	933	933	933
q18	7288	6916	6839	6839
q19	1713	1576	1494	1494
q20	522	321	316	316
q21	4447	4149	4094	4094
q22	497	386	381	381
Total cold run time: 115905 ms
Total hot run time: 49545 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4325	4309	4290	4290
q2	322	229	224	224
q3	4172	4140	4115	4115
q4	2729	2743	2719	2719
q5	7163	7113	7076	7076
q6	231	120	118	118
q7	3242	2848	2841	2841
q8	4387	4488	4451	4451
q9	16965	16653	16599	16599
q10	4209	4234	4255	4234
q11	769	724	667	667
q12	1029	856	852	852
q13	6504	3760	3738	3738
q14	439	422	431	422
q15	498	457	456	456
q16	758	701	674	674
q17	3908	3906	3862	3862
q18	8761	8750	8814	8750
q19	1682	1693	1613	1613
q20	2402	2210	2098	2098
q21	8424	8541	8636	8541
q22	1026	945	998	945
Total cold run time: 83945 ms
Total hot run time: 79285 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.80% (8076/21363)
Line Coverage: 29.45% (65931/223871)
Region Coverage: 28.92% (33940/117376)
Branch Coverage: 24.78% (17418/70300)
Coverage Report: http://coverage.selectdb-in.cc/coverage/aa0e49de391fc9fc35bc602f3dde5049b7060890_aa0e49de391fc9fc35bc602f3dde5049b7060890/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 203065 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit aa0e49de391fc9fc35bc602f3dde5049b7060890, data reload: false

query1	935	386	382	382
query2	6533	2772	2757	2757
query3	6918	197	194	194
query4	20100	17990	17917	17917
query5	19716	6510	6480	6480
query6	301	214	235	214
query7	4151	296	302	296
query8	272	251	235	235
query9	3110	2687	2634	2634
query10	421	286	295	286
query11	11568	10702	10619	10619
query12	117	79	68	68
query13	5596	705	692	692
query14	17806	13387	13559	13387
query15	356	217	222	217
query16	6474	276	255	255
query17	1731	1449	881	881
query18	2301	403	425	403
query19	209	144	147	144
query20	73	79	70	70
query21	190	92	95	92
query22	5303	5092	4986	4986
query23	32539	32011	32236	32011
query24	7001	6517	6569	6517
query25	528	425	411	411
query26	525	166	154	154
query27	1851	303	296	296
query28	6172	2364	2337	2337
query29	2805	2652	2658	2652
query30	240	159	164	159
query31	897	770	718	718
query32	67	59	53	53
query33	398	251	257	251
query34	848	480	482	480
query35	1107	938	949	938
query36	1312	1142	1225	1142
query37	90	63	59	59
query38	3065	2945	2879	2879
query39	1372	1328	1351	1328
query40	192	94	97	94
query41	42	35	34	34
query42	84	81	83	81
query43	732	666	716	666
query44	1248	722	733	722
query45	243	225	224	224
query46	1234	988	951	951
query47	1829	1761	1677	1677
query48	1014	734	716	716
query49	624	353	394	353
query50	880	628	626	626
query51	4808	4690	4664	4664
query52	84	73	76	73
query53	449	331	324	324
query54	2675	2404	2450	2404
query55	86	81	75	75
query56	211	207	202	202
query57	1162	1096	1137	1096
query58	214	209	195	195
query59	4017	4192	3991	3991
query60	209	189	188	188
query61	88	81	82	81
query62	843	489	547	489
query63	484	344	340	340
query64	2448	1587	1331	1331
query65	3646	3525	3537	3525
query66	768	368	384	368
query67	15624	14780	15515	14780
query68	10207	657	644	644
query69	573	321	359	321
query70	1749	1373	1307	1307
query71	419	296	312	296
query72	6537	3427	3401	3401
query73	734	310	315	310
query74	6349	6023	5877	5877
query75	5481	3774	3701	3701
query76	6404	1190	1208	1190
query77	1099	255	247	247
query78	12582	12223	13737	12223
query79	9037	621	641	621
query80	660	397	393	393
query81	487	230	234	230
query82	810	108	96	96
query83	177	131	144	131
query84	252	68	69	68
query85	753	296	287	287
query86	324	309	293	293
query87	3182	3041	3003	3003
query88	4274	2321	2324	2321
query89	408	294	289	289
query90	1895	206	189	189
query91	170	136	136	136
query92	57	54	53	53
query93	4670	545	577	545
query94	633	197	202	197
query95	1105	1060	1080	1060
query96	659	328	324	324
query97	6439	6616	6415	6415
query98	188	169	164	164
query99	2426	869	896	869
Total cold run time: 311676 ms
Total hot run time: 203065 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.9 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit aa0e49de391fc9fc35bc602f3dde5049b7060890, data reload: false

query1	0.02	0.02	0.02
query2	0.06	0.03	0.02
query3	0.24	0.04	0.04
query4	1.79	0.06	0.07
query5	0.53	0.53	0.52
query6	1.23	0.65	0.61
query7	0.01	0.01	0.01
query8	0.04	0.03	0.02
query9	0.52	0.48	0.47
query10	0.54	0.54	0.54
query11	0.12	0.09	0.08
query12	0.12	0.09	0.09
query13	0.61	0.62	0.61
query14	0.77	0.79	0.78
query15	0.78	0.76	0.76
query16	0.37	0.36	0.37
query17	0.99	1.02	1.01
query18	0.21	0.27	0.25
query19	1.91	1.77	1.83
query20	0.02	0.01	0.02
query21	15.47	0.54	0.54
query22	2.31	2.21	1.73
query23	17.33	0.90	0.92
query24	5.07	1.25	1.11
query25	0.35	0.09	0.05
query26	0.60	0.16	0.15
query27	0.04	0.04	0.03
query28	7.75	0.78	0.74
query29	12.61	2.38	2.34
query30	0.58	0.51	0.49
query31	2.80	0.39	0.37
query32	3.42	0.50	0.50
query33	3.08	3.11	3.07
query34	15.29	4.78	4.79
query35	4.86	4.84	4.83
query36	1.06	1.00	1.02
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.01	0.01
query40	0.17	0.15	0.14
query41	0.07	0.02	0.01
query42	0.02	0.01	0.02
query43	0.02	0.01	0.02
Total cold run time: 103.9 s
Total hot run time: 30.9 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit aa0e49de391fc9fc35bc602f3dde5049b7060890 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.5 seconds inserted 10000000 Rows, about 465K ops/s

@zzzxl1993
Copy link
Contributor Author

run p0

@zzzxl1993
Copy link
Contributor Author

run p0

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented May 2, 2024

clang-tidy review says "All clean, LGTM! 👍"

@zzzxl1993
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.81% (8078/21363)
Line Coverage: 29.45% (65937/223871)
Region Coverage: 28.92% (33947/117376)
Branch Coverage: 24.78% (17418/70300)
Coverage Report: http://coverage.selectdb-in.cc/coverage/719037f91a6e78aea356359594f4049ebb7cea2a_719037f91a6e78aea356359594f4049ebb7cea2a/report/index.html

@zzzxl1993
Copy link
Contributor Author

run feut

@doris-robot
Copy link

TPC-H: Total hot run time: 50224 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 719037f91a6e78aea356359594f4049ebb7cea2a, data reload: false

------ Round 1 ----------------------------------
q1	17631	4350	4330	4330
q2	2055	149	146	146
q3	10577	1899	1936	1899
q4	10369	1255	1315	1255
q5	8419	3929	3943	3929
q6	230	123	124	123
q7	2061	1647	1604	1604
q8	9300	2728	2737	2728
q9	10565	10417	10303	10303
q10	8704	3585	3575	3575
q11	428	246	253	246
q12	469	295	301	295
q13	18366	3963	4116	3963
q14	363	342	338	338
q15	511	463	454	454
q16	701	612	603	603
q17	1122	1016	954	954
q18	7754	7091	7204	7091
q19	1822	1592	1557	1557
q20	548	334	290	290
q21	4738	4149	4180	4149
q22	494	392	418	392
Total cold run time: 117227 ms
Total hot run time: 50224 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4363	4335	4310	4310
q2	310	218	221	218
q3	4140	4165	4109	4109
q4	2726	2724	2740	2724
q5	7201	7130	7094	7094
q6	236	119	113	113
q7	3260	2921	2824	2824
q8	4315	4457	4500	4457
q9	16881	16866	16672	16672
q10	4250	4270	4254	4254
q11	765	678	681	678
q12	1032	845	842	842
q13	6490	3714	3759	3714
q14	451	423	428	423
q15	500	462	448	448
q16	754	676	670	670
q17	3764	3846	3879	3846
q18	8760	8734	8721	8721
q19	1697	1703	1632	1632
q20	2373	2140	2100	2100
q21	8476	8589	8480	8480
q22	1014	983	934	934
Total cold run time: 83758 ms
Total hot run time: 79263 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 202961 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 719037f91a6e78aea356359594f4049ebb7cea2a, data reload: false

query1	915	386	376	376
query2	6556	3075	2595	2595
query3	6923	201	203	201
query4	20427	17902	17869	17869
query5	19720	6498	6535	6498
query6	309	218	234	218
query7	4318	291	303	291
query8	247	220	243	220
query9	3093	2689	2616	2616
query10	433	273	305	273
query11	11466	10711	10784	10711
query12	121	71	69	69
query13	5578	670	675	670
query14	18054	13200	13604	13200
query15	362	219	222	219
query16	6451	269	258	258
query17	1534	1462	861	861
query18	2275	409	398	398
query19	204	139	145	139
query20	78	75	73	73
query21	188	96	90	90
query22	5312	5127	5009	5009
query23	32425	31804	31784	31784
query24	6718	6499	6487	6487
query25	498	421	419	419
query26	526	164	160	160
query27	1869	292	287	287
query28	6236	2371	2342	2342
query29	2781	2644	2714	2644
query30	239	158	161	158
query31	911	728	710	710
query32	71	58	58	58
query33	395	244	255	244
query34	836	474	474	474
query35	1118	893	903	893
query36	1203	1300	1031	1031
query37	88	61	54	54
query38	3117	2943	2898	2898
query39	1367	1308	1302	1302
query40	201	95	84	84
query41	38	39	35	35
query42	90	79	83	79
query43	738	816	758	758
query44	1123	724	735	724
query45	241	226	228	226
query46	1221	958	954	954
query47	1833	1766	1846	1766
query48	1033	702	702	702
query49	610	370	359	359
query50	874	631	595	595
query51	4829	4608	4704	4608
query52	98	73	84	73
query53	440	309	315	309
query54	2626	2481	2449	2449
query55	81	89	77	77
query56	214	199	204	199
query57	1214	1096	1098	1096
query58	214	198	198	198
query59	4046	3978	4015	3978
query60	209	186	191	186
query61	88	81	80	80
query62	825	526	468	468
query63	471	333	329	329
query64	2329	1575	1469	1469
query65	3678	3550	3568	3550
query66	761	368	378	368
query67	16379	16320	15619	15619
query68	10062	666	680	666
query69	597	361	361	361
query70	1765	1323	1370	1323
query71	416	304	303	303
query72	6514	3438	3400	3400
query73	744	328	323	323
query74	6436	5829	5840	5829
query75	5477	3811	3843	3811
query76	6262	1146	1191	1146
query77	1086	254	251	251
query78	12732	11640	11627	11627
query79	7036	685	648	648
query80	1241	409	409	409
query81	488	233	229	229
query82	1674	96	95	95
query83	173	134	131	131
query84	257	74	67	67
query85	860	296	302	296
query86	336	295	298	295
query87	3295	3056	3089	3056
query88	4730	2328	2320	2320
query89	387	323	283	283
query90	2048	214	211	211
query91	163	148	131	131
query92	60	54	52	52
query93	6030	576	557	557
query94	726	204	205	204
query95	1100	1078	1070	1070
query96	641	335	318	318
query97	6498	6380	6355	6355
query98	195	179	179	179
query99	2960	900	878	878
Total cold run time: 314455 ms
Total hot run time: 202961 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.54 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 719037f91a6e78aea356359594f4049ebb7cea2a, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.25	0.04	0.06
query4	1.79	0.08	0.07
query5	0.53	0.52	0.52
query6	1.38	0.61	0.62
query7	0.01	0.01	0.01
query8	0.03	0.02	0.02
query9	0.52	0.48	0.49
query10	0.54	0.54	0.55
query11	0.12	0.09	0.08
query12	0.12	0.09	0.10
query13	0.63	0.62	0.61
query14	0.77	0.80	0.77
query15	0.80	0.75	0.75
query16	0.36	0.38	0.36
query17	1.00	1.00	1.04
query18	0.24	0.26	0.26
query19	1.93	1.88	1.78
query20	0.01	0.01	0.01
query21	15.47	0.58	0.56
query22	1.98	2.05	1.60
query23	17.16	0.98	1.11
query24	5.55	0.90	1.48
query25	0.33	0.11	0.06
query26	0.67	0.14	0.15
query27	0.05	0.03	0.04
query28	7.48	0.73	0.70
query29	12.71	2.28	2.35
query30	0.54	0.52	0.52
query31	2.82	0.38	0.37
query32	3.39	0.49	0.49
query33	3.06	3.04	3.07
query34	15.27	4.76	4.78
query35	4.84	4.82	4.82
query36	1.07	1.01	1.01
query37	0.06	0.04	0.05
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.13	0.14
query41	0.07	0.01	0.02
query42	0.03	0.02	0.02
query43	0.02	0.01	0.02
Total cold run time: 103.91 s
Total hot run time: 30.54 s

@zzzxl1993
Copy link
Contributor Author

run p0

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 719037f91a6e78aea356359594f4049ebb7cea2a with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.1 seconds inserted 10000000 Rows, about 473K ops/s

@zzzxl1993
Copy link
Contributor Author

run p0

@xiaokang xiaokang merged commit 13d4f79 into apache:branch-2.0 May 3, 2024
23 of 25 checks passed
@xiaokang xiaokang changed the title [opt](inverted index) the "unicode" tokenizer can be configured to select stop words [opt](inverted index) the "unicode" tokenizer can be configured to select stop words #33982 May 3, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@zzzxl1993
[opt](inverted index) the "unicode" tokenizer can be configured to di…
4b06ac2 
…sable stop words apache#33982 (apache#34376)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
area/planner Issues or PRs related to the query planner kind/test
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@zzzxl1993 @doris-robot @xiaokang