Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[debug](ES Catalog)Add FE open API for ES Catalog (#40444) #40527

Merged
merged 1 commit into from
Sep 9, 2024
Merged

[debug](ES Catalog)Add FE open API for ES Catalog (#40444) #40527

merged 1 commit into from
Sep 9, 2024

Conversation

qidaye
Copy link
Contributor

@qidaye qidaye commented Sep 9, 2024

bp #40444

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@qidaye
Copy link
Contributor Author

qidaye commented Sep 9, 2024

run buildall

@qidaye
[feature](ES Catalog)Add FE open API for ES Catalog (apache#40444)
6e7f603 
Add api to get info from Elasticsearch, it's very useful for debugging.
1. get index mapping api
```
GET /rest/v2/api/es_catalog/get_mapping?catalog=xxx&table=xxx
```

2. search index api
```
POST /rest/v2/api/es_catalog/search?catalog=xxx&table=xxx
{
    request_body
    ...
}
```
@qidaye qidaye force-pushed the pick_add_http_action_for_es_2.0 branch from accb40a to 6e7f603 Compare September 9, 2024 04:16
@qidaye
Copy link
Contributor Author

qidaye commented Sep 9, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 48823 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6e7f603968ab53ea8aebabffbf1eb8ebbacfecc8, data reload: false

------ Round 1 ----------------------------------
q1	17894	4396	4360	4360
q2	2072	156	144	144
q3	10269	1859	1950	1859
q4	10129	1259	1314	1259
q5	8579	3841	3855	3841
q6	226	124	125	124
q7	2054	1639	1597	1597
q8	9517	2734	2721	2721
q9	13257	9769	9793	9769
q10	8661	3552	3504	3504
q11	417	242	253	242
q12	468	297	289	289
q13	18341	3924	4049	3924
q14	367	332	324	324
q15	503	455	453	453
q16	531	443	457	443
q17	1132	956	931	931
q18	7242	6765	6856	6765
q19	1661	1524	1473	1473
q20	544	301	306	301
q21	4369	4122	4103	4103
q22	497	397	400	397
Total cold run time: 118730 ms
Total hot run time: 48823 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4301	4334	4271	4271
q2	313	234	219	219
q3	4159	4160	4130	4130
q4	2763	2765	2734	2734
q5	7151	7056	7105	7056
q6	235	117	120	117
q7	3223	2887	2826	2826
q8	4373	4477	4486	4477
q9	13754	13776	13586	13586
q10	4226	4279	4216	4216
q11	767	706	692	692
q12	1042	850	836	836
q13	7052	3750	3748	3748
q14	456	425	434	425
q15	496	478	466	466
q16	649	579	568	568
q17	3818	3923	3844	3844
q18	8753	8763	8789	8763
q19	1704	1657	1660	1657
q20	2403	2114	2099	2099
q21	8524	8489	8426	8426
q22	1012	967	946	946
Total cold run time: 81174 ms
Total hot run time: 76102 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 211874 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6e7f603968ab53ea8aebabffbf1eb8ebbacfecc8, data reload: false

query1	926	391	406	391
query2	6566	2322	2169	2169
query3	6922	214	209	209
query4	23274	21498	21717	21498
query5	19733	6507	6510	6507
query6	291	221	232	221
query7	4326	305	305	305
query8	235	276	262	262
query9	3037	2647	2577	2577
query10	486	315	291	291
query11	15508	14846	15279	14846
query12	130	74	75	74
query13	1043	459	429	429
query14	17415	13319	13133	13133
query15	364	223	238	223
query16	6480	283	262	262
query17	1908	925	909	909
query18	901	323	315	315
query19	204	155	153	153
query20	81	78	78	78
query21	192	100	103	100
query22	5164	5053	4997	4997
query23	34226	33427	33522	33427
query24	7886	6349	6239	6239
query25	532	444	411	411
query26	1268	162	159	159
query27	2453	302	293	293
query28	6135	2285	2233	2233
query29	2920	2825	2670	2670
query30	242	178	169	169
query31	951	746	730	730
query32	72	56	63	56
query33	455	253	260	253
query34	860	469	492	469
query35	1143	956	926	926
query36	1391	1271	1174	1174
query37	170	62	60	60
query38	3075	2878	2967	2878
query39	1389	1320	1318	1318
query40	318	94	95	94
query41	39	39	45	39
query42	83	83	82	82
query43	609	639	572	572
query44	1179	721	712	712
query45	245	234	233	233
query46	1243	968	956	956
query47	1822	1747	1815	1747
query48	515	413	404	404
query49	657	392	385	385
query50	858	594	579	579
query51	4715	4660	4679	4660
query52	84	86	70	70
query53	231	191	182	182
query54	2632	2471	2468	2468
query55	95	84	85	84
query56	219	204	220	204
query57	1201	1241	1100	1100
query58	233	212	201	201
query59	3498	3171	3239	3171
query60	220	223	207	207
query61	101	99	100	99
query62	851	498	465	465
query63	204	177	181	177
query64	3582	1612	1296	1296
query65	3664	3545	3590	3545
query66	770	445	422	422
query67	15776	16151	15578	15578
query68	9158	669	668	668
query69	477	270	270	270
query70	1492	1355	1454	1355
query71	383	305	308	305
query72	6817	4817	4625	4625
query73	750	332	325	325
query74	6320	5870	5883	5870
query75	4811	3726	3650	3650
query76	4892	1188	1204	1188
query77	692	264	253	253
query78	12321	11495	11850	11495
query79	7258	659	657	657
query80	2062	384	377	377
query81	499	246	239	239
query82	1696	100	98	98
query83	172	131	135	131
query84	257	74	71	71
query85	1180	323	322	322
query86	332	301	345	301
query87	3198	3021	3008	3008
query88	4840	2315	2312	2312
query89	407	287	292	287
query90	1799	216	221	216
query91	160	125	126	125
query92	60	56	54	54
query93	5390	568	545	545
query94	709	214	219	214
query95	2031	1987	1993	1987
query96	628	334	335	334
query97	6431	6385	6396	6385
query98	236	209	203	203
query99	2873	938	858	858
Total cold run time: 316329 ms
Total hot run time: 211874 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.97 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6e7f603968ab53ea8aebabffbf1eb8ebbacfecc8, data reload: false

query1	0.02	0.02	0.02
query2	0.06	0.03	0.03
query3	0.25	0.05	0.05
query4	1.79	0.09	0.10
query5	0.54	0.52	0.53
query6	1.24	0.62	0.63
query7	0.02	0.01	0.01
query8	0.04	0.03	0.02
query9	0.52	0.49	0.48
query10	0.54	0.53	0.54
query11	0.11	0.09	0.09
query12	0.13	0.09	0.10
query13	0.63	0.61	0.60
query14	0.79	0.78	0.79
query15	0.78	0.76	0.77
query16	0.38	0.36	0.39
query17	1.02	1.02	1.00
query18	0.22	0.26	0.24
query19	1.87	1.88	1.86
query20	0.02	0.01	0.01
query21	15.48	0.55	0.56
query22	1.93	2.60	1.76
query23	17.40	0.99	0.99
query24	7.96	1.11	0.87
query25	0.37	0.16	0.05
query26	0.79	0.16	0.15
query27	0.05	0.04	0.04
query28	5.24	0.73	0.77
query29	12.77	2.40	2.36
query30	0.62	0.54	0.53
query31	2.80	0.39	0.37
query32	3.39	0.50	0.50
query33	3.07	3.10	3.03
query34	15.27	4.77	4.79
query35	4.86	4.84	4.84
query36	1.06	1.01	1.02
query37	0.06	0.05	0.04
query38	0.03	0.02	0.02
query39	0.02	0.02	0.02
query40	0.16	0.14	0.15
query41	0.07	0.01	0.02
query42	0.02	0.02	0.01
query43	0.03	0.02	0.02
Total cold run time: 104.42 s
Total hot run time: 30.97 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 6e7f603968ab53ea8aebabffbf1eb8ebbacfecc8 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.9 seconds inserted 10000000 Rows, about 456K ops/s

@xiaokang xiaokang changed the title [feature](ES Catalog)Add FE open API for ES Catalog (#40444) [debug](ES Catalog)Add FE open API for ES Catalog (#40444) Sep 9, 2024
@xiaokang xiaokang merged commit d6c073b into apache:branch-2.0 Sep 9, 2024
22 of 24 checks passed
@qidaye qidaye deleted the pick_add_http_action_for_es_2.0 branch September 9, 2024 06:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
kind/test
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@qidaye @doris-robot @xiaokang