Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](row-count-cache) use cached row count for show #33911

Merged
merged 1 commit into from
Apr 20, 2024

Conversation

morningman
Copy link
Contributor

@morningman morningman commented Apr 19, 2024

Proposed changes

Followup #33449

Some operation, like show tables status may traverse all tables and call table's getRowCount() method.
This method may trigger the row count cache refresh async, and may take all executor thread and block other
query requests.

This PR change the logic, for some show request like show table status, use a new method getCachedRowCount().
This method will get row count from cache if present, and will not trigger cache refresh if cache miss.

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39001 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cfde1e300992ccf0ff1a0f4d037fb792a77a1478, data reload: false

------ Round 1 ----------------------------------
q1	17719	6362	4410	4410
q2	2532	192	191	191
q3	12925	1135	1179	1135
q4	10746	740	757	740
q5	8213	2742	2713	2713
q6	230	138	141	138
q7	1035	631	616	616
q8	9426	2104	2076	2076
q9	7523	6620	6552	6552
q10	8441	3534	3523	3523
q11	454	240	229	229
q12	405	223	217	217
q13	17782	2939	2969	2939
q14	272	230	231	230
q15	528	499	481	481
q16	521	384	384	384
q17	965	747	731	731
q18	7415	6938	6711	6711
q19	4856	1520	1495	1495
q20	638	316	307	307
q21	3530	2934	2876	2876
q22	357	311	307	307
Total cold run time: 116513 ms
Total hot run time: 39001 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4348	4241	4209	4209
q2	382	270	274	270
q3	3011	2770	2755	2755
q4	1880	1608	1582	1582
q5	5306	5292	5299	5292
q6	216	125	124	124
q7	2263	1875	1903	1875
q8	3218	3360	3354	3354
q9	8611	8543	8564	8543
q10	3895	3712	3727	3712
q11	592	478	489	478
q12	783	574	587	574
q13	17349	2887	2971	2887
q14	300	272	276	272
q15	505	476	477	476
q16	456	416	416	416
q17	1752	1471	1471	1471
q18	7645	7578	7339	7339
q19	1637	1504	1516	1504
q20	1946	1748	1746	1746
q21	4925	4727	4807	4727
q22	553	471	456	456
Total cold run time: 71573 ms
Total hot run time: 54062 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184073 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cfde1e300992ccf0ff1a0f4d037fb792a77a1478, data reload: false

query1	927	369	371	369
query2	6495	2673	2445	2445
query3	6656	204	207	204
query4	22825	21426	21276	21276
query5	4135	401	400	400
query6	267	169	180	169
query7	4598	286	288	286
query8	244	196	189	189
query9	8701	2341	2325	2325
query10	601	262	289	262
query11	14768	14159	14145	14145
query12	141	95	88	88
query13	1640	364	356	356
query14	9956	7858	7483	7483
query15	270	183	190	183
query16	8175	258	252	252
query17	1993	594	551	551
query18	2106	271	266	266
query19	334	149	147	147
query20	91	88	86	86
query21	195	124	122	122
query22	5041	4799	4838	4799
query23	33880	33351	33280	33280
query24	11032	2854	2947	2854
query25	635	361	381	361
query26	1516	151	148	148
query27	2918	309	317	309
query28	7618	2010	1997	1997
query29	954	612	593	593
query30	304	168	172	168
query31	982	735	714	714
query32	86	51	55	51
query33	750	246	239	239
query34	1029	466	478	466
query35	834	674	681	674
query36	1064	834	897	834
query37	129	68	72	68
query38	3342	3239	3145	3145
query39	1573	1563	1545	1545
query40	306	125	121	121
query41	46	43	44	43
query42	106	100	93	93
query43	564	528	537	528
query44	1189	720	719	719
query45	272	268	278	268
query46	1078	698	710	698
query47	1934	1873	1826	1826
query48	369	282	290	282
query49	1144	380	375	375
query50	754	367	377	367
query51	6829	6756	6804	6756
query52	98	86	88	86
query53	346	290	279	279
query54	288	245	222	222
query55	75	71	69	69
query56	237	216	218	216
query57	1210	1132	1179	1132
query58	228	190	193	190
query59	3473	3319	3059	3059
query60	243	224	224	224
query61	89	87	86	86
query62	639	454	437	437
query63	299	275	271	271
query64	6218	3767	3804	3767
query65	3082	3052	3038	3038
query66	1394	342	325	325
query67	15665	14862	15061	14862
query68	5325	521	536	521
query69	485	299	297	297
query70	1270	1186	1093	1093
query71	1390	1266	1256	1256
query72	6412	2597	2411	2411
query73	700	314	313	313
query74	6777	6444	6456	6444
query75	3358	2673	2574	2574
query76	3403	1029	932	932
query77	381	259	257	257
query78	10980	10238	10137	10137
query79	2796	520	523	520
query80	1941	421	421	421
query81	524	242	242	242
query82	773	103	98	98
query83	262	166	169	166
query84	266	90	89	89
query85	2038	270	306	270
query86	495	305	305	305
query87	3481	3317	3319	3317
query88	4405	2314	2300	2300
query89	467	375	362	362
query90	2015	179	180	179
query91	129	96	96	96
query92	64	46	48	46
query93	2833	506	493	493
query94	1219	176	172	172
query95	384	291	294	291
query96	578	264	266	264
query97	3128	2952	2947	2947
query98	241	219	221	219
query99	1226	857	853	853
Total cold run time: 286726 ms
Total hot run time: 184073 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.27 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cfde1e300992ccf0ff1a0f4d037fb792a77a1478, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.03	0.04
query3	0.23	0.04	0.05
query4	1.66	0.07	0.08
query5	0.50	0.49	0.52
query6	1.44	0.72	0.71
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.49	0.51
query10	0.54	0.55	0.54
query11	0.16	0.11	0.12
query12	0.14	0.12	0.12
query13	0.59	0.59	0.59
query14	0.76	0.80	0.76
query15	0.81	0.82	0.81
query16	0.36	0.36	0.37
query17	0.97	0.97	0.94
query18	0.20	0.26	0.20
query19	1.81	1.69	1.84
query20	0.01	0.00	0.01
query21	15.42	0.67	0.65
query22	4.27	7.77	1.87
query23	18.26	1.34	1.30
query24	1.65	0.27	0.20
query25	0.14	0.08	0.09
query26	0.27	0.17	0.17
query27	0.08	0.07	0.08
query28	13.41	1.01	0.98
query29	12.66	3.29	3.29
query30	0.27	0.06	0.05
query31	2.88	0.38	0.37
query32	3.28	0.46	0.46
query33	2.80	2.86	2.88
query34	17.00	4.40	4.43
query35	4.45	4.55	4.46
query36	0.65	0.46	0.45
query37	0.18	0.15	0.16
query38	0.16	0.14	0.15
query39	0.04	0.04	0.04
query40	0.16	0.14	0.14
query41	0.10	0.05	0.04
query42	0.06	0.04	0.05
query43	0.04	0.03	0.04
Total cold run time: 109.14 s
Total hot run time: 30.27 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit cfde1e300992ccf0ff1a0f4d037fb792a77a1478 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      33 seconds loaded 861443392 Bytes, about 24 MB/s
Insert into select:       13.5 seconds inserted 10000000 Rows, about 740K ops/s

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 20, 2024
Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xiaokang xiaokang merged commit 2107bc5 into apache:master Apr 20, 2024
26 of 28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.3-merged p0_b reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants