Huge defrag smooth linear threshold #9825

robdrynkin · 2024-09-26T16:22:59Z

Not for changelog (changelog entry is not required)

github-actions · 2024-09-26T16:26:35Z

⚪ 2024-09-26 16:26:35 UTC Pre-commit check linux-x86_64-relwithdebinfo for fb951d0 has started.
⚪ 2024-09-26 16:26:44 UTC Artifacts will be uploaded here
⚪ 2024-09-26 16:29:57 UTC ya make is running...
⚫ 2024-09-26 16:40:09 UTC Check cancelled

github-actions · 2024-09-26T16:27:17Z

⚪ 2024-09-26 16:27:17 UTC Pre-commit check linux-x86_64-release-asan for fb951d0 has started.
⚪ 2024-09-26 16:27:27 UTC Artifacts will be uploaded here
⚪ 2024-09-26 16:30:36 UTC ya make is running...
⚫ 2024-09-26 16:40:08 UTC Check cancelled

github-actions · 2024-09-26T16:49:28Z

⚪ 2024-09-26 16:49:27 UTC Pre-commit check linux-x86_64-release-asan for 2dcbac4 has started.
⚪ 2024-09-26 16:49:39 UTC Artifacts will be uploaded here
⚪ 2024-09-26 16:52:47 UTC ya make is running...
🟡 2024-09-26 19:18:35 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
11941	11725	0	47	48	121

🟢 2024-09-26 19:19:40 UTC Build successful.
🟢 2024-09-26 19:20:14 UTC ydbd size 5.7 GiB changed* by +25.3 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `c0ce5ac`	merge: `2dcbac4`	diff	diff %
ydbd size	6 085 664 896 Bytes	6 085 690 760 Bytes	+25.3 KiB	+0.000%
ydbd stripped size	1 521 101 936 Bytes	1 521 105 136 Bytes	+3.1 KiB	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-09-26T16:58:57Z

⚪ 2024-09-26 16:58:56 UTC Pre-commit check linux-x86_64-relwithdebinfo for 2dcbac4 has started.
⚪ 2024-09-26 16:59:45 UTC Artifacts will be uploaded here
⚪ 2024-09-26 17:03:37 UTC ya make is running...
🟡 2024-09-26 18:49:57 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
41343	35764	0	11	5459	109

⚪ 2024-09-26 18:53:39 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-09-26 19:05:06 UTC Tests successful.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
119 (only retried tests)	29	0	0	0	90

🟢 2024-09-26 19:05:12 UTC Build successful.
🟢 2024-09-26 19:05:52 UTC ydbd size 8.5 GiB changed* by +16.4 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `c0ce5ac`	merge: `2dcbac4`	diff	diff %
ydbd size	9 083 040 816 Bytes	9 083 057 656 Bytes	+16.4 KiB	+0.000%
ydbd stripped size	490 936 488 Bytes	490 937 576 Bytes	+1.1 KiB	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

va-kuznecov · 2024-09-27T13:49:57Z

ydb/core/protos/config.proto

+            Description: "Huge garbage threshold = DefaultHugeGarbagePerMille * min((free space share) / (this param), 1)",
+            MinValue: 1,
+            MaxValue: 1000,
+            DefaultValue: 130 }];


@alexvru, we have a problem

If we set 130 here, it means that on clusters, where color is CYAN, defragmentation will stop.

if we set 260 here, defragmentation on CYAN will be the same as now, but for clusters with occupancy > 74% new defragmentation will start and it may lead to significant performance degradation.

what do you think?

alexvru · 2024-10-16T07:23:39Z

ydb/core/blobstorage/vdisk/defrag/defrag_actor.cpp

+    bool HugeHeapDefragmentationRequired(
+        ui32 hugeCanBeFreedChunks,
+        ui32 hugeTotalChunks,
+        float defragThreshold) {


May be "double" here to make it uniform?

github-actions · 2024-10-21T16:57:23Z

⚪ 2024-10-21 16:57:22 UTC Pre-commit check linux-x86_64-release-asan for fcb7c33 has started.
⚪ 2024-10-21 16:57:41 UTC Artifacts will be uploaded here
⚪ 2024-10-21 17:01:04 UTC ya make is running...
🟡 2024-10-21 18:57:53 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
11736	11482	0	81	61	112

🟢 2024-10-21 18:58:53 UTC Build successful.
🟢 2024-10-21 18:59:24 UTC ydbd size 5.7 GiB changed* by -10.1 MiB, which is <= 0 Bytes vs main: OK

ydbd size dash	main: `989a4fa`	merge: `fcb7c33`	diff	diff %
ydbd size	6 109 603 216 Bytes	6 099 030 176 Bytes	-10.1 MiB	-0.173%
ydbd stripped size	1 522 866 288 Bytes	1 522 362 288 Bytes	-492.2 KiB	-0.033%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-10-21T16:57:24Z

⚪ 2024-10-21 16:57:23 UTC Pre-commit check linux-x86_64-relwithdebinfo for fcb7c33 has started.
⚪ 2024-10-21 16:58:05 UTC Artifacts will be uploaded here
⚪ 2024-10-21 17:01:25 UTC ya make is running...
🟡 2024-10-21 18:27:24 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
41552	35947	0	7	5502	96

⚪ 2024-10-21 18:31:02 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-10-21 18:42:31 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
101 (only retried tests)	4	0	4	0	93

⚪ 2024-10-21 18:42:40 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-10-21 18:53:55 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
97 (only retried tests)	1	0	4	0	92

🟢 2024-10-21 18:54:02 UTC Build successful.
🟢 2024-10-21 18:54:24 UTC ydbd size 2.8 GiB changed* by +9.9 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `9865b79`	merge: `fcb7c33`	diff	diff %
ydbd size	3 011 980 040 Bytes	3 011 990 200 Bytes	+9.9 KiB	+0.000%
ydbd stripped size	478 072 600 Bytes	478 073 688 Bytes	+1.1 KiB	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-10-22T16:22:53Z

⚪ 2024-10-22 16:22:53 UTC Pre-commit check linux-x86_64-release-asan for b74c20d has started.
⚪ 2024-10-22 16:22:57 UTC Artifacts will be uploaded here
⚪ 2024-10-22 16:26:22 UTC ya make is running...
🟡 2024-10-22 18:17:10 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
11689	11489	0	43	50	107

🟢 2024-10-22 18:18:11 UTC Build successful.
🟢 2024-10-22 18:18:42 UTC ydbd size 5.7 GiB changed* by +24.7 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `7a1e697`	merge: `b74c20d`	diff	diff %
ydbd size	6 099 697 000 Bytes	6 099 722 312 Bytes	+24.7 KiB	+0.000%
ydbd stripped size	1 522 515 568 Bytes	1 522 518 384 Bytes	+2.8 KiB	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-10-22T16:25:23Z

⚪ 2024-10-22 16:25:22 UTC Pre-commit check linux-x86_64-relwithdebinfo for b74c20d has started.
⚪ 2024-10-22 16:25:34 UTC Artifacts will be uploaded here
⚪ 2024-10-22 16:28:56 UTC ya make is running...
🟡 2024-10-22 17:59:37 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
41554	35945	0	8	5503	98

⚪ 2024-10-22 18:03:14 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-10-22 18:14:27 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
106 (only retried tests)	9	0	3	1	93

⚪ 2024-10-22 18:14:35 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-10-22 18:25:44 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
96 (only retried tests)	1	0	3	0	92

🟢 2024-10-22 18:25:51 UTC Build successful.
🟢 2024-10-22 18:26:13 UTC ydbd size 2.8 GiB changed* by +9.7 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `7a1e697`	merge: `b74c20d`	diff	diff %
ydbd size	3 012 201 312 Bytes	3 012 211 248 Bytes	+9.7 KiB	+0.000%
ydbd stripped size	478 100 408 Bytes	478 101 432 Bytes	+1.0 KiB	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-11-06T15:15:47Z

⚪ 2024-11-06 15:15:46 UTC Pre-commit check linux-x86_64-relwithdebinfo for 9782b73 has started.
⚪ 2024-11-06 15:16:00 UTC Artifacts will be uploaded here
⚪ 2024-11-06 15:19:30 UTC ya make is running...
🟡 2024-11-06 16:47:11 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
41749	36084	0	4	5558	103

⚪ 2024-11-06 16:50:51 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-11-06 17:02:08 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
106 (only retried tests)	9	0	0	0	97

🟢 2024-11-06 17:02:15 UTC Build successful.
🟢 2024-11-06 17:02:35 UTC ydbd size 2.8 GiB changed* by +13.7 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `e6ad659`	merge: `9782b73`	diff	diff %
ydbd size	3 044 956 248 Bytes	3 044 970 296 Bytes	+13.7 KiB	+0.000%
ydbd stripped size	481 937 784 Bytes	481 942 904 Bytes	+5.0 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-11-06T15:16:31Z

⚪ 2024-11-06 15:16:31 UTC Pre-commit check linux-x86_64-release-asan for 9782b73 has started.
⚪ 2024-11-06 15:16:49 UTC Artifacts will be uploaded here
⚪ 2024-11-06 15:20:16 UTC ya make is running...
🟡 2024-11-06 17:00:07 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
10695	10612	0	31	16	36

🟢 2024-11-06 17:01:05 UTC Build successful.
🟢 2024-11-06 17:01:41 UTC ydbd size 5.7 GiB changed* by +24.8 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `e6ad659`	merge: `9782b73`	diff	diff %
ydbd size	6 161 849 648 Bytes	6 161 874 992 Bytes	+24.8 KiB	+0.000%
ydbd stripped size	1 536 453 712 Bytes	1 536 456 592 Bytes	+2.8 KiB	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions bot added the not-for-changelog label Sep 26, 2024

va-kuznecov reviewed Sep 27, 2024

View reviewed changes

alexvru requested changes Oct 16, 2024

View reviewed changes

alexvru previously approved these changes Oct 18, 2024

View reviewed changes

robdrynkin added 4 commits October 18, 2024 14:02

Huge defrag smooth linear threshold

121b558

Add defrag threshold sensor

bd817f4

float -> double

51157be

Decrease number of inflyght chunks to defrag while disk space green

1ab736a

robdrynkin dismissed alexvru’s stale review via 1ab736a October 21, 2024 16:55

robdrynkin force-pushed the defrag_actor_threshold branch from 77c9665 to 1ab736a Compare October 21, 2024 16:55

remove verify

ead5a96

alexvru previously approved these changes Oct 23, 2024

View reviewed changes

continue

1f63b0c

robdrynkin dismissed alexvru’s stale review via 1f63b0c November 6, 2024 15:15

va-kuznecov approved these changes Nov 7, 2024

View reviewed changes

robdrynkin merged commit 8f0d66f into ydb-platform:main Nov 7, 2024
10 checks passed

GrigoriyPA mentioned this pull request Nov 7, 2024

YQ RD optimized purecalc memory usage #11394

Merged

niksaveliev mentioned this pull request Nov 8, 2024

Fix read session correct close ut #11407

Merged

This was referenced Nov 10, 2024

YQ RD enable LLVM in purecalc filters #11442

Merged

YQ WM fixed race with actor context #11576

Merged

niksaveliev mentioned this pull request Nov 14, 2024

Commit for autopartitioned topics #11629

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Huge defrag smooth linear threshold #9825

Huge defrag smooth linear threshold #9825

robdrynkin commented Sep 26, 2024

github-actions bot commented Sep 26, 2024 •

edited

Loading

github-actions bot commented Sep 26, 2024 •

edited

Loading

github-actions bot commented Sep 26, 2024 •

edited

Loading

github-actions bot commented Sep 26, 2024 •

edited

Loading

va-kuznecov Sep 27, 2024

alexvru Oct 16, 2024

github-actions bot commented Oct 21, 2024 •

edited

Loading

github-actions bot commented Oct 21, 2024 •

edited

Loading

github-actions bot commented Oct 22, 2024 •

edited

Loading

github-actions bot commented Oct 22, 2024 •

edited

Loading

github-actions bot commented Nov 6, 2024 •

edited

Loading

github-actions bot commented Nov 6, 2024 •

edited

Loading

Huge defrag smooth linear threshold #9825

Huge defrag smooth linear threshold #9825

Conversation

robdrynkin commented Sep 26, 2024

github-actions bot commented Sep 26, 2024 • edited Loading

github-actions bot commented Sep 26, 2024 • edited Loading

github-actions bot commented Sep 26, 2024 • edited Loading

github-actions bot commented Sep 26, 2024 • edited Loading

va-kuznecov Sep 27, 2024

Choose a reason for hiding this comment

alexvru Oct 16, 2024

Choose a reason for hiding this comment

github-actions bot commented Oct 21, 2024 • edited Loading

github-actions bot commented Oct 21, 2024 • edited Loading

github-actions bot commented Oct 22, 2024 • edited Loading

github-actions bot commented Oct 22, 2024 • edited Loading

github-actions bot commented Nov 6, 2024 • edited Loading

github-actions bot commented Nov 6, 2024 • edited Loading

github-actions bot commented Sep 26, 2024 •

edited

Loading

github-actions bot commented Sep 26, 2024 •

edited

Loading

github-actions bot commented Sep 26, 2024 •

edited

Loading

github-actions bot commented Sep 26, 2024 •

edited

Loading

github-actions bot commented Oct 21, 2024 •

edited

Loading

github-actions bot commented Oct 21, 2024 •

edited

Loading

github-actions bot commented Oct 22, 2024 •

edited

Loading

github-actions bot commented Oct 22, 2024 •

edited

Loading

github-actions bot commented Nov 6, 2024 •

edited

Loading

github-actions bot commented Nov 6, 2024 •

edited

Loading