Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Keep the log fd, don't re-open logfile in every logs #906

Open
wants to merge 3 commits into
base: unstable
Choose a base branch
from

Conversation

enjoy-binbin
Copy link
Member

@enjoy-binbin enjoy-binbin commented Aug 14, 2024

Currently in serverLog, we fopen and fclose the logfile in every
call, if the log prints too much, it will waste performance. Also
frequent open may cause blocking in some caes, like doing a big
rename in the same dir (like rename the bgsave temp RDB).

In this PR, we keep the log fd, and only re-open it when dir changed.
This also improve performance, using benchmark DEBUG log shows that
it improves the performance by 3 times.

Currently in serverLog, we fopen and fclose the logfile in every
call, if the log prints too much, it will waste performance. Also
frequent open may cause blocking in some caes. We have encountered
open blocking for hundreds of milliseconds to seconds.

In this PR, we keep the log fd, and only re-open it when dir changed.
This also improve performance, using benchmark DEBUG log shows that
it improves the performance by 3 times.

Signed-off-by: Binbin <[email protected]>
Signed-off-by: Binbin <[email protected]>
Signed-off-by: Binbin <[email protected]>
Copy link

codecov bot commented Aug 14, 2024

Codecov Report

Attention: Patch coverage is 90.32258% with 3 lines in your changes missing coverage. Please review.

Project coverage is 70.35%. Comparing base (370bdb3) to head (3c37799).
Report is 42 commits behind head on unstable.

Files Patch % Lines
src/config.c 60.00% 2 Missing ⚠️
src/server.c 95.83% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff            @@
##           unstable     #906   +/-   ##
=========================================
  Coverage     70.34%   70.35%           
=========================================
  Files           112      112           
  Lines         61490    61492    +2     
=========================================
+ Hits          43258    43265    +7     
+ Misses        18232    18227    -5     
Files Coverage Δ
src/debug.c 54.37% <100.00%> (+0.39%) ⬆️
src/server.h 100.00% <ø> (ø)
src/server.c 88.56% <95.83%> (+0.02%) ⬆️
src/config.c 78.93% <60.00%> (+0.23%) ⬆️

... and 9 files with indirect coverage changes

@enjoy-binbin
Copy link
Member Author

@madolson @zuiderkwast @PingXie do you guys want to takes a look with this? i would like to include it in the RC3

Copy link
Contributor

@zuiderkwast zuiderkwast left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we want it in 8.0, it's better in rc2 than later.

@madolson
Copy link
Member

This is a breaking change IIRC, if there are existing mechanisms for log rotation we need to send a signal to the process to close and re-open the log. We discussed this at length in Redis, but never committed to it.

@zuiderkwast zuiderkwast added the breaking-change Indicates a possible backwards incompatible change label Aug 27, 2024
@zuiderkwast
Copy link
Contributor

Good point @madolson. I think we can't do this change without a more advanced solution then.

We can use ionotify on linux and kqueue on BSD/MacOS to watch the file and be notified when it is renamed. Then we close and reopen it. Fallback to the old behaviour if no watch method is available.

@enjoy-binbin
Copy link
Member Author

ok, i see the point, indeed in this case, we need to find a way to re-open the file.

@PingXie
Copy link
Member

PingXie commented Aug 28, 2024

This is a breaking change IIRC, if there are existing mechanisms for log rotation we need to send a signal to the process to close and re-open the log. We discussed this at length in Redis, but never committed to it.

That is great information. I think it would be very helpful if we could leave this context in serverLogRaw.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking-change Indicates a possible backwards incompatible change
Projects
Status: Next Major release
Development

Successfully merging this pull request may close these issues.

4 participants