Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add negative cache for folders #2530

Merged
merged 35 commits into from
Oct 9, 2024
Merged

Add negative cache for folders #2530

merged 35 commits into from
Oct 9, 2024

Conversation

Tulsishah
Copy link
Collaborator

@Tulsishah Tulsishah commented Sep 24, 2024

Description

We're adding a negative stat cache for folders to reduce GCS calls for negative entries. Although we initially implemented functions for this, we encountered corner cases that needed addressing, which didn't fit within the release timeframe. Therefore, @ankitaluthra1 and I decided to postpone this feature as a post-launch item, as it's not a release blocker.

Scenarios:
BqdmEy5DnjUtiP5
We can say that StatObject(A) will cache a negative folder entry for "A" because "A" can only exist as an object, not a folder. Similarly, GetFolder(A/) will cache a negative object entry for "A/" as it can only exist as a folder, not an object, in HNS with GCSFuse.

Other changes

  • Replaced the negative cache entry with an invalidation step in the DeleteFolder function. This change aligns DeleteFolder with the existing DeleteObject function, which also uses invalidation. (Because adding negative cache was changing behaviour compare to object deletion)
  • Added logic to erase cache entries for renamed folders as part of the stale data cleanup process. Added stat call to fill negative cache in composite test.
  • Implicit_dir test changes - The test were failing due to all the tests were performing operations on single folder. One test were filling negative cache and second one try to access it. So I created separate folder for each test to avoid failure.

Link to the issue in case of a bug fix.

NA

Testing details

  1. Manual - NA
  2. Unit tests - NA
  3. Integration tests - Automated

@Tulsishah Tulsishah added the execute-integration-tests Run only integration tests label Sep 25, 2024
Copy link

codecov bot commented Sep 25, 2024

Codecov Report

Attention: Patch coverage is 83.33333% with 4 lines in your changes missing coverage. Please review.

Project coverage is 78.22%. Comparing base (4f40a8f) to head (f395dc5).
Report is 19 commits behind head on master.

Files with missing lines Patch % Lines
internal/storage/caching/fast_stat_bucket.go 76.47% 2 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2530      +/-   ##
==========================================
- Coverage   78.83%   78.22%   -0.61%     
==========================================
  Files         106      107       +1     
  Lines       11700    11800     +100     
==========================================
+ Hits         9224     9231       +7     
- Misses       1995     2076      +81     
- Partials      481      493      +12     
Flag Coverage Δ
unittests 78.22% <83.33%> (-0.61%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Tulsishah Tulsishah marked this pull request as ready for review October 3, 2024 07:31
@Tulsishah Tulsishah requested a review from a team as a code owner October 3, 2024 07:31
@kislaykishore kislaykishore requested review from a team, BrennaEpp and gargnitingoogle and removed request for a team and BrennaEpp October 3, 2024 07:32
@Tulsishah Tulsishah merged commit f2d240d into master Oct 9, 2024
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
execute-integration-tests Run only integration tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants