Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove remote-store runs fro nyc_taxis and so #4021

Merged
merged 1 commit into from
Sep 14, 2023

Conversation

rishabh6788
Copy link
Collaborator

@rishabh6788 rishabh6788 commented Sep 14, 2023

Description

After adding nyc_taxis and so workload runs for remote-store feature the nighty benchmark job is having failures while deleting the stacks after the benchmark run has finished.
During remote-store enabled cluster stack creation an S3 bucket is created with DESTROY deletion policy. In a normal scenario when the stack is deleted, a lambda, created during stack creation, deletes all the contents of the S3 bucket. Once the lambda has deleted the contents, it gets deleted first and then the S3 bucket gets deleted later on.

With remote-store enabled run it seems once the lambda has deleted contents and got deleted, new data gets pushed into the S3 bucket and when cloud-formation tries to delete the resource it is thrown Bucket not empty error, which means even 30-40mins after indexing has finished there is some data that is still getting pushed to S3 bucket. This caused 100s of stacks to be stuck in DELETE_FAILED state and had to be deleted manually.

We are seeing these failures with nyc_taxis and so workload runs mostly, therefore removing them as precautionary measure.

Issues Resolved

List any issues this PR will resolve, e.g. Closes [...].

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@codecov
Copy link

codecov bot commented Sep 14, 2023

Codecov Report

Merging #4021 (177197c) into main (7e1ab90) will decrease coverage by 0.29%.
Report is 1 commits behind head on main.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #4021      +/-   ##
==========================================
- Coverage   92.10%   91.82%   -0.29%     
==========================================
  Files         187      187              
  Lines        5674     5674              
==========================================
- Hits         5226     5210      -16     
- Misses        448      464      +16     

see 2 files with indirect coverage changes

@rishabh6788 rishabh6788 merged commit 12ce128 into opensearch-project:main Sep 14, 2023
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants