Remove remote-store runs fro nyc_taxis and so #4021
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
After adding nyc_taxis and so workload runs for remote-store feature the nighty benchmark job is having failures while deleting the stacks after the benchmark run has finished.
During remote-store enabled cluster stack creation an S3 bucket is created with
DESTROY
deletion policy. In a normal scenario when the stack is deleted, a lambda, created during stack creation, deletes all the contents of the S3 bucket. Once the lambda has deleted the contents, it gets deleted first and then the S3 bucket gets deleted later on.With remote-store enabled run it seems once the lambda has deleted contents and got deleted, new data gets pushed into the S3 bucket and when cloud-formation tries to delete the resource it is thrown
Bucket not empty
error, which means even 30-40mins after indexing has finished there is some data that is still getting pushed to S3 bucket. This caused 100s of stacks to be stuck inDELETE_FAILED
state and had to be deleted manually.We are seeing these failures with nyc_taxis and so workload runs mostly, therefore removing them as precautionary measure.
Issues Resolved
List any issues this PR will resolve, e.g. Closes [...].
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.