Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Task: Add test for simulating OOM error during merge equality deletes #2

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

nicole-martinez
Copy link

  • This PR adds a new test (testMergeEqualityDeletesOOM) to handle out-of-memory scenarios during merge equality deletes.
  • The test generates a large dataset and attempts to merge equality deletes.
  • When we run the test, we get an out-of-memory error caused by heap space, which is expected in this scenario. However, if the print statements are removed, the test is marked as "ignored."
    image

@nicole-martinez
Copy link
Author

  • Made a new commit that adds a new test case, testMergeEqualityDeletesOOM_v2, designed to simulate memory pressure during equality delete operations by committing multiple smaller delete files rather than a single large one, as was done in testMergeEqualityDeletesOOM. Specifically, we have specified the number and size of each equality delete file in Lines 326 and 327:
    int fileCount = 10; // Number of smaller delete files
    int recordsPerFile = 1_000_000;.

Key functionalities that were not in the original test include: file size logging and memory consumption logging

  • Each equality delete file's size is checked and logged to provide insight into how this approach impacts the memory consumption. This is done in lines 350-353: deleteFile.length() returns the file size in bytes and then we divide by (1024 * 1024) to convert bytes to megabytes.
  • We logged the memory usage (Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory()) before and after the simulated SELECT * read operation. We opted for this method of memory consumption check instead of using the Profiler Tool in IntelliJ because not everyone has access to Profiler on our team.

Current Issue:

  • Note: we set VM memory to 1024 MB
  • Getting negative consumption in some test cases. I have attached some screenshots below of both positive and negative consumption examples.
PNG image PNG image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant