Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(chunker): reduce overall memory use #649

Merged
merged 7 commits into from
Aug 21, 2024
Merged

Commits on Aug 1, 2024

  1. Reduce overall chunker memory use

    When the chunker is not able to fill a chunk with data, it allocated a new buffer for the partial chunk of data. With many files this results in allocation of many small buffers of varying sizes, leading to heap fragmentation. This PR allocates a new buffer from the pool only if doing so would save space, otherwise it uses an partially filled (over-allocated) chunk. This makes all chunker allocation sizes be powers of 2.
    
    Heap fragmentation is reduced at the cost of some temporary over-allocation. The advantage is that the overallocation is much shorter lived than the heap fragmentation.
    gammazero committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    76b3134 View commit details
    Browse the repository at this point in the history
  2. update changelog

    gammazero committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    32a7fbc View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2024

  1. Configuration menu
    Copy the full SHA
    30beb29 View commit details
    Browse the repository at this point in the history

Commits on Aug 5, 2024

  1. Configuration menu
    Copy the full SHA
    5f2dbb0 View commit details
    Browse the repository at this point in the history

Commits on Aug 6, 2024

  1. Configuration menu
    Copy the full SHA
    83c9669 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    b425ba2 View commit details
    Browse the repository at this point in the history

Commits on Aug 20, 2024

  1. Configuration menu
    Copy the full SHA
    a5a4645 View commit details
    Browse the repository at this point in the history