Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fixed chunk size #173

Merged
merged 13 commits into from
Mar 6, 2024
Merged

fixed chunk size #173

merged 13 commits into from
Mar 6, 2024

Commits on Mar 4, 2024

  1. WIP: fixed chunk size

    We have a fair amount of complexity (EqualSplit etc) involved in dynamically
    splitting files down into chunks based on available concurrency.
    
    We could be less clever and use a fixed chunk size, with a fixed buffer size to
    go with it.  This would dovetail nicely with a sync.Pool of fixed-size buffers,
    and also with a configuration option to limit the total allocated
    RAM (implemented as a semaphore guarding how many buffers can be fetched from
    the sync.Pool).
    
    This commit only does consistent hashing mode; we should do buffer mode as well
    if we do this work.
    philandstuff committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    99c28d5 View commit details
    Browse the repository at this point in the history
  2. Move constants in downloads to common.go

    For const and var shared between buffer and consistent-hash mode move these
    constants to a dedicated .go file. These are two distinct downloaders
    and neither should be responsible for explicitly shared elements.
    tempusfrangit committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    9862405 View commit details
    Browse the repository at this point in the history
  3. Wire up --chunk-size to cli options

    Wire up the --chunk-size option and deprecate --minimum-chunk-size.
    tempusfrangit committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    6f9c682 View commit details
    Browse the repository at this point in the history
  4. Cleanup Linting

    The concurrency calculation is no longer used in consistent-hashing.
    Remove it.
    tempusfrangit committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    d700109 View commit details
    Browse the repository at this point in the history
  5. Update Buffer to use ChunkSize

    Update Buffer downloader to use ChunkSize instead of MinChunkSize.
    Options have been modified to only contain ChunkSize now as the root and
    multifile commands no longer reference MinChunkSize.
    
    This change is in support of moving towards simplifying the chunksize
    logic. Additionally this will make reuse of buffers significantly easier
    as the next phase.
    tempusfrangit committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    efe1ac0 View commit details
    Browse the repository at this point in the history

Commits on Mar 5, 2024

  1. Failing test case

    If chunkSize doesn't divide contentSize evenly, and chunkSize < 0.5*contentSize,
    we try to download a chunk bigger than chunkSize.  This should not be possible.
    philandstuff committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    31bc26d View commit details
    Browse the repository at this point in the history
  2. Fix buffermode

    Every chunk except the last is now the same chunkSize, which is the intention of
    the code.  We no longer dynamically change chunk size.
    
    Also, simplify the consistent hashing code to be more similar to buffer mode,
    and remove the dead EqualSplit code.
    philandstuff committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    7ec1a9b View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    870dcd8 View commit details
    Browse the repository at this point in the history
  4. Update README.md

    philandstuff authored Mar 5, 2024
    Configuration menu
    Copy the full SHA
    6517274 View commit details
    Browse the repository at this point in the history
  5. Update README.md

    philandstuff authored Mar 5, 2024
    Configuration menu
    Copy the full SHA
    401e975 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    fae7259 View commit details
    Browse the repository at this point in the history
  7. Cleanup Aliasing

    Aliases in Cobra and Viper are relatively complex and
    viper.RegisterAlias does not do what would be expected as ENV handling
    is a very special case. This commit fixes up the handling of aliases
    specifically for chunk size and minimuim chunk size.
    
    * CLI Option Set
    * NEW ENV VAR
    * Deprecated ENV Var
    * Default Value
    
    For ChunkSize and MinimumChunkSize we now explicitly disallow setting
    both --chunk-size and --minimum-chunk-size as CLI options.
    
    If PGET_CHUNK_SIZE is set, we utilize that value except if the cli
    option(s) are set.
    
    If PGET_MINIMUM_CHUNK_SIZE is set
    * If PGET_CHUNK_SIZE is set to a non-default value a warning is emitted,
      and PGET_CHUNK_SIZE is used..
    
    * If PGET_CHUNK_SIZE is the default value, PGET_MINIMUM_CHUNK_SIZE is
      set as PGET_CHUNK_SIZE and a deprecation warning is emitted.
    
    Additionally, viper calls now all reference config.OptChunkSize
    tempusfrangit committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    e71b6b2 View commit details
    Browse the repository at this point in the history

Commits on Mar 6, 2024

  1. more aliasing cleanup

    Simplify conditional.
    
    Also, if both PGET_MINIMUM_CHUNK_SIZE and PGET_CHUNK_SIZE are set to the same
    value, don't emit a warning.
    philandstuff committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    a41a4c4 View commit details
    Browse the repository at this point in the history