Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add max_concurrency option to limit number of concurrent BlobClient connections #288

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Add max_concurrency option to limit number of concurrent BlobClient connections #288

wants to merge 3 commits into from

Commits on Nov 9, 2021

  1. Add max_concurrency option to limit concurrency 

    This adds a new keyword to AzureBlobFileSystem to limit the number of
concurrent connectiouns. See pangeo-forge/pangeo-forge-recipes#227 (comment)
for some motivation. In that situation, we had a single FileSystem
instance that was generating many concurrent write requests through
`.pipe`. So many, that we were seeing memory issues from creating all
the BlobClient connections simultaneously.

This adds an asyncio.Semaphore instance to the AzureBlobFilesSytem that
controls the number of concurrent BlobClient connections. The default of
None is backwards-compatible (no limit)
    Tom Augspurger committed Nov 9, 2021
    Configuration menu
    Copy the full SHA
    6cc062d View commit details
    Browse the repository at this point in the history

  2. Update docs

    Tom Augspurger committed Nov 9, 2021
    Configuration menu
    Copy the full SHA
    14c3fe1 View commit details
    Browse the repository at this point in the history

  3. compat

    Tom Augspurger committed Nov 9, 2021
    Configuration menu
    Copy the full SHA
    1b82b09 View commit details
    Browse the repository at this point in the history