Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reuse s3 boto client? #168

Open
theogaraj opened this issue Dec 22, 2023 · 1 comment
Open

Reuse s3 boto client? #168

theogaraj opened this issue Dec 22, 2023 · 1 comment
Labels
question ❓ Further information is requested

Comments

@theogaraj
Copy link

I am starting to use upath a lot, it makes things so much easier! This is not an issue/bug, just a question (this github repo doesn't seem to have a "discussions" section).

I imagine that for S3 paths, somewhere under the hood upath is using boto3 and creating an S3 boto3 client.

If I create multiple UPaths, is each one creating a separate boto3 client? Or does it have a mechanism to reuse a client?

I am using both upath as well as smart_open, and the latter supports the ability to pass in your own boto3 client (described here). I couldn't find anything in the upath documentation, is this handled implicitly or just not supported?

@ap--
Copy link
Collaborator

ap-- commented Jan 24, 2024

Hello @theogaraj,

universal_pathlib's s3 support depends on s3fs which in turn uses aiobotocore

You can pass keyword arguments down to aiobotocore via client_kwargs in s3fs: https://github.com/fsspec/s3fs/blob/34a32198188164fd48d4d1abcb267f033d1d1ce1/s3fs/core.py#L205

this would look like this:

aioboto_client_kwargs = {'somesetting': ...}

UPath("s3://mybucket/somefile.tar", client_kwargs=aioboto_client_kwargs)

Let me know if that helps,
Andreas

@ap-- ap-- added the question ❓ Further information is requested label Jan 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question ❓ Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants