put_directory creates a new s3_client for each file uploaded

[mattiamatrix]

Hello all,

I am executing build_from_flow as below to upload all my Prefect files into an S3Bucket.

Deployment.build_from_flow(
    flow=...,
    skip_upload=False,
    storage: Block = S3Bucket.load(...),
    ...,
)

It works, but the result is that upload_to_storage takes a very long time. I have about 400 files to upload and it takes around 1 second per file.
I looked at the source code and the botocore output and it looks like a boto3 client and credentials are checked for every files.

version: prefect-aws==0.4.1

Expectation / Proposal

Uploading a bunch of files to S3 should very fast.

Traceback / Example

• I would like to help contribute a pull request to resolve this!

[rudeb0y]

Not sure if there is much of a difference but ours looks like:

from prefect.filesystems import S3

default_storage = S3.load("flow-storage")

Deployment.build_from_flow(
    storage=default_storage,

...

[mattiamatrix]

I realized what is happening here, and I started working on #361.

The _write_sync function that actually talks to S3, is within a loop. Here, the function calls s3_client = self._get_s3_client() which generates a new S3 client every time.

The problem is that the s3_client should not be created as a new object but instead created once and reused across multiple calls.

The improvement with the changes is substantial. In my setup, with 420 files, the upload goes from 5-7 minutes to 2 minutes.