How should I handle aiobotocore S3 client instances when I have to read from multiple buckets from an opt-in region?

[adarmiento]

Hello

My service reads file from S3 via an aiobotocore S3 client:

async with session.create_client('s3') as s3_client:
  ...
  ...
  s3_client.XXX()

This works most of the time, however, if I deploy my service in a opt-in region, the inner region routing will not work and I have instead to create my client with explicit region of the bucket I want to read from.
For example: async with session.create_client('s3', region) as s3_client:

My problem is that in my application I do not know where I am reading from, and in fact I receive requests containing a bucket name and an object key, and I do the rest. I may have to read from N buckets potentially in N regions.

I wanted to create a simple cache of S3Client instances and use the right one when I need it, but S3Client is a short lived object that I can only create in a async with block, so my idea of having a:

 self._region_client_map: dict[str, Client] = {}
 self._bucket_region_map: dict[str, str] = {}
 
 ...
 bucket_name = 'xxx'
 client = self._region_client_map.get(self._bucket_region_map(bucket_name))
 client.YYY()

cannot work.

Is there a proper pattern or strategy to follow in this case? Thank you!

[thehesiod]

I'm imagining something like this:

class S3ClientCache:
    def __init__(self, session: aiobotocore.session.AioSession):
        self._session = session
        self._cache = {}
        self._exit_stack = contextlib.AsyncExitStack()
        
    async def __aenter__(self):
        pass
    
    async def __aexit__(self, exc_type, exc_val, exc_tb):
        await self._exit_stack.__aexit__(exc_type, exc_val, exc_tb)
        
    async def get_client(self, region_name: str):
        if client := self._cache.get(region_name):
            return client
        
        self._cache[region_name] = await self._exit_stack.enter_async_context(
            self._session.client('s3', region_name=region_name)
        )
        return self._cache[region_name]

[adarmiento]

Hello @thehesiod thanks for the reply.
Even with the clients created with the AsyncExitStack, I see I get
HTTPClientError("An HTTP Client raised an unhandled exception: _patch_task.<locals>.task_new_init() got an unexpected keyword argument 'eager_start'")
When I attempt to call an api (e.g. await client.get_bucket_location(Bucket=my_bucket))
I saw that this error appears when you are using the client outside its context. Am I doing it wrong?

[thehesiod]

Please post a list of modules and versions and sample script. That doesn't sound like a (aio)botocore issue