Iterating over paginated API methods

[llucax]

Coming from:

• Introduce simple reporting client with support for single metric historical data frequenz-client-reporting-python#16 (comment)
• Introduce simple reporting client with support for single metric historical data frequenz-client-reporting-python#16 (comment)
• Introduce simple reporting client with support for single metric historical data frequenz-client-reporting-python#16 (comment)

I suggest making gRPC methods that retrieve pages return a wrapper object with 2 ways to iterate over the results:

1. By pages
2. By individual items

For example, for the reporting API:

async for page in client.list_microgrid_components_data(...).pages:
    ...

async for sample in client.list_microgrid_components_data(...).samples:
    ...

So list_microgrid_components_data() could return a Response object with properties like (pseudo-code using the current method names):

@property
async def samples(self) -> AsyncIterator[Sample]:
    return self.client._iterate_single_metric(...)

@property
async def pages(self) -> AsyncIterator[ComponentsDataPage]:
    return self.client._iterate_components_data_pages(...)

[david-natingga-frequenz]

In Weather API first we implemented the pagination by iterating over individual items. Then we changed this to iterate over the pages of data on the grounds of a better efficiency since the iterator is asynchronous.

You can check the history of this file for both solutions: https://github.com/frequenz-floss/frequenz-api-weather/blob/v0.x.x/py/frequenz/client/weather/_historical_forecast_iterator.py

[cwasicki]

For the Reporting API client we decided to use a private method to iterate over pages. It returns a dedicated page object, which provides an iterator over individual samples . For now we only expose a public method to iterate over samples (which calls the method to iterate over pages), but it would be trivial to expose both. As long as there is no need for access the pages, there are no plans to expose it though.

[cwasicki]

From frequenz-floss/frequenz-client-reporting-python#16 (comment)

[llucax]

Just as a reminder, the main point of iterating over pages is performance (iterating over individual items asynchronously) is ~3x slower in Python 3.11:

from typing import AsyncIterator

async def pages() -> AsyncIterator[list[int]]:
    for i in range(10):
        yield [i] * 1000

async def items() -> AsyncIterator[int]:
    async for page in pages():
        for j in page:
            yield j

async def iter_pages():
    total = 0
    async for page in pages():
        for item in page:
            total += item

async def iter_items():
    total = 0
    async for item in items():
        total += item

Run:

$ python3.11 -m timeit -s 'import t; import asyncio' 'asyncio.run(t.iter_items())'
500 loops, best of 5: 962 usec per loop
$ python3.11 -m timeit -s 'import t; import asyncio' 'asyncio.run(t.iter_pages())'
1000 loops, best of 5: 311 usec per loop

And more than 4x slower in 3.12:

$ python3.12 -m timeit -s 'import t; import asyncio' 'asyncio.run(t.iter_items())'
200 loops, best of 5: 1.3 msec per loop
$ python3.12 -m timeit -s 'import t; import asyncio' 'asyncio.run(t.iter_pages())'
1000 loops, best of 5: 271 usec per loop

Iterating over pages only adds only a very minor inconvenience, so if we will provide only one way to iterate, I'm more inclined to provide pages instead, as it is the safest approach to avoid the users ending up with performance issues. I would only provide iteration over individual items for convenience for small scripts, but probably it is not worth it because of the extra complexity.

It might also be that the performance penalty is negligible, we'll run some tests after we have an example working with a lot of data, using both iteration methods to see if there is any real-world difference in performance.