Problems when handle API long pagination using durable function

[sasoripathos]

Background
Here is one scenario where we need to call an API which responds with pagination (i.e. like many List APIs provided by Microsoft Graph). Obviously the pagination mechanism requires us to perform sequential API requests to get all data we want. For some large datasets, we need to request for thousands of pages sequentially.

Since the normal Azure function has a timeout limit, we come up with the following design with Durable Functions:
In the Orchestrator Function, we have something like:

string nextLink = storedDeltaLink;
HashSet<JToken> results = new HashSet<JToken>();
while (!string.IsNullOrEmpty(nextLink))
{
        GraphApiResponse response = await context.CallActivityAsync<GraphApiResponse>("CallGraphApi", nextLink);
        nextLink = response.NextLink;
        results.UnionWith(response.Values);
}

where the Activity Function "CallGraphApi" would just perform one API request and return the results.

We tried this design in some of our development in the past year, we had the following observations.

Observations

1. In a local test run (function app runs locally with a real storage account), we noticed some "super long duration" for orchestrator that runs for large datasets. This case starts happening after the durable function runs for a while. We have some logs copied here:
   

2. Early this year, we had a test deployment for a similar Durable Function app in Consumption plan. The durable function overall took much longer time to finish querying a large dataset, comparing with a console app. Besides, we also found the Function App had some "unknown pause" during the execution. See the gap below:
   

Results
We gave up using Azure Durable Function for the API pagination case since it tremendously increased the time to fetch a large dataset. Instead we chose Azure Webjobs.

What we want to get from this discussion
Generally speaking, we want to understand what caused the "super long duration" of Orchestrators and the "unknown pause" during the execution. Especially:

• Are the two problems we observed the same problem?
• What are the causes of "super long duration of orchestrator" and "unknown pause"
• If the design of the orchestrator has problem, what's the suggested design for this API pagination case?
• Would changing a Function app plan (like premium plan) help to solve some cases?

Side Notes
We didn't keep all the logs/deployments/investigation history since some of them was nearly 1 year ago. We are MSFT internal teams. Please let us know if there is anything we can do to help the investigation.

[olitomlinson]

@sasoripathos

There have been a few occasions where large data sets have caused memory problems, and this would be my first thought.

Example #1878

@ConnorMcMahon

This looks like the kind of pattern that may benefit from this feature I requested a while ago #1682

[cgillum]

There are a couple issues I can think of that would make this scenario unnecessarily slow:

1. Each activity call involves two queue messages, one for calling the activity and passing the function input and one for sending the response with the output to the orchestrator. This is going to add latency for each page iteration. If you have a lot of pages, the impact will be significant.
2. If your function inputs or outputs are large (>40 KB) then the messages I referred to will be even slower because they have to be compressed/uploaded to blob storage and downloaded/decompressed on the other side. This is the same issue mentioned by @olitomlinson.

An always-running WebJob will always be faster for this kind of scenario, but you're basically sacrificing durability for speed. Depending on the requirements of your scenario, this may be okay. If you wanted to try using DF for this application again, two suggestions I would recommend are to enable extended sessions (which can dramatically reduce the impact of large message processing) and having each activity function process multiple pages instead of just one, reducing overall I/O being done by your app.

It's hard to know what might be causing the pauses without having access to recent telemetry. One possibility I can think of is that you might be running out of memory and the function app might be recycling. The consumption plan only allows 1.5 GB and is aggressive about recycling the host process if that limit is reached or exceeded. This can cause long delays in orchestrations due to the default 5 minute visibility timeout of the underlying Azure Storage queue messages that get locked and then abandoned.

[sasoripathos]

Thank you @cgillum and @olitomlinson for looking into this.

For the local test run, there were about 4500 pages to fetch. I had the delta query return only IDs for each piece of data. I don't think each activity function's input/output data will be > 40KB (no testhubname-largemessages container created). I didn't recall how much memory the local program used. Also I think my local test should not have the 1.5G memory limitation.
Also, in the test run logs (as the screen shot I provided), I notice there are still some orchestrator executions with normal duration (about thousands ms) between two super long duration orchestrator executions. Can you add some more insights about this?
(PS: if you need to look at the history table for this test run, I can provide the storage account name)

As for the pause problem, I agree it would be better to look into a recent telemetry. I will try to see if my team can have another test deployment about such a durable function to reproduce the pause problem. I will update this thread accordingly.