SubOrchestrators Fan out/Fan In gets stuck on Task.WhenAll #2169
Replies: 1 comment
-
Hi @KonradCzarny, Thank you for reaching out. I'll start by saying that I'm a little lost on how your 4th Activity works, but from your description it seems we're stuck in the 3rd Activity so I won't worry about that too much yet :-) . Whether 3k Activities in parallel is too much is hard to answer without your app's details, but it definitely has the potential of causing some problems. Just as a first step, to gather more context: can you try splitting batching these 3k activities into smaller collections, and sending them over to sub-orchestrators to process? Each sub-orchestrator could then fan-out-fan-in over these activities, essentially achieving the same parallelism you're looking for, while also placing a lesser burden on your top-level orchestrator's internal History data structure. If this alleviates your problem, then we would have evidence that your orchestrator's History data structure was simply growing too large (either in number of entries, object memory footprint, or both). If this doesn't work, we would need your application name, an incident timerange, and the instanceId of the affected orchestrator to dive deeper. Thanks! |
Beta Was this translation helpful? Give feedback.
-
I have an orchestrator with 4 activites:
For activities 3rd each item in collection is 1 activity call, for activity 4th it is batched in collection of 50 items per 1 activity call,
which are being awaited by Task.WhenAll. - here we would have 3k items batched into collection of 50 that are being then loaded
into 4th activity, so about 60 tasks.
Service plan is P2v2.
Everything is working fine on local environment, but on azure, orchestrator seems to get stuck on 3rd activity. I get around 2k requests in logs for SavePromoProductFromACPSActivity, and after that all I get is occasionally "Executing XYZ orchestrator", and after few minutes "Executed XYZ orchestrator", with no activity calls between those messages. Orchestrator seems to get stuck after those 2k requests.
I'm using simple fan out/fan in strategy for 3rd and 4th activities, so something like this:
Is the number of tasks the problem here? Or am I missing something else?
Beta Was this translation helpful? Give feedback.
All reactions