MS Azure Durable functions freeze in Pending status for long period of time

[maxtsaryk]

Hi, I'm faced with freezing my MS Azure Durable functions with pending runtime status. It happens time to time and with random functions. Time to time means that if I catch a freezing function and then repeat all the same steps from start to call this function it will work as expected without any delays.
how these functions are called: I have a trigger function - it is storage queue function. I call orchestrations functions during storage queue function execution. For my business logic I have to call some chain of orchestration functions and with specific order. It means I have to wait until one orchestration function complete and after I know the result I call second orchestration or stop the execution at all.
As a result of such delays my storage queue function catch the exception Microsoft.Azure.WebJobs.Host.FunctionTimeoutException and restarts the whole process. In the end I get 100% execution of all chain items but total delay could be 10-15 min that is not acceptable with my business requirement and some duplicates of orchestrations functions also appear(like sending emails)

Here are some logs(start reading from the end)

5/3/2023, 1:01:40 PM TRACE Executed 'WorkflowExecutor' (Succeeded, Id=e749bf69-a683-4303-99fb-e5841f558ad9, Duration=37011ms) Severity level: Information

5/3/2023, 1:01:40 PM <= CUSTOM LOG (after got a complete orchestration status)
TRACE
RunSendEmail completed successfully, WorkflowItemId: 5e2c3691-xxx.
Severity level: Information

5/3/2023, 1:01:40 PM
TRACE
Executed 'RunSendEmail' (Succeeded, Id=8d5fa41c-773f-4c90-b1ca-b99d2f7b8151, Duration=278ms)
Severity level: Information

5/3/2023, 1:01:40 PM <= CUSTOM LOG (the log inside orchestration func)
TRACE
Sending email to [email protected] completed successfully.
Severity level: Information

5/3/2023, 1:01:04 PM
TRACE
1ff5cefeb56d4021818030ea8f03b411: Function 'SendEmail (Orchestrator)' scheduled. Reason: NewInstance. IsReplay: False. State: Scheduled. HubName:...funcappprod. AppName: ...-func-app-prod. SlotName: Production. ExtensionVersion: 2.9.3. SequenceNumber: 2.
Severity level: Information

5/3/2023, 1:01:03 PM <=CUSTOM LOG (one line beforestart orchestration func in second storage queue func run)
TRACE
RunSendEmail is started, WorkflowItemId: 5e2c3691-xxx.
Severity level: Information

5/3/2023, 1:01:03 PM <= CUSTOM (very start of the second run of storage queue func. It restarts automatically after exception bellow appear)
TRACE
Executing with 95xx WorkflowId is started.
Severity level: Information

5/3/2023, 12:53:55 PM
TRACE
Executed 'RunSendEmail' (Succeeded, Id=82e36fe7-ac1a-476f-98b9-cf80d8016706, Duration=254ms)
Severity level: Information

5/3/2023, 12:53:55 PM <= CUSTOM LOG (the log inside orchestration func)
TRACE
Sending email to [email protected] completed successfully.
Severity level: Information

5/3/2023, 12:50:55 PM <= CUSTOM LOG
TRACE
RunSendEmail failed, WorkflowItemId: 5e2c3691-xxx.
Severity level: Information

5/3/2023, 12:50:55 PM <= CUSTOM LOG (the confirmation that orchestration func Runtime status is Pending after 10 min)
TRACE
Runtime execution status for orchestration SendEmail with 65c7806759f74add92b426b873099abb Id is Pending.
Severity level: Information

5/3/2023, 12:50:55 PM
EXCEPTION
Timeout value of 00:10:00 was exceeded by function: WorkflowExecutor
Problem Id: Microsoft.Azure.WebJobs.Host.FunctionTimeoutException at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__35.MoveNext

5/3/2023, 12:50:55 PM
TRACE
Executed 'WorkflowExecutor' (Failed, Id=099e57fb-27d5-45b1-8c63-1688df1588ac, Duration=600007ms)
Severity level: Error

5/3/2023, 12:50:55 PM
EXCEPTION
Timeout value of 00:10:00 was exceeded by function: WorkflowExecutor
Problem Id: Microsoft.Azure.WebJobs.Host.FunctionTimeoutException at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__35.MoveNext

5/3/2023, 12:50:55 PM
TRACE
Timeout value of 00:10:00 exceeded by function 'WorkflowExecutor' (Id: '099e57fb-27d5-45b1-8c63-1688df1588ac'). Initiating cancellation.
Severity level: Error

5/3/2023, 12:40:55 PM
TRACE
65c7806759f74add92b426b873099abb: Function 'SendEmail (Orchestrator)' scheduled. Reason: NewInstance. IsReplay: False. State: Scheduled. HubName: ...funcappprod. AppName: ...-func-app-prod. SlotName: Production. ExtensionVersion: 2.9.3. SequenceNumber: 9.
Severity level: Information

5/3/2023, 12:40:55 PM <= CUSTOM LOG (one line before call orchestration function - starter.StartNewAsync("SendEmail"))
TRACE
RunSendEmail is started, WorkflowItemId: 5e2c3691-xxx.
Severity level: Information

5/3/2023, 12:40:55 PM <= CUSTOM LOG (very start of storage queue func)
TRACE
Executing with 95xx WorkflowId is started.
Severity level: Information

[maxtsaryk]

Updates:
Delays have disappeared and now there are no any pending status freezing. The problem was with function app name. The resource name in azure was too long. It turned out that it is not recommended to create name more then 32 symbols. By this name MS generates some ids for internal usage. In my case I had 38 symbols and their names had only diff with postfix in the end of the name like -dev/-test/-prod.