Need help to figure out the sudden stop dequeue of control queue

[siemenstan]

Hi,
I would like to get some help how to figure out the reason of sudden stop dequeue of control queue at 12-Jan-2023 19:03:47 UTC. My app is using durable function extension v2.5.1, and deployed into Azure Kubernetes. From the App Insights, I am able to see the trace of "Stopped listening for messages on queue" but there not seeing any exception or trace message that indicate the reason of the sudden stop, appreciate if anyone can suggest what is any other traces that can help to figure out root cause of such issue. Thank you in advance.

Below are some error info and screenshot of app insights trace:
Azure Region: West US
Timestamp (UTC): 2023-01-12T19:03:47.4552886Z
HostInstanceId: a4ffcd8d-4b6e-4192-b645-bbbdb3a1ddf7
appId: 8f2c3085-e255-4aa8-bb45-175531af3e9e
Function App: C# .NET 6.0 with Durable function extension v2.5.1

[cgillum]

There are usually two reasons for an app instance to stop listening for messages on a control queue:

1. The functions host is shutting down.
2. One of the partitions has been stolen by / rebalanced to another instance.

If you're observing that you have app instances running and there are no app instances reading from a particular queue, then it's possible you've encountered a bug that can cause app instances to unexpectedly stop listening for queue messages. I recommend you update to the latest extension version (currently v2.9.0) to make sure you have all the bug fixes we've made in this area to rule out this possibility.

[siemenstan]

Checked that, there is no "Started listening for messages on queue" nor "shutdown" message around 19:03:47 UTC. "Starting listening for messages" trace seen only after the production team restarted the app service around 20:47 UTC since we pick up high queue count and delays

Below is the app insights screenshot

[davidmrdavid]

Interesting. It does appear to me like no other worker picked up the control queue until the restart.

I'm also surprised that we see no logs for "shutdown" since a restart would count as one. I wonder if the search is case sensitive, as I noticed you capitalized "shutdown" in your query. I'll defer again to @cgillum for further advice.

[siemenstan]

I did query with "shutdown" first, but surprised to see there is no shutdown trace nor stopped listening for messages upon the app containers restart, that why I run the query with "Shutdown" after that.

[davidmrdavid]

I see. So just so I'm clear: you found no logs with "shutdown" but you found results with "Shutdown", correct?
Also a ping to @cgillum in case he has anything to add here.
@siemenstan: have there been any developments / new findings since your last update or not?

[siemenstan]

@davidmrdavid No, I didn't find any shutdown trace with "shutdown" or "Shutdown" keyword. There is no findings on the reason of the sudden dequeue stop, and the app has been running ok after the restart so far.