Function w/ Service Bus Trigger not Scaling down

[JFolberth]

We have a function which is being triggered off a Service Bus. There is a steady stream of messages coming into the queue; however, historically never enough to need more than one instance of the function running.

A few months ago, our function app scaled out unexpectedly to ~60 instances and never scaled back in.

We placed a max scale limit and it happened again.

We'd like to know if there is a way to mitigate this from happening. We do not believe setting a max scale of 2 is appropriate as this would be counter intuitive to having consumption-based function apps. Does anyone have any additional ideas or experience with this issue?

[olitomlinson]

@JFolberth just for clarity, are you using Durable Functions in your project?

[JFolberth]

Hey @davidmrdavid apologies for being a little unresponsive to this one. The event last occurred on 10/10.

[davidmrdavid]

Thanks @JFolberth, for the ping. I also just returned from vacation so my apologies for the delayed response.
@cgillum: before diving too deeply, have you seen the behavior described here before? Do any theories come to mind?

[cgillum]

This sounds more like an issue with the Consumption plan scaling infra rather than Durable Functions. If the maximum instance count setting is not being honored, then it's best to open a support ticket to have the Azure Functions scale team have a look. It would be good to also mention that scaling in is not happening. This is unfortunately an issue that many users have seen when there's a steady stream of Service Bus messages.

[JFolberth]

@cgillum The function app is stopping at the max; however, if this is not setup it will scale out more than is required and never scale back in. (Talking 12 instances when there are clearly no messages in the queue type scenario) We did test with a non-durable function and got the same result when there is a Service Bus trigger w/ a steady stream of messages.

Posted on here based on your feedback on any potential mitigation or work arounds the engineering team may have when the function is "over-scaled".

[cgillum]

@JFolberth, you mentioned that there are no messages in the queue, which is interesting. A long time back we did see issues where apps would fail to scale in if there were deferred messages in the Service Bus queues, even though there weren't any receivable messages. Could that be the issue you're running into?

[mluker]

I wanted to add a little more context to this as I was part of the original issue/troubleshooting and working directly with @JFolberth.

A change I made that seemed to produce more reliable scaling for our function was to grant 'Manage' rights for the trigger on the service bus. This stopped the function from immediately scaling out to ~65 instances. Before making that change, the scale out would happen even with with no messages or trigger executions. It would just scale out for no good reason and never scale back in.

It ran well for awhile, maybe a few weeks, without scaling but then we had a scale out to ~10 instances (I had set the scale to max at 10 just incase) with basically no traffic and it didn't scale back in.

Based on MS docs Listen rights should be enough, Manage rights will only cause difference in the sensitivity of scale controller's scale out/in behavior which is mentioned to be "more efficient" in official document. The Manage rights are supposed to just bring in one more scale in/out rule to the equation. From my experience, this is not the case. Without Manage, it will scale out and not back in.

[cgillum]

@mluker that's a really interesting observation. Would you mind opening an Azure Functions support request and sharing this info so that the platform engineering team can take a look? It's possible there may be some regression that went into the platform recently.

[JFolberth]

@cgillum Would it be alright to leverage the existing ticket? I believe there is some history and communication of what Matt has explained in the ticket as well as additional supporting documentation.

[cgillum]

@JFolberth yes, if you have a ticket already, that should be fine.

[JFolberth]

@cgillum Ticket 2108120040003836. The tech currently assigned has been helpful and aligned that there are some anomalies going on that we haven't been able to explain.

[davidmrdavid]

Great! I just want to re-emphasize that, from what I understand about the symptoms and @cgillum's assessment so far, this doesn't seem to be something the Durable Extension would manage. That's to say that we might not be the right team to diagnose this issue.

@JFolberth, my recommendation would be to continue working with your support engineer for the time being, especially if the current assigned person is moving the investigation forward. They also have a means to reach out to the appropriate product group if that's required for your case. Finally, a big part of the Functions ecosystem is open source, so if the support engineer can identify the faulty component, there is likely a public repo we can use to reach out to the appropriate team; but we have to get to that point first.