Understanding 'consolidationPolicy: WhenEmpty'

[jderieg]

Hi there, I've been testing the WhenEmpty policy, but I think I may not be understanding the algorithm correctly. My disruption settings look like this:

    disruption:
      consolidationPolicy: WhenEmpty
      consolidateAfter: 10m
      expireAfter: 360h

As a test, I scale up a deployment to a large number of pods in my nodegroup so that Karpenter spins up a new node. That works fine. When I scale the deployment back down to 0, I would expect Karpenter to scale down (remove) the Karpenter node after 10m of that deployment no longer needing it. That never happens. I've let it sit for hours and it never removes the node, even though there aren't anymore workloads added to the node to keep it alive. The strange thing is, if I set that consolidateAfter value to 2m or under, it works as I would expect, and removes the node. I'm running Karpenter v0.37.

[jderieg]

I created a bug for this: #1647