feat: implement graceful error handling in operator package

[danxwang22]

Fixes #N/A

Description
This implements fail open logic within the operator package
How was this change tested?

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: danxwang22 / name: Daniel Wang (a06f481, 389512e, 2c6b0c7, 2a99070, d0b1cd0, de21216, ef0b7a0, e02d3c0)
• ✅ login: njtran / name: Nick Tran (c411e02, 8749826)
• ✅ login: jmdeal / name: Jason Deal (7162f00)

[k8s-ci-robot]

Welcome @danxwang22!

It looks like this is your first PR to kubernetes-sigs/karpenter 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/karpenter has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @danxwang22. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[coveralls]

Pull Request Test Coverage Report for Build 11074781335

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

• For more information on this, see Tracking coverage changes with pull request builds.
• To avoid this issue with future PRs, see these Recommended CI Configurations.
• For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

• 0 of 52 (0.0%) changed or added relevant lines in 1 file are covered.
• 81 unchanged lines in 4 files lost coverage.
• Overall coverage decreased (-0.1%) to 80.649%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pkg/operator/operator.go	0	52	0.0%

Files with Coverage Reduction	New Missed Lines	%
pkg/controllers/disruption/consolidation.go	4	87.25%
pkg/controllers/provisioning/provisioner.go	18	78.86%
pkg/controllers/node/termination/controller.go	26	69.63%
pkg/controllers/state/cluster.go	33	91.71%

Totals
Change from base Build 10975568388:	-0.1%
Covered Lines:	8448
Relevant Lines:	10475

---

💛 - Coveralls

[jmdeal]

/lgtm

[jmdeal]

/lgtm

[njtran]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: danxwang22, njtran

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [njtran]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment