Fix race condition in leader election for OCP

[SchSeba]

In OCP we have two steps to prepare a node before we drain it. first we get leader election and annotate the node with Draining and then we pause the MCP and mark the node as MCP paused.

if the config_daemon get a reset between the fist part to the second it will get stuck because one node will take the leader election BUT it will not mark the node as Draining as there is another node already draining. and the node with the draining label will try to get the drain lock again but the first node has it.

with this change if the node as Draning or MCP pause label it will not try to take the lock again and just continue after the reset.

[github-actions]

Thanks for your PR,
To run vendors CIs use one of:

• /test-all: To run all tests for all vendors.
• /test-e2e-all: To run all E2E tests for all vendors.
• /test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

• /skip-all: To skip all tests for all vendors.
• /skip-e2e-all: To skip all E2E tests for all vendors.
• /skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
  Best regards.

[github-actions]

Thanks for your PR,
To run vendors CIs use one of:

• /test-all: To run all tests for all vendors.
• /test-e2e-all: To run all E2E tests for all vendors.
• /test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

• /skip-all: To skip all tests for all vendors.
• /skip-e2e-all: To skip all E2E tests for all vendors.
• /skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
  Best regards.

[coveralls]

Pull Request Test Coverage Report for Build 5257619445

• 0 of 11 (0.0%) changed or added relevant lines in 1 file are covered.
• 16 unchanged lines in 4 files lost coverage.
• Overall coverage decreased (-0.2%) to 25.767%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pkg/daemon/daemon.go	0	11	0.0%

Files with Coverage Reduction	New Missed Lines	%
api/v1/helper.go	3	41.92%
pkg/utils/openshift_context.go	3	10.34%
pkg/daemon/daemon.go	4	42.68%
controllers/sriovibnetwork_controller.go	6	62.26%

Totals
Change from base Build 5143456346:	-0.2%
Covered Lines:	1957
Relevant Lines:	7595

---

💛 - Coveralls

[adrianchiris]

if the config_daemon get a reset between the fist part to the second it will get stuck because one node will take the leader election BUT it will not mark the node as Draining as there is another node already draining. and the node with the draining label will try to get the drain lock again but the first node has it.

marking the node as draining (via node annotation) is done during leader election.
what may happen is that another node got leader elected and has outdated caches then it would also mark its own node as draining.

in this case, i think in getDrainLock we should allow to continue if node already has draining annotation. and not rely on drainable attr.

in such "edge" cases we may get more than one node draining but its better than deadlock.

driving this whole process from controller should solve all of these cases
#427 is a good start

[github-actions]

Thanks for your PR,
To run vendors CIs use one of:

• /test-all: To run all tests for all vendors.
• /test-e2e-all: To run all E2E tests for all vendors.
• /test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

• /skip-all: To skip all tests for all vendors.
• /skip-e2e-all: To skip all E2E tests for all vendors.
• /skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
  Best regards.

[SchSeba]

Hi @adrianchiris please take another look when you have time :)

[adrianchiris]

LGTM