add support for mellanox/rdma-cni

[tariq1890]

In addition to sriov-cni, we need to deploy rdma-cni. It would be great if we can add this as a component to the sriov-config-daemon DS

[github-actions]

Thanks for your PR,
To run vendors CIs use one of:

• /test-all: To run all tests for all vendors.
• /test-e2e-all: To run all E2E tests for all vendors.
• /test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

• /skip-all: To skip all tests for all vendors.
• /skip-e2e-all: To skip all E2E tests for all vendors.
• /skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
  Best regards.

[adrianchiris]

to fully support RDMA namespaces in sriov-network-operator it would entail the following:

1. configure rdma subsystem to exclusive mode on node boot[1]
2. deploy rdma-cni[2]
3. SriovNetwork to support chaning rdma-cni (can be done using meta plugins[3] so no change needed here)
4. IbSriovNetwork to support rdmaIsolation ib-sriov-cni attribute [4]

for 2. we should probably look into moving rdma-cni to npwg first as now also Intel supports RDMA.

[1] #156
[2] https://github.com/Mellanox/rdma-cni
[3] https://github.com/k8snetworkplumbingwg/sriov-network-operator#chaining-cni-metaplugins
[4] https://github.com/k8snetworkplumbingwg/ib-sriov-cni#configuration-reference

[adrianchiris]

converting this one into an issue as it entails several changes and perhaps a design doc to align on behaviour.

[adrianchiris]

#481