Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support & documentation on using Azure CNI with capz #467

Closed
3 of 5 tasks
CecileRobertMichon opened this issue Mar 20, 2020 · 58 comments
Closed
3 of 5 tasks

Add support & documentation on using Azure CNI with capz #467

CecileRobertMichon opened this issue Mar 20, 2020 · 58 comments
Assignees
@nawazkh
Labels
kind/feature Categorizes issue or PR as related to a new feature. lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor. parity Used to track feature parity with other Azure provisioning tools (AKS, AKS Engine, etc) priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.
Milestone
v1.10

Comments

@CecileRobertMichon
Copy link
Contributor

CecileRobertMichon commented Mar 20, 2020 •
edited
Loading

/kind feature

Describe the solution you'd like
[A clear and concise description of what you want to happen.]
https://github.com/Azure/azure-container-networking/blob/master/docs/cni.md

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

A little lower priority issue, but must be solved to support multiple NICs on control plane.

Environment:

  • cluster-api-provider-azure version:
  • Kubernetes version: (use kubectl version):
  • OS (e.g. from /etc/os-release):
@k8s-ci-robot k8s-ci-robot added the kind/feature Categorizes issue or PR as related to a new feature. label Mar 20, 2020
@justaugustus justaugustus added this to the v0.5 milestone Mar 30, 2020
@richardcase
Copy link
Member

This would be good for installing the CNI: kubernetes-sigs/cluster-api#3050

@CecileRobertMichon
Copy link
Contributor Author

@mboersma is this one you'd be interested in working on? It's a relatively hi-pri one because it's a blocker for IPv6 (cc @jsturtevant)

@mboersma
Copy link
Contributor

@CecileRobertMichon yes indeed, I can probably start on this tomorrow.

@CecileRobertMichon CecileRobertMichon added the parity Used to track feature parity with other Azure provisioning tools (AKS, AKS Engine, etc) label Jun 4, 2020
@CecileRobertMichon
Copy link
Contributor Author

/priority important-soon

@k8s-ci-robot k8s-ci-robot added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Jun 4, 2020
@CecileRobertMichon CecileRobertMichon modified the milestones: v0.5, v0.4.x Jul 10, 2020
@CecileRobertMichon CecileRobertMichon modified the milestones: v0.4.7, v0.4.8 Aug 4, 2020
@CecileRobertMichon
Copy link
Contributor Author

/assign

@CecileRobertMichon
Copy link
Contributor Author

/unassign
/milestone next

@k8s-ci-robot k8s-ci-robot modified the milestones: v0.4.8, next Aug 26, 2020
@jackfrancis
Copy link
Contributor

An implementation question: is the version of Azure CNI currently configurable?

cc @craiglpeters

@CecileRobertMichon
Copy link
Contributor Author

@jackfrancis not sure I understand the question? There is no Azure CNI implementation currently.

@jackfrancis
Copy link
Contributor

A reminder to use the "transparent" mode configuration when implementing a capz + Azure CNI scenario. This AKS Engine PR configures Azure CNI for "transparent" mode by default:

Azure/aks-engine#3958

The vanilla installation method that AKS Engine follows (download the public release tarball and untar/gzip it) delivers a default configuration of "bridge" mode. AKS Engine modifies this to "transparent" by sed replacing the appropriate value:

https://github.com/Azure/aks-engine/blob/master/parts/k8s/cloud-init/artifacts/cse_config.sh#L288

Not pretty, but there you have it.

@CecileRobertMichon
Copy link
Contributor Author

ref https://github.com/Azure/azure-container-networking/blob/master/acncli/deployment/manager.yaml

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 27, 2021
@fejta-bot
Copy link

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Feb 26, 2021
@fejta-bot
Copy link

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

@dthorsen
Copy link
Contributor

dthorsen commented Nov 2, 2022

@CecileRobertMichon @nawazkh I have pushed my commit. The tests are passing and I spun up a few workload clusters from tilt with various privateIPConfiguration counts, public IP allocation enabled/disabled, etc. I have to sign off for the night but let me know what you think tomorrow.

@dthorsen
Copy link
Contributor

Here is a link to the Azure CNI Manager daemonset that can be used to install the Azure CNI quite easily.

https://github.com/Azure/azure-container-networking/blob/ef1bff6046156dec56b46664f516921bd423e9e0/tools/acncli/deployment/manager.yaml

@rbtr
Copy link
Contributor

rbtr commented Nov 17, 2022

@dthorsen this is not a supported tool, don't recommend it for this use please.

@dthorsen
Copy link
Contributor

dthorsen commented Nov 17, 2022 •
edited
Loading

That is disappointing, because it seems to do exactly what is required. It was very easy to get up and running this way. Is there a way it could become a supported tool? Otherwise it may be something that makes sense for CAPZ to fork and support, but that seems somewhat redundant if there is already a tool that does this. The functionality needed is pretty trivial, basically just install the CNI binaries, drop the config, and sleep forever. It seems strange to support the CNI, but not a k8s native installation method.

@jackfrancis
Copy link
Contributor

@rbtr @tamilmani1989 can you provide more information? In lieu of an official solution, why is wrapping a daemonset around the published manifest in the official Azure CNI repo not recommended?

@rbtr
Copy link
Contributor

rbtr commented Nov 17, 2022

@jackfrancis @dthorsen in short, the image referenced by that manifest is unmaintained and not prod-ready.

We built it for our CI/nonprod debugging with no intent of public use. It is technically capable of installing the CNI binary, but I can't support CAPZ using/recommending it for use. Maybe most importantly, we have moved past needing it internally and it isn't getting updates.

I did previously try to engage here to offer what will be an official supported solution but that didn't seem to get anyone's attention. This tool is being used for CNI install in AKS already and could probably trivially be extended for CAPZ, but I need some clarification on the requirements to make that happen. I suspect this will be sufficient, but if some CAPZ folks would like to contribute, we accept PRs 🙂

@jackfrancis
Copy link
Contributor

It seems based on @dthorsen's experience that the now-abandoned installer image can be used as the set of functional requirements. @dthorsen did that container image simply install the ipam stuff with a sufficient amount of UX configuration to fulfill all of the needs of Azure CNI v1 on a capz-built node? E.g.:

env:
            - name: AZURE_CNI_OS
              value: <insert value here>
            - name: AZURE_CNI_TENANCY
              value: <insert value here>
            - name: AZURE_CNI_MODE
              value: <insert value here>
            - name: AZURE_CNI_IPAM
              value: <insert value here>
            - name: AZURE_CNI_EXEMPT
              value: <insert value here>

@rbtr would it be possible to extend the tool AKS is using to (1) include v1 of the CNI components with (2) the configuration above?

@nawazkh
Copy link
Member

nawazkh commented Feb 14, 2023

Sorry, I haven't been active at posting updates on this test effort but I am actively working on this!
So far, Cecile and I have tested Dane's PR of allocating PrivateIPConfigs to a VM, and it works as expected.
I now am working on getting Azure CNI up-and-running in a test cluster.

@CecileRobertMichon
Copy link
Contributor Author

/milestone v1.8

@k8s-ci-robot k8s-ci-robot modified the milestones: next, v1.8 Feb 15, 2023
@CecileRobertMichon
Copy link
Contributor Author

/lifecycle active

@k8s-ci-robot k8s-ci-robot added the lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor. label Feb 15, 2023
This was referenced Feb 27, 2023
Closed
Merged
@nawazkh
Copy link
Member

nawazkh commented Mar 9, 2023

/milestone next

@k8s-ci-robot
Copy link
Contributor

@nawazkh: You must be a member of the kubernetes-sigs/cluster-api-provider-azure-maintainers GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your Cluster API Provider Azure Maintainers and have them propose you as an additional delegate for this responsibility.

In response to this:

/milestone next

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@mboersma mboersma modified the milestones: v1.8, v1.9 Mar 9, 2023
This was referenced Mar 21, 2023
Closed
Closed
@nawazkh
Copy link
Member

nawazkh commented Mar 21, 2023 •
edited
Loading

I created two other issues to track the branched out effort to enable Azure CNI on CAPZ

Enabling Azure CNI v1

Azure CNI v1 with one NIC per node

High Priority

Azure CNI v1 with multiple NIC per node

High Priority

Low Priority

@nawazkh nawazkh mentioned this issue May 1, 2023
Merged
4 tasks
@mboersma mboersma modified the milestones: v1.9, v1.10 May 3, 2023
This was referenced Jun 7, 2023
Merged
Closed
@nawazkh
Copy link
Member

nawazkh commented Jun 7, 2023

closing this issue in favor of epic tracking this effort #3611
/close

@k8s-ci-robot
Copy link
Contributor

@nawazkh: Closing this issue.

In response to this:

closing this issue in favor of epic tracking this effort #3611
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nawazkh nawazkh

Labels
kind/feature Categorizes issue or PR as related to a new feature. lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor. parity Used to track feature parity with other Azure provisioning tools (AKS, AKS Engine, etc) priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.
Projects
CAPZ Planning
Archived in project
Milestone
v1.10
Development

No branches or pull requests