Cilium CNI Config

[Thesykan]

Summary

Cilium CNI config files are not propagated to attached nodes by default. Instead only the node the addon install command is run on is updated.

What Should Happen Instead?

Either we should update the CNI configure on each node on addon install or allow the CNI plugin to managed the CNI configure directory instead of updating on the addon install.

Reproduction Steps

1. Create fresh multi-node setup,
2. Enable cilium plugin
3. Deploy sample app onto an attached node, not the node that the install took place in.
4. Pod will fail to create, PodSandbox will fail to be initialized, error showing Calico isn't available.

Can you suggest a fix?

Allow CNI installation to write the correct config files onto each node on startup. In the helm chart configure updating the cni.customConf flag to false will allow Cilium to write to the config path already configured

Are you interested in contributing with a fix?

Yes, I can push an additional PR for this update. In addition or separately from #116

[Thesykan]

As a side note, I'm having a bit of a time configuring this chart. If I want to modify the values for the helm chart, It requires me to disable and re-enable cilium with an updated enable script. I don't see a way around this currently as the helm chart isn't persisted and only the final templated yaml is keep. We could persist the helm chart instead allow custom helm values configured dynamically either through arguments passed to enable or another method if there is a preferred way.