kubelet-rubber-stamp initialization chicken and egg problem

[edgan]

I did successfully get kubelet-rubber-stamp working with a kube cluster I started with regular certificates, and then transitioned to auto generated kubelet server certificates. I am now trying to bring up a cluster from scratch, and finding that there is a chicken and the egg problem. My CNI, kube-router, expects the kubelet to have a valid server certificate, but gets the error below. Then kubelet-rubber-stamp expects the CNI to be running before it will start.

Edit:
The error below is actually about the retrieval of the logs from that kubelet, but it still stands that the cluster won't come up without kubelet-rubber-stamp working.

Error:
Error from server: Get https://kube-cluster-worker-01.inf.us-west-1.aws.acme.com:10250/containerLogs/kube-system/kube-router-44wjt/kube-router: remote error: tls: internal error

[edgan]

I worked around the chicken and the egg problem by starting kubelet-rubber-stamp outside a container. I copied the binary out, and ran the commands below. I had to create a special kubeconfig with a user that has cluster-admin privileges.

After starting kubelet-rubber-stamp, it did successfully approve the certificate requests.

Commands:

export WATCH_NAMESPACE=""
export OPERATOR=kubelet-rubber-stamp
./kubelet-rubber-stamp -logtostderr -kubeconfig /etc/kubernetes/kubelet-rubber-stamp.kubeconfig

[edgan]

The actual kubelet error is below:

Mar 11 01:10:55 kube-frodo-master-01 kubelet[21378]: E0311 01:10:55.312832   21378 kuberuntime_gc.go:153] Failed to stop sandbox "6cfbb9f819b87cea54bfc00a52d1d04ea859b4ddf6e126ad7a68d098228b4068" before removing: rpc error: code = Unknown desc = NetworkPlugin cni failed to teardown pod "kubelet-rubber-stamp-665fbb9bbb-67pdx_kube-system" network: no IP ranges specified

I was able to workaround this by removing "--network-plugin=cni" from the kubelet. Which allowed kubelet-rubber-stamp to start. I think the next problem it had was it needed admin privileges, because it is crashing.

[edgan]

Looks like kube-router is having trouble, but still kubelet insists that the cni should be up before starting a container.

Error:
cni.go:302] CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "ca0879fffe8373e880aa8ea5738f44225fa635149a708569ac614a2f2d6d7c49"

[edgan]

I switched back to a dedicated kubelet certificate for the kubelets on the masters. This allows kube-router and kube-rubber-stamp to run normally. While still allowing the use of TLS bootstrapping for kubelets on workers/nodes.

[jnummelin]

Hmm, the problem is that kubelet does not post the Ready condition for the node until CNI is proven to be up.

I haven't been using kube-router at all, but one alternative might be to run kubelet-rubber-stamp with hostNetwork: true so that it does not need CNI at all. That should allow rubber-stamp to boot-up before CNI.

[invidian]

@edgan you could also try to manually approve the certificate when bootstrapping first node. hostNetwork: true should also do the trick.