-
Notifications
You must be signed in to change notification settings - Fork 92
Tainting Nodes to Fix Infrastructure Issues
Kubernetes allows you to taint
nodes so that only certain pods can run on them.
This can allow you to do things such as set custom kubelet arguments, make sure all your pods land on a node with certain network/performance guarantees, and if nothing else, fence out other containers from competing with network and local disk bandwidth, which can be key performance bottlenecks.
Alpine DNS is parallelized, and thus if you have a wildcard resolver in your data center, the internal DNS names used by Black Duck's containers can get into a split-brain state, which will ultimately make your deployment non-viable. In Kubernetes versions after 1.7, hostAliases can be used to hardcode IP endpoints for well known hosts, such as kb.blackducksoftware.com, but it is difficult to use, because your service IP addresses aren't made until after you deploy Black Duck, and may change if you delete a service.
One way to prevent this is to inject your own resolv.conf file into each pod. (Note: There is pending upstream work in Kubernetes to enable this, although it is not done yet.) In the meantime, you could follow these steps:
- Taint a node to only run Black Duck containers:
kubectl taint nodes mynode.mycompany.com blackduck=true:NoSchedule
This will make sure that only Black Duck containers, and no other containers, run on this node. That way, any modifications you make to the node won't effect the behavior of other pods on the cluster. - Add tolerations to Black Duck deployment pods for
blackduck=true
.
You'll need to add the following to your pod definitions. Since every container needs to talk to something else on the internal Kubernetes network, you'll need to do this for every pod.
apiVersion: apps/v1beta1
kind: Deployment
metadata:
...
spec:
replicas:
...
template:
metadata:
...
spec:
nodeSelector:
...
tolerations:
key: blackduck
operator: Equal
value: true
- Modify your kubelet startup flags to inject a custom resolv.conf, for example:
KUBELET_ARGS="--pod-manifest-path=/etc/kubernetes/manifests
--cadvisor-port=0
--pod-infra-container-image=gcr.io/google_containers/pause-amd64:3.0
--node-status-update-frequency=10s
--docker-disable-shared-pid=True
--client-ca-file=/etc/kubernetes/ssl/ca.pem
--tls-cert-file=/etc/kubernetes/ssl/node-k8s-master-1.pem
--tls-private-key-file=/etc/kubernetes/ssl/node-k8s-master-1-key.pem
--anonymous-auth=false
--read-only-port=10255
--cgroup-driver=cgroupfs
--cgroups-per-qos=True
--fail-swap-on=True
--enforce-node-allocatable="" --cluster-dns=10.233.0.3 --cluster-domain=kubernetes.tdlab --resolv-conf=/etc/resolv.conf
In your resolv.conf, you can entirely remove your wildcard DNS settings. To see what a custom resolv.conf should look like, copy it out of a running pod, and simply delete the lines referencing your data center's wildcard DNS.
This will ensure that all your Black Duck containers land only on the node which you have tainted.
After this, if some external hosts like kb.blackducksoftware.com
aren't resolved in containers, you can modify hostAliases
to manually inject them into etc/hosts
of your containers.