Daemonset not deploying pod to each node

[adityatech12]

I have deployed a AWS x-ray daemonset on our cluster without any node selector or tolerations , but its deployed only on few nodes, not every node in the cluster , I also don't see any pods in pending state if the resources are insufficient, Wanted to know thoughts regarding what can be the issue. screenshot

[osterman]

Without seeing anything on the cluster, my guess is that the DaemonSet for xray-daemon was launched after the Nodes were already full.

A DaemonSet launches a Daemon Pod on every Node, so any Pod scheduler or autoscaler will not create a new Node to accommodate a DaemonSet. By default, no Pods will be kicked off Nodes to make room for a new DaemonSet Pod, so you won't get DaemonPods added to existing, full Nodes.

If you bring up new Nodes, Karpenter should allocate a new Node that is big enough for all the DaemonSet Pods plus whatever new Pods triggered the scale up, so this is a sort of cold-start issue.

Kubernetes does provide an official mechanism for managing this sort of thing, called PriorityClass. You create a class with a given priority value (higher numbers have priority over lower numbers, and user-assigned numbers are anything less than or equal to 1 billion). Setting this value along with setting:

preemptionPolicy: PreemptLowerPriority

This tells the Kubernetes scheduler to evict lower priority Pods to make room for these pods, if needed. DaemonSets are the main use case for this, because any other kind of Pod would just trigger the creation of a new Node to handle it.

cc @Nuru