docs: update karpenter docs around soft affinity behavior

website/content/en/docs/concepts/scheduling.md

@@ -191,6 +191,14 @@ Then the pod can declare that custom label.
 
 See [nodeSelector](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector) in the Kubernetes documentation for details.
 
+## Preferences
+
+Karpenter is aware of preferences (node affinity, pod affinity, pod antiaffinity, and pod topology) and treats them as requirements in almost all circumstances. As a result, Karpenter also treats preferred affinities as required affinities when constructing requirements for a pod. However, If these requirements can not be met, preferences are relaxed where possible and the requirements are tried again.
+
+{{% alert title="Warning" color="warning" %}}
+Karpenter interprets preferred affinities as a soft affinity when constructing topology requirements for scheduling to a node. Preferred affinities are ignored in regards to topology zones, so if these preferences are necessary, required affinities should be used [as documented in Node Affinity](#node-affinity).
+{{% /alert %}}
+
 ### Node affinity
 
 Examples below illustrate how to use Node affinity to include (`In`) and exclude (`NotIn`) objects.
@@ -200,6 +208,10 @@ When setting rules, the following Node affinity types define how hard or soft ea
 * **requiredDuringSchedulingIgnoredDuringExecution**: This is a hard rule that must be met.
 * **preferredDuringSchedulingIgnoredDuringExecution**: This is a preference, but the pod can run on a node where it is not guaranteed.
 
+{{% alert title="Note" color="primary" %}}
+Karpenter uses preferred as if it was required when determining if a pod can schedule on a node (absent topology requirements), or when determining if a pod can be shifted to a new node. However, if the preferred affinities can not be satisfied by existing nodes or any possible nodeclaim configuration then Karpenter will relax the preferences and only consider required affinities. In practice, this behavior may result in more nodes being created than expected, as Karpenter will prefer to create new nodes to satisfy preferences, rather than relax it's requirements to fit pods onto unpreferred but otherwise valid nodes with spare capacity.
+{{% /alert %}}
+
 The `IgnoredDuringExecution` part of each tells the pod to keep running, even if conditions change on the node so the rules no longer matched.
 You can think of these concepts as `required` and `preferred`, since Kubernetes never implemented other variants of these rules.
 
@@ -260,7 +272,7 @@ If they all fail, Karpenter will fail to provision the pod.
 Karpenter will backoff and retry over time.
 So if capacity becomes available, it will schedule the pod without user intervention.
 
-## Taints and tolerations
+### Taints and tolerations
 
 Taints are the opposite of affinity.
 Setting a taint on a node tells the scheduler to not run a pod on it unless the pod has explicitly said it can tolerate that taint. This example shows a NodePool that was set up with a taint for only running pods that require a GPU, such as the following:
@@ -307,7 +319,7 @@ spec:
 ```
 See [Taints and Tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) in the Kubernetes documentation for details.
 
-## Topology Spread
+### Topology Spread
 
 By using the Kubernetes `topologySpreadConstraints` you can ask the NodePool to have pods push away from each other to limit the blast radius of an outage. Think of it as the Kubernetes evolution for pod affinity: it lets you relate pods with respect to nodes while still allowing spread.
 For example:
@@ -354,7 +366,7 @@ See [Pod Topology Spread Constraints](https://kubernetes.io/docs/concepts/worklo
 NodePools do not attempt to balance or rebalance the availability zones for their nodes. Availability zone balancing may be achieved by defining zonal Topology Spread Constraints for Pods that require multi-zone durability, and NodePools will respect these constraints while optimizing for compute costs.
 {{% /alert %}}
 
-## Pod affinity/anti-affinity
+### Pod affinity/anti-affinity
 
 By using the `podAffinity` and `podAntiAffinity` configuration on a pod spec, you can inform the Karpenter scheduler of your desire for pods to schedule together or apart with respect to different topology domains. For example:
 
@@ -384,7 +396,7 @@ The anti-affinity rule would cause it to avoid running on any node with a pod la
 
 See [Inter-pod affinity and anti-affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity) in the Kubernetes documentation for details.
 
-## Persistent Volume Topology
+### Persistent Volume Topology
 
 Karpenter automatically detects storage scheduling requirements and includes them in node launch decisions.