How I can apply node selector to all pods?

[LucienShui]

I have many CPU and GPU nodes in my k3s cluster, I want to install milvus cluster with helm into fixed node using nodeSelector.

The problem is, I applied nodeSelector to all component inside helm config yaml, but there still have some components was deployed to some node which is unexpected with null nodeSelector, like milvus-pulsar-recovery-0 and milvus-pulsar-bookie-init-j5297.

How I can do that?

Here is my helm-milvus.yaml.

rootCoordinator:
  nodeSelector:
    node-type: cpu
    region: hangzhou
  replicas: 1
  resources: 
    limits:
      cpu: 1
      memory: 2Gi

queryCoordinator:
  nodeSelector:
    node-type: cpu
    region: hangzhou
  replicas: 1
  resources: 
    limits:
      cpu: "1"
      memory: 1Gi

dataCoordinator:
  nodeSelector:
    node-type: cpu
    region: hangzhou
  replicas: 1
  resources: 
    limits:
      cpu: "1"
      memory: 1Gi

proxy:
  nodeSelector:
    node-type: cpu
    region: hangzhou
  replicas: 1
  resources: 
    limits:
      cpu: 2
      memory: 8Gi

queryNode:
  nodeSelector:
    node-type: cpu
    region: hangzhou
  replicas: 36
  resources: 
    limits:
      cpu: 16
      memory: 128Gi

dataNode:
  nodeSelector:
    node-type: cpu
    region: hangzhou
  replicas: 1
  resources: 
    limits:
      cpu: 2
      memory: 8Gi

indexNode:
  nodeSelector:
    node-type: cpu
    region: hangzhou
  replicas: 4
  resources: 
    limits:
      cpu: 8
      memory: 16Gi

etcd:
  nodeSelector:
    node-type: cpu
    region: hangzhou
  autoCompactionMode: revision
  autoCompactionRetention: "1000"
  extraEnvVars:
  - name: ETCD_QUOTA_BACKEND_BYTES
    value: "4294967296"
  - name: ETCD_HEARTBEAT_INTERVAL
    value: "500"
  - name: ETCD_ELECTION_TIMEOUT
    value: "25000"
  - name: ETCD_SNAPSHOT_COUNT
    value: "10000"
  - name: ETCD_ENABLE_PPROF
    value: "true"
  persistence:
    accessMode: ReadWriteOnce
    enabled: true
    size: 30Gi  #SSD Required
    storageClass:
  replicaCount: 3
  resources:
    limits:
      cpu: 8
      memory: 16Gi
    requests:
      cpu: 8
      memory: 16Gi

pulsar:
  nodeSelector:
    node-type: cpu
    region: hangzhou
  enabled: true

  recovery:
    nodeSelector:
      node-type: cpu
      region: hangzhou
    initContainers:
      nodeSelector:
        node-type: cpu
        region: hangzhou

  bookie:
    nodeSelector:
      node-type: cpu
      region: hangzhou
    initContainers:
      nodeSelector:
        node-type: cpu
        region: hangzhou

  proxy:
    nodeSelector:
      node-type: cpu
      region: hangzhou
    resources:
      requests:
        memory: 18Gi
        cpu: 8
    replicaCount: 2
    configData:
      PULSAR_MEM: >
        -Xms8192M
        -Xmx8192M
        -XX:MaxDirectMemorySize=8192M
      httpNumThreads: "100"

  zookeeper:
    nodeSelector:
      node-type: cpu
      region: hangzhou
    volumes:
      persistence: true
      data:
        name: data
        size: 20Gi   #SSD Required
        storageClassName:
    resources:
      requests:
        memory: 8Gi
        cpu: 4
    configData:
      PULSAR_MEM: >
        -Xms4096M
        -Xmx4096M

  bookkeeper:
    nodeSelector:
      node-type: cpu
      region: hangzhou
    volumes:
      journal:
        name: journal
        size: 440Gi
        storageClassName:
      ledgers:
        name: ledgers
        size: 1100Gi  #SSD Required
        storageClassName:
    resources:
      requests:
        memory: 64Gi
        cpu: 8
    configData:
      PULSAR_MEM: >
        -Xms16384M
        -Xmx16384M
        -XX:MaxDirectMemorySize=32768M

  broker:
    nodeSelector:
      node-type: cpu
      region: hangzhou
    component: broker
    podMonitor:
      enabled: false
    replicaCount: 2
    resources:
      requests:
        memory: 72Gi
        cpu: 8
    configData:
      PULSAR_MEM: >
        -Xms16384M
        -Xmx16384M
        -XX:MaxDirectMemorySize=32768M
  
minio:
  nodeSelector:
    node-type: cpu
    region: hangzhou
  resources:
    limits:
      cpu: 8
      memory: 32Gi
  persistence:
    storageClass:
    accessMode: ReadWriteOnce
    size: 6730Gi

Here is the result of kubectl describe pod milvus-pulsar-recovery-0, you can see Node-Selectors: <none>.

$ kubectl describe pod milvus-pulsar-recovery-0 -n milvus
Name:             milvus-pulsar-recovery-0
Namespace:        milvus
Priority:         0
Service Account:  default
Node:             gpu-1/192.168.0.31
Start Time:       Fri, 23 Aug 2024 14:55:23 +0800
Labels:           app=pulsar
                  apps.kubernetes.io/pod-index=0
                  cluster=milvus-pulsar
                  component=recovery
                  controller-revision-hash=milvus-pulsar-recovery-65f7f8755b
                  release=milvus
                  statefulset.kubernetes.io/pod-name=milvus-pulsar-recovery-0
Annotations:      prometheus.io/port: 8000
                  prometheus.io/scrape: true
Status:           Running
IP:               10.42.1.24
IPs:
  IP:           10.42.1.24
Controlled By:  StatefulSet/milvus-pulsar-recovery
Init Containers:
  pulsar-bookkeeper-verify-clusterid:
    Container ID:  containerd://eede809e4b645f9a0354ecffd068c84dc03eda0d2d75b803bc70de7474ce5f5c
    Image:         apachepulsar/pulsar:2.9.5
    Image ID:      docker.io/apachepulsar/pulsar@sha256:9bcbad78a65a09b3fe5d3acd158574ad0eda4531cc8c49db16760a2ed1364070
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
    Args:
      bin/apply-config-from-env.py conf/bookkeeper.conf;until bin/bookkeeper shell whatisinstanceid; do
        sleep 3;
      done;
      
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 23 Aug 2024 14:55:24 +0800
      Finished:     Fri, 23 Aug 2024 14:55:26 +0800
    Ready:          True
    Restart Count:  0
    Environment Variables from:
      milvus-pulsar-recovery  ConfigMap  Optional: false
    Environment:              <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zf7sk (ro)
Containers:
  milvus-pulsar-recovery:
    Container ID:  containerd://c54c5e72a74a76474995eeea07ccf8dfcda3f2108bb55bacbb5d55c0752784dd
    Image:         apachepulsar/pulsar:2.9.5
    Image ID:      docker.io/apachepulsar/pulsar@sha256:9bcbad78a65a09b3fe5d3acd158574ad0eda4531cc8c49db16760a2ed1364070
    Port:          8000/TCP
    Host Port:     0/TCP
    Command:
      sh
      -c
    Args:
      bin/apply-config-from-env.py conf/bookkeeper.conf;
      OPTS="${OPTS} -Dlog4j2.formatMsgNoLookups=true" exec bin/bookkeeper autorecovery
      
    State:          Running
      Started:      Fri, 23 Aug 2024 14:55:26 +0800
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:     1
      memory:  512Mi
    Environment Variables from:
      milvus-pulsar-recovery  ConfigMap  Optional: false
    Environment:              <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zf7sk (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 
Volumes:
  kube-api-access-zf7sk:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

[xiaofan-luan]

/assign @LoveEachDay
please help on it. My guess is that helm is overwritten somewhere

[LucienShui]

@xiaofan-luan @LoveEachDay Hi, sorry to bother, is there any progress?

[xiaofan-luan]

I have many CPU and GPU nodes in my k3s cluster, I want to install milvus cluster with helm into fixed node using nodeSelector.

The problem is, I applied nodeSelector to all component inside helm config yaml, but there still have some components was deployed to some node which is unexpected with null nodeSelector, like milvus-pulsar-recovery-0 and milvus-pulsar-bookie-init-j5297.

How I can do that?

Here is my helm-milvus.yaml.

rootCoordinator:
  nodeSelector:
    node-type: cpu
    region: hangzhou
  replicas: 1
  resources: 
    limits:
      cpu: 1
      memory: 2Gi

queryCoordinator:
  nodeSelector:
    node-type: cpu
    region: hangzhou
  replicas: 1
  resources: 
    limits:
      cpu: "1"
      memory: 1Gi

dataCoordinator:
  nodeSelector:
    node-type: cpu
    region: hangzhou
  replicas: 1
  resources: 
    limits:
      cpu: "1"
      memory: 1Gi

proxy:
  nodeSelector:
    node-type: cpu
    region: hangzhou
  replicas: 1
  resources: 
    limits:
      cpu: 2
      memory: 8Gi

queryNode:
  nodeSelector:
    node-type: cpu
    region: hangzhou
  replicas: 36
  resources: 
    limits:
      cpu: 16
      memory: 128Gi

dataNode:
  nodeSelector:
    node-type: cpu
    region: hangzhou
  replicas: 1
  resources: 
    limits:
      cpu: 2
      memory: 8Gi

indexNode:
  nodeSelector:
    node-type: cpu
    region: hangzhou
  replicas: 4
  resources: 
    limits:
      cpu: 8
      memory: 16Gi

etcd:
  nodeSelector:
    node-type: cpu
    region: hangzhou
  autoCompactionMode: revision
  autoCompactionRetention: "1000"
  extraEnvVars:
  - name: ETCD_QUOTA_BACKEND_BYTES
    value: "4294967296"
  - name: ETCD_HEARTBEAT_INTERVAL
    value: "500"
  - name: ETCD_ELECTION_TIMEOUT
    value: "25000"
  - name: ETCD_SNAPSHOT_COUNT
    value: "10000"
  - name: ETCD_ENABLE_PPROF
    value: "true"
  persistence:
    accessMode: ReadWriteOnce
    enabled: true
    size: 30Gi  #SSD Required
    storageClass:
  replicaCount: 3
  resources:
    limits:
      cpu: 8
      memory: 16Gi
    requests:
      cpu: 8
      memory: 16Gi

pulsar:
  nodeSelector:
    node-type: cpu
    region: hangzhou
  enabled: true

  recovery:
    nodeSelector:
      node-type: cpu
      region: hangzhou
    initContainers:
      nodeSelector:
        node-type: cpu
        region: hangzhou

  bookie:
    nodeSelector:
      node-type: cpu
      region: hangzhou
    initContainers:
      nodeSelector:
        node-type: cpu
        region: hangzhou

  proxy:
    nodeSelector:
      node-type: cpu
      region: hangzhou
    resources:
      requests:
        memory: 18Gi
        cpu: 8
    replicaCount: 2
    configData:
      PULSAR_MEM: >
        -Xms8192M
        -Xmx8192M
        -XX:MaxDirectMemorySize=8192M
      httpNumThreads: "100"

  zookeeper:
    nodeSelector:
      node-type: cpu
      region: hangzhou
    volumes:
      persistence: true
      data:
        name: data
        size: 20Gi   #SSD Required
        storageClassName:
    resources:
      requests:
        memory: 8Gi
        cpu: 4
    configData:
      PULSAR_MEM: >
        -Xms4096M
        -Xmx4096M

  bookkeeper:
    nodeSelector:
      node-type: cpu
      region: hangzhou
    volumes:
      journal:
        name: journal
        size: 440Gi
        storageClassName:
      ledgers:
        name: ledgers
        size: 1100Gi  #SSD Required
        storageClassName:
    resources:
      requests:
        memory: 64Gi
        cpu: 8
    configData:
      PULSAR_MEM: >
        -Xms16384M
        -Xmx16384M
        -XX:MaxDirectMemorySize=32768M

  broker:
    nodeSelector:
      node-type: cpu
      region: hangzhou
    component: broker
    podMonitor:
      enabled: false
    replicaCount: 2
    resources:
      requests:
        memory: 72Gi
        cpu: 8
    configData:
      PULSAR_MEM: >
        -Xms16384M
        -Xmx16384M
        -XX:MaxDirectMemorySize=32768M
  
minio:
  nodeSelector:
    node-type: cpu
    region: hangzhou
  resources:
    limits:
      cpu: 8
      memory: 32Gi
  persistence:
    storageClass:
    accessMode: ReadWriteOnce
    size: 6730Gi

Here is the result of kubectl describe pod milvus-pulsar-recovery-0, you can see Node-Selectors: <none>.

$ kubectl describe pod milvus-pulsar-recovery-0 -n milvus
Name:             milvus-pulsar-recovery-0
Namespace:        milvus
Priority:         0
Service Account:  default
Node:             gpu-1/192.168.0.31
Start Time:       Fri, 23 Aug 2024 14:55:23 +0800
Labels:           app=pulsar
                  apps.kubernetes.io/pod-index=0
                  cluster=milvus-pulsar
                  component=recovery
                  controller-revision-hash=milvus-pulsar-recovery-65f7f8755b
                  release=milvus
                  statefulset.kubernetes.io/pod-name=milvus-pulsar-recovery-0
Annotations:      prometheus.io/port: 8000
                  prometheus.io/scrape: true
Status:           Running
IP:               10.42.1.24
IPs:
  IP:           10.42.1.24
Controlled By:  StatefulSet/milvus-pulsar-recovery
Init Containers:
  pulsar-bookkeeper-verify-clusterid:
    Container ID:  containerd://eede809e4b645f9a0354ecffd068c84dc03eda0d2d75b803bc70de7474ce5f5c
    Image:         apachepulsar/pulsar:2.9.5
    Image ID:      docker.io/apachepulsar/pulsar@sha256:9bcbad78a65a09b3fe5d3acd158574ad0eda4531cc8c49db16760a2ed1364070
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
    Args:
      bin/apply-config-from-env.py conf/bookkeeper.conf;until bin/bookkeeper shell whatisinstanceid; do
        sleep 3;
      done;
      
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 23 Aug 2024 14:55:24 +0800
      Finished:     Fri, 23 Aug 2024 14:55:26 +0800
    Ready:          True
    Restart Count:  0
    Environment Variables from:
      milvus-pulsar-recovery  ConfigMap  Optional: false
    Environment:              <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zf7sk (ro)
Containers:
  milvus-pulsar-recovery:
    Container ID:  containerd://c54c5e72a74a76474995eeea07ccf8dfcda3f2108bb55bacbb5d55c0752784dd
    Image:         apachepulsar/pulsar:2.9.5
    Image ID:      docker.io/apachepulsar/pulsar@sha256:9bcbad78a65a09b3fe5d3acd158574ad0eda4531cc8c49db16760a2ed1364070
    Port:          8000/TCP
    Host Port:     0/TCP
    Command:
      sh
      -c
    Args:
      bin/apply-config-from-env.py conf/bookkeeper.conf;
      OPTS="${OPTS} -Dlog4j2.formatMsgNoLookups=true" exec bin/bookkeeper autorecovery
      
    State:          Running
      Started:      Fri, 23 Aug 2024 14:55:26 +0800
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:     1
      memory:  512Mi
    Environment Variables from:
      milvus-pulsar-recovery  ConfigMap  Optional: false
    Environment:              <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zf7sk (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 
Volumes:
  kube-api-access-zf7sk:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

I belive those are one time pod and after pulsar start those pod will be terminated automatically.
Does that matters?

[LucienShui]

for milvus-pulsar-recovery-0 and milvus-pulsar-bookie-init-j5297 they are one time init pod so I guess it's not important.

Yeah, I got it, thanks!

Other pod should follow the node selector.

What’s the best practice for specifying a nodeSelector in Milvus Helm? Are there any available docs on this?

[LoveEachDay]

@LucienShui We'll add a section to describe how to use nodeSelector for all milvus components and dependencies in the milvus-helm chart documentation.

[LucienShui]

@LoveEachDay Thank you very much, I would really appreciate it if you could let me know when the documentation is ready.

[LoveEachDay]

@LoveEachDay Thank you very much, I would really appreciate it if you could let me know when the documentation is ready.

Ok, I'll link the doc here.

[LoveEachDay]

@LucienShui https://github.com/zilliztech/milvus-helm/tree/master/charts/milvus#node-selector-affinity-and-tolerations-configuration-guide

Here we add a guide for nodeSelector, affinity and toleration setup for milvus components and all dependencies.