Jupyterlab spawned server properties

[gmaze]

hi !

Question 1

I wonder what are the possible values for the kubespawner_override properties that one can set in the qhub-config.yml file ?

For instance, by default, I have

profiles:
  jupyterlab:
  - display_name: Small Instance
    description: Stable environment with 1 cpu / 1 GB ram
    default: true
    kubespawner_override:
      cpu_limit: 1
      cpu_guarantee: 1
      mem_limit: 1G
      mem_guarantee: 1G
      image: quansight/qhub-jupyterlab:1abd4efb8428a9d851b18e89b6f6e5ef94854334

I added a new entry with these modifications:

  - display_name: Larger Instance (Test)
    description: Stable environment with 2 cpu / 2 GB ram
    kubespawner_override:
      cpu_limit: 2
      cpu_guarantee: 2
      mem_limit: 2G
      mem_guarantee: 2G
      image: quansight/qhub-jupyterlab:1abd4efb8428a9d851b18e89b6f6e5ef94854334

but jupyterhub throws this when this server profile was spawned:

Event log
Server requested
2021-01-05 13:48:24+00:00 [Warning] 0/3 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match node selector.
2021-01-05 13:48:26+00:00 [Normal] pod didn't trigger scale-up (it wouldn't fit if a new node is added): 1 max node group size reached, 2 Insufficient cpu

Question 2

Do the values of cpu/mem limits/guarantee are constrained by the image or by the worker or user instance set in the kubernetes nodes (I deployed on GCP)

I modified the default values to:

google_cloud_platform:
  project: <MY-PROJECT>
  region: us-central1
  zone: us-central1-c
  availability_zones:
  - us-central1-c
  kubernetes_version: 1.14.10-gke.31
  node_groups:
    general:
      instance: n1-standard-2
      min_nodes: 1
      max_nodes: 1
    user:
      instance: n1-standard-4
      min_nodes: 1
      max_nodes: 50
    worker:
      instance: n1-standard-2
      min_nodes: 1
      max_nodes: 100

but I don't have a clear understanding of the detailed interactions between the properties of nodes, pods, images, workers, etc ... throughout kubernetes, gcp, jupyterhub, dask...

thanks for your help/explanation !

[costrouc]

what are the possible values for the kubespawner_override properties that one can set in the qhub-config.yml file ?

Anything in https://jupyterhub-kubespawner.readthedocs.io/en/latest/spawner.html c.KubeSpawner.*. This should provide you full capabilities to modify the pod spec of the jupyterlab instance.

Do the values of cpu/mem limits/guarantee are constrained by the image or by the worker or user instance set in the kubernetes nodes (I deployed on GCP)

limits/guarantees are Kubernetes terminology. guarantees are what the kubernetes scheduler uses to schedule pods. If a given node has 4 GB of RAM and 3 GB is already guaranteed for other pods. A pod requesting a guarentee of 1.1 GB would not be able to be scheduled on the node but if it were requesting a guarantee of 0.9 GB it would be. limits are imposed by cgroups and linux https://en.wikipedia.org/wiki/Cgroups and enforce the maximum memory/cpu that a given container can consume. So imagine the pod had a request of guarantee 0.9 GB and limit 3 GB it would be scheduable on the node and could theoretically use 3 GB of RAM. But if it consumed more than 3 GB of RAM it would be terminated.

[gmaze]

Thanks for your quick answer !

[gmaze]

hi @costrouc
so, if a pod exceed limits, this should trigger the instantiation of a new node (Q2) ? within the limits of max_nodes GCP mode properties ?
this is the scale-up step, right ?
g

[costrouc]

limits are enforced by the linux kernel and will terminate the running container and will not trigger a scale up. Scale ups happen when there are more scheduled pods than available resources. Kubernetes will then create new nodes to satisfy the demand within the min_nodes and max_nodes in the node_group. In the case of jupyterlab instances this is the general node_pool.

[Adam-D-Lewis]

I'm closing old discussions in a repo cleanup effort. Feel free to re-open if needed.