This project capitalizes on the Kubernetes scheduler extender model to enhance the functionality of the scheduler. The Kubernetes default scheduler offers a lot of value in terms of the vast number of predicates and priorities it implements. Moreover, it is highly configurable, in the sense that we can change the default configuration to one that better matches our needs. However, there is still room for improvement, and this improvement can be implemented in the form of extensions.
We will use this project as an incubator for new Kube scheduler extension ideas. The base scheduler extender is built on the Kubernetes scheduler example.
We provide a solution for allowing the expansion of the scheduler extender to incorporate additional predicates and priority functions. We refer to a collection of predicates and priority functions which tackle a particular objective as an Agent. In particular, we include in this project the following agents:
- Safe balancing and overloading agent (safe)
- Policy-based optimizing agent (pigeon)
- Node-congestion aware agent (congestion)
- Template agent (foo)
- Reward biased scheduler extender (kubeRL)
kubeRLis a software framework implementing a reward biased scheduler extender in Kubernetes.kubeRLadopts the idea from reinforcement learning to adaptively learn the failures or performance issues of containers and model their runtime performance on nodes as rewards. kubeRL then adaptively prevents scheduling pods on nodes that give low rewards.kubeRLprovides one-click deploymentmake deploy-onlyand a demo tutorial. To usekubeRL,please skip the following steps.
We seek an architecture which (1) allows ease of expandability and introducion of new agents, (2) provides a development environment for an agent, in isolation of other agents, and (3) enables the selective deployment of agents as scheduler extenders. A detailed description is provided here.
Briefly, the Kubernetes scheduler extender works as follows.
The above is packaged into one pod with multiple containers. One container runs the (default) scheduler. Additional container(s) run the extender(s). Each extender consists of an http server and an agent, implementing its corresponding predicates and priority functions. Each extender container will use a different port. An example with two extender containers follows.
In this project, several examples of agents are provided, as well as a template agent. To add an agent, say foo, one needs to:
-
Create a file foo.go in package main. Its sole purpose is to register the agent predicates and prioritiy functions with the router created by main.go as depicted below.
Further, the top line in the file specifies to the go builder a tag name with the name of the agent foo (a tag all is also added, allowing the building of all the agents if needed) as depicted below. This way the building step of an agent is isolated from any potential issues with other agents.
-
Create a package foo which includes the implementation of the agent predicates and prioritiy functions. Additional packages may be added.
-
Create a Docker file to build the image for the agent. Notice the inclusion of the tag foo in the go install step.
-
Create a deployment yaml file. This file serves several objectives:
- define the scheduler name (my-scheduler)
- define the extender configuration, specifying the port number, the predicated and priority functions, as well as their weights.
- define the pod deployment
- define the (default) scheduler container
- Define one or more extender containers, each specifying the name of the agent image and the container port. An example of a pod with a scheduler conainer and two extender containers is here. Note that container 1 runs the safe agent extender on port 5401, whereas container 2 runs the congestion agent extender on port 5402. Environment variables for each agent are included, as well as the environment variable HTTP_PORT which defines the port number (default is 80).
- A note on deployment in clusters with RBAC security: The scheduler needs to run with some privileges in kube-system. A yaml which sets up a service account by the name extender-serviceaccount with the required privileges is provided service-account. If used, one needs to add
serviceAccountName: extender-serviceaccountto the spec.template.spec of the Deployment in the extender yaml file. The deployment of the service account yaml should precede the deployment of the extender yaml.
Now, we provide instructions to build an agent image, deploy the scheduler extender, and test by placing a pod using the scheduler extender. In the following xxx refers to a particular agent of interest. For example, substitutions such as safe, pigeon, congestion, and foo are possible.
$ docker build -f dockerfiles/Dockerfile.xxx -t kube-sched-ext-xxx:0.0 .
$ kubectl create -f yamlfiles/extender.xxx.yaml
If multiple extenders are needed to run simultaneously, then use the corresponding yaml file
$ kubectl create -f yamlfiles/extender.xxx.yyy.yaml
$ kubectl create -f yamlfiles/test-pod.yaml
$ kubectl describe test-pod
Name: test-pod
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 42s my-scheduler Successfully assigned default/test-pod to worker-node1
Normal Pulling 41s kubelet, worker-node1 pulling image "nginx"
Normal Pulled 41s kubelet, worker-node1 Successfully pulled image "nginx"
Normal Created 41s kubelet, worker-node1 Created container
Normal Started 40s kubelet, worker-node1 Started container
The my-scheduler pod contains two or more containers. One container, named my-scheduler-ctr, runs the default Kubernetes scheduler. Additional containers, named my-extender-ctr-nnn, where nnn is 1, 2, ... and so on, each runs an extender (http server) and one or more agents, implementing the related predicates and priority functions.
The following command serves viewing the logs of the scheduler.
kubectl logs -f <my-scheduler-...> -c my-scheduler-ctr -n kube-system
And, the following command serves viewing the logs of a particular extender agent (substitute nnn by the container number of interest). The log level for the extender is controlled by the LOG_LEVEL environment variable. Possible values are: TRACE, DEBUG, INFO, WARNING, ERROR, and ALERT. The default value is INFO.
kubectl logs -f <my-scheduler-...> -c my-extender-ctr-nnn -n kube-system
The following files have been removed from the vendor folder as they contain authentication credentials. You may download them from k8s.io/kubernetes to your forked repository.
vendor/k8s.io/kubernetes/test/e2e/common/runtime.go
vendor/k8s.io/kubernetes/test/e2e_node/runtime_conformance_test.go