Scheduling Policies

Kubernetes Scheduling Policies

Scheduler configuration has changed in Kubernetes v1.19. For older clusters, skip to the section configuring the kube-scheduler.

On Kubernetes v1.19 or later

Kubernetes v1.19 supports configuring multiple scheduling policies with a single scheduler. We are using this to define a bin-packing scheduling policy in all v1.19 clusters by default.

To use that scheduling policy, all that is required is to specify the scheduler name bin-packing-scheduler in the Pod spec. For example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 5
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      schedulerName: bin-packing-scheduler
      containers:
      - name: nginx
        image: nginx:1.17.8
        resources:
          requests:
            cpu: 200m

The pods of this deployment will be scheduled onto the nodes which already have the highest resource utilisation, to optimise for autoscaling or ensuring efficient pod placement when mixing large and small pods in the same cluster.

If a scheduler name is not specified then the default spreading algorithm will be used to distribute pods across all nodes.

Configuring the kube-scheduler

Note: These instructions are for Kubernetes v1.18 and earlier.

Setting non-default kube-scheduler policies requires some manual configuration on the master node.

$ ssh core@cluster-pj2n3k4nz5qi-master-0
$ sudo su
# vi /etc/kubernetes/scheduler

In this file, change the KUBE_SCHEDULER_ARGS line to add these two policy parameters:

KUBE_SCHEDULER_ARGS="--leader-elect=true --policy-configmap=scheduler-policy --policy-configmap-namespace=kube-system"

The scheduler policy must be provided in a json format, and stored in a ConfigMap with this name and namespace. A default policy can be taken from the Kubernetes examples, and an example including the ConfigMap is provided in the following section.

If the predicates, priorities, or any other sections of the policy configuration are not specified they will assume the default values. But, if the predicates section is given, then all the desired predicates must be listed. It is not possible to only add or remove single predicates. This also applies to the priorities section.

Once the ConfigMap has been created the kube-scheduler must be restarted on the master node in order to use the new configuration.

# systemctl restart kube-scheduler.service

This step must be repeated if the ConfigMap is changed.

Packing policy

The default kube-scheduler priorities include the priority LeastRequestedPriority, which gives preference to scheduling on nodes which have less CPU/memory usage than others. This causes pods to be spread out evenly among all nodes.

For cluster autoscaling it is better to pack pods together onto the fewest possible number of nodes, so that the autoscaler can remove nodes which are not needed. To achieve this, the LeastRequestedPriority can be changed for MostRequestedPriority. An example scheduler policy which has this change, and has a higher weighting for MostRequestedPriority than other priorities, can be taken from here:

$ cat scheduler-policy.yaml

kind: ConfigMap
apiVersion: v1
metadata:
  name: scheduler-policy
  namespace: kube-system
data:
  policy.cfg: |
      {
        "kind": "Policy",
        "apiVersion": "v1",
        "priorities": [
          {"name": "MostRequestedPriority", "weight": 100},
          {"name": "ServiceSpreadingPriority", "weight": 1},
          {"name": "EqualPriority", "weight": 1},
          {"name": "ImageLocalityPriority", "weight": 1},
          {"name": "SelectorSpreadPriority", "weight": 1},
          {"name": "InterPodAffinityPriority", "weight": 1},
          {"name": "LeastRequestedPriority", "weight": 1},
          {"name": "BalancedResourceAllocation", "weight": 1},
          {"name": "NodePreferAvoidPodsPriority", "weight": 1},
          {"name": "NodeAffinityPriority", "weight": 1},
          {"name": "TaintTolerationPriority", "weight": 1}
        ]
      }

kubectl create -f scheduler-policy.yaml