Setting up Over-Provisioning
It's considered a best practice to create appropriate PriorityClass for your applications. Now, let's create a global default priority class using the field globalDefault:true. This default PriorityClass will be assigned pods/deployments that don’t specify a PriorityClassName.
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: default
value: 0
globalDefault: true
description: "Default Priority class."
We'll also create PriorityClass that will be assigned to pause pods used for over-provisioning with priority value -1.
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: pause-pods
value: -1
globalDefault: false
description: "Priority class used by pause-pods for overprovisioning."
Pause pods make sure there are enough nodes that are available based on how much over provisioning is needed for your environment. Keep in mind the —max-size parameter in ASG (of EKS node group). Cluster Autoscaler won’t increase number of nodes beyond this maximum specified in the ASG
apiVersion: apps/v1
kind: Deployment
metadata:
name: pause-pods
namespace: other
spec:
replicas: 2
selector:
matchLabels:
run: pause-pods
template:
metadata:
labels:
run: pause-pods
spec:
priorityClassName: pause-pods
containers:
- name: reserve-resources
image: registry.k8s.io/pause
resources:
requests:
memory: "6.5Gi"
In this case we're going to schedule a single pause pod requesting 7Gi of memory, which means it will consume basically entire m5.large instance. This will result in us always having a "spare" worker node available.
Apply the updates to your cluster: