Kube Startup CPU Boost for Containers Initiating with Hunger
We deploy an application container in Kubernetes cluster. It might get issue related to resource usage and provisioning — two scenarios we’re interested in:
- Immediate resource spike: The container gets terminated due to a sudden increase in CPU usage exceeding the node’s capacity, or it hit the configured resource value
- Temporary high usage: The container exhibit a burst of high CPU consumption for a short initial period, followed by a decrease and stabilization
In general, we tend to add node spec and/or change the resourcerequest value and leave it which will reserve the compute resource forever that won’t be used after some time. But the application container is requesting big resource only for very minimal time at startup.
Google Cloud Run already has this feature if you have created any service in the Serverless platform.
This option lets the container to consume more CPU at startup and for additional 10 seconds afterwards, which is billed accordingly. The increased difference is based on allocated CPU as below:
Kubernetes World
Let’s get into how this can be done in Kubernetes environment. Kube Startup CPU Boost to the rescue! What the booster does is changes the CPU resource value to higher and as specified percentage increase for a duration. Afterwards, the resource spec fallback to default value. The CPU boost is open source and is under google’s github org.
A clean config will make it more clear:
apiVersion: autoscaling.x-k8s.io/v1alpha1
kind: StartupCPUBoost
metadata:
name: booster
selector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values: ["my-demo-app"]
spec:
resourcePolicy:
containerPolicies:
- containerName: my-demo-app
percentageIncrease:
value: 80
durationPolicy:
podCondition:
type: Ready
status: "True"With above definition, the booster will increase the resource of pod with label app.kubernetes.io/name=my-demo-app by 80 percent till the pod is not in Ready state. Likewise, these conditions could be changed also, like, increasing the resource by multiple.
spec:
containerPolicies:
- containerName: my-demo-app
fixedResources:
requests: "1"
limits: "2"And for a fixed duration too(apart from earlier Ready state)!
spec:
durationPolicy:
fixedDuration:
unit: Seconds
value: 30Installation steps and requirements
The installation is quite simple by applying the latest Kubernetes manifest from releases page.
kubectl apply -f https://github.com/google/kube-startup-cpu-boost/releases/latest/download/manifests.yamlIt will create CRDs in kube-startup-cpu-boost-system namespace along with some ValidatingWebhookConfiguration and deployments.
The cluster should have InPlacePodVerticalScaling feature gate enabled which is available on Kubernetes 1.27 or newer version. If you are using Kind cluster, the feature gate needs to be enabled while creating cluster.
Behind the scene architecture
Kube Startup CPU Boost temporarily increases CPU resources of pod even without restarting the containers, thanks to a feature in Kubernetes called “In-place Resource Resize for Kubernetes Pods”. This makes containers which are CPU intensive in startup time spin-up faster and not reserving unused resources later.
Here’s how it works:
- Webhook Triggers: When a new container (pod) starts in the system, Kube Startup CPU Boost automatically receives a notification.
- Boost Configuration Check: The notification triggers the Boost Manager to search for specific settings related to CPU boosts for this new container.
- Resource Adjustment: If matching settings are found, the Boost Manager instructs the webhook to increase the container’s CPU requests and limits as defined in the configuration.
- Temporary Boost: Once the container reaches its desired operational state, the Boost Manager automatically adjusts the CPU resources back to their original values.
There should be room for extra CPU for this booster to work, otherwise the pod will not be scheduled.
