Kubernetes Autoscaling Under Load

Autoscaling in Kubernetes is a multi-layered coordination system that operates on two levels: Horizontal Pod Autoscaling (HPA) scales the number of pod replicas based on resource demand, while the Cluster Autoscaler (CA) scales the underlying node infrastructure when resource limits are reached.

Mental Model

When user traffic surges, pod CPU/memory utilization rises, but autoscaling is delayed by metric collection, reconciliation, scheduling, image startup, and readiness. The cAdvisor agent on each node collects usage metrics, which are polled periodically by the Metrics Server. The Horizontal Pod Autoscaler (HPA) queries the Metrics Server and calculates the required pod replica count based on a target utilization threshold. If existing nodes lack capacity to host the new replicas, the pods are scheduled but remain in a Pending state. The Cluster Autoscaler detects these pending pods and commands the cloud provider to provision new virtual machine nodes. Once nodes join the cluster, pending pods can schedule, pull their images, boot, pass readiness checks, and join the Ingress routing pool to handle the traffic.

Components and Why They Exist

Traffic Generator

The traffic generator simulates external user requests hitting the cluster at configurable rates (RPS). It directs baseline or burst traffic to the Ingress tier, causing requests to be dropped with 503 errors on the telemetry chart whenever existing pods saturate before new replicas finish booting.

Ingress Controller

The Ingress controller is the entry point for external HTTP traffic into the cluster. It performs load balancing by routing incoming request particles round-robin across pods currently in the Ready state.

Horizontal Pod Autoscaler (HPA)

The Horizontal Pod Autoscaler (HPA) automates pod scaling to handle varying load dynamically. It periodically polls the Metrics Server for CPU utilization and raises the desired replica counter once observed CPU usage exceeds the target threshold.

Cluster Autoscaler

The Cluster Autoscaler resolves node-level resource exhaustion by adjusting the VM node count. It detects pods stuck in a Pending state due to insufficient CPU or memory on existing machines and triggers cloud provisioning for new VM nodes, or terminates empty nodes after scale-down cooldowns.

VM Nodes

The VM nodes are the virtual or physical host machines that provide CPU and memory capacity to run containers. Each node hosts pod replicas up to its maximum resource allocation, displaying spinning provisioning loaders during cloud boot and live capacity bars once active.

Pods

Pods are the smallest deployable units in Kubernetes that run the application containers. Each pod progresses through distinct lifecycle states—Pending (scheduling), ImagePulling (downloading), Initializing (booting and readiness checks), Ready (serving traffic), and Terminating—changing badge colors as it moves toward active service.

Edge Cases and Misconceptions

• Misconception: Autoscaling is instantaneous: There is a cumulative lag consisting of metrics scrape delays (cAdvisor/Metrics Server), HPA calculation loops, cloud VM provisioning delays (which can take minutes), and pod boot times.
• Misconception: Setting CPU limits high prevents pod crashes: If a pod runs at 100% CPU capacity for too long under load, it can become unresponsive; CPU limits do not create capacity or guarantee prompt replacement. Runtimes might crash or experience cgroups OOM kills, leading to a cascading crash loop as traffic redirects to surviving pods.
• Edge Case: Scale-Down Cooldown (Thrashing): If the scale-down stabilization window is too short, brief fluctuations in traffic will cause the cluster to repeatedly add and delete pods (thrashing), wearing out resources and degrading performance.
• Edge Case: Karpenter vs Cluster Autoscaler: Unlike CA which waits for pods to be Pending and spins up nodes in minutes, advanced schedulers like Karpenter bypass the scheduler queues to provision right-sized nodes in seconds, reducing outage duration.

Open the Kubernetes Autoscaling simulator sandbox after the walkthrough when you want to experiment with load rates, probe settings, node capacity constraints, and cooldown policies.