EKS Kubernetes Cost Optimization: Why AWS Clusters Leave Compute Idle

EKS gives platform teams a wide range of compute choices, but that flexibility hasn't translated into efficient use of cluster capacity. In the 2026 State of Kubernetes Optimization Report, average CPU utilization across measured Kubernetes fleets on AWS, Azure and Google Cloud fell to 8% in 2025, while CPU overprovisioning rose to 69%. The figures point to an operational problem that node autoscaling alone can't solve: clusters can add the right kind of capacity and still pay for far more than applications consume.

The report draws on measurements from tens of thousands of clusters and millions of compute resources, captured before automated optimization was activated. Across the fleets, average memory utilization was 20%, down from 23%, while memory overprovisioning reached 79%. GPU utilization averaged 5%, including on EKS. Those measurements describe the environments studied, rather than every Kubernetes deployment, but they highlight the gap between what teams reserve and what workloads use.

Autoscaling follows requests, not actual demand

Karpenter can provision EC2 capacity in response to pending pods and their scheduling requirements. That makes it more flexible than relying on fixed node groups, but it doesn't know whether a pod's resource requests reflect what the application actually uses. Like other Kubernetes autoscalers, it makes decisions based on declared requests.

Consider a service configured to request four vCPUs and 8 GiB of memory. If it uses only 500 millicores of CPU and 1.5 GiB of memory, the request still shapes where Kubernetes can place it and what capacity the cluster needs. Karpenter can select a suitable instance; it can't, by itself, correct an inflated request.

There are understandable reasons for the gap. Engineers may set requests high to guard against CPU throttling or latency problems, and memory limits may be padded to reduce the chance of an out-of-memory termination. Those choices can persist in Helm charts and Terraform templates long after the workload's traffic or behavior changes. The result is a cluster that satisfies its configuration while carrying capacity applications rarely need. On EKS with self-managed EC2 nodes, the organization pays for that unused headroom.

GPU sharing can change the cost equation

Low GPU utilization has a different cost profile from idle CPU capacity: an unused accelerator can cost dollars per hour. The report puts average GPU utilization at 5%. It also notes that AWS raised H200 Capacity Block prices by 15% in January 2026, citing supply and demand imbalances. When supply is tight and accelerator prices rise, leaving most of a GPU idle becomes especially expensive.

Many inference services see bursts of traffic followed by quiet periods. Assigning a dedicated GPU instance to each endpoint means paying for the full instance during those lulls. Sharing accelerators through time-slicing, then placing workloads efficiently, can increase useful work per physical GPU.

ALLEN Digital, an edtech platform, ran seven machine learning models on AWS EKS, including BGE-M3, LlamaGuard and Multilingual E5 Large. The company had previously run dedicated instances on Amazon SageMaker, where intermittent demand left hardware idle. After moving to EKS, it used GPU time-slicing, automated bin-packing and a 50/50 mix of Spot and on-demand capacity. The report says the change cut total compute costs by more than 70% compared with SageMaker, with no compromise in inference latency.

Spot economics depend on region and workload

EC2 Spot Instances can cost up to 90% less than on-demand capacity, yet fewer than 2% of GPUs in the report ran on Spot. The risk of interruption, which can come with two minutes' notice, is a practical barrier for production teams. But the report's regional data suggests that treating Spot as uniformly risky can leave savings unused.

For NVIDIA T4 g4dn.xlarge instances, the report found that Spot survival probability in eu-west-3 remained above 90% over a 24-hour period. In us-east-1 and eu-central-1, it fell below 20% over the same period. Those differences make region choice part of capacity planning, especially for workloads that can tolerate interruption or resume elsewhere.

Price also varied substantially by location. For p4d.24xlarge instances with A100 GPUs, Spot prices differed by a factor of two to nearly five among us-east-1, us-east-2 and us-west-2. The report says shifting workloads in response to market conditions delivered savings of up to 80% against the average Spot price. Iterable, a marketing platform, used automated hourly Spot rebalancing across its AWS footprint. The reported results were 2,000 fewer CPUs, a 10% reduction in node count and a 75% cut in compute spending while supporting changing real-time traffic.

Graviton adoption fits a largely stateless fleet

ARM CPU nodes have grown 3.5 times faster than x86 nodes since the second quarter of 2024, and now account for 9% of the Kubernetes CPU fleet in the report. One factor behind the trend is that 93% of observed deployments were stateless. Stateless APIs, microservices and background workers can often run on linux/arm64 without the persistent storage constraints that may complicate a migration.

Moving a workload to Graviton can lower per-core costs. But changing processor architecture won't remove waste caused by oversized requests. Rightsizing first, then placing workloads on appropriate Graviton instances, addresses both the allocation and the compute price. That sequence matters: otherwise, a team may run an unnecessarily large workload on a cheaper processor and still carry avoidable capacity.

Static headroom doesn't guarantee reliability

Teams often justify broad overprovisioning as protection for service-level objectives. The report's findings challenge that assumption. In one production environment, out-of-memory kills averaged 40 to 50 per measurement window and sometimes exceeded 80, despite extensive resource padding.

A fixed buffer can leave quiet services with more resources than they use while failing to keep pace with a growing or memory-hungry workload. Automated rightsizing can reduce allocations when observed use stays below requests and raise limits when a workload shows memory pressure. In the environment described, this approach brought OOM kills close to zero while reducing provisioned CPU by about 50%.

Rightsize before making long-term commitments

Savings Plans and Compute Reserved Instances can reduce rates, but committing before removing excess capacity can lock the waste into a longer-term bill. The report says organizations managing commitments manually may waste up to 35% of prepaid resources as workloads and traffic shift. A discounted rate doesn't help much if the underlying capacity isn't being used.

A practical sequence starts with workload requests grounded in production consumption, then packs nodes efficiently using dynamic provisioning and suitable instance families such as Graviton. Interruption-tolerant workloads can move to Spot with availability-zone diversity and on-demand fallback. Only after those changes should teams commit to the baseline capacity that remains; the report describes a target of keeping commitment utilization near 98%.

The same utilization challenge appears across EKS, AKS and GKE in the report. The provider's compute catalog and autoscaler matter, but neither can keep resource requests aligned with changing application behavior on its own. For larger fleets, efficiency depends on ongoing measurement and adjustment of workload sizing, node placement, accelerator sharing and capacity purchasing. The useful next step for EKS teams is to compare declared requests with production consumption, then make those adjustments continuously as the workload changes.