Kubernetes Adds Gang Scheduling and Finer GPU Control for AI Workloads
Kubernetes already runs much of the world's AI inference. The Cloud Native Computing Foundation's January 2026 Annual Cloud Native Survey found that 66% of organizations hosting generative AI models use Kubernetes for inference, and that's before counting teams running other AI workloads on it. At KubeCon + CloudNativeCon North America 2026 in Salt Lake City, the CNCF and its vendors laid out what comes next: smarter scheduling, bigger control planes, faster container startup, and tighter limits on autonomous agents.
The nearest milestone is Kubernetes 1.38, due December 16. It advances gang scheduling and additional accelerator-allocation capabilities toward stable status. But the conference message reached past one release. Karpenter, the open-source node lifecycle manager that provisions and terminates compute in response to unscheduled pod requests, is getting its own improvements, alongside several emerging CNCF projects.
Control Planes Built for Scale
"AI workloads can spin up 10s of 1000s of GPU instances per cluster, and things break at that scale," said Mike Stefaniak, AWS's Head of Containers, in his keynote. His prescription was to push fixes discovered in enormous production deployments upstream instead of leaving them in a provider's private fork.
AWS hit bottlenecks while pushing past Kubernetes' conventional 5,000-node design envelope toward 100,000 nodes. In a million-Pod benchmark, an API-server watch-cache lock kept writers waiting more than half a second. A lazy-snapshot implementation cut that wait to two milliseconds. Other changes removed serialization from Horizontal Pod Autoscaler processing and replaced repeated scheduler volume-reference calculations with incremental updates. Stefaniak reported scheduling rates above 300 Pods per second under sustained backlog in AWS's largest clusters, and said those fixes already shipped in Kubernetes 1.37.
For 1.38, AWS co-authored a proposal to compress API objects before they're stored in etcd, the backing data store. "It's accepted for 1.38 alpha, and the first code was actually merged just last week," Stefaniak said. The immediate target is storage pressure in very large clusters, a problem that only shows up once you're operating at that size.
Scheduling Whole AI Jobs
The bigger shift is treating distributed AI workloads as coordinated groups instead of unrelated Pods. Gang scheduling does exactly that, and the Workload and PodGroup APIs provide native scheduling primitives for those groups.
The payoff is practical. A training job with only some of its workers placed can sit on expensive GPUs while making no progress. Gang scheduling holds the whole group until its required minimum membership fits. It coordinates admission rather than promising every container starts at the same instant, but it stops GPUs from idling while they wait on workers that can't be placed yet.
The workload-aware scheduling roadmap also targets beta maturity for topology-aware placement and CompositePodGroups. Those address communication-sensitive placement and workloads made of multiple coordinated groups, such as distributed training and disaggregated inference. Dynamic Resource Allocation (DRA) fills in the device side by describing accelerator capabilities and capacity more precisely. In 1.38, DRA features for partitionable devices, consumable capacity, fractional capacity ranges, and device binding conditions are slated for general availability. That moves scheduling well past counting whole GPUs.
Getting GPUs Online Faster
Karpenter now models devices on the instance types it can provision, Stefaniak said. That closes the gap between DRA's device inventories and just-in-time node creation. "GPU capacity can scale from zero and be optimally placed," he said. To be clear, this is Karpenter work rather than a Kubernetes 1.38 feature.
Container startup is the next wall. AI images can reach tens of gigabytes, and parallel downloads just move the delay to unpacking layers. Stefaniak said an upstreamed parallel-unpack implementation in containerd 2.2 cut registry-to-node time for a 10GB image from two minutes to consistently under 45 seconds in AWS testing. Multi-gigabyte containers still feel excessive to me, but that's the workload now.
Agents Need Efficiency Too
Efficiency now covers agents, not just models. Jonathan Bryce, CNCF Executive Director, framed the goal at the press briefing: "This is how do we take the same footprint, the same power consumption, the same set of hardware, and get more out of it."
Bryce pointed to OpenShell's default-deny approach to agent permissions and Agent Substrate's handling of idle agents. He cited a demonstration that went from one agent per node to 65 by sleeping and waking agents around their work. That's still in progress, but a density jump like that is the kind of number infrastructure teams pay attention to.
Portability is still unfinished. Dilpreet Bindra, VMware Senior Director of Engineering, said, "Our goal is to define and verify how workloads should behave across any environment, reducing the fragmentation and ensuring portability."
My read is that AI is pulling Kubernetes toward coordinating workloads, hardware, and security as one problem. Nothing here is final until the release process says so, and making these pieces hold up in production will take longer than a single December release. But if they land, teams should get more work out of the same clusters, and that's the outcome that matters most right now.