Buoyant Kinference Adds Kubernetes Control for vLLM AI Inference
Control over inference becomes a Kubernetes problem
GPU capacity can be scarce even as much of it sits idle. That mismatch points to an operations problem as much as a hardware shortage: teams may have GPUs assigned to workloads without a clear way to keep those processors busy across changing demand. Buoyant’s open-source Kinference project targets that gap with a control plane for AI inference workloads running on vLLM across Kubernetes clusters. Its focus is operational: direct workloads according to performance, cost and compliance requirements, while making better use of GPUs already assigned to them.
That approach gives infrastructure teams a way to run models on systems they control and manage inference as a shared cluster resource. Buoyant CEO William Morgan said self-hosted deployments can replace token-based cloud charges with infrastructure allocated to specific workloads. The trade-off is that teams take on responsibility for operating that infrastructure, including how capacity is divided when demand rises. They can also set the quality of service each workload receives, which makes allocation policy part of the serving system rather than an informal decision made after congestion appears.
Using vLLM’s serving and memory model
Kinference works with vLLM, the inference system developed by researchers at the University of California, Berkeley’s Sky Computing Lab. vLLM combines an inference server that handles network traffic with an inference engine that uses PagedAttention to improve memory throughput. Those pieces address different parts of model serving: one accepts and manages requests, while the other works to use memory more effectively during inference.
The control plane adds coordination across distributed Kubernetes environments. It can use GPU cache data and queue status when managing workloads, giving teams more information to guide how inference requests are handled. That context matters because routing based only on whether a GPU is technically available can miss the queue and memory conditions that shape actual response times. Morgan said this approach could help raise GPU utilization across multiple workloads; he described typical utilization as being in the single digits, even amid a broader GPU shortage. The numbers here are underwhelming, and they help explain why scheduling has become part of the inference conversation.
Local capacity with conditional cloud routing
Self-hosting appeals to organizations that handle sensitive data or need tighter control over where it’s processed. It can also reduce latency for iterative applications, Morgan said, where repeated requests make response time a practical concern. Keeping a model close to the application may simplify that path, though the benefit depends on how the organization has built and connected its systems.
Kinference lets teams define when requests should leave their own GPU pool. Routing conditions can include a saturated pool, usage by a particular tenant or characteristics of the inference request. That gives organizations a way to keep routine or sensitive work on infrastructure they manage while sending selected requests to cloud providers when local capacity is constrained. Here's the thing: a policy for handing work off is only useful if teams can explain which requests qualify and understand what happens when a threshold is reached. Cloud routing can add capacity, but it also makes the boundary between local and external serving an operational choice that needs to be visible.
The project also tracks GPU consumption by tenant, group or team. That attribution can help operators understand which workloads are using capacity and where infrastructure costs are accumulating. (And this is the part most people overlook.) A shared pool can make hardware easier to allocate, but without useful attribution, teams may struggle to connect usage patterns to the applications and groups driving them.
Why workload policy matters
AI applications are adding a new class of demand to Kubernetes clusters that already run other services. The number of AI workloads on those systems isn’t specified, but their growth puts pressure on teams to allocate finite compute resources across different needs. Inference also behaves differently from a simple background task: requests arrive over time, and queues or service expectations can affect the experience of the application using the model.
Kinference’s practical proposition is to make those allocation decisions more explicit: teams can set operating boundaries, monitor GPU use and define conditions for cloud spillover. If you’re working in this space, the appeal is a clearer way to connect serving behavior with workload priorities. That said, policy can’t create capacity where none exists, and the project’s value will depend on whether its controls fit the operational demands of a given cluster. More visibility. More deliberate allocation.
This is more significant than it looks: as AI workloads take a larger share of cluster capacity, managing who gets access to existing GPUs may matter as much as adding more. Kinference frames inference as a resource coordination problem that Kubernetes teams can address through scheduling, usage tracking and routing rules. Whether that translates into higher utilization in practice will depend on the workloads and policies operators bring to it.
