Kubernetes GPU Capacity: Measure Useful AI Throughput Before Scaling

Oct 24, 2026 582 views
Monolithic concrete tunnel with circular arches and high-contrast shadows representing scalable AI compute infrastructure.

For teams operating AI workloads on Kubernetes, a full GPU pool and a lengthening job queue can make adding hardware feel inevitable. Yet those signals alone don’t tell you whether more GPUs will improve results. The queue may reflect jobs that could start later, a serving configuration that needs tuning, or a mismatch between the model and its workload.

“An allocated GPU, a busy GPU, and a workload producing valuable results are not the same thing.”

That distinction is the useful starting point. Infrastructure teams can make better capacity decisions when they connect resource use to deadlines, application quality, latency, and cost. The objective isn’t to keep every accelerator busy; it’s to deliver the required results reliably at a cost the organization can explain.

Define success in terms of the workload

Start by specifying what each workload must produce and when. An always-on customer service chatbot has different operating needs from an evaluation run whose results are needed before an internal release meeting. Labels such as “inference” and “training” don’t tell you which job is urgent. Batch inference might tolerate a delayed start, while a training run may have a fixed delivery date.

For each workload, record its deadline, expected runtime, resource needs, and tolerance for interruption. Then define a useful result. An inference service might need to meet a task-specific quality threshold and a latency objective. Returning a successful HTTP response or valid JSON doesn’t establish that the model gave an acceptable answer.

Keep two scheduling questions distinct: can the job wait before it begins, and can it be interrupted after it starts? A workload may be flexible on start time yet require uninterrupted execution. Treating these as the same setting can lead to poor scheduling decisions.

Use deadlines to smooth GPU demand

Consider four compatible GPUs available for batch work between noon and 5 p.m. Four independent evaluation jobs arrive at noon. Each needs two GPUs for an hour, and the results are due by 5 p.m. Starting every job at once would require eight GPUs. Running two jobs at noon and the other two at 1 p.m. finishes the entire batch by 2 p.m. on the four GPUs already available.

With runtimes and supporting resources held constant, both approaches use eight GPU-hours, excluding startup overhead. Staggering changes peak demand without increasing the work or missing the deadline. That’s why the shape of a queue matters: an immediate start may be convenient, but it may not be necessary.

“Staggering reduces peak demand. The jobs still finish on time without expanding the pool.”

Kueue offers an admission control point for Kubernetes workloads, applying configured quotas and policies to decide when jobs enter the system. The platform doesn’t define an organization’s deadlines or resource requirements for it. Teams must represent those needs in their policies and workload configurations.

Queueing work with flexible start times can help teams plan capacity and avoid launching jobs prematurely, only to preempt them when higher-priority work arrives. The practical question is how much schedule flexibility exists before the deadline becomes a real constraint.

Test serving capacity under realistic conditions

Scheduling can reduce pressure on shared pools, but inference services also need a direct capacity measurement. High GPU activity doesn’t reveal how much additional throughput the system can sustain. Nvidia’s profiling documentation describes how streaming multiprocessor activity can include warps waiting on memory. A busy accelerator therefore isn’t, by itself, proof that the deployment is delivering useful work at its limit.

Interpret hardware signals alongside serving metrics: running and waiting requests, cache occupancy, and preemption counts can help explain what the service is doing. In vLLM, KV-cache usage reports the share of cache blocks in use. That provides context about cache pressure; it doesn’t directly report total serving capacity.

A representative load test provides a more useful estimate. Hold the model, GPU, serving configuration, and request mix steady. Increase the request arrival rate and track throughput, output quality, errors, and latency. The highest sustained rate that still satisfies the workload’s requirements is a practical capacity figure. Specify the latency percentiles and error limits that matter rather than relying only on average completion time.

For interactive generation, measure time to first token and the pace of subsequent token delivery. DistServe illustrates why serving capacity should be judged against application latency objectives. A system can produce an acceptable average while still delivering a poor experience for a meaningful share of requests.

Compare production demand with the tested rate. Low cache occupancy can prompt investigation, but it can’t support a claim of tenfold throughput without a throughput test. A short-lived queue may come from a burst; a queue that persists may indicate a capacity or configuration problem. Neither signal identifies the bottleneck on its own.

Make cost accountable to verified results

The FinOps community has framed AI efficiency in terms of cost and useful outcomes. For AI workloads, that means linking infrastructure spending to application results, not stopping at a token count or utilization chart.

OpenCost’s allocation API can provide workload cost data over defined intervals. Join it to serving telemetry and application outcome records using workload identities and timestamps. OpenCost also has opt-in inference cost tracking, including cost per million tokens.

For cost per outcome, include relevant GPU, CPU, memory, and other infrastructure charges. Document how shared and idle costs are assigned, and account for costs outside OpenCost separately to avoid double counting. Include retries, failures, startup time, and idle periods when they contribute to the total. Count the final business result once, even if several attempts were needed to produce it.

A useful measure is cost per verified outcome: attributable cost divided by results that meet the quality bar and arrive on time. If correctness comes from sampled reviews, record the sampling method and its uncertainty. OpenCost supplies allocation data; the application must supply the definition and evidence for a successful result.

Keep capacity measurement separate from cost allocation. OpenCost proposal 3915 explores adding serving-engine telemetry to the cost model. It remains a draft and doesn’t provide a released saturation or correctness score.

Compare configurations by the outcomes they preserve

Imagine a service handling 1,000 tasks per hour with a 98% correctness requirement. A small, inexpensive model that reaches only 90% doesn’t meet the workload’s needs, even if it keeps its GPU busy. Utilization can’t compensate for results that fail the required quality threshold.

A stronger model might deliver 990 correct, timely results for $2 per hour while using only part of its tested serving capacity. That works out to roughly $2.02 per 1,000 qualifying outcomes. Spare capacity may be reasonable when it protects quality and absorbs bursts. The next question is whether a smaller GPU, fewer replicas, or a different batching configuration can preserve the same service level.

If a tested configuration produces the same 990 qualifying results for $1.50, the cost falls to about $1.52 per 1,000. That reduction comes from using fewer resources to deliver equivalent results. Raising utilization on the original GPU without improving cost or outcomes would not offer the same value.

“The target is a deployment that delivers the required outcomes at a defensible cost, with enough headroom to meet its requirements.”

When two models both meet the minimum bar, better quality may justify extra expense if it prevents consequential errors or reduces human review. Compare that benefit with the actual workload. Cost per correct result is most informative when the results have similar value and the errors carry similar consequences.

Change one major variable at a time and compare cost, verified outcomes, latency, and deadline performance under representative demand. Preserve enough headroom for bursts and failures, and verify that apparently idle capacity is available to use. A GPU assigned exclusively to one Pod can’t be reassigned to another until the allocation is released.

The evidence may point to delayed batch jobs, different batching, a resized deployment, another model, or additional GPUs. The decision should follow measured workload requirements: deliver the right results on time, understand their cost, and maintain the capacity needed to keep doing so.

The post Before you expand your GPU pool, measure the outcome appeared first on The New Stack.

Source: Danielle Zeis · thenewstack.io

Comments

Sign in to comment.
No comments yet. Be the first to comment.

Related Articles

Before you expand your GPU pool, measure the outcome