Kubernetes AI Inference: Measure Latency, Quality and GPU Capacity

Oct 30, 2026 954 views
Data center corridor lined with glowing server racks as teal energy streams pass, evoking busy AI inference workloads

A Kubernetes cluster can report healthy infrastructure while an AI service gives users a poor experience. GPU utilization won’t tell operators whether answers arrive quickly, finish at an acceptable pace, or meet quality expectations. The more useful operational view connects those outcomes to the model server, Kubernetes, accelerators, storage, databases and network.

That connection starts with a baseline. Before launch, teams need to measure the service under representative prompts, context sizes, output lengths and traffic patterns. After launch, they can compare live performance with those results and investigate changes across the stack. In the fourth installment of a HPE-sponsored series on AI and Kubernetes, The New Stack spoke with Marius Bogoevici, Senior Principal Product Manager for HPE Morpheus Software; Mahesh Ramachandran, Vice President of Product Management for HPE OpsRamp Software; and Karthik Subramanian, Principal Product Manager for HPE Morpheus Software.

The series has also examined Kubernetes self-service, ownership and governance, and the gap between green dashboards and user-visible latency:

Measure the whole response

For a conversational service, a useful scorecard can include time to first token, time per output token, complete-response latency, reliability, capacity, resource efficiency, cost and answer quality. The right thresholds depend on what the product promises. A quick first token can create the impression of speed even if generation then slows to a crawl. A fast response can still fail if it’s inaccurate or irrelevant.

“A quick first token can mask slow subsequent generation; a fast complete response can still be inaccurate or unhelpful.”

Time to first token captures when generation begins; it says little about the rest of the answer. Inter-token latency and time per output token help teams understand the pace after that first result. Operators should define where each interval starts and ends, then examine results under steady load and during traffic spikes. A single average can hide slow requests and changing behavior across workloads.

Ramachandran’s advice is to profile the serving system before customers depend on it. Use realistic prompt and context lengths, expected output volumes and traffic levels. Record token throughput, latency and service-level measures under those conditions. That baseline makes later changes visible and helps teams distinguish a demand increase from a performance regression.

The metrics also need to connect with infrastructure capacity. If throughput falls as requests rise, operators need enough context to determine whether the serving layer is saturated, a dependency is slowing down, or capacity is unavailable. As Ramachandran puts it, teams should “profile it before you launch.”

Answer quality calls for a separate evaluation. Bogoevici says teams should test whether the service produces proper answers or hallucinates. A platform metric can confirm that a request completed; it can’t decide whether the response meets a domain’s standards. Data science and subject matter specialists need representative examples and clear acceptance criteria.

The ebook Kubernetes in 2027: Building platforms for AI, autonomy, and scale recommends assessing output quality alongside performance and repeating those checks after changes to the model, prompt, retrieval system or serving configuration.

GPU activity doesn’t identify the bottleneck

A fully occupied GPU may indicate heavy demand, but it doesn’t prove that users are receiving timely responses or that resources are being used efficiently. Bogoevici describes the mistaken conclusion this way: “Our GPUs are maxed at 100%. So we’re doing well.” Operators need to look beyond that utilization number to request routing, pod scaling, model behavior and the dependencies each request relies on.

Bogoevici describes the mistaken conclusion this way: “Our GPUs are maxed at 100%. So we’re doing well.”

Ramachandran recommends checking whether inference pods scale against metrics that reflect the workload, whether requests reach the intended model, and whether related components can be scheduled when demand changes. Models can have different latency and capacity profiles. A shift in the request mix can therefore change response times even when overall accelerator activity appears steady.

Subramanian points to network configuration and database or vector-database response times as other places to investigate. He also asks, “Is the GPU partition too small to serve that model?” Those are diagnostic possibilities, not universal explanations. Teams should correlate each signal with affected requests and the relevant time window before changing the deployment.

Kubernetes’ Horizontal Pod Autoscaler supports configured custom or external metrics, so inference services can use workload-specific signals instead of relying on CPU utilization alone. The metrics pipeline, scaling limits and available accelerator capacity still need to be configured and tested. An autoscaler can request more pods; it can’t make additional GPU capacity appear immediately.

Connect operations data across the stack

HPE describes HPE Morpheus Software as a platform for governed provisioning and automation across hybrid environments. Its Advanced and Enterprise versions include HPE Kubernetes Service (HKS); Enterprise extends that operating model across hybrid and public-cloud environments. Platform teams can define catalog items, blueprints, access controls and approvals for supported environments, and manage virtual machines and Kubernetes clusters through a shared approach. Those capabilities support provisioning and governance; they don’t benchmark a model or assess the quality of its answers.

HPE OpsRamp Software provides monitoring and observability across multi-vendor hybrid infrastructure and applications. Its capabilities include service maps, AI-powered event correlation and operational automation. Ramachandran frames the roles this way: Morpheus is the provisioning engine, while OpsRamp provides observability within HPE’s cloud operations approach.

Ramachandran says HPE’s combined solution includes Morpheus, OpsRamp and runtimes. In model-serving environments, he says, it can measure token service rates; in agentic workloads, it can measure token request rates. HPE is extending OpsRamp’s AI workload observability to connect model and agent activity, token volumes, serving rates, inference latency and time to first token with infrastructure telemetry.

The value depends on whether teams can use those connections to narrow down a slowdown. They need to know which component emits a metric, how it reaches the observability system and whether the application needs extra instrumentation. With that context, teams can investigate whether a slow response originates in the application, serving layer or infrastructure, then use demand data to inform capacity planning and cost accountability.

For agentic applications, planned observability enhancements extend this view to AI traces, tool calls, errors and quality signals, alongside Kubernetes, CPU, GPU, memory, storage and network telemetry. Lightweight eBPF and OpenTelemetry-based instrumentation are also planned to add visibility while limiting collection overhead. Those signals can guide investigation, but domain-specific answer evaluations and security controls remain necessary.

Telemetry coverage depends on deployed agents, integrations, permissions and application instrumentation, as the previous article in this series explains. For an inference service, teams should connect user-facing measures with model-server metrics and traces, Kubernetes events, and relevant network, database and accelerator signals. They should also identify where monitoring ends, particularly when requests cross into an external model or service.

Make routing and scheduling workload-aware

Ramachandran urges teams planning toward 2027 to treat autoscaling, model routing and scheduling as deliberate design choices for AI workloads. He points to content-aware routing, co-scheduling related components and disaggregated serving as areas to evaluate. Their value depends on the architecture and traffic; they aren’t automatic features of every Kubernetes installation or HPE product.

An experimental Gateway API proposal describes pre-routing filters and cites the Inference Extension’s Body-Based Router as an example. The router can extract a model name from a request body and use it to guide routing. Teams need a compatible implementation and configuration, then should measure the effect against their own models, traffic and hardware.

Set data boundaries and shared launch criteria

Observability can capture sensitive information along with performance data. Prompts, responses and retrieved documents may end up in logs or traces. Bogoevici asks, “What information is actually captured and where does it go?” Teams should decide what to collect, where it’s stored, how long it’s retained and who can access it. Prompts are data, and troubleshooting needs to account for that.

Ramachandran expects operations and security observability to move closer together as AI agents create new events and access records. He says OpsRamp is preparing to interpret existing telemetry from a security perspective, while agent activity may require additional instrumentation. Teams still need to design and test their own security controls and incident response.

Before launch, Ramachandran recommends that business and IT stakeholders agree on the service’s expected value, including a plausible inference volume tied to a business outcome. “There has to be measurable business value,” he says.

“The goal is not simply busy GPUs, but reliable AI services with performance and resource use aligned to business needs.”

Subramanian says sign-off can’t rest with one person: “It can’t be a single person.” Platform administrators, machine learning and data science teams, security staff and the business owner should share the decision. They need representative load and failure tests, along with evidence about capacity, answer quality, cost and recovery.

A strong launch decision ties the user experience to agreed business targets and gives each team a clear role when the service misses them. Keep comparing inference behavior with infrastructure conditions after launch; that’s how operators spot regressions, trace problems across dependencies and adjust capacity as demand changes.

The post Busy GPUs can hide slow AI inference on Kubernetes appeared first on The New Stack.

Source: Chris J. Preimesberger · thenewstack.io

Comments

Sign in to comment.
No comments yet. Be the first to comment.

Related Articles

Busy GPUs can hide slow AI inference on Kubernetes