How Agentic AI Is Exposing Kubernetes Gaps in GPU Scheduling and Operations

Agentic AI is putting pressure on Kubernetes in ways that expose old coordination and visibility problems. The useful question isn’t simply whether AI can run on Kubernetes; it’s whether the systems around it can keep up when workloads restart, demand GPUs in bursts, and produce evidence that standard metrics miss.
This is another edition of Road to KubeCon, following developments across Kubernetes and cloud native ahead of KubeCon + CloudNativeCon North America 2026, scheduled for November 9–12 in Salt Lake City.
This week’s stories span automated pod-restart diagnosis, better coordination between Kubernetes scheduling and autoscaling, and the push to make distributed AI inference production-ready. They also raise a less comfortable set of questions: what operational context is missing, which failures disappear between metric scrapes, and whether the AI agents already in developers’ hands can do useful SRE work.
HPE tackles pod restarts in AI development
Long-running AI and machine learning jobs don’t always fit neatly into Kubernetes’ ephemeral container model. Enterprises using Kubernetes at scale are running into pod restarts, a problem HPE has seen among customers hosting HPE Private Cloud AI (PCAI) workloads. These AI/ML workloads can be demanding to troubleshoot when a cluster is large and the cause of a restart isn’t immediately obvious.
Hewlett Packard Enterprise (HPE) is a presenting sponsor of Road to KubeCon. HPE Software helps IT organizations modernize infrastructure, streamline operations, and accelerate AI initiatives across hybrid, multi-vendor environments.
Kubernetes includes troubleshooting commands, but using them still means operators have to investigate manually. HPE system software engineer Guoping Jia says that approach becomes hard to sustain in PCAI’s large clusters. “This operational workflow does not scale effectively in PCAI’s large-cluster environment,” he writes on the HPE blog.
Jia’s approach uses the open-source k8s-pod-restart-info-collector to collect restart details automatically and post diagnostics to Slack. It’s a practical way to reduce the work required to investigate a common failure. The broader point is that, as AI workloads expand, operational usability can matter as much as raw compute capacity.
Scheduling and autoscaling need to coordinate
GPU workloads expose a mismatch between two Kubernetes control loops. The scheduler places pods on available nodes; the autoscaler provisions additional nodes. If those systems don’t share a view of capacity, pods can remain pending while expensive GPUs sit unused. Reservations, device topology, and GPU sharing make the coordination problem harder.
AWS senior software development managers Sajjan Gundapuneedi and Neelendra Bhandari laid out the issue in a post about AI’s effect on Kubernetes autoscaling. Their sharpest observation is that accelerator capacity only helps when scheduling and provisioning agree on how it will be used:
“A scarce accelerator is only as useful as the agreement between the loop that schedules it and the loop that provisions it.”
Inference makes this more than an architecture discussion. Serving traffic can arrive in bursts, and agentic workloads can’t always wait for a new node to start. As the authors put it, “Inference is what makes this pressing rather than academic, since bursty serving and agentic workloads have no patience for a cold node.”
Several pieces of a solution are already emerging: dynamic resource allocation (DRA)-aware autoscalers, workload-aware scheduling, and provisioning requests that can connect the two loops. The authors expect the autoscaler to become “a more scheduler-aware partner.” A public proposal in the SIG-Autoscaling group goes further, sketching APIs for the scheduler to view provisionable capacity, state placement preferences, and request capacity directly.
llm-d v1.0 targets production inference
The CNCF sandbox project llm-d reached v1.0 on Tuesday at PyTorch Conference North America. The release aims to connect developers in the PyTorch ecosystem with the cloud-native infrastructure needed to serve AI applications in production.
For Carlos Costa, llm-d co-founder and lead maintainer, and a distinguished engineer on IBM’s hybrid cloud platform for AI, version 1.0 marks a change in emphasis. “This is the moment open source AI serving crosses from experimental projects into dependable infrastructure you can build on,” he told The New Stack. Earlier releases demonstrated performance; this one, he says, is about maintaining it in production.
That claim has some real-world context. llm-d has been used at Tesla, AWS, Oracle, Google, and elsewhere. Costa also points to an IBM Research demonstration in which an agent factory exploring quantum error correction hardware ran open models at 11.5 times lower inference cost than an all-proprietary configuration.
Agentic workloads are where Costa believes llm-d stands out: it can absorb bursty traffic, and cache reuse reached about 98% in IBM’s test. Those results are promising, though a single demonstration doesn’t tell us how the system will perform across the variety of production environments teams need to support.
Operating frontier intelligence at scale
A recent conference talk in Shanghai offers a preview of the operational questions likely to surface at KubeCon North America. At last month’s KubeCon + CloudNativeCon + OpenInfra Summit + PyTorch Conference China 2026, CNCF CTO Chris Aniszczyk and Xiao Zhang discussed “Operating Frontier Intelligence at Scale.”
Aniszczyk’s argument is that the industry’s focus is shifting from building increasingly capable models to running them together, at scale, across different environments. “AI is essentially moving from just purely about building the greatest models [to] how do we actually make this run at scale, stitch it all together and run in different types of environments,” he said.
That shift is already adding pressure to online services. Cloudflare CEO Matthew Prince recently said AI-driven bot traffic had surpassed human traffic online for the first time. Aniszczyk connected the rise in agent activity to strain on systems, including recent GitHub outages. If inference becomes the dominant workload, the infrastructure assumptions built around older patterns of use will have to change.
“It’s the craziest kind of technology change that I’ve seen probably in my career.”
The consequences reach beyond software: companies need infrastructure and data centers, and they need enough power to run them. Aniszczyk sees that as a difficult but worthwhile challenge for the cloud-native community.
Can your coding agent handle SRE work?
AI SRE products are multiplying, but Roy Libman, CPO at Radar from Skyhook.io, argues that teams may already have a useful starting point: the coding agent their engineers use. His caveat is the important part. The agent needs operational context.
“The coding agent your engineers use, whether that’s Claude Code, Codex or Cursor, can already investigate a Kubernetes cluster,” Libman writes on the CNCF blog. “What it usually lacks is context: what’s failing right now, and what changed before it broke.”
Libman tested Claude Code against 50 injected faults on a live cluster. With read-only context supplied through MCP tools, it diagnosed faults slightly more accurately than it did with shell and kubectl access. The larger difference was speed: the agent reached the right diagnosis within two minutes for 86% of faults with context, compared with 60% when using a shell.
That’s a useful result, but it doesn’t establish that an existing coding agent can replace an SRE platform. It does suggest that context may be the higher-value upgrade—and that some teams could improve incident triage without paying for another AI subscription.
Industry take: what Kubernetes fails to show
A previous Road to KubeCon examined what’s missing from OpenTelemetry and Prometheus. One overlooked category is state that doesn’t naturally appear as a time series: slowlog entries, client lists, and access control list (ACL) changes.
Kristiyan Ivanov, a former engineering manager at Redis, a current contributor to Valkey, and founding CTO of BetterDB, says that evidence can disappear between scrapes. “Metrics count, but they don’t explain.”
Ivanov says Redis and Valkey keep useful evidence in in-memory buffers, while Prometheus captures it as lengths and counters. Kubernetes makes the problem worse: pod restarts, failovers, and ephemeral pod IP addresses all complicate the job of following what happened.
“Almost everything in the ecosystem is an INFO scraper,” he says. What’s missing, in his view, includes durable history through restarts and failovers, attribution to a service or ACL user, an audit trail for ACL and configuration changes, and aggregation across shards.
Better detection in Valkey and OpenTelemetry eBPF Instrumentation can help operators see more. But eBPF has limits: Ivanov says it can’t detect evictions, ACL denials, or configuration changes. Those events need to be collected inside the database engine.
Kubernetes is changing, and so are the demands on the teams operating it. Presenting sponsor HPE offers software for virtualization, cloud management, observability, and automation to help teams manage that complexity.
Many teams can’t measure AI agents well
AI monitoring is a stated priority for 87% of respondents to the Splunk State of Observability 2026. Yet more than half say they can’t effectively assess AI quality, including the quality of LLM responses, or control AI costs. The report surveyed 2,050 IT operations and engineering professionals and was released October 20.
There’s another warning in the numbers: 83% are concerned about unintended consequences from AI agents. Agents can take unexpected actions and generate bursty traffic, both of which create new demands on infrastructure. Cisco president and chief product officer Jeetu Patel said in the report’s press release, “Most companies will soon have far more agents at work than people.” Meanwhile, 81% of respondents said AI monitoring is prompting greater collaboration between observability and security teams.
More updates from the cloud-native community
Cloud-native projects and companies keep moving. Here are more community, project, and vendor updates from the past week or so.
- Anish Bista is the first Golden Kubestronaut from Nepal. That’s a personal milestone with a wider signal: the Kubernetes community’s reach keeps expanding.
- Kubernetes v1.38.0 is scheduled for Wednesday, December 16. Meanwhile, Kubernetes 1.34 reaches end of life on October 27, according to the release schedule. If you’re still running 1.34, that date deserves a place on your upgrade calendar.
- Fairwinds has shared what it learned from running n8n in production on Amazon EKS.
- Broadcom’s AAI v26 adds cost tracking and Kubernetes support; EMA lays out the changes in its impact brief.
- Splunk now supports remote configuration of OpenTelemetry Collectors on Kubernetes.
- New Relic’s Kubernetes integration v4.8.2 is out with dependency updates.
One more stop on the Road to KubeCon
Road to KubeCon is an eight-part series presented by HPE. Before you go, see how HPE Software helps IT teams do more with less complexity.
There’s one edition left in the series. We’ll return next week with a special sign-off; until then, you can revisit earlier installments in the Road to KubeCon archive.
Bill Doerrfeld, who writes the series, welcomes community pitches, whether you have an interesting perspective, a new release, or a tip for a good dive bar in Salt Lake City. Send him a direct message on LinkedIn or use his personal contact page.
The post AI jobs are waiting while GPUs sit idle. Kubernetes has a new plan. appeared first on The New Stack.