GKE Dataplane V2 Scales Network Identity for High-Density AI Agents
At high agent concurrency, Kubernetes networking can become a bottleneck before compute does. When sandboxes start and stop by the thousands, every delay in provisioning leaves valuable GPUs and TPUs waiting. Google’s approach on GKE targets one source of that delay: the network identities Dataplane V2 assigns to pods. The issue is easy to miss because it sits between workload scheduling and network policy, rather than in the accelerator itself. Yet a pod that’s ready to run still can’t do useful work if the network setup needed to isolate it is holding up startup.
Agent Sandbox provides isolated execution environments for dynamic workloads. For systems with millions of concurrent idle sessions, Agent Substrate builds on those primitives with higher density and sub-second state rehydration. Both depend on keeping sandbox startup fast as workloads scale. That dependency matters because these systems have to manage more than active compute: they also need to create, suspend, and restore isolated environments as demand changes. A delay repeated across many short-lived sandboxes can become a capacity problem of its own.
Why pod labels can slow startup
GKE Dataplane V2 uses eBPF to enforce network policy. It maps pod labels to numeric identities, which the dataplane uses to apply security controls. That model can strain agent workloads when orchestration systems add unique labels to each pod, such as an agent ID, timestamp, or claim UID. Labels are useful for describing and selecting workloads, but their role in identity calculation means that seemingly routine metadata can affect the network control plane too.
Each distinct label combination can require its own identity. During rapid pod churn, serial identity allocation may briefly hold up sandbox creation while the system updates eBPF maps. At larger scale, workloads can also approach the cluster-wide limit of 65,536 network identities. The mechanism creates a potentially awkward coupling: adding unique descriptive details to pods can increase the number of network identities, even when those details don't represent a meaningful difference in the security policy applied to the workloads.
The challenge is especially visible in reinforcement learning, where thousands of isolated environments may be created and removed every minute. Faster provisioning helps keep accelerators busy, while limiting identity growth helps sustain that speed as concurrency rises. This is more significant than it looks: infrastructure teams often focus on accelerator utilization, but the path to keeping those accelerators occupied includes scheduling, isolation, and network setup. A delay at any of those stages can leave expensive resources waiting.
Keep identity focused on security
GKE Dataplane V2 offers two ways to reduce identity churn while retaining network controls. First, move metadata that isn't needed for service selection or resource grouping from pod labels to annotations. Dataplane V2 calculates identities from labels, so annotations can carry high-cardinality details without triggering a separate identity for every pod. The distinction is practical: labels can still describe properties that matter to policy or workload organization, while annotations can hold per-instance details that don't need to shape network identity.
That change requires teams to understand how their own controllers and tooling use metadata. A label may look like incidental bookkeeping but still be used for selecting pods or grouping resources; moving it blindly could change behavior elsewhere in the cluster. If you're working in this space, the useful question is whether a field needs to participate in policy or selection, not simply whether it changes from pod to pod. (And this is the part most people overlook.)
Second, operators can configure Cilium label exclusions. The cilium-config-emergency-override ConfigMap in kube-system can specify rules such as !agents.x-k8s.io/sandbox-name-hash or !timestamp. These tell the dataplane to ignore selected dynamic labels when calculating security identities. That can also help when a developer framework adds high-cardinality labels automatically. It gives operators a way to address identity growth at the dataplane configuration level, though the choice of excluded labels still needs to reflect what the network policy depends on.
With those labels removed from identity calculation, pods running the same agent type can share a Cilium identity. The stated result is support for up to 300 fully isolated sandboxes per second at sub-second latency, without identity counts growing in step with every unique pod. That’s a specific operational claim, and its value is clearest in workloads where sandbox creation happens frequently enough for small delays to accumulate. It doesn't mean every workload will see the same rate; the reported figure describes the stated result of this approach.
Scaling beyond Kubernetes scheduling
Google cites a customer deployment of Agent Substrate that orchestrated hundreds of thousands of secure environments across clusters, with spikes above 30,000 sandboxes on a single cluster. For workloads that need still more density, Agent Substrate incorporates the Dataplane V2 recommendations and works around Kubernetes scheduling limits. The distinction matters: reducing identity churn addresses one source of provisioning delay, while higher density and scheduling workarounds address other constraints that emerge as the number of environments grows.
In benchmark validation with 200,000 agents, Agent Substrate sustained 500 activations per second, with suspend and resume latency below 500 milliseconds. The platform can run on existing GKE clusters, and its fast state restoration is intended to reduce the compute spent on idle agents. Those figures give readers a sense of the scale Google is targeting, but they describe benchmark validation rather than a guarantee for every deployment. Workload shape, configuration, and the surrounding cluster can affect how a system behaves in practice.
What this means for agent infrastructure
The wider lesson is that high-concurrency agent systems expose costs that are easy to overlook when each pod is considered in isolation. Network identity is one such cost: when it tracks every unique metadata combination, the control plane has more distinct identities to manage. Keeping identity tied to security-relevant labels can help prevent descriptive details from becoming an additional scaling burden. The benefit is operational predictability, not a claim that networking alone determines sandbox performance.
Here's the thing: configuration advice is most useful when it helps teams identify which metadata genuinely affects security policy. Exclusions and annotations can reduce identity growth, but they don't remove the need to understand how labels are used across a cluster. For teams running high-churn workloads, that review can clarify whether pod metadata is carrying policy meaning or merely recording instance-specific information. A cleaner boundary can make scaling behavior easier to reason about.
GKE Dataplane V2 addresses a specific scaling pressure by keeping network identity tied to security-relevant labels rather than per-pod metadata. For teams building high-churn agent systems, that can mean more predictable sandbox startup and less accelerator time lost to infrastructure overhead. The approach is a targeted optimization, and its practical impact will depend on whether identity allocation is a meaningful bottleneck in a given deployment. That said, as agent systems create and restore more isolated environments, attention to these less visible startup steps is likely to matter. Learn more about Agent Sandbox and Agent Substrate on GKE.