ZenteiQ Uses GKE Inference Gateway to Generate 5.97 Billion Daily Tokens
ZenteiQ’s experience points to a practical constraint in training scientific foundation models: accelerator capacity is only useful when the data pipeline can keep pace. By routing inference requests according to live vLLM queues and cached prompt context, the startup raised aggregate throughput 4.07x and increased daily generation from 1.47 billion to 5.97 billion tokens per 1,000 accelerators. That extra capacity helped its scheduled pre-training runs start on time.
The company, spun out of the Indian Institute of Science and selected under the IndiaAI Mission, is developing scientific models designed to work with physical laws, governing equations and numerical solvers. Its target workloads include thermodynamics, computational fluid dynamics and electromagnetics. The data pipeline uses generated reasoning traces and solver validation problems alongside curated scientific literature and simulation data.
That work demands hundreds of billions of reasoning tokens for pre-training. ZenteiQ says it built a batch pipeline to generate, evaluate and filter candidate derivations before passing verified datasets to MaxText training jobs. Since the company operates with a fixed compute allocation, slow data generation can leave multi-host accelerator clusters waiting instead of training.
Why round-robin routing fell behind
ZenteiQ separated prompt preparation from model serving. Worker pods expand scientific seed concepts into prompts, while vLLM models running across multi-node accelerator pools generate responses. The batch then passes through scoring by evaluator models and guardrail checks using a separate model family. Verified material is packaged for pre-training.
The original setup used Kubernetes ClusterIP round-robin balancing across inference endpoints. That approach treated serving instances as interchangeable, even though their workloads and available capacity varied. As ZenteiQ scaled to support 9-billion- and 35-billion-parameter model schedules, two specific inefficiencies began to limit output.
- Requests piled up unevenly. Prompts varied widely in complexity and response length. Round-robin routing could send a new request to a pod already handling a long derivation, while another node had room to serve it.
- Shared prompt context was recomputed. ZenteiQ estimates that 70% to 80% of prompts in a generation batch share instructional context and task specifications. Sending similar prompts to different instances meant each could have to process the same prefix again, rather than use a warm cache in accelerator memory.
The combined effects made generation throughput fluctuate by as much as half during a run, according to ZenteiQ. That variation stretched data preparation and threatened the timing of pre-training jobs. The team wanted routing that could account for live vLLM queue depth and prompt cache state, without building and maintaining a scheduler of its own.
A shared gateway for the inference stages
ZenteiQ already runs data preparation and MaxText pre-training on Google Kubernetes Engine (GKE). It added GKE Inference Gateway, powered by the open-source llm-d distributed inference framework, in front of its vLLM accelerator pools. The change introduced a routing layer while leaving the company’s scientific methods and pipeline structure in place.
Worker pods retrieve seed documents and domain taxonomies from Cloud Storage, then turn them into batches of multi-step prompts. A single gateway deployment manages traffic across the model families and serving nodes. Generation requests, evaluation queries and guardrail checks each go to their corresponding model pools through that gateway. Once responses pass evaluation and guardrails, workers deduplicate and package the resulting datasets for MaxText jobs on GKE.
This arrangement gives ZenteiQ one routing layer across several distinct inference tasks. The reported gains come from how that layer directs requests to serving nodes, rather than from changing the models or the data-generation stages.
GKE Inference Gateway routes batch generation, evaluation, and guardrail requests across multi-node accelerator pools running vLLM before verified datasets are fed into MaxText pre-training.
Routing by load and cache locality
GKE Inference Gateway monitors real-time queue depth and key-value (KV) cache utilization across ZenteiQ’s vLLM endpoints. When one node is busy decoding a long equation derivation, the gateway can direct new work to a node with more available capacity. That helps avoid the pattern in which requests stack up behind a busy replica while another has idle resources.
The gateway also considers whether a serving node already has a prompt prefix in its KV cache. Requests with matching scientific instructions can be sent to the node holding that warm context. Since the shared instructions account for most of each prompt while the reasoning paths differ, cache-aware placement can reduce repeated prefill work. Queue balancing matters here too: concentrating similar requests on one node could improve cache reuse while creating a new queue bottleneck.
ZenteiQ reports aggregate throughput rising from 16,964 to 69,114 tokens per second per 1,000 accelerators. It also reports daily output increasing from 1.47 billion to nearly 6 billion generated tokens, with gains across individual model pools ranging from 3.5x to 5.5x. The prefix-cache hit rate improved by 14 percentage points.
For the teams running the pipeline, the operational benefit was not just a higher peak rate. More predictable data production meant verified datasets were ready when planned, allowing multi-host pre-training runs to begin on schedule. In a fixed compute environment, that coordination connects inference efficiency directly to the amount of time available for training.
Next, post-training and solver evaluation
ZenteiQ plans to apply the same GKE Inference Gateway and llm-d setup as it scales models beyond 35 billion parameters. Its proposed uses include post-training reinforcement learning and automated numerical solver evaluation. Those workloads involve multi-turn rollouts, where each turn builds on earlier context. That makes both available queue capacity and reuse of warm prefixes relevant to keeping inference efficient.
The case offers a useful takeaway for teams building large batch pipelines: request routing can affect the productivity of an entire training schedule. When prompts vary in cost and repeat much of their context, balancing queues and placing related requests near cached prefixes can help turn accelerator capacity into more consistent dataset output. ZenteiQ’s next test will be whether those gains carry into longer, context-dependent post-training loops.