Prime Intellect has launched Prime Inference, a serving platform built to put frontier open-source models to work. The platform combines serverless endpoints with reserved capacity on Prime’s own GPUs across multiple datacenters, giving developers two ways to run demanding AI workloads.
There is serious scale behind the release. Before opening Prime Inference to users, Prime processed nearly a trillion tokens per day inside its own systems, turning the platform into the serving layer of Prime Intellect’s open training stack.
A Serving Platform Built for Open Models
Prime Inference is live with serverless and reserved serving for open models, and it is OpenAI compatible. That compatibility gives developers a familiar way to connect applications, while the two service modes target different needs: serverless endpoints offer flexible access, and reserved capacity provides a more controlled slice of GPU resources.
Prime’s hardware lineup includes NVIDIA Blackwell, with Vera Rubin coming soon. The serving stack combines NVIDIA Dynamo, vLLM, Mooncake, and FlashInfer, creating a system designed around the pressure points that emerge when AI models handle long prompts, repeated context, and tool calls.
Prime reports that its GLM-5.3 endpoint ranks among the fastest on OpenRouter. The company also reports a near-zero tool-call error rate and 100% uptime since launch, two measurements that matter when models move beyond chat and begin acting inside software systems.
Prime Inference targets agentic workloads. A typical agent turn adds about 6K tokens to a 140K-token prompt, so every response can bring a large context window, new instructions, and another tool request. Prime benchmarks this workload with SemiAnalysis AgentX and injected cold arrivals, testing how the platform responds when fresh demand hits the system.
Splitting the Work to Cut Agent Latency
Prime separates prefill and decode across different GPU groups. Dynamo handles routing, vLLM runs the model, Mooncake adds a second key-value cache tier in host DRAM, and FlashInfer supports the serving stack. This division lets each part of the system focus on a specific stage of generation instead of forcing every GPU group to handle the same job.
Dynamo’s KV-aware router weighs cached prefix overlap against queued work. That choice matters because agents often reuse large parts of a prompt, yet a router that focuses only on cache hits can leave requests waiting too long. Prime reports nearly 40% lower p90 inter-token latency in its tests.
The GLM-5.3 benchmarks on GB200 NVL72 show how Prime is targeting sustained agent traffic. With a 1:4 prefill/decode ratio, the system served 66 sessions per prefill group at 101 tokens per second per user, against a 100 end-to-end tokens per second target.
- NVFP4 KV cache capacity rose from 1.09M to 1.63M tokens per decoder.
- Halving tokens per step from 8K to 4K cut median queue wait from 550 ms to 110 ms.
- A native sparse-MLA kernel reached about 12.0 μs at 15 query tokens, compared with 17.7 μs staged and 13.7 μs FP8.
- Transfer descriptors in BLHNC KV layout fell from 19,559 to about 1,940.
- Mean transfer time dropped from 146 ms to 78 ms.
These numbers point to a clear goal: keep long-context agents moving even when many sessions compete for the same serving infrastructure. The platform is not only chasing raw generation speed; it is tuning the full path from prompt arrival to tool-backed response.
Tool Reliability Becomes the Next Battleground
Agents fail when tool calls carry wrong names or broken arguments. Prime addressed that problem by contributing a structural-tag builder to Dynamo for GLM’s tool format, while vLLM uses xgrammar to mask tokens that violate the tool schema and fixed parsing bugs.
That work connects the infrastructure to the behavior developers actually need. An agent that generates tokens at high speed but calls the wrong tool still breaks the task, so schema control and parsing reliability become part of serving performance.
Prime Inference enters a field that includes Together AI, Fireworks AI, and Baseten. Prime has not yet published the GLM-5.3 price per 1M tokens, but reserved capacity is available, and dedicated model endpoints are on the roadmap. Batch inference and 1-click dedicated deploys are next.
Google also unveiled Gemini 4 Argon on October 2, 2026, creating a wider test for the AI market: can new models reach the frontier, and can developers deploy them at production scale? The model is rolling out in phases, starting with trusted cybersecurity partners while Google works with the U.S. government on pre-release safety evaluations.
Artificial Analysis Intelligence Index places Gemini 4 behind only Claude Opus 5.5 and Claude Sonnet 5.5. Tim Law, director of research for AI at IDC, said Gemini 4 shows advanced reasoning on critical tasks, including legal reasoning, finance, enterprise knowledge work, and long-running tasks. Lian Jye Su, chief analyst at Omdia, said the model takes Google to the frontier in AI, particularly in cybersecurity.
Google has also used Gemini 4 internally to optimize memory at its datacenters, freeing hundreds of terabytes without buying additional hardware. The company has given no indication when the model will receive a broad public release, while Gemini 3.5 Pro is no longer planned after Sundar Pichai originally said it would arrive in June 2026.
That contrast puts Prime Inference’s launch in sharp focus. Model quality drives attention, but serving determines whether agents can handle real workloads. With open models, reserved GPUs, tool-call safeguards, and more deployment options on the way, Prime Intellect is pushing the next contest toward infrastructure that can keep AI agents running when the prompts get long and the workloads get real.
Based on
- Prime Intellect Launches Prime Inference: Serverless and Reserved Serving for Frontier Open Models — marktechpost.com
- Does Google’s new model catch up to OpenAI, Anthropic at the frontier? — cnbc.com
- Google finally updates Gemini with new top model – but says it is so dangerous only select users can have it | The Independent — independent.co.uk
