AI hardware is entering a new phase, and the biggest surprise may be the return of the CPU. Edge devices are exposing the limits of a single forward pass, while agentic AI systems keep generating new work long after the first answer appears.
The shift reaches from a phone in your hand to the largest server fleets. When an agent decides, calls a tool, reads the result, and decides again, the workload becomes a loop—and that loop is putting power, heat, scheduling, and CPU capacity at the center of AI design.
Edge AI Has Outgrown the Single-Pass Benchmark
The hardware conversation about edge AI has become a great deal more honest in the last year. Edge devices are thermal-limited, not MIPS or compute-limited, so power now takes top billing, with raw throughput coming last.
That matters because every item in the hardware hierarchy is budgeted against an assumed workload, and the workload almost everyone is still budgeting against is a single forward pass. A model processes a prompt, produces tokens, and the benchmark ends. Agentic AI does not end there.
An agent decides, calls a tool, reads what comes back, and decides again. A single agent can fire off 300 actions an hour, creating a workload that keeps the processor active across many rounds of reasoning, tool use, parsing, and response handling.
In OpenAI’s Agents SDK, the runner runs a loop until a turn limit is exceeded or disabled. “The only thing that stops it is a turn limit — exceed max_turns and you get an exception — and the documentation notes that you can pass max_turns=None to disable the limit entirely.”
On a server, the cost of that run is a billing decision. On a device, the loop length is a thermal decision because duty cycle is passive cooling, and “duty cycle is the one variable that passive cooling cannot argue with.”
Phones Reveal What Long AI Runs Really Cost
A March 2026 benchmark tested four platforms with a 1.5-billion-parameter model, a 258-token prompt, and twenty back-to-back runs. The iPhone 16 Pro peaked at 40.35 tokens per second, but its performance degraded to 22.56 tokens per second within two inferences.
That represented a 44 percent reduction in tokens per second, and the phone stayed throttled for 65 percent of the benchmark. The Galaxy S24 Ultra imposed a hard GPU frequency floor at 78.3°C, stopping inference.
“Degradation showed up within two inferences. It settled at 22.56 tokens per second — a 44 percent reduction — and stayed throttled for 65 percent of the benchmark.” The shape of the result is what matters: peak performance can hide what happens when an agent keeps working.
An edge device that looks strong on one forward pass may deliver a different experience across a long agentic loop. Each decision, tool call, returned result, and new decision adds work, while sustained activity pushes the system toward its thermal limits.
Why Agentic AI Is Bringing CPUs Back
The AI boom drove demand for GPUs and later memory, while CPUs were mostly left out of AI model inference workloads because of their lack of parallelization. Agentic AI systems are changing that narrative by spawning many agents and sub-agents that make API calls and talk to more agents.
“Many components of an agentic AI task are inherently CPU based jobs,” said Souvik Kundu, senior staff research scientist at Intel. The CPU handles parsing output, figuring out which tool to invoke, making API calls, collecting results, and feeding them back into the system.
Madhu Rangarajan, vice president of compute and enterprise AI products at AMD, said, “Seven of the eight stages in realistic agentic AI pipelines run entirely on the CPU.” Safety guardrails on agents often run on the CPU to minimize latency, adding another CPU-based task to the loop.
Tokenization creates another pressure point. It is a key first step in LLM inference and can become a bottleneck, especially with long sequences, because tokenization of large sequences increases time-to-first-token latency.
Euijun Chung, a PhD student at Georgia Tech, described the problem this way: “If you have an ongoing sequence of, say, 100,000 tokens, and you have a tool result of 1,000 tokens, the tokenizer will have to tokenize the whole sequence again.” Agentic AI workloads can involve sequences of 500,000 to a million tokens.
In test runs at longer sequence lengths, increasing CPU core counts reduced time-to-first-token latency by roughly 1.5 to 7 times. Scheduling optimizations can also deliver up to a 1.8-times reduction in end-to-end latency.
“In the world of agentic AI, the average sequence length will grow and grow, so I’m expecting this problem to get worse in future workloads,” Chung said. Larger models are expected to experience less dramatic tokenization bottlenecks because they create higher GPU demand, but the CPU still owns much of the work around each inference.
A New CPU Race Is Already Underway
The demand is moving through the market. AWS experienced an explosion in wait times for CPU server capacity because of AI workloads, while Intel has sold out of server CPUs through at least the end of the year.
AMD has doubled its server CPU forecast. Arm and Qualcomm have both announced new CPUs designed to accelerate agentic AI, and Nvidia has prioritized Vera, its Arm-based CPU for agentic AI, as part of Nvidia’s Vera Rubin platform.
Matt Kimball, vice president and principal data center analyst at Moor Insights & Strategy, said 2026 has brought a spike in CPU demand, much of it due to agentic AI. The change is not a retreat from GPUs; it is a recognition that agents need a complete system around the model.
The next AI benchmark will not stop after one forward pass. It will measure whether a device can sustain the loop, whether a server can schedule its expanding CPU workload, and whether tokenization, tool calls, guardrails, and GPU inference move together without creating a new bottleneck.
That is the CPU comeback: not a return to the old computing order, but a new role in which processors keep the agent moving from one decision to the next.
Based on
