The Hidden Cost of Real-Time State in AI Inference

Real-time AI inference carries a cost that a memory bill alone cannot show. The model must fit on the available hardware, but the system also needs room to handle demand, meet latency targets, and serve multiple requests at once.

That makes production inference a larger engineering decision than choosing a model. Teams must match the model, deployment approach, accelerator, capacity, serving configuration, and management boundary to the workload. The central question is not only whether a model runs, but whether it runs at the speed, scale, and cost the application needs.

Memory changes the shape of the workload

Memory capacity determines whether a model fits on a configuration and how many GPUs it requires. More memory can let a larger model run on fewer GPUs, leaving room for batching. Memory bandwidth also affects how efficiently data moves through the accelerator, which influences concurrency, throughput, latency, and the economics of each request.

Long-context inference makes this tradeoff harder. Model weights and the key-value, or KV, cache compete for memory, so a configuration that fits the model may still lack enough room for real-time state and additional requests. This is the cost hidden behind the simple question of whether a model can load.

The AMD Instinct MI300X shows why hardware details matter. Each GPU provides 192 GB of HBM3 high-bandwidth memory and up to 5.3 TB/s of memory bandwidth. An eight-GPU platform offers 1.5 TB of memory, giving larger open models more room for batching and long-context workloads.

That extra capacity does not guarantee the best result for every application. Teams still need to measure their own workload, because accelerator benchmarks say little about whether a configuration fits a production application.

Dedicated capacity turns inference into an operating decision

Pay-per-token inference suits workloads that need flexible consumption without dedicated infrastructure. Other workloads may need dedicated capacity because demand stays high, latency and throughput must remain predictable, GPU availability matters, or the team needs a specific open-model and accelerator combination.

Managed Compute in Microsoft Foundry offers one way to handle that choice. Once a workload runs on dedicated infrastructure, teams must estimate demand, provision capacity, and choose an operating point that balances latency with throughput. Interactive applications may favor response time, while batch and high-volume workloads may favor the amount of work completed.

The economics change with that decision. Instead of treating inference only as a per-token charge, teams must consider GPU-hour cost, utilization, throughput, and completed work. Those factors shape the effective cost per token or inference request, even when the memory price itself looks clear.

Foundry’s managed approach keeps key workload decisions with the customer. Teams control model and accelerator selection, the runtime and container environment, provisioned capacity, infrastructure tuning, and the operating point between latency and throughput. Microsoft manages much of the serving environment, including software and runtime updates, capacity scaling, vulnerability fixes, and security patching.

That boundary matters because production teams do not always want responsibility for every layer of the serving stack. With managed compute in Foundry, developers can continue using familiar inference APIs, authentication, networking, SDKs, and observability across deployment types.

Benchmarks must match the real application

AMD’s ROCm ecosystem provides the open software and optimization layer around AMD Instinct hardware, with support for open models and open-source AI frameworks. AMD Performance Labs testing found that full-stack optimization in ROCm 10 on an 8× AMD Instinct MI355X delivered an average 3.3× higher inference throughput across GLM-5, Kimi-K2.5, and DeepSeek-R1-0528 than a ROCm 7.0 baseline.

That result shows what software optimization can deliver, but it does not remove the need for workload-level testing. Different applications place different demands on memory, concurrency, response time, and throughput.

GLM 5.3 Flash offers a clear example. The 320B-parameter model has 18B active per token and a 512K context window. It comes in three configurations designed for AMD Instinct MI300X: one four-GPU setup for compact chat, coding, and agent workloads, plus two eight-GPU setups using tensor, data, and expert parallelism.

Those eight-GPU configurations target different priorities. One focuses on concurrency and time to first token, while the other provides memory headroom for longer-context workloads. In tests using a conversational workload, the eight-GPU TP4 + DP2 + EP configuration delivered approximately 1.8× the output throughput of the four-GPU configuration and cut average time to first token by more than half.

Other workloads produced smaller gains. That is why teams need to benchmark each configuration against their own traffic instead of choosing from a headline GPU result.

Production begins after model selection

AT&T used managed compute in Foundry across multiple open models and GPU architectures, with about 530 GPUs, including around 430 AMD Instinct MI300X GPUs. Phi-4 alone processed more than 700 billion tokens a month for data preparation and synthetic data generation.

That scale highlights the difference between selecting a model and operating one. A production system must account for memory requirements, expected demand, latency, throughput, capacity, and the level of infrastructure ownership the team is prepared to manage.

For dedicated infrastructure, the goal is balance: enough control over the model, accelerator, capacity, and performance without requiring the team to own every part of the serving stack. Managed compute in Foundry provides one path to that balance, while the memory capacity of AMD Instinct MI300X GPUs can help large open models run on fewer GPUs.

Stay connected via Google News

Follow us for the latest AI News