Large-model training hits a memory wall fast. Scaling laws dictate that pre-training or full fine-tuning of multi-billion-parameter foundation models requires clusters of H100s connected by 3.2 Tbps InfiniBand interconnects. That is not a modest workstation project, regardless of how many reassuring diagrams appear in the announcement.
The constraint becomes obvious with a 7B parameter model. Its static weights occupy 14 GB of VRAM in standard FP16 or BF16, while AdamW optimizer states require approximately 56 GB and backward-pass gradient tensors add another 14 GB in FP16. A device with 24 GB to 48 GB of VRAM cannot hold the full training state without careful memory management or distributed systems.
Three ways to reduce the training burden
Quantized Low-Rank Adaptation, including QLoRA and DoRA, freezes base model weights in a 4-bit representation and injects trainable low-rank matrices stored in full precision. The base parameters use 4-bit NormalFloat, or NF4, while Double Quantization quantizes the quantization constants themselves and saves an additional 0.37 bits per parameter.
That compression does not mean the model computes directly on tiny integers. During the forward pass, dynamic dequantization converts the base weights into BF16 for computation. DoRA adds another architectural choice by decoupling magnitude updates from directional updates, giving the adaptation process separate controls for those two changes.
The tradeoff is throughput. Dynamic, on-the-fly dequantization introduces compute overhead that degrades training throughput by 20% to 35%—a useful reminder that saving memory can move the bill to compute time.
GaLore takes a different route by shrinking the optimizer state rather than quantizing the base model. Its Memory-Aware Low-Rank Optimizers project high-dimensional gradient matrices into a low-rank subspace, then track momentum and variance for projected matrices P ∈ ℝ m × r, where r ≪ min(m, n).
GaLore applies SVD or randomized orthogonal projections to the gradient tensor G ∈ ℝ m × n. The projections update every T steps instead of every iteration, and the method can support full-parameter pre-training or domain adaptation. Periodic SVD factorizations still create compute stalls and step-latency spikes, because mathematics remains unimpressed by scheduling preferences.
Distributed memory solves one problem and creates another
FSDP and ZeRO-3 shard optimizer states, gradients, and model parameters across device VRAM and host RAM, paging tensors across PCIe buses when needed. Under ZeRO-Stage 3 or FSDP Full Shard, each GPU holds only 1/N of the complete model state during idle intervals.
Host-offload mode places non-active parameter shards and optimizer states in pinned host CPU RAM, then streams them over PCIe asynchronously. On consumer PCIe Gen4 or Gen5 lanes, that movement creates severe I/O bottlenecks and can push GPU compute utilization below 30%. The model fits, technically. The hardware then spends its time waiting for the model to arrive.
These methods expose the central design choice in limited-hardware training: reduce stored precision, reduce optimizer memory, or distribute state across devices and host memory. Each approach changes a different part of the workload, and each carries a cost in throughput, latency, or system complexity.
Inference costs now depend on product design
The same efficiency problem continues after training. Token prices for frontier models fell roughly 80% in one year, from $18.40 to $6.07 per million tokens, yet 73% of enterprises reported that their AI costs exceeded original projections in 2026.
Enterprise AI budgets grew from $1.2 million in 2024 to $7 million in 2026, but cheaper tokens did not prevent overspending. Reasoning models generate internal “thinking tokens” that companies pay for but do not see, while agentic workflows can trigger chains of calls without modeling total token cost upfront.
Teams also stop trimming irrelevant history from context windows when tokens look cheap, and many enterprises default to the most visible model—often paying 15 to 30 times more than necessary for a task. The “Big Model Fallacy” is the assumption that every task needs a frontier model.
Dr. Jon Krohn described the calculation this way: “cost per token, multiplied by how much background thinking the agent does, multiplied by how many data points it creates. ‘All these multiples mean token usage is exploding,’ Krohn says.” Treating AI costs solely as a finance problem misses the influence of engineering decisions. The most expensive mistake is using the wrong model for the job by default.
Inference gateways such as OpenRouter, LiteLLM, and Portkey decide which requests go to which model, cache repeated queries, and show spending by team and task. That turns a model switch into a configuration change instead of a quarter-long migration, while cost now scales directly with usage patterns baked into the product’s architecture rather than a fixed number negotiated at contract signing.
The warnings from inside large companies are blunt. Uber burned through its 2026 AI coding budget in four months and capped employee AI spending at roughly $1,500 per month per coding agent after exhausting its annual AI budget in months. Microsoft has begun pulling back internal Claude Code licenses due to overruns.
The hardware and inference lessons match: efficiency is not one trick. It is a chain of choices covering quantization, gradient projection, memory sharding, model selection, context management, and routing. Ignore any link, and the bill finds it.
Based on
