NVIDIA just rewrote the rules on low-precision AI training. Their new 4-bit floating-point format, dubbed NVFP4, shatters the conventional wisdom that you need at least 8-bit precision to train massive language models without crashing and burning. They proved it by training a 12-billion-parameter hybrid Mamba-Transformer on a staggering 10 trillion tokens—the longest 4-bit precision training run anyone’s published.

Why does this matter? Floating-point precision is the currency of AI training. Lower bits mean faster compute and less memory, but also more noise and errors. Previous attempts at 4-bit training stumbled because the dynamic range shrinks and quantization errors balloon over time. NVFP4 tackles this head-on with a clever microscaling scheme that breaks 16-element blocks into finely tuned scales, giving a tighter grip on numerical range. The trick: a two-level scaling system that pairs per-block precision with a per-tensor adjustment. This combination lets NVFP4 pack near 8-bit precision for key values while cutting memory roughly in half against FP8.

On NVIDIA’s Blackwell GPUs, this translates to FP4 matrix multiplications running up to six times faster than BF16 on the latest hardware—and two to three times faster than FP8. That’s not just a speed bump; it’s a leap that could redefine how we train giant models. The memory savings alone open doors for bigger batches or larger models on the same hardware.

But NVFP4 isn’t a free lunch. Only certain parts of the model run in 4-bit: linear layers’ forward, backward, and weight gradient matrix multiplications. Key components like embeddings, attention mechanisms, normalization, and optimizer states stay at BF16 or FP32 to avoid precision pitfalls. This selective quantization keeps the training stable without sacrificing speed.

To prevent early divergence—a classic headache with low-bit training—NVIDIA layered in four stabilizers. First, they keep the first two and last eight linear layers in BF16. The last layers need more dynamic range and precision to converge properly. Second, they apply a Random Hadamard Transform on weight gradients. This spreads out erratic outliers into a smooth Gaussian distribution, taming quantization noise without math hacks in the core multiplications. Third, weights are scaled in 16×16 blocks instead of 1×16 to maintain consistency between forward and backward passes, sealing a chain rule loophole. Fourth, they use stochastic rounding on gradients to erase systematic bias introduced by rounding, but only on gradients, not activations.

The results speak volumes. Their 12B Mamba-Transformer achieved nearly identical MMLU-Pro 5-shot accuracy—62.58% versus 62.62% for the FP8 baseline—after a marathon 10 trillion token training run. That’s a clear signal low-bit precision can scale to serious model sizes without a catastrophic hit to quality.

This breakthrough rides on the back of NVIDIA’s broader software stack too. NVFP4 is baked into the Transformer Engine, their optimized library for transformer workloads, and plays nicely with Megatron-LM, NVIDIA’s flagship framework for training massive language models. Megatron-LM’s modular design and GPU-centric optimizations make it a natural match for these new data types and scaling tricks.

NVFP4 isn’t just about squeezing out speed. It’s a strategic pivot toward making massive AI training more accessible. Less memory, faster compute, and stable convergence at 4-bit precision could mean fewer GPUs or less expensive hardware for future projects. The industry has chased FP8 as the low-bit sweet spot for years. NVIDIA’s research says 4-bit can join that club without breaking your model.

Of course, this isn’t a plug-and-play fix. NVFP4 demands careful engineering and tuning, and it’s currently validated on a specific large model architecture with a precise training recipe. But the implications are clear: the frontier of AI training precision is shifting. With hardware designed around these formats and smart algorithms mitigating quantization woes, the era of ultra-low-bit training just arrived.