Large language models (LLMs) can now understand and generate text across hundreds of thousands of tokens. But there’s a big catch. The memory they need to keep track of all this context grows fast. This memory, called the key-value (KV) cache, stores intermediate data as the model reads or writes. It helps the model remember what came before without redoing all the math every time.

The problem is the KV cache can become huge. For example, a 70-billion-parameter model using 16-bit precision might need more than 300 gigabytes of memory to hold a million tokens in context. That’s more than twice the size of the model’s own weights. This makes running very long conversations or working with big codebases costly and slow.

To tackle this, AI researchers have focused on compressing the KV cache. The goal is to shrink the memory it takes up without losing model accuracy. Over the last decade, improvements in KV cache compression have cut the per-token memory by roughly 100 times. Meanwhile, GPU memory sizes have grown only 18 times. This means smarter math and algorithms have done more to expand context length than hardware alone.

TurboQuant: Compression Without Retraining

One breakthrough is TurboQuant, developed by Google DeepMind. It compresses the KV cache by six times with zero loss in accuracy. What’s more, it doesn’t need any retraining of the model. You can apply it to existing models and get massive memory savings instantly.

TurboQuant works by transforming the data. It first rotates the vectors randomly to make their components independent and easier to quantize. Then it applies a special mathematical transform called Quantized Johnson–Lindenstrauss (QJL) to approximate attention scores with minimal error. This two-step process lets TurboQuant represent vectors using as little as 3 bits per value, compared to the usual 16 or 8 bits.

This means GPUs can process tokens faster because less data moves in and out of memory. On NVIDIA H100 GPUs, TurboQuant has shown up to eight times faster throughput. This is a big deal for data centers and cloud providers, as it lowers costs and increases the number of conversations a single GPU can handle.

OSCAR: Attention-Aware Compression for Real Deployment

Together AI took a different approach with OSCAR. They realized that when you compress down to just 2 bits per value, you can’t rely on blind rotations. Instead, OSCAR uses an attention-aware rotation. It aligns keys and values with the patterns seen in actual query data. This helps it better preserve the important signals during extreme compression.

OSCAR combines this with a mixed-precision cache. Recent tokens remain in full precision, while older tokens compress aggressively. This layered approach lets it maintain quality even at huge context lengths, like 128,000 tokens. On some models, it cuts KV cache memory by about eight times and speeds up decoding by three times.

OSCAR also ships as a complete system, with optimized GPU kernels and precomputed rotations for popular models. This makes it easier to adopt in production environments without heavy engineering work.

The Long Game: Bigger Contexts, Better Agents

Memory compression isn’t just about saving hardware costs. It changes how people use AI models. Longer contexts let coding assistants read entire repositories, lawyers analyze stacks of contracts, and support agents remember full case histories. This reduces the need for awkward workarounds like chunking text or deleting previous messages.

Developers no longer have to design around short context windows. They can build products that handle complex, multi-document workflows. The user experience improves because AI remembers more and feels more continuous and smart.

Behind the scenes, researchers have combined compression with architectural tricks. For example, some models use linear attention in some layers to keep fixed-size state instead of growing KV caches. Others use grouped-query attention to share memory more efficiently across heads. These advances let context length scale from a few thousand tokens to over a million.

Still, more tokens don’t always mean better answers. Models can forget details buried deep in the context or overweight recent information. So developers often combine long context with retrieval, ranking, and summarization methods to keep responses accurate.

Looking ahead, AI memory efficiency will keep getting better. The math is improving and hardware is catching up. The biggest wins come when both move together. This lets AI models grow smarter without exploding memory needs.

For anyone building or using AI today, these compression tricks unlock new possibilities. They make longer, richer AI conversations practical. And they help bring powerful AI tools to devices with limited memory, like smartphones.