Large language models, or LLMs, like GPT and PaLM, are changing how we work and communicate. They power chatbots, coding helpers, and more. But running these models isn’t cheap. It costs way more than simple keyword searches—up to ten times more in some cases. A big reason for this high cost is how these models manage memory.

The Hidden Memory Problem in LLMs

At the core of LLMs is the Transformer model. It creates text one word at a time, but it needs to remember what came before. This memory is stored in something called the Key-Value (KV) cache. Think of it as the model’s short-term memory for a conversation.

The issue is, the KV cache is huge and changes size depending on the task. Current systems store all of this in a single block of memory. That causes two big problems. First, memory gets fragmented inside, meaning a lot of space is wasted. Systems pre-allocate a large chunk assuming the longest possible output, like 2048 tokens. But if the output is shorter, much of that space isn’t used.

Second, because different requests reserve different amounts of memory, the GPU ends up scattered with tiny gaps. This makes it hard to fit new requests into the available space. Studies show that only about 20 to 38 percent of this memory is actually used for storing token data—the rest is wasted.

The second problem is that the current setup makes sharing memory between different tasks difficult. When a decoding process generates multiple outputs, it could reuse parts of the KV cache. But because each sequence’s cache is stored separately in a big block, sharing isn’t easy. This limits how many requests can be processed at once, slowing everything down.

The Inspiration from Operating Systems: PagedAttention

To fix these issues, researchers created PagedAttention. The idea comes from operating systems, which use virtual memory and paging to manage physical memory efficiently.

Imagine dividing the KV cache into small, fixed-size blocks called pages. Instead of holding all data in one big chunk, each block stores part of the token data for a set number of tokens. Requests are managed like processes, with their own “logical” blocks mapped onto “physical” blocks in GPU memory. This makes memory management more flexible.

With PagedAttention, memory is allocated only when needed, so internal fragmentation is almost eliminated. Since all blocks are the same size, external fragmentation—tiny unusable gaps—disappears. This setup allows the system to dynamically assign memory to different requests without wasting space.

Another big advantage is that it enables sharing of KV blocks across multiple sequences. For example, in parallel sampling or beam search, multiple outputs can reuse the initial prompt’s KV cache. The system even uses a copy-on-write mechanism, borrowed from OS concepts, which allows different sequences to modify shared blocks safely without making redundant copies.

Introducing vLLM: Faster and More Efficient LLM Serving

Building on PagedAttention, vLLM is a system designed for high-speed LLM serving. It manages memory at the block level and includes a smart scheduler that works seamlessly with PagedAttention.

The result is a system that wastes almost no memory and can share KV cache data across different requests. This boosts throughput—how many tokens or requests it can handle per second—by 2 to 4 times compared to current top systems like FasterTransformer and Orca. This boost is even bigger with longer sequences, larger models, or complex decoding techniques.

For instance, when serving a 13-billion-parameter model, vLLM can handle more requests simultaneously than even an “oracle” version of Orca, which assumes perfect knowledge of how long outputs will be. It can process roughly twice as many requests at once and significantly reduces memory usage during tasks like parallel sampling and beam search.

The Future of LLM Deployment

By borrowing ideas from operating systems, PagedAttention and vLLM are making large language models much more efficient. This means lower costs for cloud providers and faster, more responsive AI tools for users everywhere. They address a major bottleneck in AI deployment, paving the way for smarter, more accessible AI services in the future.

This shift could lead to cheaper cloud hosting and more powerful AI applications, making advanced language models easier to deploy at scale and more affordable.

Inspired by

• https://www.infoworld.com/article/4055048/unlocking-llm-superpowers-how-pagedattention-helps-the-memory-maze.html

Sources

• gmpg.org
• podigee.com
• podigee.com