Memorilla Shrinks Massive Document Stores to 16 Tokens

Long documents just became a 16-token problem. Stanford researchers have released Memorilla, an open-source memory system that reads an entire document collection and gives a frozen large language model 16 memory tokens. Code, training recipes, and checkpoints are public under Apache 2.0.

The system targets a basic limit of frozen LLMs: they can only use information inside their context window. Full-context prompting becomes slower as history grows, while retrieval pastes chunks that resemble the question and can miss weaker clues. Memorilla takes a third route by learning what each question needs from the whole document store.

The work comes from Jure Leskovec’s SNAP group, with co-authors from Toronto, Penn, and MIT. Its central claim is simple enough to test: compress a collection into a fixed set of learned vectors without forcing the language model to read every document at answer time.

How Memorilla Compresses a Collection

Memorilla uses three steps. First, a frozen Qwen3-Embedding-4B encodes each document once, so new documents do not force old documents through the encoder again. Next, 16 learned slots attend to both the question and the full store through two cross-attention layers with eight heads.

Finally, the module projects those slots into Qwen3-8B’s input embedding space and prepends them to the question. The encoder and decoder remain frozen; only the memory module trains, using 194,123,776 parameters for the 8B decoder.

Training runs in three stages: about 1.98 million Wikipedia paraphrase-reconstruction examples, followed by about 1.34 million examples from 10 question-answering and summarization datasets, then fine-tuning on all eight benchmarks. The full recipe takes about 19 hours on four H100 GPUs.

On FactKG, a frozen Qwen3-8B reached 84.33% accuracy with Memorilla, compared with 80.43% when every document sat in context and 80.38% with MemoRAG. Memorilla reached that score using 108 times fewer document tokens. A tidy result, at least until the hardware bill enters the room.

Strong Results, Uneven Trade-offs

Memorilla posted the highest reported average across eight benchmarks, scoring 49.68 and taking the top score on five of eight. On PersonaMem-v2, it reached 44.60% accuracy, ahead of 27.92% for full context and 30.90% for xRAG, the next best method.

It also recorded a 57.00 exact-match score on TriviaQA, just ahead of MemoRAG at 56.60, and the best LaMP-4 and LaMP-7 ROUGE-L scores at 18.15 and 48.37. The overall average compared with 41.92 for xRAG, 40.75 for LLMLingua-2, and 38.02 for Mem0.

The system loses on three benchmarks. PubMedQA is the clearest miss: Memorilla scored 59.00%, while LLMLingua-2 reached 69.00%; those abstracts total only 324 tokens, leaving compression with little to gain. MemoRAG led PersonalizationV4 at 67.88% versus 66.65%, and NarrativeQA at 20.51 versus 19.31 ROUGE-L.

The compression numbers remain striking. On NarrativeQA, 16 memory tokens replace 67,385 document tokens, a 4,212-fold cut, while MemoRAG used about 690 times and 50 times more document tokens on the two benchmarks where it led. The research team says all baselines used the same frozen Qwen3-8B, and the scores are self-reported.

Memorilla also produced a gain as agent memory: eight memory tokens raised a Qwen3-4B TextWorld agent’s average success from 18.6% to 24.2%. That result suggests the module is not limited to question answering, though the reported evidence still comes from the listed benchmarks and setup.

Deployment Has Serious Hardware Requirements

Memorilla runs on Linux with CUDA GPUs and Python 3.11. The quickstart needs one GPU with 80 GB, and the recipes ran on H100 80GB hardware; no quantized variants have been disclosed.

On one H100, Memorilla answered in 0.74 seconds with 512 documents, or about 128K tokens. Full-context inference rose from 0.75 seconds to 19.9 seconds, making it 27 times slower, while the memory module itself took about 2 milliseconds.

An optional memory cache precomputes cross-attention keys and values. On one B200, a question took 8.1 milliseconds at 1 million documents and 33 milliseconds at 4 million; with torch.compile, the 1-million-document case dropped to 5.0 milliseconds. The cache costs 80 GiB at 4 million documents, because apparently “memory” remains a physical concept.

Installation uses uv on Linux, followed by MemoryVLLM.from_pretrained. The first run downloads about 25 GB of models. Checkpoints sit in memorilla/Memorilla-Qwen3-8B folders labeled Stage 1, Stage 2, Stage 3, PV4, and PMv2, and each checkpoint must run with the decoder it was trained with.

Custom data requires parquet files with collection_id, question, answer, and documents columns. The research team also released PersonalizationV4, containing 149 synthetic users with 200 conversations each, under CC BY 4.0.

Memorilla’s appeal is clear: fixed-size memory, strong results on several long-context tasks, and public code that others can inspect and retrain. Its limits are just as clear—an 80 GB GPU for the quickstart, 194 million trainable parameters, and uneven performance when the source material is already short.

Stay connected via Google News

Follow us for the latest AI News