Speed and cost are big hurdles for companies using large language models (LLMs). These models often generate thousands of tokens per query, but current hardware struggles to keep up. Now, researchers have found a way to make these models faster by modifying how they predict tokens, without needing extra draft models or complex decoding methods. This breakthrough could help businesses run AI systems more efficiently and affordably.

How Traditional Language Models Work

Most LLMs generate one token at a time, in sequence. This serial process limits how fast the model can produce results. For tasks like reasoning or “chain of thought” prompts, models need to produce many tokens, which takes time and costs more in GPU resources. To speed things up, researchers aimed to enable models to predict multiple tokens simultaneously in a single pass.

The challenge is maintaining coherence. If the model predicts multiple tokens independently, it might produce nonsensical or inconsistent outputs. To prevent this, the team used a teacher-student setup, where a larger, more accurate “teacher” model guides a smaller “student” model to generate multiple tokens that make sense together. The student learns to produce chunks of tokens that are both fast and accurate, guided by the teacher’s evaluations.

The Multi-Token Prediction Technique

The key innovation is a special training method that turns standard next-token models into parallel decoders. This involves adding a unique mask token and an online self-distillation process. During training, the student model predicts several tokens at once, and its predictions are scored by the teacher model, which ensures they are coherent. This process helps the student model learn to generate more than one token in a single pass without sacrificing quality.

At inference time, the system uses a dynamic decoding strategy called ConfAdapt. This method adjusts how many tokens are predicted at each step based on the model’s confidence. When the model is sure, it predicts larger chunks of tokens to speed up processing. When it’s less confident, it reduces the chunk size to maintain accuracy. This balance helps achieve faster responses without significant errors.

Performance and Impact

In tests on math reasoning benchmarks, the new approach delivered more than three times the speed of traditional models. For example, an 8-billion-parameter model showed a speed increase with less than a 3% drop in accuracy. Smaller models also benefited, reaching similar speed gains with minimal performance loss. Importantly, the final models retain the same core architecture as the original pretrained models, making deployment straightforward without extra verification steps or specialized code.

This technique could be a game-changer for enterprises that want to deploy reasoning AI systems more cost-effectively. By embedding acceleration directly into the model weights, companies can reduce GPU costs and improve response times, making large language models more practical for real-world applications.

Overall, this multi-token prediction method offers a promising way to boost AI performance without adding complexity. It combines clever training techniques with adaptive decoding, providing faster and cheaper inference for demanding AI workloads.

Inspired by

• https://www.infoworld.com/article/4136453/multi-token-prediction-technique-triples-llm-inference-speed-without-auxiliary-draft-models.html