Google has introduced a new way to make its AI models faster without losing quality. The latest Gemma 4 models now include a feature called Multi-Token Prediction (MTP) that can triple their speed during tasks. This update is a big step toward making local AI more practical and accessible for users running AI on their own hardware.

How MTP Improves AI Performance

Traditional AI models generate text one token at a time, which can slow things down, especially on regular hardware. Each token requires the model to do a lot of calculations, and moving data between memory and processing units takes time. Google’s new approach uses MTP to guess multiple tokens ahead of time with a smaller, lightweight draft model. This speculative step helps the larger model work more efficiently by verifying these guesses in parallel.

The draft models are much smaller—only around 74 million parameters—but they’re designed to quickly produce predictions that the main model can confirm. They share memory caches with the main model, which reduces redundant calculations. The process involves generating draft tokens, then verifying them all at once with the main model, allowing for faster output without sacrificing accuracy.

Real-World Gains and Practical Uses

Google says that with MTP, the speed of its Gemma models can increase up to three times. In tests on various hardware, smaller models on Pixel phones saw nearly threefold improvements, while larger models on Apple’s M4 chips gained about 2.5 times in speed. This means users can run powerful AI models on their personal devices more smoothly, saving time and energy.

One big advantage is that faster local AI can lead to better battery life on mobile devices and make it easier to run advanced models without needing expensive cloud infrastructure. Since the process doesn’t reduce output quality, users can expect the same reliable performance while enjoying quicker responses. The new features are available under an open license, making it easier for developers to incorporate them into different frameworks and tools.

Overall, Google’s MTP technology promises to make local AI faster and more efficient. This could spark more innovation in edge AI, where users want powerful tools that work quickly without relying on the internet. As hardware continues to improve, such techniques will help bring advanced AI closer to everyday use.

Inspired by

• https://arstechnica.com/ai/2026/05/googles-gemma-4-open-ai-models-use-speculative-decoding-to-get-up-to-3x-faster/

Sources

• blog.google
• huggingface.co
• vllm.ai
• sglang.io
• ollama.com