Google Launches MTP Drafter for Gemma 4, Boosting Speed 3x

Google has unveiled a new Multi-Token Prediction (MTP) drafter for its Gemma 4 family of open-source AI models, delivering up to 3x faster inference speeds through a speculative decoding architecture. The announcement, published in a blog post on May 5, marks a significant leap in making large language models more practical for real-world deployment — especially on consumer-grade hardware.

Gemma 4, currently Google's most capable open-source model family, has already surpassed 60 million downloads in just weeks since its initial release. The MTP drafter is designed to push inference efficiency to its limits without compromising output quality or reasoning capability.

Key Takeaways

• Google's MTP drafter achieves up to 3x faster inference for Gemma 4 models
• The system uses speculative decoding to predict multiple future tokens simultaneously
• Gemma 4 has already exceeded 60 million downloads since launch
• Benchmarks on Apple Silicon show approximately 2x speedup with batch sizes of 4 to 8
• Output quality and reasoning logic remain unchanged despite the speed gains
• The lightweight drafter pairs with heavy target models like Gemma 4 27B and 12B

How Speculative Decoding Solves the Memory Bandwidth Bottleneck

Standard large language model inference suffers from a fundamental hardware limitation: memory bandwidth. During autoregressive generation, processors must repeatedly transfer billions of parameters from GPU memory (VRAM) to compute units for every single token generated. This creates severe latency bottlenecks and leaves expensive computational resources sitting idle.

Google's blog post explains that this memory-bound nature of inference means that even the most powerful GPUs spend more time waiting for data than actually performing calculations. The result is painfully slow token-by-token generation that frustrates users and drives up serving costs for companies deploying these models at scale.

The MTP drafter addresses this core pain point by introducing a two-model architecture. A lightweight drafter model runs alongside the heavy target model (such as Gemma 4 27B), using the idle compute capacity to predict multiple future tokens in rapid succession. The target model then verifies these predictions in parallel, confirming or rejecting them in a single forward pass.

When predictions are accepted — which happens frequently due to the drafter's training — the system effectively generates an entire sequence of tokens in the time it would normally take to produce just one. This is the essence of speculative decoding, and it represents one of the most promising approaches to making LLM inference economically viable.

Real-World Performance Gains on Consumer Hardware

Google's benchmark results demonstrate tangible improvements across different hardware configurations. On Apple Silicon chips — the processors powering MacBook Pro and Mac Studio machines — the Gemma 4 26B model achieved approximately 2x speedup when batch sizes were set between 4 and 8.

These results are particularly significant for several reasons:

• Consumer accessibility: Apple Silicon is widely available in consumer laptops, not just data center GPUs
• Batch size flexibility: The speedup scales effectively across practical batch size ranges
• No quality tradeoff: The verification step ensures every accepted token matches what the target model would have generated independently
• Resource efficiency: The drafter utilizes compute cycles that would otherwise go to waste during memory-bound operations

The up to 3x speed improvement figure represents peak performance under optimal conditions, but even the more conservative 2x gains observed on Apple Silicon represent a transformative improvement for local inference workflows. For developers running models on their own machines, this could mean the difference between a usable interactive experience and an frustratingly slow one.

Technical Architecture Behind the MTP Drafter

The MTP drafter represents a carefully engineered balance between prediction accuracy and computational overhead. Unlike traditional single-token prediction, where the model generates one token at a time in a strictly sequential fashion, the MTP approach trains a smaller model to anticipate multiple tokens ahead in the sequence.

Google's implementation pairs the drafter with specific target models in the Gemma 4 family. The drafter shares embedding layers and architectural patterns with its target model, allowing it to make informed predictions about what the larger model would generate. This architectural alignment is critical — a poorly matched drafter would see most of its predictions rejected, negating any speed benefits.

The verification process works as follows:

• The MTP drafter generates a candidate sequence of N future tokens
• The target model processes all N tokens in a single forward pass
• Each predicted token is compared against the target model's actual output distribution
• Matching tokens are accepted; the first mismatched token triggers a fallback to standard generation
• The process then restarts from the last accepted position

This approach guarantees that the final output is mathematically identical to what the target model would produce on its own. The speculative decoding framework introduces zero degradation in output quality — it purely optimizes the speed at which that identical output is generated.

Industry Context: The Race for Inference Efficiency

Google's MTP drafter arrives at a pivotal moment in the AI industry. While much attention has focused on training ever-larger models, the practical challenge of serving those models efficiently has become the dominant concern for companies deploying AI at scale.

Meta's Llama 3 series, Mistral's models, and other open-source competitors have all been exploring various optimization techniques. Quantization, pruning, knowledge distillation, and now speculative decoding represent a toolkit of approaches that model providers are combining to reduce inference costs.

The timing is also notable because inference costs now represent the largest ongoing expense for companies deploying LLMs in production. Training a model is a one-time cost, but serving millions of user queries daily creates a continuous financial burden. Any technique that doubles or triples throughput without requiring additional hardware directly impacts the bottom line.

Compared to approaches like NVIDIA's TensorRT-LLM optimizations or vLLM's PagedAttention, speculative decoding offers a complementary advantage. These techniques can often be combined, meaning the MTP drafter's benefits could stack on top of other inference optimizations for even greater cumulative speedups.

What This Means for Developers and Businesses

The practical implications of the MTP drafter extend across multiple use cases and deployment scenarios. For individual developers running Gemma 4 locally, the speedup makes interactive applications like coding assistants, chatbots, and writing tools significantly more responsive.

For businesses deploying Gemma 4 in production environments, the efficiency gains translate directly into cost savings. Serving the same number of requests with fewer GPU hours — or serving more requests with existing infrastructure — represents a compelling economic argument for adoption.

Key practical benefits include:

• Lower latency for user-facing applications, improving user experience
• Reduced serving costs by maximizing throughput per GPU
• Edge deployment viability on devices with limited memory bandwidth like laptops and mobile devices
• Identical output quality means no revalidation or fine-tuning is needed
• Drop-in compatibility with existing Gemma 4 deployment pipelines

The fact that Gemma 4 has already reached 60 million downloads suggests a massive installed base of developers who can immediately benefit from this optimization. Google's decision to release the MTP drafter as part of the open-source ecosystem rather than keeping it proprietary reflects the company's strategy of building developer loyalty around the Gemma brand.

Looking Ahead: The Future of LLM Inference Optimization

Google's MTP drafter for Gemma 4 signals a broader industry trend toward inference-time optimizations becoming as important as model architecture innovations. As open-source models approach the capability levels of proprietary alternatives from OpenAI and Anthropic, the competitive battleground is shifting from raw intelligence to practical deployability.

We can expect several developments in the coming months. First, other model providers will likely adopt similar speculative decoding approaches for their own model families. Second, hardware manufacturers like NVIDIA and AMD may begin designing chips with speculative decoding workloads specifically in mind. Third, the community will likely develop general-purpose drafters that work across multiple model families, rather than requiring model-specific training.

The 3x speedup ceiling may also rise as Google and other researchers refine drafter architectures. Techniques like tree-based speculative decoding, where multiple candidate sequences are evaluated simultaneously, could push throughput gains even higher.

For now, Google's MTP drafter establishes Gemma 4 as not just a capable open-source model, but an efficiently deployable one — a distinction that matters enormously as AI moves from research demos to production-grade applications serving millions of users worldwide.