We’re thrilled to announce a significant leap forward for the Gemma 4 family! Our latest versions are now meticulously optimized with Quantization-Aware Training (QAT). This powerful enhancement dramatically slashes memory requirements and supercharges on-device performance, making AI more accessible than ever before.
Since the initial launch of Gemma 4 just two months ago, we’ve been relentlessly innovating to expand its capabilities. We first introduced Multi-Token Prediction (MTP) to accelerate inference speeds, and then quickly followed up with a 12B model to perfectly bridge the gap between our E4B and 26B MOE models.
Unlocking Efficiency with Quantization-Aware Training (QAT)
Today’s release marks another major milestone: new checkpoints that leverage Quantization-Aware Training (QAT). This groundbreaking optimization makes Gemma 4 incredibly efficient, empowering you to run these advanced models directly on everyday edge devices and consumer GPUs.
So, what exactly is QAT? It’s a sophisticated technique that simulates the quantization process during the model’s training phase. By doing this, QAT minimizes any potential quality loss that typically occurs when a model is compressed, ensuring you get top-tier performance without the heavy memory footprint.
This release includes QAT checkpoints for the widely popular Q4_0 quantization format. But we didn’t stop there; we also developed a novel, specialized quantization format tailored specifically for mobile use cases. Utilizing this mobile-first format, we’ve managed to shrink the memory footprint of Gemma 4 E2B to an astonishing 1GB.
QAT vs. Post-Training Quantization (PTQ)
Quantization is an essential technology for running large language models on consumer hardware. It dramatically reduces memory usage and accelerates decoding speeds, but traditional methods often come with trade-offs.
Standard Post-Training Quantization (PTQ) applies compression after the model has already been trained, which can sometimes lead to a noticeable drop in performance. In contrast, QAT integrates the quantization process directly into the training regimen. While PTQ is already effective, our QAT results consistently yield even higher overall quality compared to standard PTQ baselines.
We’ve meticulously applied this QAT recipe to the popular Q4_0 format across all Gemma 4 models to maximize their performance. For our edge models (E2B and E4B), we completely re-imagined the approach to quantization, developing a special, purpose-built mobile-specialized schema.
This innovative approach ensures that Gemma 4 performs smoothly and efficiently, even on resource-constrained mobile processors. Traditional compression formats often struggle on these devices, so our custom schema makes a significant difference.
Optimized for Every Device
Want to reduce your memory footprint even further? Many use cases don’t require both audio and vision encoders. By deploying only the modalities you need, you can significantly optimize your memory usage.
For example, the Gemma 4 E2B text-only model (without Per-Layer Embeddings) astonishingly requires less than 1 GB of memory. This level of efficiency opens up new possibilities for AI applications on a vast array of devices.
To ensure these highly optimized models are easily integrated into your existing workflows, we’ve partnered with popular developer tools across the ecosystem. Starting today, these tools will seamlessly support the new Gemma 4 QAT checkpoints, making deployment straightforward and hassle-free.
The approximate memory requirements for loading these models are now dramatically reduced, thanks to these innovations. We’re incredibly excited to see the groundbreaking applications and experiences you’ll create with Gemma 4, now running locally and more efficiently than ever before!
Source: Google Blog (The Keyword)
