Google Releases Open-Source DiffusionGemma 26B Model

Google DeepMind has released DiffusionGemma, a new family of open-source language models that explores a novel architecture for generating text. The first release is an instruction-tuned, 26-billion-parameter model, signaling Google's continued investment in both the Gemma family and alternative AI architectures.

A Different Approach to Generation

Unlike most large language models like GPT-4 or Llama 3, which are autoregressive and generate text one token at a time, DiffusionGemma employs a diffusion-based method. This technique, widely used in popular image generation models, starts with a complete, noisy sequence of text and iteratively refines it into a coherent output. This marks a notable departure from the standard Transformer architecture that has dominated natural language processing for years.

The newly released model is a Mixture-of-Experts (MoE) variant, which means that while it has 26 billion total parameters, only 4 billion are active for any given input. This design makes inference more computationally efficient than a dense model of the same size. According to its official model card, the DiffusionGemma 26B-A4B-it model is designed for a variety of image-to-text and text-to-text tasks.

By open-sourcing DiffusionGemma under a permissive Apache 2.0 license, Google is enabling the wider AI community to experiment with this less-common approach. The exploration of non-autoregressive models could lead to new discoveries in text generation, potentially offering different trade-offs in speed, coherence, and controllability compared to today's leading models.