DiffusionGemma: The Developer Guide

Short summary

DiffusionGemma is an experimental 26B Mixture-of-Experts diffusion-based language model built on the Gemma 4 backbone, activating only 3.8B parameters at inference. It shifts the generation bottleneck from memory bandwidth to compute, achieving up to 700+ tokens/sec on RTX 5090 and 1000+ on H100. The guide includes a Sudoku-solving fine-tuning recipe demonstrating bidirectional context propagation and self-correction via re-noising.

• •26B MoE diffusion LLM (3.8B active params) fitting in 18GB VRAM quantized
• •Up to 4x faster token generation by shifting bottleneck from memory bandwidth to compute
• •Fine-tuning recipe on Sudoku demonstrates bidirectional reasoning and parallel self-correction

Generated with AI, which can make mistakes.