Diffusion Nemotron: Nvidia's "Two Brain" Diffusion Model

Short summary

NVIDIA's Nemotron "Two Tower" diffusion model freezes one 52-layer context tower and retrains a second as a parallel denoiser generating 16-token blocks concurrently—bypassing autoregressive next-token prediction's compute wall. Quality retention reaches 98.7% of the original, with cross-attention "sky bridges" binding the towers. Tradeoffs are steep: math/code performance drops and brittleness at different block sizes, indicating diffusion LLMs aren't production-ready yet.

• •NVIDIA's Nemotron uses dual towers: frozen context encoder + retrained parallel denoiser (16-token blocks)
• •Achieves 98.7% quality vs autoregressive while generating tokens concurrently
• •Current limitations: math/code performance loss, architectural brittleness at different block sizes

Generated with AI, which can make mistakes.