Gradient Smoothing: Coupling Layer-wise Updates for Improved Optimization

Short summary

arXiv paper introduces Gradient Smoothing, a depth-wise optimization method that improves training of deep networks like transformers and vision models by applying structured smoothing to layer-wise updates. The technique works with any base optimizer (SGD, Adam, Muon) and shows consistent improvements across language model pretraining, RL post-training, and diffusion modeling without modifying architectures. Method treats cross-depth structure as a preconditioning opportunity, promoting more stable representation evolution.

• •Introduces Gradient Smoothing: a depth-wise optimization technique for deep networks that improves training across diverse architectures
• •Compatible with any base optimizer (SGD, Adam, Muon) with minimal computational overhead and no model/training changes needed
• •Evaluated across language models, diffusion models, and vision transformers; shows consistent improvements in optimization and generalization

Generated with AI, which can make mistakes.