Tilde Research Introduces Aurora: A Leverage-Aware Optimizer That Fixes a Hidden Neuron Death Problem in Muon

Short summary

Tilde Research introduced Aurora, a leverage-aware neural network optimizer designed to fix a critical architectural flaw in Muon: the silent death of a significant fraction of MLP neurons during training, which permanently impairs model performance. Aurora achieves state-of-the-art results on 1.1B parameter pretraining tasks. The research addresses a hidden structural problem that degraded training quality without obvious symptoms.

• •Aurora fixes neuron death problem in Muon optimizer
• •Achieves state-of-the-art on 1.1B parameter pretraining
• •Addresses hidden architectural flaw in neural network training

Generated with AI, which can make mistakes.