Back to feed
arXiv cs.LG
arXiv cs.LG
8/5/2026
Sphere Retraction Normalizations

Sphere Retraction Normalizations

Short summary

This paper unifies Euclidean residual connections and Geodesic Normalization under a single framework of retraction maps on a hypersphere, showing all retractions collapse to one scalar design choice. It introduces Proj-SpheretNorm and Cay-SpheretNorm as norm-preserving, algebraically simple alternatives, both members of a one-parameter family p-SpheretNorm. On nanoGPT, all three proposed methods outperform existing lightweight schemes, with optimal performance at finite p rather than at the exponential map limit.

  • •Unifies Euclidean residual connections and GeoNorm as retraction maps on a hypersphere
  • •Introduces p-SpheretNorm family with Proj and Cay variants that are exactly norm-preserving
  • •nanoGPT experiments show all three methods beat existing schemes, optimal at finite p

Generated with AI, which can make mistakes.

#research-breakthrough
Read full article at arXiv cs.LG

Is this a good recommendation for you?

Comments

Failed to load comments. Please try again.

Explore more

Mechanisms of Width Scaling in Normalized Residual Networks: The Effective Alignment Dimension
arXiv cs.LG
LieBN: Batch Normalization over Lie Groups
arXiv cs.LG
The Active Ingredient in Muon's Grokking
arXiv cs.LG
Between Gradient and Natural Gradient: A Continuum of LoRA Initializations
arXiv cs.LG