Singular Learning and Occam's Razor in Deep Monomial Networks

Short summary

Researchers analyze critical points in deep neural networks with monomial activations using polynomial algebra and Mason's Theorem. They show that network criticality occurs exactly when some neurons become inactive or redundant, with this effect strengthening at higher activation degrees. The work explains the implicit bias of deep networks toward simpler, more generalizable functions.

• •Critical points in deep networks with monomial activations correspond to inactive or redundant neurons
• •Effect increases with higher activation degrees (mathematical proof using Mason's Theorem)
• •Provides theoretical foundation for implicit bias toward simpler functions in deep learning

Generated with AI, which can make mistakes.