Feature Repulsion and Spectral Lock-in: An Empirical Study of Two-Layer Network Grokking

Short summary

Empirical validation of feature repulsion theory in neural grokking: sign-pattern predictions hold robustly (96.5–100% match), but spectral signatures in parameter updates are strongly activation-dependent. Power-law activations show clear eigengap separation (229× magnitude increase); ReLU activations show no spectral signal, despite maintaining identical sign structure. Result: repulsion exists in feature space regardless of activation, but translates to weights only for power-law derivatives.

• •Sign-structure predictions validated: feature repulsion rules match theory 96.5–100% across 5 seeds on modular addition
• •Spectral signature is activation-dependent: squared activation detects eigengap reliably; ReLU spectrum remains rank-1
• •Key mechanism: feature repulsion depends on correlation structure, but weight-update transmission depends on activation derivatives

Generated with AI, which can make mistakes.