S-GAI: Spectral Geometry-Aware Initialization for Sigmoidal MLPs -- From Dataset Geometry to Network Weights

Short summary

S-GAI proposes a spectral geometry-aware initialization for one-hidden-layer sigmoid networks that encodes dataset geometry into network weights via SVD. Instead of random weights, it represents each principal direction as sigmoid gates, creating a hidden layer directly informed by the training data's class structure. On MNIST and CIFAR-10, S-GAI-initialized networks reach comparable final accuracy to standard training while starting from a more informative state.

• •Novel initialization method encodes class-wise spectral geometry into sigmoid network weights
• •Uses SVD to extract principal directions and represents them as sigmoid gates
• •Reaches comparable accuracy to full training with better initialization than Xavier method

Generated with AI, which can make mistakes.