Training transformers where every layer W = V·Uᵀ from initialization reveals a corpus-determined optimal rank - looking for arXiv endorser (cs.LG) [D]

Short summary

Independent researcher introduces Native Factorized Weights (NFW), replacing dense transformer weights with low-rank factorizations W=V·U^T trained from initialization. On WikiText-2, NFW achieves better generalization than dense models with 5-10x fewer parameters by operating within a corpus-dependent optimal rank band that structurally prevents memorization. Full code, results, and training scripts shared on GitHub; author seeking arXiv endorsement for peer review.

• •Native Factorized Weights replaces dense layers with low-rank decomposition from initialization, not post-hoc
• •Models operate in a corpus-determined 'generalization band' where rank boundary prevents both underfitting and memorization
• •Achieves superior performance to dense baselines with 5-10x fewer parameters; code available on GitHub

Generated with AI, which can make mistakes.