The Kinetics of Training: A Driven-Nucleation Rate Law for Emergence, Plasticity Loss, and Circuit Control in Language Models

Short summary

This paper models capability emergence in language models as a driven-nucleation rate process where all circuit parts must align simultaneously. It shows no-partial-credit joint alignment is the rate-limiting step, with ablating one circuit part leaving only 17% of capability versus the 50-83% partial credit predicts. The framework enables predicting capability ignition, diagnosing plasticity loss, and restoring learnability by re-initializing query-key slices.

• •Capability emergence follows a nucleation rate law: all circuit parts must align at once
• •Ablating one circuit part leaves 17% of capability, far below the 50-83% partial credit predicts
• •Re-initializing query-key slices restores learnability after plasticity loss (6/6 models)

Generated with AI, which can make mistakes.