SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervision

Short summary

SEAD improves on-policy model distillation by adapting supervision to student competence, automatically skipping ~50% of redundant tokens through entropy-guided selection. Combines dynamic token weighting, KL divergence annealing, and competence-gated curriculum learning to achieve +4.8% accuracy on OLMo-3 (7B-32B) across math benchmarks, with confirmed synergistic interactions.

• •Uses entropy to detect and skip redundant training tokens (~50% reduction in gradient computation)
• •Integrates token selection, KL annealing schedule, and curriculum learning as interdependent components
• •+4.8% average accuracy gain on OLMo-3 models across six math benchmarks with ablation-confirmed synergy

Generated with AI, which can make mistakes.