Shared SFT Lessons Across Alignment, Model Organisms, and Toy Models

Short summary

This paper shows that supervised fine-tuning lessons transfer across alignment training, model organisms, and toy models. Training on the reason for a behavior generalizes better than training on examples alone. Mixing on-model data with off-model outputs prevents capability damage during alignment SFT. However, follow-up benign SFT can erase alignment behaviors while preserving capabilities, showing robustness requires more than capability preservation.

• •SFT on reasoning behind behaviors generalizes better than training on examples alone
• •Mixing on-model data with off-model outputs prevents capability damage during alignment
• •Follow-up benign SFT can erase alignment behaviors, proving capability preservation ≠ robustness

Generated with AI, which can make mistakes.