Why teaching AI agents to use tools keeps blowing up in training

Short summary

Multi-step agent training collapses when control-token probabilities spike unexpectedly, scrambling execution scaffolding even though underlying skills remain intact. The remedy: interleave supervised learning with reinforcement learning to keep control tokens stable during training. The trade-off: relying on supervised examples may reduce adaptability on out-of-distribution tasks, but the technique addresses a critical bottleneck in shipping reliable agents.

• •Control-token probability spikes cause multi-step agent training collapse, not skill loss
• •Interleaving supervised learning with RL stabilizes training by managing control tokens
• •Trade-off: supervised examples reduce out-of-distribution generalization but improve reliability

Generated with AI, which can make mistakes.