Predicting LLM Safety Before Release by Simulating Deployment

Short summary

Deployment Simulation is a pre-release safety methodology that replays real conversations with candidate models to predict behavioral changes. In a GPT-5.4 study, it predicted behavioral shift directions 92% of the time versus 54% for traditional challenge-based evaluations. The approach complements traditional safety evals and handles agentic tool use by simulating realistic tool responses.

• •Deployment Simulation replays real conversations with new LLM candidates to test safety behavior before release
• •92% accuracy predicting behavioral changes in GPT-5.4 study, vs 54% for traditional challenge-based evals
• •Addresses agentic tool use by simulating tool responses, complements but doesn't replace traditional safety testing

Generated with AI, which can make mistakes.