Building and evaluating model diffing agents

Short summary

Google DeepMind researchers propose 'diffing agents'—LLM auditors that intelligently search for behavioral differences between two models rather than relying on static test suites. This approach surfaces subtle behavioral changes more effectively than traditional single-model auditing. The method includes ground-truth evaluations and demonstrates results on real model pairs.

• •Diffing agents use intelligent prompt crafting to find behavioral differences between LLMs
• •Outperforms standard auditing agents at detecting subtle behavioral changes
• •Includes rigorous ground-truth evaluations for validating model diffing effectiveness

Generated with AI, which can make mistakes.