Training Therapeutic Judges and Multi-Agent Systems for Human-Aligned Mental Health Support

Short summary

Researchers introduce TheraJudge, an open-source therapeutic evaluator trained on human-annotated data to assess LLM mental health responses across 7 psychological dimensions with high clinician agreement (ICC 0.87-0.95). TheraAgent, a multi-agent system with specialized roles, acts on these evaluations to improve response quality by +0.43 points on a 5-point scale. Open-source code and framework enable reproducible therapeutic AI alignment.

• •Open-source therapeutic evaluator (TheraJudge) validates LLM responses against 7 psychological dimensions with 87-95% clinician agreement
• •Multi-agent system (TheraAgent) uses evaluation signals to refine responses, improving quality +0.43 points
• •Demonstrates that human-aligned evaluation driving refinement yields better therapeutic outcomes than generation strength alone

Generated with AI, which can make mistakes.