Scoring AI Agents: Deterministic Metrics + an LLM Judge

Short summary

Build an evaluation framework combining deterministic metrics (accuracy, timeout rate, reproducibility) with optional Claude judges to assess subjective agent qualities. Run agents in subprocess isolation against fixtures, then use schema-validated LLM feedback to propose specific prompt fixes. Feed results into an automated improvement loop that mutates and tests candidate prompts, tracking drift over time.

• •Framework scores agents on deterministic metrics first, reserves LLM judging for qualitative dimensions
• •Subprocess isolation ensures reproducibility; identical outputs judged once to bound cost
• •Automated loop mutates prompt candidates, persists improvements, closes the quality-measurement feedback loop

Generated with AI, which can make mistakes.