Evaluating LLM Apps in Java

Short summary

Testing LLM apps requires golden datasets, programmatic scoring for deterministic checks, and LLM judges for open-ended rubrics. The post provides Java patterns using Anthropic SDK: structured EvalCase records, ProgrammaticScorer logic, and LLM-as-judge harnesses that catch regressions before production. A solid eval set balances common cases, known-breaking edges, adversarial inputs, and expected refusals.

• •Golden datasets are hand-curated (input, expected-behavior) pairs covering production use patterns, edge cases, and adversarial inputs
• •Programmatic scoring handles exact matches and fact-checking without calling an LLM; LLM judges evaluate open-ended criteria like tone and completeness
• •Evaluation harness runs on every change, fails the build on score drop, and keeps golden sets small enough to iterate fast

Generated with AI, which can make mistakes.