The original title is "From weeks to a day: how we made LLM evaluation fast enough to iterate on"

Short summary

Airbnb Engineering details their four-layer approach to making LLM evaluation fast enough for same-day iteration. The core challenge is non-determinism: judges disagree with themselves, references regenerate as different strings, and small score movements are ambiguous. They introduce 'dual indeterminacy' framing to separate epistemic uncertainty (model/judge limits) from aleatoric uncertainty (task ambiguity), each requiring different detection and fix strategies.

• •LLM evaluation friction is primarily an infrastructure problem, not a model quality problem
• •Four-layer stack: diagnostics, evaluation, model mutation, and end-to-end validation
• •Dual indeterminacy framework separates judge drift from task ambiguity to enable trustworthy comparisons

Generated with AI, which can make mistakes.