Eval-Gated AI Releases: Treating Retrieval Quality Like Unit Tests

Short summary

The author built an eval-gated CI pipeline for an enterprise RAG copilot (Atlas) that blocks merges on quality or cost regressions, treating retrieval metrics like unit tests. The pipeline uses RAGAS faithfulness, answer relevancy, context recall, citation correctness, and adversarial injection tests with absolute floors plus regression bands. A real QLoRA fine-tuning example shows the gate both promoting a candidate (citation validity 0→0.955, cost −79%) and blocking on faithfulness trade-off.

• •Eval-gated CI blocks merges on RAGAS quality regressions and cost increases using floors plus regression bands
• •Deterministic GPU-free evals run against committed cassettes for fast reproducible gates
• •A QLoRA fine-tuning case study demonstrates the gate promoting a 79% cost reduction while flagging a faithfulness trade-off

Generated with AI, which can make mistakes.