We gated CI on six open-source LLM eval frameworks. Only two survived the merge queue.

Short summary

Author wired six open-source LLM eval frameworks into a real GitHub Actions merge queue over eight months, testing determinism, speed, cost, wiring effort, and signal quality. Only Promptfoo and DeepEval survived as reliable CI gates because they support deterministic exit-code-based checks. The key lesson: LLM-as-judge metrics should be non-blocking signals, not gating thresholds, because judge scores drift and will block clean PRs.

• •Tested 6 open-source LLM eval frameworks in a real merge queue over 8 months
• •Promptfoo and DeepEval survived due to deterministic exit-code checks
• •LLM-as-judge scores should be non-blocking signals, not CI gates

Generated with AI, which can make mistakes.