A RAG evaluator that admits what it can't judge

Short summary

rag-triad is a local RAG evaluator that replaces overconfident LLM grading with deterministic checks and honest abstention. It diagnoses three distinct failure modes (bad retrieval, hallucination, off-topic response) and validates itself before deployment. The tool prioritizes calibration over raw capability—a model that abstains is safer than one that confidently hallucinates.

• •Deterministic groundedness checking prevents fabricated citations from passing evaluation
• •Diagnoses three failure modes separately: retrieval miss, hallucination, off-topic answer
• •Open-source, runs locally on Ollama, emphasizes honest abstention over confident guessing

Generated with AI, which can make mistakes.