RAG for developers who aren't AI engineers: what actually matters

Short summary

An experienced developer shares hard-won lessons from building a production RAG system over 62 books (~46,000 chunks), measuring every design decision against a 135-question test set. The baseline fixed-chunk top-5 setup achieved only 35.2% recall, meaning the correct passage never reached the model two-thirds of the time — yet the LLM answered confidently anyway. The article argues that building a test set with known-answer locations and unanswerable questions is the single most important step before optimizing any other component.

• •Standard RAG baseline achieved only 35.2% recall@5 — model answered confidently even when retrieval failed
• •Test sets with known answer locations and unanswerable questions are essential before tuning chunking or embedders
• •RAG failure mode is plausible fabrication, not 'no results' — only measurement reveals the problem

Generated with AI, which can make mistakes.