Building a RAG Pipeline From Scratch: What SmartQueue Taught Me About Retrieval

Short summary

The author built SmartQueue's RAG pipeline using BM25 instead of ChromaDB to simplify deployment on free-tier infrastructure, sharing concrete tuning decisions (k=4 docs, temp=0.2) with production reasoning. Every AI endpoint includes non-LLM fallbacks for graceful degradation—a principle often overlooked but critical in production systems.

• •Switched from ChromaDB to simpler BM25 search for reliable container deployment
• •Shared tuning decisions with rationale: k=4 retrieved docs, temperature=0.2, rate-limiting, token budgets
• •Emphasized designing fallback paths when the LLM fails—graceful degradation matters more than raw retrieval quality

Generated with AI, which can make mistakes.