Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40%

Short summary

A detailed engineering deep-dive on rebuilding a production RAG retrieval layer from first principles, achieving 95% recall@10 and 40% latency reduction. Covers document-type-specific chunking strategies (recursive, semantic, agentic), hybrid retrieval with BM25 + vector + cross-encoder reranking using Reciprocal Rank Fusion, and LLM-based query transformation. Includes production config tables with measured recall metrics per document type.

• •Document-type-aware chunking: recursive for legal, semantic for tickets, agentic for wikis — each tuned with measured recall@10
• •Hybrid retrieval via parallel vector + BM25 with Reciprocal Rank Fusion, then cross-encoder rerank (50→5 funnel adds 15% recall for 50ms)
• •LLM query expansion transforms poorly-phrased user questions into multiple targeted search queries before retrieval

Generated with AI, which can make mistakes.