Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40%

Short summary

A detailed engineering breakdown of rebuilding a RAG retrieval pipeline from first principles, moving from naive 512-token chunking to document-type-specific strategies (recursive, semantic, agentic) achieving 91-97% recall@10. The approach combines hybrid retrieval (vector + BM25 via Reciprocal Rank Fusion) with cross-encoder reranking, cutting latency 40% while improving relevance. Query transformation using LLMs further improves retrieval quality for poorly-formed user questions.

• •Document-type-specific chunking (recursive, semantic, agentic) beats fixed 512-token chunks, reaching 91-97% recall@10
• •Hybrid retrieval with RRF fusion and cross-encoder reranking (50→5 funnel) gains 15% recall for 50ms cost
• •Query transformation via LLM expands poorly-formed user queries into multiple search strategies

Generated with AI, which can make mistakes.