Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40%

Short summary

A detailed engineering walkthrough of rebuilding a production RAG retrieval layer, covering chunking strategies (recursive, semantic, agentic) tuned per document type, hybrid retrieval combining vector search and BM25 with reciprocal rank fusion, and cross-encoder reranking that improved recall@10 to 95% while cutting latency 40%. The article includes concrete code, a comparison table of chunking configs by document type, and query transformation patterns for handling poorly-formed user questions.

• •Recursive chunking by document type outperforms fixed 512-token splits — legal contracts hit 94% recall, internal wikis 97% with agentic chunking
• •Hybrid retrieval (vector + BM25 + cross-encoder rerank) via reciprocal rank fusion eliminates the exact-match vs semantic-match tradeoff
• •Cross-encoder reranking costs 50ms but gains 15% recall; bi-encoder similarity correlates only 0.75 with relevance vs 0.92 for cross-encoders

Generated with AI, which can make mistakes.