I added a reranker to my RAG pipeline — it broke everything, then I fixed it

Short summary

When adding a cross-encoder reranker to optimize RAG retrieval, the model often scores dense/tabular data poorly despite its accuracy on natural language passages, undoing gains from hybrid search. After testing 7 approaches (score blending, rank fusion, larger pools), the author found that cross-encoder scores are too negative to compensate. The fix: reserve guaranteed slots from first-stage results and use the reranker only to fill remaining candidates.

• •Cross-encoders trained on natural language passages actively reject dense/tabular data despite containing correct answers
• •Rank fusion and score blending fail when underlying model distributions are fundamentally misaligned
• •Hybrid approach works: preserve top first-stage results (FAISS + BM25) and let reranker fill only remaining slots

Generated with AI, which can make mistakes.