Transforming LLMs into Efficient Cross-Encoders via Knowledge Distillation for RAG Reranking

Short summary

This paper fine-tunes LLaMA 3 8B as a drop-in reranker for RAG pipelines using LoRA-based supervised fine-tuning followed by 4-bit quantization, replacing traditional cross-encoders that have quadratic inference costs. The resulting model is integrated into a dual-retriever pipeline (BM25 + dense search) and evaluated with RAGAS, achieving 14-21% gains across answer relevancy, context precision, similarity, and correctness over the cross-encoder baseline. The approach demonstrates that instruction-tuned LLMs can serve as efficient, accurate rerankers without quadratic complexity.

• •Fine-tunes LLaMA 3 8B with LoRA + 4-bit quantization as a RAG reranker
• •Achieves 14-21% gains over cross-encoder baseline on RAGAS metrics
• •Eliminates quadratic inference cost of traditional cross-encoders

Generated with AI, which can make mistakes.