Autoregressive retriever training raises BEIR scores

Short summary

DREAM proposes an alternative to contrastive learning for dense retrievers: deriving supervision from a frozen LLM's next-token prediction objective. Tested on backbone sizes of 0.5B to 3B parameters, the method consistently outperforms existing baselines (DPR, ANCE, ColBERT) on BEIR and RTEB benchmarks by injecting retriever scores into select attention heads. If validated at scale, this approach could become the new baseline for retrieval-augmented generation, eliminating expensive contrastive data mining pipelines.

• •DREAM uses LLM next-token prediction instead of contrastive mining for retriever training
• •Outperforms existing methods (DPR, ANCE, ColBERT) on BEIR and RTEB across 0.5B, 1B, and 3B parameter models
• •Could simplify RAG pipelines by eliminating costly positive-negative pair mining stages

Generated with AI, which can make mistakes.