Back to feed
arXiv cs.CL
arXiv cs.CL
7/27/2026
A Consensus-Based Framework for Relative Preference Evaluation of Large Language Models

A Consensus-Based Framework for Relative Preference Evaluation of Large Language Models

Short summary

This paper introduces a consensus-based evaluation framework where a panel of diverse LLMs ranks anonymized candidate responses to measure relative preference rather than absolute correctness. Using five state-of-the-art models across programming, knowledge, safety, reasoning, and math, responses are aggregated into a Relative Intelligence Index (RII). The authors find consistent inter-model preference patterns but caution these reflect model alignment, not human judgment or objective correctness.

  • •Proposes consensus-based LLM evaluation using inter-model agreement as a proxy for response quality
  • •Introduces Relative Intelligence Index (RII) from structured peer voting across five SOTA models
  • •Results show consistent preference patterns but are not directly aligned with human evaluation

Generated with AI, which can make mistakes.

#research-breakthrough#ai-tools
Read full article at arXiv cs.CL

Is this a good recommendation for you?

Comments

Failed to load comments. Please try again.

Explore more

Comprehensive Evaluation of Large Language Model Responses: A Multi-Factor Scoring System
arXiv cs.CL
RIMS: Preference Optimization via Smoothed Multi-pair Aggregation for Small-Scale LLM Retrieval-Augmented Generation
arXiv cs.CL
CogArena: A Multimethod Evaluation of Cognitive Ability Structure in Large Language Models
arXiv cs.CL
The original title is "Some Large Language Models Exhibit Consistent Risk Attitudes"
arXiv CS.AI